Kevin Smith has finally started blogging again - I've been harassing him for a while now. He's kicked it off with a great discussion on Large Interchanges (Batches of messages), We're seeming customers increasingly hitting this problem and it's largely unknown so definately worth a read.
Splitting messages in an Orchestration is a common solution by customers but you have to make sure you get the design of the "splitter" right, typically customers implement it in a number of ways and the example I posted a while back is in my opinion far faster then the typical approach that developers end up doing that involves using Atomic Scopes for each iteration of the message splitting which could be hundreds, that's a whole lot of persistence points.
That said the example above introduces a high number of persistence points as well, each Send Shape will cause a Persistence Point as it commits the message to the Message Box. And when we're dealing with hundreds of thousands, and perhaps millions you really want to reduce the persistence points as much as you can.
The best way to alleviate this is to use an Atomic Scope to batch persist multiple messages at the same time, so instead of persisting each message individually you will use an Atomic Scope with multiple sends inside which will cause all messages to be commited at the same time.
You can acheive this quite easily with the example above by wrapping all of the logic inside an Atomic Scope, but when your dealing with millions of messages you probably want to persist the messages in smaller batches so processing of them can start whilst you continue to process the other messages - In that case you would then use a new atomic scope say every 100 messages - I'll update my example to show how to do this shortly.
Kevin is part-through writing a posting about the wonders of BAM, Hopefully he'll post this soon otherwise I'll be writing it 🙂 BAM is my favourite BizTalk technology - and most customers don't appreciate the power of it.