Looping around Message Elements


 I had loads of fun a few weeks back with coming up with some options inside BizTalk on how to separate multiple elements inside a Message for individual processing.

e.g: Retrieving all of the items individually in this Order message

            <Order>
               
<Items>
                  
<Item Code=”ITEM001” Price=”23.00” Qty=1>
                  <Item Code=”ITEM002” Price=”23.00” Qty=1>
                  
<Item Code=”ITEM003” Price=”23.00” Qty=1>
               
</Items>
            
</Order

Now, BizTalk has the ability to “shred” these recurring elements into subsequent messages which is very cool – you simply define a Envelope Schema which in this case would just encompass the <Order> and <Items> element, and also define a Body Schema that would define the Item elements.  Typically you would reference the Item schema from the Element as an unbounded child of the Items Node.

To enable the “shredding” to work you need to set two properties on the Envelope Schema:

  • Click on the Schema Element of your Envelope Schema in the BizTalk Editor and set the Envelope Property to Yes
  • Click on the Root element of your Envelope Schema ( <Order> in the example above), and set the “Body XPath” property to point at the node above your repeating elements (<Items> in this case>

You can then create a Custom Pipeline and drag on the XML Disassembler (is it just me that always gets the Assembler/Disassembler the wrong way round J).  Click on the Dissasember component and view the properties pane, Point the Envelope schema at the appropriate schema and point the Document Schema at the Item schema in this case.

If you need to have promoted properties in the Body/Document Schema let me know – it’s not straight forward right now.

Any messages now flowing through this pipeline will be “shredded”, n Item Messages will arrive inside the MessageBox ready to be consumed – all this with no code!  You do however “loose” the original message in this context which may or may not be a problem – sometimes there is some data in the “Envelope” section of each message, it could be that the split messages need to have that data to aid with the processing.   In this case I think you might be able to use an Inbound Map on your Receive Port to transform the message to add the elements – need to try this though as I’m not sure if the Map is run first or not.

This works a treat, but what happens if one of these <Item> elements is malformed or has some problem with it?  Well in this case the Pipeline is going to thrown an exception and you’ll end up not getting any <Item> messages created, now this in some cases might be reasonable (if you’ve defined and agreed a schema and have no requirement to make best endeavors with processing the message). 

However if you want more control over the splitting and any errors that might occur you end up with an Orchestration solution.  So – How do you “loop” around multiple elements of a BizTalk Message in an Orchestration?

Based on some internal material I managed to do this by utilising a XmlNodeList variable that took the result of a xpath statement in a Expression shape, retrieved a Enumerator by using the GetEnumerator method on the XmlNodeLIst and iterating around this inside BizTalk using a Loop shape, each iteration cast the Element to a BizTalk Message variable which was then sent via a Direct port back to the MessageBox for further processing.  This worked great (and I have detailed steps on doing this if anyone wants it) but all of this has to be done inside an Atomic scope as the XmlNodeList and IEnumerator types are not Serializable. 

However a very observant customer spotted that this solution didn’t fit their requirements, they needed to loop around all the recurring elements, assigning each one to a BizTalk Message and doing stuff – this had to be done within an atomic scope and therefore they couldn’t handle any exceptions that might be thrown by their .NET component or by BizTalk itself if one of the recurring element was malformed, so if something bad happened the Atomic scope would be rolled back and any Messages “sent” to the Direct port would not be committed to the MessageBox leaving us in the same situation as the pipeline solution.

Much head scratching and discussion with a colleague later we came up with a much nicer solution that allows exceptions to be caught so you can effectively loop around all elements “skipping” bad ones and avoiding Atomic scopes completely!.  Basically we use the xpath function exposed by the Orchestration runtime to “count” the number of elements in the message, we then loop around each of these again using the xpath function to retrieve the message and cast it to a Message.

The xpath built-in function is very powerful and most people don’t seem to know about it – you can read up on it here

Dead simple when you see it – but it was hard to get there!  Here’s the sample to save you the pain!

Comments (26)

  1. xai says:

    I’ve had problems with this, and ended up just mapping the documents to another schema with the promoted properties to work around the issue. So how do you do it??

  2. Darren Jefford says:

    Can you drop me a mail please? (via the Contact hyperlink on the top left)

  3. Gregor Hohpe says:

    This is a very elegant implementation of my Splitter pattern: http://www.eaipatterns.com/Sequencer.html

  4. Dixie's blog says:

    <a href="http://blogs.msdn.com/darrenj/archive/2004/08/30/222614.aspx">Looping around Message Elements</a> – this is another fine approach to split messages in BTS. Also, Darren Jefford explains how to loop in message in BTS orchestration. Take a look at

  5. I recently needed a means to split these inbound XML documents into multiple outbound documents where the number of outbound documents and the content of each document is determined by one of the fields in the inbound document. For lack of a better term I’m going to call this a “Logical Splitter” pattern.

  6. Abhilash M S says:

    One of the fairly common issue that most Biztalk designers face is the issue of Debatching and then Batching…

  7. Abhilash M S says:

    One of the fairly common issue that most Biztalk designers…

  8. Abhilash M S says:

    &amp;nbsp;

    One of the fairly common issue that most Biztalk designers face is the issue of Debatching and…

  9. Abhilash M S says:

    One of the fairly common issue that most Biztalk designers face is the issue of Debatching and then Batching…

  10. Working with XPath inside Orchestrations is a powerful and simple feature of BizTalk 2004. The help guide

  11. crazyhat says:

    I followed the steps you have listed but get the error message:  

    There was a failure executing the receive pipeline:

    Reason: No Disassemble stage components can recognize the data.

    The envelope schema property is mapped to the schema with the root Order.

    the document schema property is mapped to the schema with the root Item.

    And i do not have the Items node in my xml file.

    <Order>

    <Item Code=”ITEM001” Price=”23.00” Qty=1/>

    <Item Code=”ITEM002” Price=”23.00” Qty=1/>

    <Item Code=”ITEM003” Price=”23.00” Qty=1/>

    </Order>

    Thanks!

  12. crazyhat says:

    Please ignore my previous post. I had a namespace issue.

  13. yuri.khenokh says:

    Darren,

    I have read you book (Professional BizTalk 2006). Thank you for the tons of usefull BizTalk material you have published!

    I’m trying to split (or as some people say debatch) an incoming message using an envelope and a document schema in BizTalk 2006 R2. The problem is that I need to retain some data contained in the envelope by copying it down to every debatched message.

    In the post you mention, quote:

    "… You do however “loose” the original message in this context which may or may not be a problem – sometimes there is some data in the “Envelope” section of each message, it could be that the split messages need to have that data to aid with the processing.   In this case I think you might be able to use an Inbound Map on your Receive Port to transform the message to add the elements – need to try this though as I’m not sure if the Map is run first or not. …"

    As far as I know, the inbound map runs after the receive pipeline and therefore cannot be used. However a custom orchestration solution described in the post seems to be an overkill for a problem as simple as copying some data from the envelope.

    Richard Seroter in his post "BizTalk Aggregation Pattern for Large Batches" (http://blogs.msdn.com/richardbpi/archive/2006/05/08/592476.aspx) seems to imply that simply promoting corresponding properties in the envelope and in the document schema will do the trick, however this approach also does not seem to work. (I tested it with xmldasm tool and output messages did not contain envelope values).  

    This looks like a very common business problem that I would think many people should encounter. Still I could not find any answers in the book or on the net.

    Is custom pipeline disassembler component the only way to do it or is there a simpler approach?

    Thank you,

    Yuri

  14. Working with XPath inside Orchestrations is a powerful and simple feature of BizTalk 2004. The help guide

  15. Jaeger6669 says:

    So I’ve been working on this and I’ve run into a snag.

    If the "Order" message has a namesapce defined (as good biztalk messages do, ie uri:MyOrders) then the Item message doesn’t contain the namespace when you assign it using Xpath.

    If you then try to send the Item message, you end up getting an error about not being able to determin message type by "Item"

    Has anyone found a way around this?

  16. rachanadikshit@hotmail.com says:

    Hello,

    Has anyone found out a way to use data that is in the “Envelope” section of each message, it could be that the split messages need to have that data to aid with the processing?

    If not then Please suggest to me any article with the explanation of the custom pipeline disassembler component .

    Also , I am planning to use the XPATH approach.Is it a good way to use for Large messages? I am not concerned with how long the message will take to process.

  17. eliassal says:

    Hi,

    Hi I’m working with a flat file that has several records which needs to be debatched to multiple output file. The questio is how we can aggregate the result of individual messages to multiple files and not only one based on a special code inside each record in the input message. What I mean that I have the input message with several messges, each message has a customer code, I want to group all messages relevant to customer 1 in output file 1, messages relevant tocustomer 2 to output file 2 and so on.

    So if I have n messages in my input, then individual elements should be debatched and aggregated in several files. If the n messages represent 10 messages, 2 of which are for cust1, 3 for cust 2, 3 forcust 3, 2 for cust 4, then then 4 output files should be produced by the process.

  18. eliassal says:

    Hi, I am trying to download the sample but it seems the link is not good. can you please send me the sample code, thanks salam.elias at free.fr