Continuation scenarios with mixed of TPE-enabled and non-TPE-Enabled tracking:


It has been a while since I last blogged on this.  Let’s continue from where I left off. 

 

Let’s pick two continuation scenarios: the first, a TPE segment continues to a non-TPE segment, and the reverse being the second.

 

 

1.      TPE-enabled segment continues to a non-TPE-enabled segment.

 

In this scenario, you would need only a Continuation folder. The naming of this folder is important since the concatenation of the folder name and its mapped value will form the continuation TokenID used in non-TPE-enabled segment.  For example, if the Continuation folder name is “TPE2OES” and the mapped value has “PO123” for instance, the continuation TokenID would be “TPE2OESPO123”.  Once you have mapped other data items and milestones, you will deploy* this.

 

In the related orchestration, in the process somewhere downstream, you will need to insert the following:

 

// Continuation segment using OrchestrationEventstream**

// Where ContinuationTokenID based on this instance contains the value “”TPE2OESPO123”.***

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.UpdateActivity(“PurchaseOrder”,  ContinuationTokenID,

        “Column1”, Value1,

        “Column2”, Value2);

 

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EndActivity(“PurchaseOrder”,  ContinuationTokenID);

 

 

2.      Non-TPE-enabled segment continues to a TPE-enabled segment.

 

In the reverse scenario where an activity begins from an non-TPE segment,  the following code would be required:

 

// Sample main segment of continuation using OrchestrationEventStream.***

// Activity in this instance contains: “PO124”

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.BeginActivity(“PurchaseOrder”,  ActivityID);

 

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.UpdateActivity(“PurchaseOrder”,  ActivityID,

        “Column1”, Value1,

        “Column2”, Value2);

 

// CONTINUATION LINE HERE:

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EnableContinuation(“PurchaseOrder”,  ActivityID, “OES2TPE” + ActivityID);

 

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EndActivity(“PurchaseOrder”,  ActivityID);

 

To create the continue-to segment, in your tracking profile, you would need to add a ContinuationID folder, name this folder as “OES2TPE” and map the relevant item (one that will give you the above “PO124” for this particular instance of message.  Deploy* this and drop a message and see the beauty flows 🙂

 

 

 

*   You will get a warning that the tracking profile is missing the ContinuationID or Continuation folder, but since we know what we are doing :), just ignore and proceed.

 

** Note: there is no BeginActivity call since this is a continued-to segment. 

 

***The above can easilly be changed to used DirectEventStream or BufferedEventStream. If you do so, make sure you call the Flush method explicity or set the flush threshold to 1 or greater in the constructor.


Comments (18)

  1. Peter says:

    I have the following scenario that I cannot get to work. From the BAMEndToEnd sample I used the pipeline code to start an activity, let’s call this A, from a receive pipeline. The activity is continued in an orchestration. Following the SDK sample and the guidelines you give in this blog I have created a continuation ID in which I drop the ActivationID that is created in the CustomPipeLine. In this orchestration I have a loop in the main process which I would like to BAMify as well. So I created an extra activity for this. Let’s call this B. I would like to use TPE. According to the very sparse documentation that is available on the subject of Relationship folders it is hard to figure out how to setup a relationship from a continued activity in TPE. It is not clear to me what ID to use to connect the main and sub activity with eachother. I tried the following  values for this ID.

    1. The activityID that is used to continue the activity that was started in the receive pipeline.

    2. the activityID from 1, preceeded by the prefix that is used in the ContinuationID (Orch1_ in the sample)

    3. A whole new ID.

    I cannot get it to work. In all cases I get the following error.

    Exceptions.Types.InterceptorException

    Invalid Interceptor Configuration. The first trackpoint must be TraceStartTrackpoint

           

    Exception type: BAMTraceException

    Source: Microsoft.BizTalk.Bam.EventObservation

    Target Site: Microsoft.BizTalk.Bam.EventObservation.BAMTraceFragment CreateTraceFragment()

    Help Link:

    Additional error information:

    Any suggestions?

  2. theforger says:

    solved already, it was a different problem, had to do with whe activity used in the relation folder of the looping structere, I filled it with the activityID from the main activity, whereas I should have filled it with an equal valued activityID from within the loop. MS documentation might be a bit clearer on this one.

  3. keithlim says:

    Was in class the last two days.  Great that you found the solution to the problem 🙂

  4. jason says:

    I am trying the second scenario and am having difficulties.  I followed the procedure exactly as explained but the TPE tracking is producing its own entry with a unique Activity ID.  I cant figure the cause of this as i am assiging the activity id to a predefined guid which is obtained by the message xml.  For example:  Message A has continuation enabled for (x + guid) and is called in code outside of the orchestration.  TPE has continuation id named x and set to guid.  The BAM message appears as two separate messages (TPE message has a unique activity id).  Any suggestions?

  5. keithlim says:

    Hi Jason,

    Without seeing your solution, I can only guess what went wrong.   Is the continuation segment in the Orchestration or pipeline or both?

    If the activity continues from an application to the biztalk pipeline and then to orchestration, then you would need:

    1) One ContinuationID folder for the pipeline to pair up with the continuation (EnableContinuation call) from the application.

    2) A pair of Continuation and ContinuationID folder for the pipeline and orchestration respectively.

    Could you send me an email with the artifacts via my blog page left pane email option.

    Thanks,

    Keith

  6. jason2d2 says:

    Thanks,,

    I understand your points as i have observed the same explanations across the web.  I think the issue is coming from Tracking Profile Editor.  

    Everytime i track a message through an orchestration, it generates a new random guid for the activityid.  However, I set the activityid to a unique value within the message schema (in the TPE), but for some reason it still generats a new value.  This causes a series of row entries to be displayed in the portal for a single message.  

    Are you famaliar with some kind of configuration issue?  Does this make sense to you?

  7. keithlim says:

    I think the confussion is in the TPE Select Event Source button.  When you click on this button, there are 4 options.  If you want to track anything from BizTalk orchestration, you must click on “Select Orchestration Schedule” option.  The other 3 options are applicable to Biztalk messaging (i.e. pipeline and port).

    Mapping from orchestration: once you have selected the appropriate orchestration (orchestration displayed on the right pane), to map a milestone, just drag the milestone to the appropriate activity milestone item. To map a dataitem, you would right button click on a particular shape and pick the Message Payload Schema (to map to message payload) and drag the appropriate schema item to the activity item.

    In other words, anything you want to track from the Orchestration, has to come from the Orchestration schedule, whether by dragging the shape (for milestone mapping) or dragging a schema item (for non-milestone item mapping).

    Let me know if this solves you issue.

  8. jason2d2 says:

    This was definately the problem! I was messing with the TPE and realized that you cannot use the "select event source" to select the messaging payload, then drag.  You have to right click on the orchestration, then select messaging payload, then drag.  

    Guess its another one of BizTalk’s goofy glitches.

    On another Note

    Do you happen to know if its possible to send message data within an alert.

    For instance: create an alert with a flat file transport, which includes the message "id or some unique identifier."  Also, have you heard any possibility to create custom transports?

    thanks

  9. keithlim says:

    BAM Alert and customizing message and provider.

    Out of the box, BAM Alert supports file and SMTP (email) transports.  But users are free to write custom transport.  (Please look up BAM help doc as well as SQL NS doc for writing custom transport.)    

    On customizing the alert message itself, you can customize it via the related XSLT file in the tracking folder.   By default, we provide a link in the message that you can click that will bring you to the original alert/query in the Portal UI.  Providing link rather than actual message, provides an added layer of security ensuring that only those that have the rights can see the data.

  10. Many of you guys are eagerly anticipating the release of the Microsoft BizTalk Server 2006 R2… and the

  11. Nick says:

    Hi Keith;

    Thanks for the post, I found it helpful in trying to understand continuations. I’m still having trouble grasping exactly why continuations are needed, though. Correct me if I’m wrong, but it seems to me that continuations are useful if: a)Different segments writing to the same activity don’t have access to the original ActivityID used to create the activity entry; b)An activity spans two TPE-enabled segments, or a TPE-enabled segment and a non-TPE-enabled segment (in either order); or c)Both are true.

    Let’s say that I’m using the BAM API only in custom pipeline components to create tracking for a pure messaging solution, and the string I’m using as an ActivityID is available everywhere I’m doing BAM writes. I’m not using the TPE anywhere, so I have full control over the tracking behavior. What’s the point of using continuations when I could just leave the activity active and continue writing to it using the ActivityID?

    This also brings up another question: When I’m using continuation between segments of a solution, do I call EndActivity() at the end of every segment? If continuation is enabled, is the activity actually moved into the Completed table or is it left in the Active table?

    Thanks for your help.

  12. keithlim says:

    Good questions Nick,

    Let’s say that I’m using the BAM API only in custom pipeline components to create tracking for a pure messaging solution, and the string I’m using as an ActivityID is available everywhere I’m doing BAM writes. I’m not using the TPE anywhere, so I have full control over the tracking behavior. What’s the point of using continuations when I could just leave the activity active and continue writing to it using the ActivityID?

    [KeithLim]:  If you use DirectEventStream, and write the events in correct sequence, no continuation is needed; i.e. BeginActivity,UpdateActivity and EndActivity, in that order.  And no UpdateActivity calls for that instance after that.

    For all the other asynchronous EventStreams( BufferedEventStream, MessagingEventStream and OrchestrationEventStream or TPE enable segments)  continuation is needed when there is more than one EventStream instance is used. Long answer: Reason being that these events are actually persisted in intermediate MessageBox and are sorted according to their eventstreams instance ID. Since TDDS has multiple thread dequeing and processing these events to BAMPrimaryImport database, all the events with the same instance ID are guaranteed to be process in sequence. If there is no continuation in these cases, there is possiblily of dangling event occuring.  (instances having the same activity ID but one in completed and another in active.)

    This also brings up another question: When I’m using continuation between segments of a solution, do I call EndActivity() at the end of every segment?

    [KeithLim] Yes.

    If continuation is enabled, is the activity actually moved into the Completed table or is it left in the Active table?

    [KeithLim] By enabling continuation, the system actually insert a record in the continuation table. This ensure that for the instance, eventthough it has been issued BeginActivity and EndActivity, it will not be moved to completed table.

    The full criteria for a activity to be completed:

    IsActive = 1  (synonymous to BeginActivity call)

    IsCompleted = 1 (synonymous to EndActivity call)

    And there is no correspondent entry in Continuation table.

    Hope these clarify any issues ambiguities you have.  Very good questions. 🙂

  13. Nick says:

    Thanks so much for your quick reply. Continuations (and the way the engine handles BAM in general) now makes a lot more sense to me. There’s one bit of your reply that has me a bit confused though – the part where you say you can get a "dangling event" (two instances of the same ActivityID, one in Completed and one in Active) if BAM calls are resolved by TDDS in the wrong order.

    As I was looking around for information about continuations, I found that the .NET class documentation for EnableContinuation (http://msdn2.microsoft.com/en-us/library/microsoft.biztalk.bam.eventobservation.eventstream.enablecontinuation.aspx) has a pretty good example of why you’d want to prevent calls on different instances of BufferedEventStreams from occurring out of order. That particular example, where some statistics are goofed up because certain bits of data weren’t updated in order/on time, makes sense to me. However, I don’t understand how you can get a "dangling event" – it seems like you’d just get an exception thrown instead.

    Let’s say that updates to an activity are made across multiple instances of BufferedEventStreams. They all use the ActivityID so no continuations are needed on account of that, and the activity is simply left active in each segment except for the last one, where it is ended. All these calls go to the buffer, where they unfortunately occur out of order and the buffer that ends the activity occurs before a buffer that has an update. After the buffer that ends the activity resolves, the engine then attempts to resolve the buffer that updates an activity that is not in the Active table. If an update is attempted on an ActivityID that isn’t in the Active table (even if it is in the Completed table), doesn’t BAM throw an exception? For a dangling event to occur, the resolution of that buffer would have to create the activity before updating it, which it wasn’t explicitly told to do.

    Thanks for your help!

  14. keithlim says:

    I will attempt to answer the second part first since I think it will self clarify the first part.

    Answer to 2nd part.

    [keithlim] For performance reason, BAM does check the completed table for duplicate when inserting to active table.  It will however, throw an exception when it attempts to move the record to the completed and there is a duplicate id.   As I mentioned in my previous post, the BeginActivity is synonymous to IsVisible.  Once this is called the record becomes visible in the BAM Portal.  So if you really only want user to see the record after certain info is available, you can always call BeginActivity after you have update those column.  And calling BeginActivity several time does not hurt as long as the corresponding EndActivity has not been called.  You probably notice, only the main segment of the continuation has BeginActivity, the secondary segments do not and that why you don’t see them in the Portal under their tokenized IDs.  

    Asnwer to 1st part:

    Since TDDS uses multipe threads in parallel to dequeue raw bam events off the messagebox,  it is possible for them to be processed in different order than they are inserted.  The continuation event ensures that the activity instance stays around even though the segment is completed.

  15. Nick says:

    Thanks! I didn’t know that you could do an UpdateActivity without doing a BeginActivity first. Makes perfect sense now.

  16. Ali Shahzad says:

    great article!

    I intend to modify the Bam alert’s mail message. The scenario is that my alert condition can be a combination of many conditions and is time dependent as well (uses in the last operator), so if we simply provide the recepient with the link, it might return too much info in some scenarios, while in some cases the info might be "expired" (because of in the last operator).

    so I would like to add something like an interchangeId in my mail message. its part of my tracking profiles as well. i read ur reply- "BAM Alert and customizing message and provider."

    My question is where can i locate ‘BAM help doc’ and ‘SQL NS doc’ for writing custom transport. I need to know the input to the xslt, before i can change it! Will be great to know the resources that document the internals of BAM alerts for poosible extensions

    thanks in advance

    Ali

  17. Vishal says:

    How can we use continuation in purely messaging only scenario. i.e only maps on ports are used and no custom pipelines or orchestration to call BAM API. I managed to do that somehow but end of actvity it remains in active table and doesn’t get moved to completed table.