DirectEventStream vs BufferedEventStream


Which EventStream to use:

DirectEventStream (DES) vs BufferedEventStream (BES)

Use DES when you don’t want any latency of data appearing in BAMPrimaryImport database.   Since this is synchronous call, any error will be thrown back to the calling application.

Use BES if your concern is throughput.  This is particularly suited for scenarios with spurts of large chuck of events.  A small penalty for using this is that there will be small latency between when the event is sent and the event appearing in BAMPrimaryImport database.

For more info on EventStream: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sdk/htm/frlrfmicrosoftbiztalkbameventobservationeventstreamclasstopic.asp


Comments (15)

  1. Eric Stott says:

    I have a client who is concerned about the latency and would really like to use the DES for the application that does pre-processing of the file before it is handed to BizTalk. I am actually against this approach, as I would rather have a guaranteed delivery using the BES. Once the application is through its processing, and actually hands the file off to BizTalk, BizTalk must capture the file and check BAM to make sure that no records were lost. If I have implemented the BES inside of my application, would there be a possibility of entries to the BAMPrimaryImport database not being committed before a BizTalk orchestration checks. If I were to use the BES and flushing the instance in the application before creating the file, is there ANY possibility of that entry into the database not being there when BTS actually picks it up?

    This brings me to another question, what approach does Microsoft recommends when an application hands a message (in this case a file) to BizTalk, and to ensure that nothing is lost, needs balancing designed into the two processes. In BizTalk 2004, is there an automated way to check the BAM database to verify particular information, filename/row count has been satisfied before continue processing?

  2. Eric Stott says:

    I am assuming that you would have to use the IBAMDataExtractor.GetValue Method in the BizTalk orchestration. If I have my application write the filename and rowcount to the BAM database, once BTS picks up the file, it would get the rowcount and filename and then invoke the GetValue Method to retrieve the rowcount for this filename from BAM I would assume. Alas, I am unable to find an example of how this would be done though.

    Or even better yet, would it be possible to use the GetValue method to capture information not explicitly defined in the database. Example is that my applicaion is writing informaiton about each row in the file to BAM, however when BTS picks up the file, is there a way to sum up the number of rows for that file using this method?

  3. keithlim says:

    Q: If I were to use the BES and flushing the instance in the application before creating the file, is there ANY possibility of that entry into the database not being there when BTS actually picks it up?

    A: Yes due to latency, which normally ranges from instantaneous to 1 minute. However, should there be other configuration problems such as invalid activity name, column name, type etc, the record will not be seen in the Activity table. These records will be inserted to the TDDS_FailedTracking table and errors will be logged to Windows Event log. These sort of issues are expected to be ironed out before moving to production.

    I am interested to know why would the customer want to write to BAM first ourside of the Biztalk pipelines and orchestrations, and then submit the file to Biztalk? Unless there is a good reason, the writing to BAM is better done from within Biztalk.

    On Biztalk 2006, I would recommend using TPE to BAMify Biztalk pipeline or orchestration. This requires the least of effort, not to mention less error prone and also no recompilation, undeployment and restarting of biztalk service. If you absolutely need something very specific, then you would need to write your own code, e.g. want to write to BAM from a very specific stage of the pipelines, then use MES (MessagingEventStream). In Orchestration, you should use OrchestrationEventStream.

    On checking if any specific information is fulfilled before continuing, you should design your orchestration accordingly with the correct logic, e.g. to take a different branch or to suspend or to send request, e.g. to custom asking for additional missing info. Think of BAM at this point as just to collect data.

    On IBAMDataExtractor interface, this is really only if you want to write your own custom interceptor. I can tell you that TPE internally uses this.

  4. Eric Stott says:

    "I am interested to know why would the customer want to write to BAM first ourside of the Biztalk pipelines and orchestrations, and then submit the file to Biztalk? Unless there is a good reason, the writing to BAM is better done from within Biztalk."

    We are recieving a 45mb file from a mainframe process, and to improve performance, we have a VB app that calls a stored proc and parses out only the records that are to be directed to BizTalk. If BTS were to consume the entire 45mb file, it then becomes a process that would take the flat file pipeline 20 minutes to dis-assemble, before we would even start interrogating where split the records and we only have a 30 minute window to process the file.

    We would like a central repository to be able to go to, to look at the activity to look at our KPI’s.

  5. keithlim says:

    Interesting…my turn to do ask some questions :) to help me better understand your scenario, could you elaborate on the process that takes place. What do you want to capture while the big file is being split up? Do you want to find KPI’s such as how long does it take to split up the 45 Mb, each individual files, how many good and bad files? Or can I assume always all good or if any bad considered all bad?

    Are these individual files then sent to Biztalk orchestration, e.g. into a in folder/directory for Biztalk to pickup? What kind of processing takes place in the orchestration? Where do they uliimately end up? And during this processing, what KPI’s would you like to capture?

    Thanks,

    Keith

    Will be back tomorrow.

  6. Eric Stott says:

    "On checking if any specific information is fulfilled before continuing, you should design your orchestration accordingly with the correct logic, e.g. to take a different branch or to suspend or to send request, e.g. to custom asking for additional missing info. Think of BAM at this point as just to collect data."

    If I were to use the DirectEventStream in the application then, I would know that the records were successfully inserted before I continued. I then could test, in the application, that the record was inserted and then confidently continue with my file processing. Otherwise, the VB app can have error checking logic and fail if the BAM insert fails.

    I really don’t want to create a seperate datastore location to do this ‘balancing’ when it seems like everything I want to track would be in the BAM database already.

    So there are no examples of using the IBAMDataExtractor?

  7. Eric Stott says:

    Sorry, KPI’s was incorrect.

    There is a mainframe process that will only produce 1 file a day, and it is a big file that can not be broke up. This big file needs to be interrogated: line by line to find out the ultimate destination of each line, one of two places. Then two files need to be produced, one that goes BACK to the mainframe that essentially state that the mainframe should not process these lines. The other file needs to be translated into EDI documents (one per line) and sent off to a seperate processing system that only handles EDI data. Biztalk is used to take the lines and translate them to EDI, apply modifications to the EDI document and then finally sent to the new system.

    The customer would like a single interface to go to to account what actually happened, and that I think is a good purpose in BAM, kinda shallow, but a use none-the-less.

  8. theforger says:

    What about TPE? Does it use the DirectEventStream or the BufferedEventStream

  9. keithlim says:

    Internally, TPE uses an EventStream very similar to BufferedEventStream.  

  10. While working on a BizTalk POC last week, I needed to track Business Activity Monitoring (BAM) data…

  11. keithlim says:

    Hi Richard,

    What’s your specific questions? :)

  12. Richard says:

    Hey Keith, that was a trackback from my post, not a question 😉

  13. Gil says:

    "On IBAMDataExtractor interface, this is really only if you want to write your own custom interceptor. I can tell you that TPE internally uses this."

    Can "Interceptor" capture BAM messages (milestones & data) by TPE?

    In other word, can I customise the TPE’s interceptor?

  14. polatyrrao says:

    Hi,

    I have one question about when event stream is used behind the scene when we map using tracking profile editor instead of using BAM API. Is it one of the asynchronous eventsreams or DirectEventStream. I tried serching for this but could not get much info on this.

    Thanks

    Rajeshwar Polaty

  15. keithlim says:

    Hi Rajeshwar,

    The EventStream used behind the scene when mapped using TPE, of of asynchronous type similar to BufferedEventStream.

    Thanks,

    Keith

    p/s I will be posting on BAM WCF Interceptor very soon.