Tell us what you REALLY think about adapter development!


This is an open forum for gripes, complaints, suggestions, recommendations, and any other input you want to provide from your experience with Microsoft documentation regarding adapter development. Let it all hang out! I am compiling a list of items to work on for this new calendar year regarding custom adapter development and would love to hear what you REALLY think!

Comments (34)

  1. Satch says:

    I think I’d rather poke my eyes out with sticks than attempt to create (w/o framework) a custom solicit/response adapter. What I think would be FANTABULOUS is some examples of adapters at various levels of sophistication that DO NOT use adapter framework. Not that I’m against it, but it seems bloated and overdone for what I (typically) need … that and I get lost easily in layers of classes (admitedly a personal problem!)

    Hey, you wanted it all hanging out! Maybe I’m just nuts for wanting to implement IBTxxx interfaces directly, but … there it is … I do.

  2. mmckeown says:

    thx Satch. That is good input! Do you know of many folks out there that are developing w/o Adapter Framework? Did you actually attempt it and hit a wall? That is a great idea and definitely to do on the horizon. But for now I need to focus on the adapter framework as per higher powers 🙂

  3. satch says:

    I don’t know that there are many, but I know of at least one other (hey, we’re a community!). I’ve had no real issues with several one way’s I’ve had to make, but I did hit a bit of a wall on two-way. I specifically got lost in the callback mess.

    If you are constrained to adapter framework per boss, then I think if documentation got explicit with regard to what’s happening under the sheets, that would go a long way.

  4. Mike says:

    So your manager is making you use the framework? the framework is just a set of COM intefaces into BizTalk that your adapter uses and implements. What else would you want to use? And what kind of information under the covers would you need to get access to besides what is documented in the adapter whitepapers? Tell me and I will see if I can get you an answer. Also take a look if you would off my blog site under ‘adapter documentation’ as there are lots of deep details in those papers. I’m here to help but I cant do it generically so let me know explictly the types of info you need and why the adapter framework does not quite meet your needs.

    Thanks Satch

  5. satch says:

    <snip>But for now I need to focus on the adapter framework as per higher powers</snip>

    I took this to mean that you were constrained.

  6. Lee says:

    I have recently developed a Solicit Response Adapter based on the adapter framework in the CTP of BTS2K6.

    Admittedly it was a rush job developed as part of a proof of concept. We’re working on a criminal intelligence integration project amongst the 25 Member States of the European Union.

    Anyway, the Proof of Concept was to determine the applicability of BizTalk for our project and I was able to convince the department to use it – so we’re off to a great start for 2006 ;o)

    The adapter I developed was to interface a custom in house application more than it was an implementation of a transport.

    As part of a proof of concept the development of the adapter was just that – proving that it was possible – it certainly wasn’t an enterprise class development project! It will be developed properly when it is revisited as part of the project going forward.

    My memory of the development at the moment is one of having some difficulty. This difficulty is derived from a number of sources. Firstly the whole adapter framework comprises a number of conceptually difficult components. Difficult both in themselves and in how they interact with one another. I felt this to be the main source of difficulty because I didn’t think that the whole framework is well documented at a pruely conceptual level.

    I got the solicit-repsonse adapter working by modifying the http.net sample adapter that ships with the framework.

    From this basis, technically, it wasn’t that difficult but my problem is the amount of confidence I have in the reliability of what I was able to develop. Because I still feel exposed in my conceptual understanding I’m not left with a sufficient measure of confidence to deliver what I did to production. It works. Thats all. Happy path.

    I got it working but whether I did that correctly or in the best, most efficient way I couldn’t say. It’s hard to understand exactly what goes on behind the adapter framework. What the messaging engine is up to and what at any particular stage in the process is expected of the adapter.

    This is all off the top of my head based on memories of a rushed week before christmas. When I revisit the code and with hopefully some better adapter framework documentation I will write a more considered piece.

  7. Mike says:

    Thx Lee. This is great feedback and lines up with what I have been hearing from a lot of folks including MS personnel. I hear you on the need for a deeper understanding of the framework’s interaction with the messaging engine. Additionally you used the base adapter (since you modeled after http sample adapter) I assume which is a layer of abstraction over the adapter framework. Its interaction with the framework also needs to be documented more. After RTM, my goal will be to do this using the transactional adapter sample as a basis.

    Any more specific input (such as interfaces or method calls you struggled with) would be greatly appreciated.

    Thank you

    Mike

  8. Richard Case says:

    We develop custom adapters on a reagular basis. What i would really like is the ability to customise the adapter properties UI with my own property pages. The file adapter has a custom UI for its properties so it would be good to open this up to adapter vendors.

  9. mmckeown says:

    I agree Richard. I will be working on a documenatation walkthrough after RTM that will hopefully touch on customizing property pages.

  10. Nathan says:

    I worked on BTS 2004 and wrote few custom adapters (file adapter with backup / archive functions). Also I was adding lots of custom properties (context) to each of the message that is being picked up by the different receive ports as well send thru the send ports.

    This was necessiated because the information coming from third part applications does not provide enough context properties regarding the message and we need to fill this based on the receive port it picks up.

    In previous version of BTS 2000/2002 there was a concept of envelope which can be attached to both incoming and outgoing messages at the channel level where you can create a schema and attach it to the channel and port and set the values and everytime a message passes thru that channel this custom property will also be part of the context.

    Is there anything like this available in 2006.

    This will be a real help.

  11. Nathan says:

    Few suggestions on future adapter funcations which will allow IT to embrace the product without much custom work on adapters.

    1.Major function that needs to be added to atleast file adapter is the capability of taking a backup to a different location before submitting to the BizTalk.

    2. In all adapaters there should be a way to define additional properties thru XML schema and also custom property page to configure the property values. The values could be constants or variables from the incoming message or port properties for example (%datetime[Format]%, %ReceiveportName% etc

  12. mmckeown says:

    Nathan

    Thanks for your input. I have passed it along to the product team.

    I will get back to you on your envelope question.

    thx

    Mike

  13. mmckeown says:

    Nathan

    Here is a reply from one of our folks to your question on envelopes in 2006. HTH!

    I used envelope in BTS 2004 to split an interchange into individual messages.

    <PurchaseOrders>

    <InterchangeID>I394</InterchangeID>

    <From> Partner ONE </From>

    <PO>

    <POId>1</POId>

    </PO>

    <PO>

    <POId>2</POId>

    </PO>

    <PO>

    <POId>3</POId>

    </PO>

    </PurchaseOrder>

    I would have two schemas, one for PO and other for PurchaseOrders. PurchaseOrders schema would be my envelope schema for PO schema. When a file with 10 POs come in, XML Disassmebler will use the envelope schema to split the file into individual POs. So, it generates 10 PO messages and store them in Message Box database. On send side, you will not be able to join multiple POs into one file using an envelope. But, you will be able to generate a file that looks like below.

    <POEnvelope>

    <EnvelopID>133</EnvelopeID>

    <PO>

    ………

    </PO>

    </POEnvelope>

    Customer wants set values for custom context properties defined by an envelope from Receive Port and Send Ports. As far as I know, you can only do what customer wants from an adapter or pipeline component programmatically. It is not going to be as simple as configuring though.

  14. Well i have done some adapter developement.

    I got the framework for free cause I used the BizTalk Adapter Wizard.

    However the Adapter Wizard is not completely problem free….

    One of the hardest things to tackle was making the adapter react to a stop command from bizTalk.

    I was shutting Biztalk down via the admin console while my adapter was spitting out large amounts of data. So BizTalk should send a stop to my adapter and then my adapter would stop and then Biztalk would stop.

    I have tried several things to solve this problem. Yopu can see the final solution at gotdotnet SPXII adapter that works very well.

    Shutting down BizTalk while the adapter is working will never cause oproblems anymore…….

  15. mmckeown says:

    Hi Patrick

    Be encouraged that the version of the BaseAdapter and the Adapter Framework upon which it is based for Pathfinder will be a much better solution for developers.

    I’m not fully understanding your issue.  Once I do I will also pass your input onto the BizTalk developers who worked on that framework.  Are you saying you want the adapter to still process messages while BizTalks is shutting down to allow the adapter to complete any tansfer operations it is in the middle of? Kind of like a way to block BizTalk’s shutdown process until the adapter finishes and cleans up properly?

    Also for the clarification of our blog readers, can you summarize in a few sentences what you did to fix this? Insert a few snipets of code if that works for you.  

    Thank you for your feedback Patrick

  16. Well0549 says:

    well what i did was having an xml document with let’s say 10 records in it.

    Let’s say every record is a Call to a stored procedure, so each document will do 10 database calls in one transaction.

    And lets also suppose i do it with request/response. So once the document is received by the orchestration, it tries to send it to a Req/Resp port. Then it must wait for the operation to complete to get the result.

    Then the orchestration will insert those same records in another database again, So the orchestration does about 3 roundtrips to the Sql Server but to different databases. ( Just a test Scenario )

    If I put that system under a heavy load by dropping 1000 Xml files in a directory, my laptop would be busy quite a while.. Let’s say 2 minutes of 99% processor usage.

    If i tried to stop BizTalk via the management console i would get strange com errors in the eventlog and the Stop command would timeout.

    You can download the SpxII adapter from GotDotnet and inspect the code to see how i solved the problem. With this new way of using the threads of BizTalk Stopping the adapter under the same stress would be a smooth task. The stop will take about 8 seconds and then BizTalk is really Stopped.

  17. mmckeown says:

    thx Patrick! Sounds like a clever solution to a somewhat common problem. Has anyone else had this and if so, what did you do to solve it?

  18. Well0549 says:

    It took me three month’s of testing (in spare time) to get it like this…..

  19. matt says:

    adapter development is like all BizTalk development…IT BLOWS

  20. jinishans says:

    Hi

    Last project we’ve done in Biztalk 2004 and we’ve used the Sample File Adapter and customized it, but not the Adapter Framework. But, we know about it. Why there is no support for a SQL Adapter which can support Other DB’s also. Atleast in Adapter Framework if you provide a sample to use the .NET Data Providers, it’ll be better.

    Also, if there is sample for FTP adapter, it’ll be better. We can plug our own 3rd party FTP client and use it also. We had a requirement to download the file from a FTP Server in our last project. But, FTP does not support locking of file, because of which partial files are downloaded.

    What we did is modifed the File Adapter and did it. But, it took huge memory (we’re newbies in BTS). So, we dropped that idea. Because, to solve the issue, what we’ve done is, we’ve used a 3rd party FTP Client, and issues a rename command from the adapter to the remote file. If it fail, then, the file is still not uploaded by the other application in the ftp server and we’ll try to download once we’re able to rename it.

    With Regards,

    Sankar,B

    Wipro Technologies,

    Chennai, India

  21. mmckeown says:

    Great stuff, Sankar. So you want a sample using a .Net Data Provider and an FTP adapter. I will pass this directly to the product team.  Thanks for your feeback!

    Mike

  22. Patrick Wellink says:

    Sankar,

    I totally agree with you,

    why not have a decent ODBC adapter.

    Think of the possibillities…

    We could read Excel files and DBF files with it. Wouldn’t that be nice.

    So on the wish list there must be ODBC adapter

    Oh and another on the wish list….

    ( this one made me make the SPXII adapter….)

    An adjustable transaction level in the SQL adapter…..

    Sometimes you just want to exec a stord procedure to get some stuff back… (or do some stuff) and setting the Transaction Isolation level to Serializable makes the database unresponsive and you end up with lotsa deadlocks and timeouts…

    And if you only want to execute a sproc that reads some stuff this is a very heavy penalty to pay

  23. SinStereo says:

    Adapter development lacks one basic thing… Samples are good, of course, but what about tutorials? I’d found a couple of nice documents about how to develop my own adapter, but I find it really hard to start, as I don’t know how to 🙁 I mean, I open the BaseAdapter for example, I see dozens of files and I just don’t get how everything fits together. Same case with the different samples, many files and very few comments make everything quite confusing.

    I’d read Biztalk help, "Writing Effective BizTalk Server Adapters" & "Microsoft Biztalk 2004 Adapters: A Developer’s Guide" yet I find myself nowhere closer to being able to start developing.

    Another problem is that I hadn’t found any documentation about changes between developing adapters for 2004 and 2006, so I don’t know if what I end up writing based on 2004 information around the net will work on my 2006 Beta2 server.

    Thanks a lot!

    Leandro

  24. mmckeown says:

    Thx Leandro! This is a great topic and seems to be hitting a lot of sore spots with folks.

    Funny you should mention the need for a tutorial. As of tomorrow I am starting to work on documenting exactly what you are asking for by coincidence. I will be taking the TransactionalAdapter and answering those very questions you are asking such as, "Okay there is this base adapter class and this Adapter Framework that are supposed to do so much for me and make this easy. But how do I use them? Where do  I start, what is my responsibility to code and what does it do for me that I don’t have to code, what is each file for, which do I modify, etc…"  You read blogs all over the Net of folks just scraping the whole base adapter/framework thing and writing them from scratch. And even then not too sure about how to implement what interface methods – this is crazy I agree! I too have read these fine white papers and I gained a good understanding of adapters and interfaces, how messages flow, etc. They gave me a good knowledge foundation upon which to start thinking about what I would need to write. But there is nothing to bridge that gap between head knowledge and practical application. They did not allow me to jump start my project with the base adapter or adapter framework. I was still confused.  Thus….my goal is to remedy this and publish a useful and practical walkthrough using the base adapter starting tomorrow oddly enough.

    So good timing on your post, and stay tuned!  As a developer I understand these things and will endeavor to get a good solution for you and others by RTM for Pathfinder!

    Kind regards

    Mike

  25. George says:

    Hi Mike,

    Funny enough I am looking for exactly the same type of tutorial as Leandro! How is your documenting coming along?

    Thanks,

    George

  26. mmckeown says:

    It  is coming along well. I plan to get the rough draft done by next Friday to pass it to the product team for tech review. thx for asking.

  27. mmckeown says:

    Leandro

    You can get some info on the changes in adapters from 2004 to 2006 in the latest RC .chm help file for Pathfinder.  Go to http://blogs.msdn.com/luke/archive/2006/02/03/524534.aspx

    and download it.  I have been meaning to start a new post with this link so now I have a reason to do so.  

    Once you download and extract, open the BTS2006.CHM file. Go to Getting Started What’s New in BizTalk Server 2006 New Features in BizTalk Server 2006. Scroll down to the section entitled Empowerment for Developers and you’ll see a heading under there of Enhancements in existing adapters. Although it is not in detail it highlights the differences for you.

    ** Is there anyone out there that could see the use for a detailed paper on the differences between the adapters released from 04 to 06?

  28. SinStereo says:

    Thanks everyone! I’m now into developing some adapters for testing based on the Socket Adapter & Adapter Wizard by Scott Woodgate.

    I’ve some doubts obviusly, so here I go (Should I be posting this here? I’m not quite sure, if not, I’m sorry!!) ^^

    About dynamic port binding… let me see if I understood this… You have 3 binding types, Direct, Now & Later, where does dynamic port binding fall? What I wanna do is the following: My adapter sends an XML to a server, but I may have the connection data in the XML. So I want the following to occur: If my XML contains the connection data then the adapter should send my message using the connection data from the XML, else it should use the Default Connection Data (Host / Port) which should be configurable properties of the Send Port. Any comments on this?

    I’d also like to read Schemas from the adapter, actually I have 2 XSD that will never change but I should be able to read them with the Adapter Schema Wizard and I’m clueless on this point.

    Thanks a lot!

    Leandro

  29. Sami says:

    I have created AS400 Data Queue Adapter. This adapter read messages from a DQ and submit them in BTS. It is working good but I still have a problem. When I want to stop the host, it takes a long time and some messages lost.

    How can I stop the host without any problems???

  30. mmckeown says:

    Is this DQ adapter a transactional adapter?  If not, do the messages only get removed from the DQ when the adapter receives a successful BatchComplete callback (either Submitting or suspending)?  If a non-transactional adapter deletes messages from the external store, then submits them to BizTalk there’s always a possibility of data loss.  The way around this with non-transactional is to wait for the successful BatchComplete callback before removing the data.  Consider the FILE adapter: it waits until a message has either been successfully submitted or suspended (successfully) before it will ever delete the file from the disk.  Otherwise you could run into the case where you delete the file, try to submit but BizTalk is shutting down, then the adapter cannot submit, suspend, or “un-delete” the message/file so it would be dropped.

  31. mmckeown says:

    Is this DQ adapter a transactional adapter?  If not, do the messages only get removed from the DQ when the adapter receives a successful BatchComplete callback (either Submitting or suspending)?  If a non-transactional adapter deletes messages from the external store, then submits them to BizTalk there’s always a possibility of data loss.  The way around this with non-transactional is to wait for the successful BatchComplete callback before removing the data.  Consider the FILE adapter: it waits until a message has either been successfully submitted or suspended (successfully) before it will ever delete the file from the disk.  Otherwise you could run into the case where you delete the file, try to submit but BizTalk is shutting down, then the adapter cannot submit, suspend, or “un-delete” the message/file so it would be dropped.

  32. Patrick Wellink says:

    Sami,

    I had the same problems in the beginning with my adapter. (SPXII see GotDotnet). But i fixed that.

    Have a look at the code of the SPX II cause i have solved it there. I don’t know how I did it, but it had something to do with code generated from the Adapter Wizard.

    The code in the wizard causes the behaviour you mentioned. Hard to stop and loss of messages.

    See the SPX II code and see how i fixed it.

  33. Angelo says:

    Sorry to post this here. I’ve been trying to develop a client adapter. I’ve used the wizard for 2006, and I’m very thankful.  On a Windows 32-bit platform (Win2k3), everything works fine.  The protocol I’ve been trying to implement is sound and functional.  However, when we tried porting it on a 64-bit server, it’s like the adapter is not even there.  I’ve looked at the registry file (.reg) to see if all the guids and paths are correct.  I’ve double-checked this in the registry itself, and it checks out.  There’s not even an event being fired up to say whether a dll is not found or something.  (The host instance to which the orchestration, and effectively the adapter, is bound was up and started.)

    Surprisingly, when I tried putting the adapter on a 32-bit host instance, it works!  (But we really need to have it running on 64-bit.)

    Has there been any other reported issue of any custom adapter not working on a 64-bit environment?

    Thanks!

  34. nyc_jon says:

    I worked on a custom adapter that communicate with a proprietary tcp/ip server and gets the message to BizTalk. I found the documentation is not very easy to follow and it’s a challenging exercise since examples are scarce. I ran into the same problem that it takes a long time to terminate the host instance which hosts the adapter. Any comments or thoughts to fix this behavior?

Skip to main content