Making Services Idempotent – II


This post covers the scenarios where you are building web-service gateway over legacy applications. In my previous post we covered the basic implementation for making services idempotent. We talked about the notion of persistent log that keeps track of unique request messages and the necessity to scope processing of request and log update in an atomic transaction. To re-cap, here is the pseudocode that we have seen in the earlier blog.


 


[WebMethod]


ResponseMessage DoRequest(RequestMessage)


{


Extract the request identifier from the request.


 


Check if the request has already been processed by querying the persistent log.


 


If (request processed) then return response


Else


{


//The transaction object need not be ITransaction. It could be


//ISqlTransaction or IOleDbTransaction or whatever depending on 


//the resource manager you use.


 


ITransaction trans = CreateTransaction()


 


ProcessMessage(RequestMessage, trans);


 


//Store the response along with the request identifier the persistent log


UpdateLog(responsemessage, request identifier, trans)


 


trans.Commit();


 


Return responsemessage;


}


 


}


 


What if the ProcessMessage step is simply nothing but delegating the request to some legacy system (of course after doing appropriate message transformations)? If this is the case, there is no way you can scope the two steps (ProcessMessage and LogUpdate) in an atomic transaction. Here are some tips that may help you.


 


Approach 1: Delegate the responsibility to legacy system


 


In this approach, you don’t have to do anything special in the web service layer. You just pass on the unique request identifier to the legacy system and let the legacy system do the work. But for this to work, the legacy system should have been designed for this up-front. If that is not the case, you may have to tweak the legacy system which may be impossible.


 


Approach 2: Keep track of the communication state.


 


This approach is really useful, when you are using queues to communicate with legacy system. In this approach, you have to keep the state of communication with the legacy system in question. Following are the valid states: {RequestSent, ResponseReceived} – where RequestSent implies that the request has been sent to the legacy system, but response hasn’t been received. ResponseReceived state implies that the response has been received from the legacy system.


 


Now, let’s see the pseudocode for this approach


 


<PseudoCode>


Extract request identifier from the request


 


Check the state of request by querying the persistent log


 


If (state == ResponseReceived)


{


      //This implies that the message has been processed and the response has been received.


//Just retrieve the response from log and return.


 


ResponseMessage = FetchResponseFromLog()


Return responseMessage;


}


 


if (state == ResponseSent)


{


      //This means that the service crashed before the response was received.


//implying that the response is out there in the response queue to be picked up.


Goto ReceiveResponse;


}


 


BeginTransaction1


      TransformMessage();


      EnqueueMessageToLegacySystem(requestMessage);


      UpdateLog with the state = RequestSent


CommitTransaction1


 


ReceiveResponse:


BeginTransaction2


      ResopnseMesage = WaitOnResponseQueue(RequestIdentifier)


      UpdateLog with the state = ResponseReceived. Also save the responsemessage in the log.


CommitTransaction2;


 


Return responseMessage;


</PseudoCode>


 


Main difference is that there are now two distributed transactions (I am assuming that the queue infrastructure supports XA style transactions.)


 


This algorithm can be extended for the general case.


 


PS: One reader, by the name of David Taylor, mentioned that in their domain they follow a different approach to implement idempotent behavior. Apparently, in their domain, most of the messages contain some unique message identifier. For instance, they have used timestamps as a means to uniquely identify the message. Here is his message.


 


In my industry, we use an API called EPP for Domain Name provisioning (an IETF standard).  In that case the protocol has been designed to be idempotent purely by API design (rather than sending unique identifiers). 


 


An example being that when you renew a domain, you need to supply the current expiry date, so if you send a command to renew for 1 year (and then repeat the message) you dont end up with a 2 year renewal.


 


So the API designers have specifically designed each operation as idempotent without requiring a unique client or server identifier.  It words quite well because you dont have to cache the unique ids.


 


PS: Timestamps are notoriously unreliable. So, be careful when you use timestamps.

Comments (7)

  1. Jef Newsom says:

    This was a great discussion on idempotent messaging. I added some <a href="http://integralpath.blogs.com/thinkingoutloud/2004/05/idempotent_mess.html">comments here</a>

    Thanks,

    Jef

  2. Great discussion onf Idempotent messaging… we could have used a person of your knowledge at my website.

  3. Michael Royster says:

    Are you sure this algorithm works what about the 1st and 2nd requests following in quick succession in which case both could potentially get past the “If (request processed) ” check as no transaction had started the first request will be processed the 2nd will block awaiting the transaction to complete and then continue processing the request twice.