Making Services Idempotent - II
This post covers the scenarios where you are building web-service gateway over legacy applications. In my previous post we covered the basic implementation for making services idempotent. We talked about the notion of persistent log that keeps track of unique request messages and the necessity to scope processing of request and log update in an atomic transaction. To re-cap, here is the pseudocode that we have seen in the earlier blog.
[WebMethod]
ResponseMessage DoRequest(RequestMessage)
{
Extract the request identifier from the request.
Check if the request has already been processed by querying the persistent log.
If (request processed) then return response
Else
{
//The transaction object need not be ITransaction. It could be
//ISqlTransaction or IOleDbTransaction or whatever depending on
//the resource manager you use.
ITransaction trans = CreateTransaction()
ProcessMessage(RequestMessage, trans);
//Store the response along with the request identifier the persistent log
UpdateLog(responsemessage, request identifier, trans)
trans.Commit();
Return responsemessage;
}
}
What if the ProcessMessage step is simply nothing but delegating the request to some legacy system (of course after doing appropriate message transformations)? If this is the case, there is no way you can scope the two steps (ProcessMessage and LogUpdate) in an atomic transaction. Here are some tips that may help you.
Approach 1: Delegate the responsibility to legacy system
In this approach, you don’t have to do anything special in the web service layer. You just pass on the unique request identifier to the legacy system and let the legacy system do the work. But for this to work, the legacy system should have been designed for this up-front. If that is not the case, you may have to tweak the legacy system which may be impossible.
Approach 2: Keep track of the communication state.
This approach is really useful, when you are using queues to communicate with legacy system. In this approach, you have to keep the state of communication with the legacy system in question. Following are the valid states: {RequestSent, ResponseReceived} – where RequestSent implies that the request has been sent to the legacy system, but response hasn’t been received. ResponseReceived state implies that the response has been received from the legacy system.
Now, let’s see the pseudocode for this approach
<PseudoCode>
Extract request identifier from the request
Check the state of request by querying the persistent log
If (state == ResponseReceived)
{
//This implies that the message has been processed and the response has been received.
//Just retrieve the response from log and return.
ResponseMessage = FetchResponseFromLog()
Return responseMessage;
}
if (state == ResponseSent)
{
//This means that the service crashed before the response was received.
//implying that the response is out there in the response queue to be picked up.
Goto ReceiveResponse;
}
BeginTransaction1
TransformMessage();
EnqueueMessageToLegacySystem(requestMessage);
UpdateLog with the state = RequestSent
CommitTransaction1
ReceiveResponse:
BeginTransaction2
ResopnseMesage = WaitOnResponseQueue(RequestIdentifier)
UpdateLog with the state = ResponseReceived. Also save the responsemessage in the log.
CommitTransaction2;
Return responseMessage;
</PseudoCode>
Main difference is that there are now two distributed transactions (I am assuming that the queue infrastructure supports XA style transactions.)
This algorithm can be extended for the general case.
PS: One reader, by the name of David Taylor, mentioned that in their domain they follow a different approach to implement idempotent behavior. Apparently, in their domain, most of the messages contain some unique message identifier. For instance, they have used timestamps as a means to uniquely identify the message. Here is his message.
In my industry, we use an API called EPP for Domain Name provisioning (an IETF standard). In that case the protocol has been designed to be idempotent purely by API design (rather than sending unique identifiers).
An example being that when you renew a domain, you need to supply the current expiry date, so if you send a command to renew for 1 year (and then repeat the message) you dont end up with a 2 year renewal.
So the API designers have specifically designed each operation as idempotent without requiring a unique client or server identifier. It words quite well because you dont have to cache the unique ids.
PS: Timestamps are notoriously unreliable. So, be careful when you use timestamps.