Make Services Idempotent
In the brave new world of service orientation, service requests may fail due to different reasons that may include network connectivity, resource failures (like db failure) etc. When a client receives a failure, all it can do is to retry for a while and then give up. This implies that the request may actually arrive more than once due to un-reliable nature of protocols like HTTP. Let's take an example
[WebMethod]
ResponseMessage DoRequest(RequestMessage)
{
ProcessMessage(RequestMessage);
Return responsemessage;
}
Failure can happen at any time. Failure can happen
- before the message has been processed,
- during the processing of message,
- after the processing of message but before returning the response.
Let's assume the third case. When a client re-sends this message again, then the message will be processed again. This may cause problems in many scenarios. For example, think of DebitTransaction as a service operation. You don't really want your debit transaction to be processed twice.
One of the important design principles of service orientation is that the service should be made idempotent. Idempotent means 'it is okay if the requests arrive multiple times'. As a designer of the service, you should expect the messages to be delivered more than once and you should process the message only once.
Unique Request Identifier
Traditional way to make a service idempotent is to create unique request identifier and pass the request identifier along with the message. When the client retries, client uses the same request identifier rather than creating new one. On the server side, service should inspect the request identifier, check if the request has been already processed and if so, return the response; if not - process the message, store the response in the persistent log along with the request identifier, indicating that the request has indeed been processed. Here is the pseudocode
[WebMethod]
ResponseMessage DoRequest(RequestMessage)
{
Extract the request identifier from the request.
Check if the request has already been processed by querying the persistent log.
If (request processed) then return response
Else
{
ProcessMessage(RequestMessage);
Store the response along with the request identifier the persistent log
Return responsemessage;
}
}
Will this algorithm work? Not quite. For example, consider the scenario where server crashes after processing the message and before storing the response in persistent log. This means that the request is not logged in the persistent log, even though it has been processed. So, when the request arrives again, you will process it because the request is not present in the log.
So, how do we solve the issue? The key is to use transactions. Both processing of message and log updates should happen in a single atomic transaction. Here is the modified logic.
[WebMethod]
ResponseMessage DoRequest(RequestMessage)
{
Extract the request identifier from the request.
Check if the request has already been processed by querying the persistent log.
If (request processed) then return response
Else
{
TransactionObject trans = CreateTransaction()
ProcessMessage(RequestMessage, trans);
//Store the response along with the request identifier the persistent log
UpdateLog(responsemessage, request identifier, trans)
trans.Commit();
Return responsemessage;
}
}
If your persistent log store is different from your application data store, then you may have to do distributed transaction - which is really costly. So, it is better to keep the log store and your app data store to be one and the same.
How do you pass the unique identifier?
Use SoapHeader to embed request identifier related information. Service can obtain the SoapHeader using the ASMX programming model. Please refer to ASP.NET documentation for using SoapHeaders.
If you are using WSE2.0 for the client, then you don't have to generate message id elements. It will be generated for you automatically.
What if the service implementation is simply a façade to legacy system? In that case, you can't really do a distributed transaction between legacy system and persistent log. How do you solve that? I will cover this in a later blog. It is not as difficult as you may think.
Do all the service requests need to be idempotent?
No. Certain requests are implicitly idempotent. For example, read requests are implicitly idempotent. Certain updates are also idempotent. For example, changing an address is idempotent - it doesn't matter how many times you process the request, end result is still the same. (think of it as an assignment statement i=2. This statement is idempotent.)
I will also cover how this topic is related to WS-ReliableMessaging in a later blog.