Distributed Architecture Drawbacks Revealed By Netmon(Bonus - TDS Parser Goes Public)

  • Article
 Alik Levin    Distributed architecture can mercilessly backfire at you.  In my case flexible architecture, elegant design patterns, and smart code led to abuse of the flexibility, resulting in very poor performance. Free Microsoft Network Monitor (Netmon) helped to identify the root cause of the Distributed Architecture Hits Bac  
performance hit. It showed that over-distributed-ness can cost you in terms of performance.

Customer Case Study

The customer complained about poor response times in his web application. The application's architecture was similar to the Web Application Archetype. Notice Services Agent box that connects your application to downstream services? Our assumption was that the services agents are too chatty causing the performance hit. Netmon only made it clear.

Analysis

We took captures on the application server where the Server Agents are to identify what other downstream servers are accessed. In no time we get very clear picture - the application server was accessing other three downstream resources:

 netmon distributed architecture

There are two well known ports - 443 and 1433 - so we could safely assume there is communication over SSL/HTTPS and SQL Server. The other one - 1414 - turned out to be MQ.

Next step was to identify which one of the conversations is causing us troubles the most - either by chatty communication or by just long running transaction.

Looking at Time Delta column for HTTPS stateless communications we found nothing exciting regarding the latency:

HTTPS Communications network monitor

For MQ communications we used magic ContainsBin(FrameData, 0,"StringToFindGoesHere") to identify XML messages going back and forth over MQ transport. For example, to find the beginning of the XML message we used the following filter:

Netmon ContainsBin

Similar technique was used to correlate request and response XML messages.

To identify SQL communication we used shiny new TDS parser available for free download on Codeplex - SQL Parser in Latest CodePlex Package. I particularly like this one, it shows SQL Server communication without using SQL Server Profiler:

 netmon tds parser

Conclusion

Lessons learned:

  • Distributed Architecture can mercilessly backfire at you by its distributed-ness. Use Distributed architecture to solve problems vs introducing new ones.
  • Elegant and flexible design can be abused resulting in poor performance. Review code to avoid the abuse.
  • Low level network monitoring tools such as Netmon can quickly reveal over-distributed-ness. Use it! It is free.

Related Materials

This post is made with PracticeThis.com plugin for Windows Live Writer