There is a performance problem!

  • Article

During the TechMela, i had a discussion with more than 5 people and although all of them had different questions but it all boiled down to the same problem – "the performance problem". And i overheard a few other speakers talk to customers on the same issue. When i was supporting customers developing and fixing their applications – performance issues were one of the most difficult to handle, largely because of misunderstood reasons. 90% of the work in resolving and fixing performance issues lies in tracking down the issue and pinpointing the cause. Before we stand up and start looking for answers – it is very important to have clarity around what the issue is. Here i am trying to share my experience that can possibly help developers when they seek to find a resolution to the performance issues:

  1. First make sure of what are you comparing against what? It is of almost critical importance that we compare apples to apples. Most of the time i hear that people say that it was working fast before and now suddenly it is very slow! That really does not help us help you. So what was the performance before? In seconds/minutes/transactions/cycles/etc – please choose a unit which you are measuring the performance on. Measure the "good" performance and the "bad" performance. Note down the numbers for a more than a few runs if possible.
  2. Make sure that you keep AS MUCH AS POSSIBLE of the environment constant. Apples to Apples comparison essentially means that you use the same type of hardware, OS, memory etc. if you are comparing performance on two different machines. In case of a testing two different builds or versions of the software – the recommendation is that try and run both of them on the same machine.
  3. If you are processing data from a database and you are seeing a performance problem while processing the data in two different versions of the application, make sure that you measure the performance on the same set of data, the database server is same, try and eliminate network latency for both the comparisons or at a minimum try and keep the network latency same. A common misunderstanding i have come across is that in one case the client application is in the same building as the database server and the other client application is somewhere on the other corner of the globe. Now due to any reason one of the client gives a largely different reading – Although it may not be visible, the major contributor to the difference could be the network itself.
  4. Isolate the layer which is causing the performance bottle neck – eg. the back-end, the middle-tier, the front-end. Based on where the bottle neck is different tools and troubleshooting techniques will need to be used.
  5. When comparing performance of two different versions / builds, make sure that you are comparing debug build v/s a debug build and a release build v/s a release build and that the options used for compiler and linker to make both the builds being compared are the same. Also make sure that the same versions of the compiler and linker are used in both cases.
  6. Use a profiler (eg. The inbuilt profiler that comes with Visual Studio 20005 Team System or 3rd party profilers like ANTS) to find out the bottle necks in the code. Profilers allow you to drill down to the function/ loop / line of code which is taking major chunk of the execution time.
  7. There is often a misconception that if an application is giving X performance on say 2 Ghz machine with 2 GB RAM, if you run it on a 3 Ghz machine with 3 GB RAM you should get greater than X (eg. 1.5X) performance. This is not always (usually never) true! Similarly – it is not true that the same application will run faster on a 64 bit machine v/s a 32 bit machine.
  8. A very common question when comparing performance of managed apps (.Net app) v/s native (Win32) is that which of the two would be more efficient/faster. While it is true that a 'properly' thought over, designed, architected and developed native app would beat a normal similar .Net app, but that is not the way to decide which of these two paradigms is better. Designing and developing a real world performing app in native code would take much much more time than doing the same in .Net. In my opinion the better approach (that takes care of time to market and development cost issues) is to design and develop the application in .Net and then test and profile it. If you come across bottle-necks in your application that are showing real performance issues – making your application unusable, unstable or losing to competition, then identify those bottle necks – see if optimizing them will give you benefits or if it makes more sense to write those portions in native code and have them interop with the .Net application.
  9. Usually the performance problem aggravates when the project sizes become bigger. No tool is developed to handle all scenarios and so your applications may actually be ones that pose limitations on a tool. It is always best to analyze the tool set that your team is going to use. Analyze the architecture and possible limitations that it may pose on the tool set. If possible try to select a tool that does not have that limitation or try and change the architecture to accommodate that limitation in the early stages rather hit the roadblock and then waste man-hours trying to get over it.
  10. Performance problems may have their roots in very deep, unknown locations, for example the kind of heap that is used by a library that you are using, such subtle issues may not be visible and would take a lot of digging and debugging to identify them. A methodical, patient and step-by-step approach makes it easy and fun to track the issue down.