A pattern for successful load testing: Component Load Testing

Nearly every server team at Microsoft now use Visual Studio Load Tests to test their server. I've worked with a lot of the teams, and found a common pattern that I'd recommend using.

First, load test early and often. With the cost of load testing tools today, load testing is often only done by a central team right before an app is going into production. This is too late! Many perf problems are due to architectural problems in the app, not the kind of problem that can be readily fixed just before the app is to go into production!

Second, make sure you sized your data properly. This is an obvious statement, but is often overlooked in load testing. Many perf problems only happen at large volumes of data. An important requirement to understand for a load test is how much data you'll be testing against. This is one of the most difficult parts of load testing: getting your app data populated properly. Once you do, the first set of perf problems can usually be found with no load test at all, and are due to problems in the data tier (missing indexes, too many indexes, bad query plans, etc.).

Once you have the data sized, and fix the first set of problems, you should load test components in isolation first. Go down to load testing individual transactions first, to ensure they perform well in isolation. Then create load tests to mix in additional transactions. Finally, create load tests to mix the entire set of transactions against the server.

The office team uses this technique, as does Team Foundation Server. For the office team, each of the 7 server teams run their own load tests, both component and mixes, and then they run a big test against all the servers.

This technique also works well with the central load test team model (even if the central team is using a competitor's tool). If you've worked with such a team, I'm sure there have been times when you get called in to investigate a performance problem. At that point you're working in a strange environment, sometimes locked down, and you're under the gun. Definitely not a pleasant situation! By load testing early, and load testing components, you can be sure the problem is in the configuration of the app, and not a fundamental architectural problem or a problem in your component.

So if your company has a central load testing team, don't wait to let them find your perf problems! Test early and often using Visual Studio load tests, they will save you a lot of time and trouble in the end.

Ed.