What To Watch

  • Article

[Housekeeping stuff.  I added categories.  Neat stuff.]

RichB asked me yesterday in the comments to my first post:

“What are the range of metric values I should expect to see for a given scenario. For example, how many gen1 collections per minute is OK for a heavily loaded web app?“

This, unfortunately, is not the kind of advice I can give you.  Rico or Jan are far more likely to have useful answers to that kind of question - they're design kind of guys.  Being a tester, my angle is more from the “how do I measure and keep track of this stuff“ point of view.

On with the show...

When I said I'd point you at some metrics to keep in mind, I was talking about these things:

  • Working set
  • Startup Time
  • Code execution speed (Throughput)

There are other considerations, but these are the big concerns.  Let's define, shall we?

Working Set

Working set can be defined as the number of pages your process is keeping in RAM.  This includes the space required for all the code and data you have loaded.  It also includes any dynamic data that your process allocates, including the GC heap. As a rule, measuring your working set does not require any changes to your code.

Startup Time

This is the time required to bring your app to a useful state.  Depending on what your particular scenario is, this is going to be a different measurement.  If you're concerned with time to Main(), as we are with the CLR, you'll write a test scenario that exits as soon as possible.  More likely, however, you'll find yourself wanting to time to when your app is loaded and responsive to the user.  This type of testing requires some instrumentation.

Code Execution Speed (Throughput)

When we're discussing Code Execution Speed, we're asking ourselves how long an arbitrary bit of code takes us.  Often times, it's useful to create benchmarks for your high priority code paths so you can track them independent of the entire product.  For example, if you're concerned about the execution speed of submitting an item to your shopping cart, you might write a console application which exercises the ShoppingCart::Add method. 

As a note, if you have a modification or clarification for these definitions, by all means let me know!  Going forward, these are what I'll be using.

That's all nice: So what?

Having laid some foundation, we can start discussing how to measure these things.  I realize that this may be starting slow for some of you.  I'm starting from the absolute ground up to ensure that I don't leave anyone behind.  If you find this stuff useful, great!  Please drop me a line in the comments.  If you're reading this and yawning, great!  You're ahead of the game on perf.  Drop me a line too.  Once I gauge who's reading and where the right level is, I can narrow in on more useful stuff.  Until then, I'm going to keep going along the basics for now.  Tommrow (Or maybe later tonight), we'll get into some useful tools for measuring the stuff I'm discussing.