The Performance War -- Win it 5% at a time

If it feels like getting good performance out of your application/library/service/whatever is more like "trench warfare" than it is like "shock and awe" then you're probably doing something right.

The trouble with performance work is that the easy work gets all the press.  Well, sort of.  Let me explain. 

Suppose you're an elite performance engineer. Sometimes a big ugly nasty problem gets dropped in your lap. When that happens you roll up your sleeves, apply some good techniques, and maybe you net a nice fat win for your clients.  Everyone cheers and they all rush to the store buy a superhero cape just like the one you wear.  Children ask for your autograph.  It's good times for everyone. 

OK well, maybe I'm getting a little carried away.

The thing is when something like the above happens probably you shouldn't be celebrating at all.  Giant performance wins that could not be achieved without the help of a superhero are usually a sign that something has gone terribly wrong in the design process.  Perhaps the team in question left their performance work to a point that was too late in the cycle.  Perhaps they had some basic flaws that had to be remedied, flaws that never should have crept into the codebase in the first place.  But the fact that some "superhero" was able to come along and significantly fix them points more towards mistakes in the original code than it does to any greatness on the part of the hero.

Still, sometimes that's the gig and so you do it.

Now the "real" performance work, the stuff you should be proud of because it's hard, is much less glamorous.  Basically it happens by carefully understanding your whole process, avoiding any big problems (so that they never require a superhero to come along and fix), and steadily working on the most important areas slowly but surely. 

In a mature product with a healthy process you're much more likely to see a 50% gain come in the form of many 5% gains compounding to get to your goal via sustained effort and quality control.  Those wins are largely unsung but they are the hard ones.  They are the wins that give you headroom for your new features and convert your older features from sluggish to snappy.  Every one of them is hard work.

The tragedy is that more care and effort often goes into any one of those fixes than one superhero action, but the capes get all the good press.

Huge instant performance wins are more often a sign of problems than they are of greatness.