Tuesday at CMG 2004 – "A Performance Process Maturity Model"

My favorite talk at CMG yesterday was “A Performance Process Maturity Model”, given by Michael Maddox of MCI. You’ve probably already heard of the Capability Maturity Model (CMM), which is used extensively in the software industry to measure the “quality” of an organization in terms of their development processes. Michael was inspired by a talk by Dr Connie Smith to propose a similar model that measure the ability of an organization to develop and run performant software. In other words, how well can you write code that runs fast enough for your business needs? Do you get it right every time? If not, why not?

Michael defines five levels for his Performance Process Maturity Model (PPMM) – I’ll try to summarize them here, but he has much more complete definitions in his paper, comparing them across multiple criteria. As with the CMM, higher is better!

  1. Firefighting — you do ad-hoc analysis and fixing of performance problems. This is typical of small organizations.

  2. Monitoring — you collect performance data from your line-of-business applications 24×7, and receive alerts when performance out of bounds. You do some capacity planning, but it’s often fairly simplistic. This is typical of larger organizations that use monitoring applications such as MOM.
    Michael defines three additional sub-levels of level 2, depending on whether the response times seen by users are being collected as part of the performance data:

    • You don’t measure response times — you just care about your utilization and throughput.

    • You measure response times from commercial off-the-shelf software.

    • You measure response times from all software, including your own.

  3. Performance Optimizing — you are pro-active about software performance. Your organization uses software performance engineering throughout the application development lifecycle (i.e., they don’t just start tuning an app to fix performance problems one week before release), and does comprehensive capacity planning for individual applications.

  4. Business Optimizing — you understand how relevant the performance of particular applications are to business success (i.e., which ones should you spend your time on?). An enterprise architecture is emerging in your organization, unifying the separate applications and processes.

  5. Process Optimizing — you can now improve and optimize all the processes involved in steps 1-4. Your organization has a unified enterprise architecture that is aligned with its business needs.

One of the nicer aspects of Michael’s talk was that he talked a lot about the political and inter-personal issues involved in moving an organization up a level, from the different types of players involved (management, development, and operations) to the level of management buy-in and expertise that’s required. I liked the realism displayed here!

The talk ended with what was essentially a call to action, where Michael outlined his plans for the future of the Performance Process Maturity Model. He’s starting with an exhaustive specification of the performance activities for each level at each stage of the application life-cycle, and he’ll use the CMMI and ITIL capacity-management documentation as templates (meta-models) for this specification. Next he’d like to add a quantitative document that would justify the costs involved for a software organization to change its processes and increase its PPMM level. And eventually he’d like to add certification processes, similar to those of CMM, for vendors of commercial off-the-shelf software, and possibly get the entire thing proposed as an ISO or IEEE standard.

These are huge goals, but Michael clearly has support from inside MCI. And if he achieves even half of what he’s laid out it’ll be a big step forward for the performance engineering field, so it’s not surprising that he won a best-paper award at CMG. I’ll see if I can dig up more details when I get back to the office tomorrow. In the meantime, think about your own organization. Where do you come on the PPMM? How could you do better? What’s stopping you?

Comments (1)

  1. <p>I did a conference tour the past days. It started off with Usenix’ <a href="http://www.usenix.org/events/worlds04">Workshop on Real, Large Distributed Systems</a> on Sunday. I saw a few reasonable papers but I was mainly surprised that for many in the research world real & large still appears to be <a href="http://www.planet-lab.org/">planetlab</a&gt;. Which is a selection of 200 (or 400, depending on who you talk to) nodes placed at privileged places around the world. It is certainly not my definition of large and it is real insofar it exists, but it is far from my reality. The lone voice that somewhat resonated with my ideas was <a href="http://www.sun.com/executives/perspectives/papadopoulos.html">Greg Papadopoulos</a>, Suns CTO, he was on the final panel together with <a href="http://www.isi.edu/~carl/">Carl Kesselman</a> from the Grid Forum and <a href="http://www.cs.princeton.edu/~llp/">Larry Peterson</a> from Princeton who leads the PlanetLab effort. Greg rightly observed that both the other panel members are pushing for more and more complex technologies, without any guarantees that these will result in systems that can function better at large scale in the real world, or even solve relevant problems in terms of real scale.
    I had to miss most of <a href="http://www.usenix.org/events/osdi04">OSDI</a&gt;, which followed WORLDS, because I had promised to give presentation at the <a href="http://www.cmg.org/conference/cmg2004/">Computer Measurement Groups annual conference</a>. This was the last of my Cornell commitments that carried over into Amazon.com time. The CMG audience is very targeted on tuning, capacity planning, etc. and quite a few are die-hard industrial professional, still involved with running mainframes. Very interesting folks and I enjoyed talking with them about their problems. I am not sure whether they actually appreciated my talk on web-services which seems somewhat remote from their core areas. </p><p>
    There was one thing that was a new experience for me. The attendees fill in little slips with speaker feedback. I thought they went back the organizers, but that is not the case. The room monitor collects the feedback forms and gets them to the speaker, so I ended up with 40-50 little notes with feedback about the talk or my presentation skills. Very useful.</p><p>
    <p><a href="http://blogs.msdn.com/jonathanh/">Jonathan Hardwick</a> has a number of excellent <a href="http://blogs.msdn.com/jonathanh/archive/2004/12/08/278612.aspx">posts</a&gt; from the CMG conference</p><p>
    In the evening I strolled around Las Vegas a bit, rode the bus to the north end of the strip and back to the south end. After I recovered from image overload that all the big casinos shoot at you, I am surprised how obvious the human misery is that seems to be the underlying current in Las Vegas. It is so obvious that it is all an illusion that it makes you wonder why we actually expose ourselves to it. And just outside the main casino area the presence of poverty and high-crime are so pervasive that the contrast is huge. But even though I was happy to leave town today, I do want to come back to take a bit longer look at it.</p>