Tuesday at CMG 2004 - "A Performance Process Maturity Model"

  • Article

My favorite talk at CMG yesterday was “A Performance Process Maturity Model”, given by Michael Maddox of MCI. You've probably already heard of the Capability Maturity Model (CMM), which is used extensively in the software industry to measure the “quality” of an organization in terms of their development processes. Michael was inspired by a talk by Dr Connie Smith to propose a similar model that measure the ability of an organization to develop and run performant software. In other words, how well can you write code that runs fast enough for your business needs? Do you get it right every time? If not, why not?

Michael defines five levels for his Performance Process Maturity Model (PPMM) - I'll try to summarize them here, but he has much more complete definitions in his paper, comparing them across multiple criteria. As with the CMM, higher is better!

  1. Firefighting — you do ad-hoc analysis and fixing of performance problems. This is typical of small organizations.
  2. Monitoring — you collect performance data from your line-of-business applications 24x7, and receive alerts when performance out of bounds. You do some capacity planning, but it's often fairly simplistic. This is typical of larger organizations that use monitoring applications such as MOM.
    Michael defines three additional sub-levels of level 2, depending on whether the response times seen by users are being collected as part of the performance data:
    • You don't measure response times — you just care about your utilization and throughput.
    • You measure response times from commercial off-the-shelf software.
    • You measure response times from all software, including your own.
  3. Performance Optimizing — you are pro-active about software performance. Your organization uses software performance engineering throughout the application development lifecycle (i.e., they don't just start tuning an app to fix performance problems one week before release), and does comprehensive capacity planning for individual applications.
  4. Business Optimizing — you understand how relevant the performance of particular applications are to business success (i.e., which ones should you spend your time on?). An enterprise architecture is emerging in your organization, unifying the separate applications and processes.
  5. Process Optimizing — you can now improve and optimize all the processes involved in steps 1-4. Your organization has a unified enterprise architecture that is aligned with its business needs.

One of the nicer aspects of Michael's talk was that he talked a lot about the political and inter-personal issues involved in moving an organization up a level, from the different types of players involved (management, development, and operations) to the level of management buy-in and expertise that's required. I liked the realism displayed here!

The talk ended with what was essentially a call to action, where Michael outlined his plans for the future of the Performance Process Maturity Model. He's starting with an exhaustive specification of the performance activities for each level at each stage of the application life-cycle, and he'll use the CMMI and ITIL capacity-management documentation as templates (meta-models) for this specification. Next he'd like to add a quantitative document that would justify the costs involved for a software organization to change its processes and increase its PPMM level. And eventually he'd like to add certification processes, similar to those of CMM, for vendors of commercial off-the-shelf software, and possibly get the entire thing proposed as an ISO or IEEE standard.

These are huge goals, but Michael clearly has support from inside MCI. And if he achieves even half of what he’s laid out it'll be a big step forward for the performance engineering field, so it's not surprising that he won a best-paper award at CMG. I'll see if I can dig up more details when I get back to the office tomorrow. In the meantime, think about your own organization. Where do you come on the PPMM? How could you do better? What's stopping you?