Obtaining Enterprise Architecture Metrics - Part 2

In my last post I talked about "keeping EA metrics on the trail" and what you need to think about when gathering the right metrics. By understanding what you need and pitfalls to avoid including common mistakes when gathering metric and tips on what you should be looking for when gathering these metrics. I will now talk about some ways in which you can figure out the value of your organization.

The problem you will have is that there isn't a prescriptive way you can measure your organization. There are the following factors:

• EA Charter - Not all EA organizations are defined in the same. Since there is little definition of what an EA organization should be there is a degree of  variances EA responsibilities per each organization.  
• Organizational Culture - Some cultures such as the ones within software vendors have a culture of being innovative, these cultures will push back on the slightest bit of governance. However, if you look at a bank they encourage more structure due to their culture being risk adverse in nature to run their business.
• External Forces - Reactions to the market or regulatory concerns often effect what EA's do. Be prepared to adjust your metrics.

Since we have many factors that go into this, I am going to be somewhat generic but try to give you some solid all-purpose guidance for obtaining metrics. Bare in mind, keep your ego out of the metrics discussion. You want to prove the value of enterprise architecture and not the value of Joe Smith who is an EA.

If we look back at the previous post we want to look at the three data points EA's should gather.

• Obtain the Information
• Process
• Communicate

First is to obtain the right information. But what is the right information? Should we be focusing on how to align this with the business or with IT activities? The answer to this is that it depends... It depends because these answers hinge on the maturity of your EA and organization as a whole. However, in most organizations we are on the slightly less mature side so that is what I will focus on.

EA organizations rarely report into a CEO or Business unit. There are a few companies out there that do but this ideal state is something that most business do not do. On a side note, this is unfortunate because this is ideal for EA's to report to the CEO rather than the CIO. Most EA organizations report to the CIO or below. This has implications on EA's. What it means is that your in the IT organization and at the end of the day you are measured not on the business but on the health of your IT organization. So to go back to the questions posed, you should focus on IT activities when building metrics. Does this mean you should exclude business alignment, absolutely not! However you should realize that you are ultimately accountable and responsible for IT, not the business.

Now that we have set the stage a bit,  what are some metrics you can obtain.

1. Tools are Essential -   The tools used can be simple to advanced. The key here is to have mechanisms to collect and integrate into other enterprise solutions. I've seen Excel and Word used to manage EA with a great deal of success but also with COTS based solutions as well. The key is to do what is right for your situation. Your tools should integrate into key repositories such as:
   • Application Portfolio Management
   • Business Process or Capability Repositories
   • Configuration Management and Ideally Service and Incident Management
   • Etc.
2. Architecture Decisions - It is a must to be able to document the decisions made. This needs to be a formal process in which all changes are documented and quantified. This is similar to change control but for the architecture design process rather than implementation.  But what does this mean exactly? It means that for each architecture problem you should have multiple potential architecture solution options. Each of these options have tradeoffs. Examples of data element that are included per architecture decision are:
   • Architecture Issue
   • Assumptions
   • Business Issue or Requirement
   • Potential Architecture Options
     • <#>
     • Solution Drivers
     • Risks and Implications
   • Decision
   • Implications of <#>
3. Aid on Mission Projects - Be humble and aid mission critical projects. This gives EA tangible metrics on your involvement in the success of a key business enable project. Similar and to a degree some of the same elements but at a higher level is to gather information on how you helped make the project successfully. To do this you need to quantify you architecture decisions. For Example, I formally recommended the following that was not in the current project designs:
   • Move to VM's and Blades - the result was lower electric and ventilation required in the data center, increased availably ratings of the applications, lowered the cost of disaster recovery and dramatically reduced infrastructure support staff. This resulted in a <x> million dollar cost savings.
4. Integration into Application Portfolio Management - Have an administration function that is able to monitor and track the health of the organization by taking snapshots of in time of the environment as a whole. By doing so you can track the gradual success of your EA efforts by aligning the work done on projects and IT optimization efforts.
5. Satisfaction Ratings - Solicit feedback whether it is good, bad and indifferent feedback. It should should be collected and acted upon. When you do a great job on that strategic project and the business sends you a happy gram find a way to put metrics on feedback to create satisfaction ratings. There should also be tools where people can submit anonymous comments and ratings on your performance. This can quantify the need for EA as well be demonstrating the need by the business and IT. Creditability is critical here and you must ensure to be honest with these. Even if there is a poor perception of your activities. Publish it no matter what.
6. Risk Management Ratings - Work with your Information Security and Risk Management teams to find ways to quantify how you are helping to build less risky and more secure architectures. Start simple and evolve if you do not currently have something to do this.

Second is to integrate into existing processes. You want to ensure that it isn't too intrusive to the process. Remember too much process is a bad thing. The key here is to gather key metrics from these processes. There are a ton of great metrics from process out there such as:

• ITIL or Microsoft Operations Framework (MOF) - There are some great books on the topic of ITIL metrics that you can check out. In regards to MOF there are a ton of out of the box metrics and KPI's that can be leveraged, see below:

• COBIT and SOX Processes - Likewise with these processes there are a ton of metrics that can be extracted.
• Correlate Process Metrics - Once you integrate into these process, either homegrown or acquired. You must now correlate these in a meaningful way. This takes a bit of creativity, however once you are finished it will pay off big. Just by performing this type of effort it can show immediate value to both the business and IT.
• Repeatability - Like stated in the previous post, you need to ensure that this process is a repeatable processes. By not doing this your metrics become circumstantial and invalid.

Lastly, communicate frequently. Surfacing this information is critical because what you want is awareness of your EA efforts. Overall perception is important to your organization. You should be viewed as a change agent in the organization. Tools that you can use to do so are:

1. Portals - Used for general purpose EA presence and communication. Also used for hosting of EA types of applications.
2. Forums - These are great for Q&A and keeping a dialog with the community.
3. Wiki's  - For project level activities wiki's provide yet another communication mechanism for EA's
4. Blogs - These should be used for personal opinions that document the bloggers perspective rather than a dialog like with Forums.
5. E-Mail - One way communications, be careful with this mechanism as you could be flagged as spam and ignored. I'm not a big fan of e-mail being used as the primary communication.

Each one has a specific purpose and should be leveraged for their strengths. Even though these mechanisms are "soft metrics" they can still provide you with valuable ways to quantify your efforts.

• Portals - Gather site visits and downloads of content to gauge EA interest and usage. By gathering stats from EA applications you can show a direct correlation with usage patterns.
• Forums - Measure end user interest in EA activities with user submission statistics. This is a great incentive mechanism to reward the architects in the community.  
• Wiki's  - Demonstrate your effectiveness on projects. You can correlate how significant your activities are on projects with tracking of tasks and submissions on Wiki's. This can be linked to architecture decisions and project planning tools as well.
• Blogs - Measure end user interest in EA activities with user submission statistics.

In summary, shown above are practical and realistic metrics you can use to quantify and qualify your EA efforts. Even though we didn't focus on the business directly, they benefits can be articulated in business terms as well. Once they see the successes you have achieved in IT, then and only then will you be a trusted partner to aid them directly on their efforts.

Like all EA organizations stat small and work in an iterative fashion. These activities are  rarely perfect out of the gate. However keep your eye on the future state and build or buy the tools that are needed to track these metrics.

Also keep your eyes on the organizational aspects such as:

1. Make it Public -   Ensure transparency in the process to prevent those "conspiracy theorist's" in the organization from damaging your metric gathering activities.
2. Identify Influential Stakeholders - Address their needs first, do not try to boil the ocean and provide generic metrics to address everyone. This just provides everyone with information that isn't really useful.
3. Identify Your Charter and Execute on it - Do not go beyond the scope of your EA charter. You can use the metrics acquired to expand your charter however.
4. Make it Tangible - Avoid selling your EA mission statement but rather how you are helping the community as a whole through what they already know.
5. Perspective based Communication - Since there are many ways to look at architecture, communicate in terms your audience understands. Additionally, you can communicate the metrics in different ways to varying roles as well.
6. Avoid "Selling" EA - Do not try to sell these metrics, merely post the good, bad and the ugly.

Tags: Enterprise Architecture Architecture Repositories