Security Innovation Security Engineering Study

  • Article

The Security Innovation Security Engineering study,  Comparing Security in the Application Lifecycle - Microsoft and IBM Development Platforms Compared, is timely, given the emerging industry emphasis on integrating security in the life cycle. 

My favorite quote in the study is "The patterns & practices security guidance covers the key security engineering activities better than any other resource we’ve found."  I think this reflects the fact we have more than 2,500 pages of security guidance (see Security Guidance, Security Engineering, Threat Modeling, and Improving Web Application Security) , and we've integrated our guidance into MSF/VS 2005 (see MSF/VS 2005 and p&p Integration.) 

The study was available from the MSDN Security DevCenter for a while but seems to have fallen off.  I've summarized the study here for quick reference:

Overview
Security Innovation evaluated the guidance and tools of Microsoft's and IBM's development platforms.  The study compared the support available to a development team via security guidance, documentation and security focused features in the life-cycle tool suites.  Gartner reviewed the approach.

Evaluation Criteria

  • Coverage.  How well do the provided tools and guidance cover the key set of security areas?  
  • Quality.  How effective and accurate are the tools and guidance?
  • Visibility.  How easy is it to find the tools and guidance and then apply it to your security needs?
  • Usability.  Are the tools and guidance precise, comprehensive and easy to use?

Ratings

  • Outstanding: 81-100%
  • Good: 61-80%
  • Average: 41-60%
  • Below Average: 21-40%
  • Poor: 0-20%

Scorecard Categories

  • Basic Platform Security.  When used in accordance with its documentation, a platform should be inherently secure.
  • Platform Security Services.  A mature platform should include services that make it easier for developers to implement security features in their applications.
  • Platform Security Guidance. A secure platform is much less useful if it lacks proper guidance.
  • Software Security Engineering Guidance.  It is not possible to develop a secure application unless security is a focus during every phase of the development lifecycle.
  • Security Tools.  A secure platform should include tools that make it easier to define, design, implement, test, and deploy a secure application.

Results of the Study

First, here's a couple key points, then the summaries are below:

  • Microsoft beat IBM in every category around guidance.
  • Microsoft beat IBM in three out of four categories around tools.

IBM

  1. Platform Overall
    1. Overall: 36%
    2. Coverage: 62%
    3. Quality: 70%
    4. Visibility: 17%
    5. Usability: 72%
  2. Platform Security Guidance
    1. Overall: 50%
    2. Coverage: 81%
    3. Quality: 85%
    4. Visibility: 17%
    5. Usability: 84%
  3. Security Engineering Guidance
    1. Overall: 25%
    2. Coverage: 50%
    3. Quality: 64%
    4. Visibility: 17%
    5. Usability: 69%
  4. Security Tools
    1. Overall: 32%
    2. Coverage: 55%
    3. Quality: 59%
    4. Visibility: 56%
    5. Usability: 63%

Microsoft

  1. Platform Overall
    1. Overall: 67%
    2. Coverage: 88%
    3. Quality: 85%
    4. Visibility: 61%
    5. Usability: 80%
  2. Platform Security Guidance
    1. Overall: 76%
    2. Coverage: 93%
    3. Quality: 85%
    4. Visibility: 67%
    5. Usability: 91%
  3. Security Engineering Guidance
    1. Overall: 78%
    2. Coverage: 100%
    3. Quality: 89%
    4. Visibility: 67%
    5. Usability: 79%
  4. Security Tools
    1. Overall: 47%
    2. Coverage: 71%
    3. Quality: 78%
    4. Visibility: 50%
    5. Usability: 68%

Quotes from the Study

  • Microsoft’s overall rating of 67% reflects the impressive level of focus Microsoft has applied to application security in the past several years.
  • IBM’s overall score of 36% is the result of a more disjointed approach to security. Security guidance is spread throughout the IBM web site and is difficult to discover.
  • The patterns & practices security guidance covers the key security engineering activities better than any other resource we’ve found.

More Information
For more information, see Comparing Security in the Application Lifecycle -
Microsoft and IBM Development Platforms Compared at Security Innovation's site.  They created four documents that take you through the details and results: Executive Summary, Research Overview, Full Detailed Reports and Results, and Methodology.