@Stake Security Study: .NET 1.1 vs. WebSphere 5.0

  • Article

I like competitive studies.  I'm usually more interested in the methodology than the outcome.  The methodology acts as a blueprint for what's important in a particular problem space. 

One of my favorite studies was the original @Stake study comparing .NET 1.1 vs. IBM's WebSphere security, not just because our body of guidance made a direct and substantial difference in the outcome, but because @Stake used a comprehensive set categories and an evaluation criteria matrix that demonstrated a lot of depth.

Because the information from the original report can be difficult to find and distill, I'm summarizing it below:

Overview of Report
In June 2003, @Stake, Inc., an independent security consulting firm, released results of a Microsoft-commissioned study that found Microsoft's .Net platform to be superior to IBM's WebSphere for secure application development and deployment.  @stake performed an extensive analysis comparing security in the .NET Framework 1.1, running on Windows Server 2003, to IBM WebSphere 5.0, running on both Red Hat Linux Advanced Server 2.1 and a leading commercial distribution of Unix..

Findings
Overall, @stake found that:

  • Both platforms provide infrastructure and effective tools for creating and deploying secure applications
  • The .NET Framework 1.1 running on Windows Server 2003 scored slightly better with respect to conformance to security best practices 
  •  The Microsoft solution scored even higher with respect to the ease with which developers and administrators can implement secure solutions

Approach
@stake evaluated the level of effort required for developers and system administrators to create and deploy solutions that implement security best practices, and to reduce or eliminate most common attack surfaces.

Evaluation Criteria

  • Best practice compliance.  For a given analysis topic, to what degree did the platform permit implementation of best practices?
  • Implementation complexity.   How difficult was it for the developer to implement the desired feature?
  • Documentation and examples.  How appropriate was the documentation? 
  • Implementor competence.  How skilled did the developer need to be in order to implement the security feature?
  • Time to implement.  How long did it take to implement the desired security feature or behavior? 

Ratings for the Evaluation Criteria

  1. Best Practice Compliance Ratings
    1. Not possible
    2. Developer implement
    3. Developer extend
    4. Wizard
    5. Transparent
  2. Implementation Complexity Ratings
    1. Large amount of code
    2. Medium amount of code
    3. Small amount of code
    4. Wizard +
    5. Wizard
  3. Quality of Documentation and Sample Code Ratings
    1. Incorrect or Insecure
    2. Vague or Incomplete
    3. Adequate
    4. Suitable
    5. Best Practice Documentation
  4. Developer/Administrator Competence Ratings
    1. Expert (5+ years of experience
    2. Expert/intermediate (3-5 years of experience)
    3. Intermediate
    4. Intermediate/novice
    5. Novice (0-1 years of experience)
  5. Time to Implement
    1. High (More than 4 hours)
    2. Medium to High (1 to 4 hours)
    3. Medium (16-60 minutes)
    4. Low to Medium  (6-15 minutes )
    5. Low (5 minutes or less )

Scorecard Categories
The scorecard was organized by application, Web server and platform categories.  Each category was divided into smaller categories to test the evaluation criteria (best practice compliance, implementation complexity, quality of documentation, developer competence, and time to implement).

Application Server Categories

  1. Application Logging Services
    1. Exception Management
    2. Logging Privileges
    3. Log Management
  2. Authentication and Access Control
    1. Login Management
    2. Role Based Access Control
    3. Web Server Integration
  3. Communications
    1. Communication Security
    2. Network Accessible Services
  4. Cryptography
    1. Cryptographic Hashing
    2. Encryption Algorithms
    3. Key Generation
    4. Random Number Generation
    5. Secrets Storage
    6. XML Cryptography
  5. Database Access
    1. Database Pool Connection Encryption
    2. Data Query Safety
  6. Data Validation
    1. Common Validators
    2. Data Sanitization
    3. Negative Data Validation
    4. Output Filtering
    5. Positive Data Validation
    6. Type Checking
  7. Information Disclosure
    1. Error Handling
    2. Stack Traces and Debugging
  8. Runtime Container Security
    1. Code Security
    2. Runtime Account Privileges
  9. Web Services
    1. Credentials Mapping
    2. SOAP Router Data Validation

Host and Operating System Categories

  1. IP Stack Hardening
    1. Protocol Settings
  2. Service Minimization
    1. Installed Packages
    2. Network Services

Web Server Categories

  1. Architecture
    1. Security Partitioning
  2. Authentication
    1. Authentication Input Validation
    2. Authentication Methods
    3. Credential Handling
    4. Digital Certificates
    5. External Authentication
    6. Platform Integrated Authentication
  3. Communication Security
    1. Session Encryption
  4. Information Disclosure
    1. Error Messages and Exception Handling
    2. Logging
    3. URL Content Protection
  5. Session Management
    1. Cookie Handling
    2. Session Identifier
    3. Session Lifetime

More Information
For more information on the original @stake report, see the eWeek.com article, .Net, WebSphere Security Tested.