Mark Tomlinson shared an emerging industry practice with me. Customers are setting up incremental environments. The environments are incremental steps from a developer environment to production.
- Component-level performance testing. (close to dev) The lab is setup with debuggers and profilers – anything a developer would need to investigate issues.
- Application performance testing. A single “slice” of architecture, good for scale-up and optimization/tuning); usually dedicated for optimization or tuning of a single application/system; still have debuggers and profilers setup.
- Performance integration. This is still the basic “slice” of architecture, but now bring into play other applications or systems; usually has multiple applications and supporting technologys that mutually get loaded (e.g. IIS and AD); network diagnostic tools and debuggers may be used here sometimes.
- System performance and stress. Larger performance testing with scale-out scenarios, load balancing, failover; larger sized systems get more load – so you see more stressing of entire system resources, esp. network; often just for 1 application, but also for multiple integration testing.
- Large-scale integration and performance. Multiple applications, with everything needed to prove business needs will be met; usually without some security and perhaps some 3rd-party integrations; usually not a stress testing environment – e.g. virtual users are set to generate real-world pacing and load.
- Pre-production simulation. This is just like the real thing – full sized system, with full security and network topology (only not production); used both for internally built applications, and 3rd-party solutions which must be integrated; production repro’s, patches, fixes, etc can be tested here safely.
There’s no strict rule for how many of each type of environment, and the most sohpisticated setup has multiple physical environments/labs which could be used for any of each purpose. The beauty of this approach is that instead of having a great big wall to throw your application over, it’s a series of incremental hurdles. Each hurdle represents increasing requirements and constraints.
This approach is also great for Centers of Excellence. A Center of Excellence team can build the environment to reflect and codify their practices. The Center of Excellence team can also harvest and share the lessons learned to help teams over each incremental step.