Testing Embedded Standard Systems
[The following article is authored by one of the Windows Embedded MVPs (Most Valuable Professionals). Our MVPs have a heavy background in Embedded systems and are a great repository of information on Windows Embedded products. We’re providing this space on our team blog as a service to our readers by allowing MVPs to share some of their knowledge with the rest of the community.]
It is a great feeling for any OS developer to have a nice and lean Windows Embedded Standard image up and running for the first time, after all the processing has been done during the FBA phase. What always spoils the party is the question, “How do I prove that this image fits the project requirements?”, when functionality, flexibility and security are taken into consideration. But, no worries, there is a simple answer: Testing.
Installation Test
Applications installed correctly on the image, either during FBA or in a manual phase, should naturally be running without errors. To test them, merely starting the application is not enough. Depending on the application, it is important to achieve good code coverage during such a test, because there may calls to late-bound APIs in some functionality, which might fail. Often, unit test applications that exist in the dev team can be leveraged for this.
If application logging can be set to high verbosity, it is always good to do so during the test. In addition, it is good to compare the application’s logs to those from installations running on XP Pro to detect anomalies. In most cases real errors will be quite visible, causing error message boxes, application crashes or entries to be written to the Windows event log. In addition, is it very beneficial to turn off Windows Embedded Enabling features such as “Message Box Interception” during tests or you will most probably miss quite a few errors.
Load testing
The behavior of an application or a set of applications under load is especially interesting to embedded devices that ideally need to work constantly throughout most of the known usage scenarios. It is good to pick out some sample load scenarios and to test them. For example, test if a device still can be configured via the Web UI while the data collection is busy reading in a large amount of data into a local database.
Load testing for web applications is available as a project type in Visual Studio 2008.
Regarding the load testing scenario, it is always good to not think academically, but rather to think of real world scenarios that commonly occur during normal operation. Otherwise the focus of the testing may skew towards special situations, outside the standard usage patterns.
Security
This is a sensitive topic. As we know, there is no absolutely secure device. Therefore, from a project perspective the best secure device is a device that hits the sweet spot between providing an acceptable level of security achieved with reasonable effort. This means that the security requirements can be very different depending on the environment different devices exist in. To use a standards-based approach for this challenge Microsoft offers the Baseline Security Analyzer. This application checks systems against a catalog of best practices. It also gets updated via Internet to provide the most current information.
It is a best practice to run MBSA locally with administrative rights on the device, because in the local situation no firewall or rights limitations block access for collecting information. Results are displayed in a report, and actions to establish countermeasures against security holes are provided. Most of the security best practices valid for XP Professional, for Firewall, Group Policies und user rights settings, apply to Windows Embedded Standard as well, if these features are part of the image.
Maintainability
Connected devices need to be maintainable with application and image updates. Do not rely solely on just the existence of related infrastructure such as DUA, SCCM Advanced Client or WSUS client in your image. These applications require configuration, testing, as well as the setup of backend systems to complete the solution. Assuring their seamless operation from the start is able to provide huge savings when devices are out in the field.
Backup
It may seem odd, but do you ever test to restore systems from backup sets? Well, if not, this is definitely recommended. Running into restore problems when systems have failed or are down just puts one deeper into trouble when it could be avoided with testing up front.
System Performance Counters
Talking about testing, the built-in diagnostics in Windows Embedded Standard should not be ignored as a testing tool. The system performance counters are quite useful for measuring and tracing systems in different environments and circumstances.
This list of test categories is far from complete, but provides a basic set to get started with. After reviewing project requirements, more categories and techniques may be added according to the importance of a needed feature or functionality.
Alexander Wechsler
Wechsler Consulting
Technorati Tags: XPe,Standard 2009