Application Compatibility Testing — Overview


Delivering a new release of Windows includes a major effort to insure that applications continue to function as well on the new release as they have on the previous release. At the PDC we talked about some of the new areas of Windows Vista that reduced this level of compatibility, such as changes we made around the OS security model. With Windows 7 we renewed our engineering efforts to maintain compatibility. As with device testing, compatibility testing is an effort that spans the entire engineering organization, though we also have a group that is dedicated to this effort. This post was authored by a set of folks and coordinated by Grant George, the corporate vice president for testing in the Windows Experience team. –Steven


We have taken a very proactive approach to Application Compatibility and our process starts from the moment we first plan our product schedule and design and check in code for Windows and runs through all of our engineering processes and disciplines leading up to our final release (and beyond).


Application Compatibility Goals for Windows 7


Our main Application Compatibility goal for Windows 7 is to make sure that most all applications which work on Windows Vista will continue to work seamlessly on Windows 7. We do have to be careful about making this claim to be universal because there is a class of applications that are always updated in tandem with a new Windows release. These applications are primarily system utilities, diagnostics, and security software—the common thread is that they make assumptions about the underlying implementation of Windows internals and thus require updates. We carefully coordinate with a large set of ISVs who provide these applications. This was talked about earlier this month as we announced our Ecosystem Readiness Program, which we will discuss more below.


At the start of our product cycle we review our new features and changed designs to ensure that every element of Windows 7 has Application Compatibility in mind. Our engineering process includes automated quality checks to assure public APIs don’t change, and our test engineers have the right tools, engineering time and information that is used to find application issues as early as possible in our development cycle. Telemetry information is collected to assess and prioritize the breadth of applications our users depend on, paired with market data and install base information, across a wide variety of software categories to make sure they work as expected in our new OS version.


Below we expand on how we deliver on this goal.


Engineering Application Compatibility


Rich telemetry from external usage and the broad software marketplace helps us to provide a list of the most popular and critical applications, updated frequently throughout our development cycle. Our engineers then acquire these applications and build automated tests that verify each one works as expected on Windows 7.


Changes that could impact application compatibility are followed up closely, for example Legacy Code Removal which involves removing code that existed in the previous release of the product is strictly guarded and tightly managed and should not happen without proper documentation and engagement with the impacted parties. For example if we need to deprecate an API call, the documentation for this API will be updated and we will wait until the impact of the removal is minimal as indicated by telemetry data unless it is required sooner to fix a security issue.


Throughout the development process we are running tests in the background creating an ongoing validation of new code relative to application compatibility. As code is getting ready to be checked into the main build, if a compatibility failure is detected in an automated regression test the checkin is halted. At that point the code is scanned for known compatibility issues and if an issue is detected the developer is asked to fix the problem. Of course we also develop new tests throughout the course of developing Windows 7 in order to broaden our coverage of third party software.


Engineering & Testing Strategy


We have several teams dedicated to application compatibility as part of the Windows development effort. These teams provides guidance on how to build in application compatibility, provides data on application dependencies, and information on what applications may be impacted for any particular change we make in the Windows platform. These teams also reviews new feature designs, as well as other planned changes, to ensure that the engineering team has fully taken application compatibility into account so that we achieve the tenet of “keeping applications working on Windows 7”.


In addition to working with the internal Windows engineering teams, we also reach out to 3rd party developers writing Windows applications to ensure these partners have all the information they need to make their solutions fully compatible with Windows Vista and Windows 7. Furthermore if we do uncover any issues that may need to be resolved by the 3rd party developers, we collect all the information, resources and guidance and engage in a conversation with those external developers to help them understand and fix these issues.


We recently announced the Windows 7 Ecosystem Readiness Program. This program provides partners with access to Windows 7 builds and tools they need to test solutions for Windows 7, Windows Vista, Windows Server 2008 and Windows Server 2008 R2. The Ecosystem Readiness Program also facilitates testing multiple components of the ecosystem together to improve the overall user experience. Rather than just focusing on getting a specific OEM product, software application, or hardware device certified, we will be bringing multiple components together to verify a rich user experience that delivers quality, reliability, and performance as well as innovation through new feature adoption.


As mentioned earlier we use telemetry data and market share information to choose the applications we directly test on as part of our compatibility efforts, below you can find the a sample of the Consumer Scenarios we focus our testing on:



  • Communications

  • Gaming

  • Fundamentals like Setup, Security and Performance

  • Memories

  • Music

  • Productivity

  • TV/Movies

  • Data Backup/Security

  • Mobility

  • Financial Management

To view a full list of Application Categories covered please take a look at the Appendix section at the end of this post. While we would love to provide the detailed list of products, several of the sources of this information are based on proprietary research from third parties.


Another very important set of technologies we test on are the middle tier technologies like Java, the .Net Framework, etc. to make sure the applications that use these technologies continue working as expected


In addition to 3rd party stand alone applications we test a subset of OEM pre-installed software and their inbox applications for compatibility. The software tested come from the engagements we have with our OEM partners and their submitted installation images. These images are tested on clean installations of Windows 7 and upgrades from Windows Vista on OPEM standard hardware. This level of coverage allows us to best replicate the initial experience with Windows 7 for many of our customers. Because many of these applications are closely aligned with the OS, hardware and drivers, it is not unusual for an OEM to provide updates to this software with a new OS release.


In addition to the above mentioned testing approaches, Microsoft IT maintains a software portfolio of approximately 1,500 applications. These applications must be tested prior to software deployments inside Microsoft.


Microsoft IT developed an application-tracking method that simplified the process of selecting applications for sample-based testing. By identifying groups of applications that have similar data processing, controls, underlying technology, and methods, Microsoft IT is able to test approximately 4 to 6 percent of the total applications and gain a reasonable assurance of compatibility for all.


For more information you can read the LOB Application Compatibility Technical White Paper.


Validation Strategy


Part of our testing process includes the creation of scenarios that we validate on 3rd party applications. We approach this by verifying the intended functionality of the application while focusing some of our attention to changes in the OS, new functionality and risky integration areas. Manual and automated test passes are scheduled to cover the identified scenarios and to verify the user experience. We cover the applications on different sets of hardware and make sure that that we cover a lot of different configurations, x86, x64, Intel, AMD, touch and multi -touch machines, etc.


We use specific categories when measuring the compatibility of the applications we test on:



  • Excellent – No compatibility problems

  • Poor – Minor functionality is not working

  • Failure (regression) – It used to work in Windows Vista now it is not working properly

  • Failures (non-regression) – The application does not work on Windows 7 but it did not work on Windows Vista either

  • Accepted Regression – We know of changes in the OS that may affect the applications and we have/are working with ISVs on mitigations for these issues. A simple example is version number. An application that is hard coded to only work on specific versions and it doesn’t know what our new operating system version is would fail.

We will cover these categories in more detail in subsequent blog posts about how we manage application compatibility.


As part of our testing we do find issues that may need to be resolved by the 3rd party developers at the companies that develop and sell these applications. We collect all of the relevant technical information, resources and guidance and engage in a conversation with those external partners to help them fix the issues. Of course we also engage them in our technical beta programs so they can test Windows 7 compatibility with their products at the same time we do.


Final Word


Application compatibility is very important to the Windows team. We are constantly working to improve your experience with applications as you move from one release of Windows to the next. We encourage you to try our Windows 7 beta release to experience the improvements in the application compatibility space and we want to hear your feedback.


It is worth mentioning the work we have done from an end-user perspective to assist in application compatibility. Many failures of application compatibility happen at install time and to assist with this, in Windows 7 we have improved the detection of failed installations and provided a step by step wizard which will help to get an application’s “compatibility mode” correctly set. We also provide real-time problem reports and solutions that can help you locate an updated version or patch. Many of you might have experienced this when trying to run Skype after an upgrade or install the current version, and in both cases you were automatically referred to the Beta download site.


Appendix


Microsoft Covered Consumer Scenarios & Application Categories


Table of Microsoft Covered Consumer Scenarios and Application Categories


Our list includes the top 50% best selling applications in the past 24 months, some of this data is collected and aggregated from several well-known third party research information providers.

Comments (52)

  1. domenico says:

    this is my feedback and my forum frends

    Beta 7000  (Italian Feedback)

    http://tinyurl.com/bqz8u5  

  2. consumer4beta@hotmail.com says:

    All my WDM softsynths fail to work on Windows 7 but work flawlessly on Windows Vista. In Vista the UI to select the softsynth was gone but with community replacements like "MIDI Device Picker" (search a bit), it was fixed. Now after installing my favorite Roland Virtual Sound Canvas VSC-MP1 or WinGroove or Yamaha XG Softsynth on Windows 7, I find they no longer show up. My sound card’s FM synthesizer also does not work. Tried the 32-bit PDC build, beta everything. Has this been fixed in RC1?

  3. I’ll be happy if I don’t have to add a line of code to say my app is W7 compatible. I had to add a meta tag to my sites for IE8 compatibility…. 🙁

  4. anonymuos says:

    Won’t removing Microsoft Agent break some apps? Moreover installing Agent from the 2000-ish redistributable succeeds however the Agent characters lose their transparency (looks very ugly). The Vista release of the Agent binaries doesn’t have this problem but it isn’t redistributable.

  5. d_e says:

    "A simple example is version number. An application that is hard coded to only work on specific versions and it doesn’t know what our new operating system version is would fail."

    I still find it very strange that Windows 7 is internally 6.1. I agree that it is indeed "6.1", but if marketing decides that the next version is called "Windows 7" the version number should reflect that.

    But you guys do a good job. This is one of the most difficult things to work on. And you’re doing great.

    Is it getting easier with each new Windows release? Is Windows 7 an "easy" release for you since there aren’t many fundamental changes?

  6. solaris says:

    The VPN client of my university by Cisco didn’t work under Windows 7. I wasn’t able to boot into Windows 7 because right before the login screen the VPN client caused a bluescreen. There wasn’t anything I could do except using one of the recovery tools Windows provides to recover an earlier state of the OS.

  7. digitalx64 says:

    The Vista version of Inkball is not compatible with Windows 7.  Inkball was excluded from the Windows 7 beta.  Inkball is a great game that should be playable on future versions of windows.  Would you reconsider including it in the final release.  Alot of people will be disappointed by it’s omission.

    Perhaps you can release downloadable non-drm x86 and x64 versions.  Thanks.

  8. TimOR says:

    Um – what about compatibility with apps that work in XP? A smooth upgrade from Vista won’t be what MS has to sell us in the corporate world, but a hassle-free move from XP. MS has to convince us to move from XP. Merely releasing 7 as an upgrade to Vista isn’t the deal.

  9. Asesh says:

    We said we just one SKU that can be modularized but here comes Microsoft with so many SKUs again. Can you guys either merge or remove Pro edition of Windows 7 and make Ultimate edition available on retail stores because like I said before we don’t wanna be a part of that PAY MORE AND GET MORE–Ultimate scheme. I am not gonna pay AGAIN by taking out my credit card again after purchasing Windows 7 Pro just to take advantages of so called enterprise features like BitLocker, AppLocker, booting from VHD when those features should be present in the consumer/business edition of Windows 7. I have Vista Ultimate x64, so upgrading to Pro means I would lose BitLocker? that really sucks. Shouldn’t AppLocker and booting from VHD be present in both the home and pro editions of Windows 7? it should be…Please do listen to us.

    There should be an option to turn the Superbar and the window border to black just like in Vista when a window is maximized as opposed to manually changing the color of glass (Aero) because I often find it very distracting when using programs like Visual Studio etc.

  10. Moneymaker says:

    Among apps, I haven’t found anything that is compatible with Vista but not compatible with Windows 7. Inkball, talked about above, is the only thing I’ve heard about.

  11. smartpatrol says:

    here comes the "Why doesn’t my 12 year old 32-bit application not work on Windows 7" comments. Their are people still lamenting the death of 16-bit application support in Windows.

  12. yipcanjo says:

    I was trying to install the latest version of ISOrecorder, but came across the "this product must be installed on Vista"-type error message.  How is Microsoft classifying errors such as these?  The product works under Win7, I’m sure, but won’t install because it detects the wrong OS.

  13. gormed says:

    i’am sorry if this dont belongs here but i fond no other place

    @solaris:

    1.  Uninstall any previous version of Cisco VPN client that you may have installed

    2.  Reboot

    3.  Download the Citrix DNE Update and install it.

    or

    3.  Download the Cisco DNE update and install it.

    4.  Reboot

    5.  Take ownership and delete ndis.sys (in c:windowssystem32drivers).

    6.  Take ownership and delete ndis.sys.mui (in c:windowssystem32driversen-us).

    7.  Install Cisco VPN Client 5.0.04.0300.

    8.  Reboot

    9.  Windows 7 will repair itself (should take a few seconds) and automatically reboot.

    10.  Cisco VPN Client should work without any other tweaks.

    this worked for me…but no guarantee 😉

  14. SteveBall says:

    "someone" wrote:  

    "All my WDM softsynths fail to work on Windows 7 but work flawlessly on Windows Vista. In Vista the UI to select the softsynth was gone but with community replacements like "MIDI Device Picker" (search a bit), it was fixed. Now after installing my favorite Roland Virtual Sound Canvas VSC-MP1 or WinGroove or Yamaha XG Softsynth on Windows 7, I find they no longer show up. My sound card’s FM synthesizer also does not work. Tried the 32-bit PDC build, beta everything. Has this been fixed in RC1?

    Tuesday, March 03, 2009 9:09 AM by someone"

    Hey "someone" — we’ve been trying to find a repro for this issue that we first heard about in Larry Osterman’s blog.  I’d love to find out more about your specific HW and config to see if we can get to the root cause?  

    -Steve Ball from the Windows Sound team.

  15. juliemad says:

    @gormed

    Thanks for the note. Please use the Send Feedback button to submit your report with the details above.

  16. fuchueh says:

    Suggestion:

    ———–

    Integration WordPad with Notepad…

    Two editors built-in Windows, too many, too stupid…

    (After Integration rename application? use a all-new name?)

    (1) Integration WordPad with Notepad

    (2) complete support Multi-Language (ANSI Code). support read SJIS, Big5, GB, KOR, etc… (MacOSX can, Windows can’t)

  17. fuchueh says:

    Suggestion:

    ———–

    Add Virtual Desktop

    http://www.windows7taskforce.com/view/1553

    http://www.windows7taskforce.com/view/143

    Like:

    http://www.youtube.com/watch?v=TM6HXoa0Lgk

    Unix/Liunx has Virtual Desktop

    MacOSX 10.4 Added Virtual Desktop

    Windows: None

  18. fuchueh says:

    Suggestion:

    ———–

    Add Toggle option: show thumbnails in ‘Details’, ‘List’, ‘Small Icons’ view mode

    in other system desktop environment,

    ‘Details’, ‘List’, ‘Small Icons’ view mode also show thumbnails.

    Includes all picture files, video files, text files, etc…

    but Windows can’t.

    add custom option by user to toggle

    show thumbnails in ‘Details’, ‘List’, ‘Small Icons’ view mode

    look screenshot:

    http://uploads.bizhat.com/view/383070

  19. domenico says:

    @fuchueh

    see Mark Russinovich Application or ….

    download this inutility software http://www.youtube.com/watch?v=1mdMSN7c27A&feature=channel_page

  20. domenico says:

    @fuchueh

    This is a news for you?

    Back in 2003

    http://www.youtube.com/watch?v=X0idaN0MY1U

  21. fuchueh says:

    Suggestion:

    ———–

    keep Start Menu and Applaction Menu style consistency

    There is no consistency between the Start Menu and the Applaction Menu.

    To keep Start Menu and Applaction Menu style consistency, like Windows XP.

    All use white color for background, light blue color for selected item.

    to improve it.

    screenshot:

    http://uploads.bizhat.com/view/387119

    http://www.badongo.com/file/13695750

  22. fuchueh says:

    @Domenico

    yeah, I know Longhorn (WinHEC 2003)

    and Longhorn concept: http://www.youtube.com/watch?v=b9ifQvQCO7Y

    but it’s freeze by MS.

  23. fuchueh says:

    @Domenico

    yeah, I know Longhorn (WinHEC 2003)

    and Longhorn concept: http://www.youtube.com/watch?v=b9ifQvQCO7Y

    but it’s freeze by MS.

  24. anonymuos says:

    @SteveBall, my HW config is as follows: HP Pavilion DV2901TU Laptop with Intel 965PM Express Chipset / 1MB L2 Cache / 667 Mhz FSB, Intel Dual-Core Processor T2390 (1.86 GHz), 3 GB PC2-5300 DDR2 (667 MHz), 160 GB SATA @ 5400 RPM), Intel Graphics Media Accelerator X3100, 14.1" WideScreen (WXGA), came with Windows Vista Home Premium. Azalia/HD Audio (Conexant codec). See the other specs here: http://www.onlinestore.net.in/hp-pavilion-laptops/pavilion-dv2901tu-laptop-.html. I connect the laptop to high-end Bose speakers via a receiver. I’m surprised the issue can’t be reproduced. Just install any of the softsynths: Yamaha XG, Roland VSC or WinGroove and try and see if they show up in tools like MIDI Device Picker (since Vista removes the UI to choose and set these as default!!). MIDI Device Picker can be obtained at http://rapidshare.com/files/205513560/MIDI_Device_Picker.exe.html.

  25. solaris says:

    @gormed: thanks for your suggestions. But I won’t experiment with it and risk another total crash of the OS.

  26. davidtan says:

    Sounds interesting and you make it sounds like I should be part of the team 🙂

  27. fuchueh says:

    Suggestion:

    ———–

    Improve Start Menu and Appliaction Menu color style (suggestion consistency)

    There is no consistency between the Start Menu and the Appliaction Menu.

    Suggestion to keep Start Menu and Appliaction Menu color style consistency, like Windows XP.

    Suggestion all use white color for background, light blue color for selected item.

    to improve Appliaction Menu color. (discriminated Appliaction Menu color very blurred)

    screenshot:

    http://uploads.bizhat.com/view/387397

    http://www.badongo.com/file/13707468

  28. steve30x says:

    Can you guys try to get Bitdefender free V10 to work with Windows 7 64 bit. I use the antivirus software and its 32 bit only. It would be nice if there was a way to get it to work in Windows 7 64 Bit.

  29. yeehaamcgee says:

    Autotune (VST version) doesn’t work in Win7

  30. last.try says:

    I thought Y2K problem was solved in 1999 (That’s why We all still alive :). But Microsoft continues support for programs solving that problem. Look at last record in Fundamentals scenario in the picture above.

    Microsoft is very serious about Y2K problem, so Y2k Solutions presents in the ‘Data Security And Backup’ scenario too.

    That’s what I call Compatibility 🙂

  31. wtroost says:

    > Their are people still lamenting the death

    > of 16-bit application support in Windows.

    Say the company sends invoices using a 16-bit application.  There is no business need to replace the application.

    Assuming they want to switch to Vista or W7, how would they go about it?

    Some people tried, and they got burned.  And that was in better economic times 🙂

  32. arachniagirl says:

    I’m wondering if it’s possible to successfully network a win98 machine to Windows 7?

    I’m looking at purchasing some PCs w/Windows 7 for a small business that presently has primarily XP Pro machines except for 2 PCs that run win98. The 98 boxes run proprietary software that controls production machines in the shop and they cannot be upgraded. They do however need access to files that would live on the Windows 7 machines.

    If anyone could provide feedback on:

    a) it is possible

    b) what extra steps would be required to get this working

    Thanks for any information you can provide!  

  33. Urvabara says:

    *IMPORTANT* *IMPORTANT*

    1. 16-bit applications do not work in 64-bit Windows 7. Can you, please, virtualize/emulate 16-bit/32-bit, so that 16-bit applications will work out-of-box?

    2. Inkball from Windows Vista does not work in Windows 7. Why not? Why did you remove the game altogether? Please, fix this.

    3. Cisco VPN Client version 5.0.02.0060-BETA does not work in 64-bit Vista. I assume that it does not work in 64-bit Windows 7 either. Please, fix this.

    4. Many old programs are missing: ClipBook Viewer, Inkball, Winpopup, Winchat, QuickView. Please, bring them back.

    *IMPORTANT* *IMPORTANT*

  34. rbrogdon says:

    It would seem easier to help test if we had access to current builds. As of now, only build 7000 is available for download. It is rumored that build 7105 should be "leaked" soon. If we are testing, it seems that it would be more relevant if we were testing using the latest build. Is there a way to get access to these newer builds?

  35. jrlopezp says:

    I have a suggestion for Microsoft. I’m testing Windows 7 RC with Windows XP Mode for app virtualization.  The concept is very useful but the fact that it runs as a remote desktop app lacks the GUI performance.  Take a look at VirtualBox and see what those guys are doing for GUI performance without the latency of a remote desktop display.  Also, I love MS products but please please simplify the line of products. One O.S. with the full features is enough.  Thanks for listening to the users.

  36. macdet says:

    Big Hands to all for a lot of inspiration.

  37. Luke Vording says:

    I have identified an alarming problem with Windows 7 release candidate… Up until now we have instructed our application’s users to select “Run as Administrator” and the application has worked well. Now that our application can be run as a Standard User some of our users are still setting it to “Run as Administrator” because that is what they are used to. This is the problem, they can never undo the “Run as Administrator” and go to back to normal. The application uses the All Users directory for the application’s common data and the path to that directory is no longer valid under this scenario. Uninstalling and reinstalling the application does not fix the problem.

  38. Research Writing says:

    The application uses the All Users directory for the application’s common data and the path to that directory is no longer valid under this scenario. Uninstalling and reinstalling the application does not fix the problem.

  39. Dissertation Service says:

    If we are testing, it seems that it would be more relevant if we were testing using the latest build. Is there a way to get access to these newer builds?

  40. Research Paper Help says:

    The 98 boxes run proprietary software that controls production machines in the shop and they cannot be upgraded. They do however need access to files that would live on the Windows 7 machines.

  41. Thesis Service says:

    Microsoft is very serious about Y2K problem, so Y2k Solutions presents in the ‘Data Security And Backup’ scenario too.

  42. Essay says:

    There is no consistency between the Start Menu and the Applaction Menu.

  43. MMOGamer says:

    Awesome stuff, i’m just concerned that compatibility won’t be as ‘good’ as microsoft claims. I guess only time will tell.

  44. Aaron says:

    halo 2 for windows vista is not compatible for my windows 7

  45. Engr. Gufran Khan says:

    dear all, I am staying in Saudi Arabia and basically I am from India, I call my home country by using Jumblo and action voip, these are all voip calls, But from 1 month these softwares are not working in my laptop, if i troubleshoot problems then it says system incompatibility, but it was working before, i dont know what to do, please anyone help me. Its a kind request, if anyone got any suggestion then please help me and mail me to mail id gufran2t@gmail.com

  46. Engr. Gufran Khan says:

    dear all, I am staying in Saudi Arabia and basically I am from India, I call my home country by using Jumblo and action voip, these are all voip calls, But from 1 month these softwares are not working in my laptop, if i troubleshoot problems then it says system incompatibility, but it was working before, i dont know what to do, please anyone help me. Its a kind request, if anyone got any suggestion then please help me and mail me to mail id gufran2t@gmail.com

  47. ardis ilene barnes says:

    MY AVAST ANTI VIRUS  DOESN't respond because of a side by side incompatability issue

  48. dave says:

    When I open this game on windows 7, the screen goes black and returns to the home page twice. There is a little icon that is at the bottom of the screen that says that the running program isn't compatible with certain visual elements of windows. What is wrong and how do I fix the problem?

  49. Need Help Please says:

    I've downloaded the Itouns program & now I get a message saying that it needs the saport area along with the help area. How do I get it so I can use the Itunes program. Can anyone help me if you can PLEASE email me at galen612@yahoo.com

  50. Blasine says:

    attempting to install Print Shop version 3.5  will not complete instllation.

  51. BARRY says:

    EVER SINCE UPDATE, MALWAYBYTES WILL NOT WORK??? HELP