Workflow of the IDE QA team
Hello everyone, I am Anca Miclea. I work as a SDET (QA) in the VC IDE team. This is my first post on VC blog and I would like to share with you how we test the IDE for VC++.
Yes, we do test the IDE and this is not an easy job 😉 For each new feature that we add to the product, we need to write a Test Plan (TP). Each TP is reviewed by the whole IDE QA Team and if the feature is a large one, like the recent “MFC” update, a review meeting will be held with the developers and the Program Managers (PMs) that work on that feature.
As soon the TP is reviewed, one or more QA folks will design each test case and implement them. In order to be able to implement our automated tests we use as much DTE available support as we can. EnvDTE is an assembly-wrapped COM library containing the objects and members for Visual Studio core automation. For the parts of the IDE that cannot be covered by DTE we use an internal API that helps us record the mouse events and generate the code that will be used by our test. This API is mostly used for black-box testing for UI features.
As our software is used by developers all over the world, it is very important that our tests are written in a way that will pass on Vista OS using a Normal User and also pass on localized OSes and VS SKUs. In order to have confirmation that a test is good and robust we need to run it in our lab environment, with different configurations, like “Win2k3 ENG x64 + VSTS ENG”, or “Vista JPN x86 + VSTS JPN”, etc. Running tests in these configurations is a way to validate not only the product itself, but also to validate the automated tests as well.
If an IDE product bug is found, usually we try to find an easier set of steps that reproduce the issue (these steps form so called “repro case”), investigate if that repro case is a regression from a previous product version and then open a bug that will be assigned to the appropriate developer. Once the bug is fixed, QA verifies the “repro case” with the version of IDE that has the fix and close the bug. It is our QA team’s responsibility to make sure a specific issue does not regress in later versions, by providing an automated test case for each product bug that is fixed.
Thanks,
Anca
Light
Dark
0 comments