Test Professional Frequently Asked Questions


Hello everyone – Please find some answers to the most common questions that we have been getting asked below.  We will try to keep this updated over time.

 

 

Automation

Q: What is the difference between what Microsoft Test Manager offers and Record & Playback?

A: Test Manager’s record/playback is intelligent and robust. We try to figure user intention based on actual captured input to reduce dependency on specific location of controls, etc… Focus is on helping manual tester accelerate through some already recorded steps.

 

Q. Can testers view and modify code created by the recorder?

 A: While the action log created by the recorder cannot be modified, you can create a Coded UI test (either C# or VB.Net code) which can be viewed or modified in the Visual Studio IDE. We made the rerecording of the actions easy enough that we don’t expect you to edit the recording.

 

Q. How does the “assisted manual testing” react to UI whose properties are dynamically generated?

 A: This should work fine as long as the generated properties are deterministic. Here is a blog that enables you to control search properties.

  

Q. How easy would it be for a developer to re-use the rich bug to test the fix?

 A: This is fairly straight forward. If the test has been automated, the developer can open the associated test case, choose to run the test case and playback the recorded actions against the fixed application.

 

 Q: Would you recommend leveraging the UI automation created in Test Manager in a performance test?

 A: While UI automation can be used in a performance test, that would not be our recommendation since you won’t be able to drive significant load using a UI tool that will bottleneck at the UI layer. Instead you might be better off using Web Performance Tests to drive the server load without the UI dependency.

 

 Q: When you convert test actions to code, what languages do you support?

 A: We generate either C# or VB.Net code from the recorded actions.

 

 Q: Can you parameterize your automated tests to run for multiple sets of data?

 A: Yes, you can parameterize your tests to be run for multiple sets of data.  If you like, the data stored on a parameterized Manual Test can  be bound to and leveraged by it’s automated counterpart.

  

Q: Can you drive your automated tests from data in a database?

A: You can bind your automated tests to a database to run them for multiple iterations of data.  Please refer to this article for more details.

 

 Code Coverage

Q: What level of support do you provide for code coverage?

 A: Users of Visual Studio are able to see the code covered by their automated tests in Visual Studio.  Unfortunately Code Coverage information is not currently collected when Manually Testing.

 

 Competition

 Q:  How does this tool compare to Selenium and other open-source testing tools?

 A:  Our offering engages testers in the overall ALM effort through integration with lab management and TFS for work-item tracking, source control, build management, and reporting.  Many of the open-source testing tools are stand alone and tend to focus on a particular platform or type of testing (e.g., Selenium is specifically for browser-based testing of web applications). 

Q: What are the key benefit this tool provides over HP’s QC/QTP?

A: We feel that Visual Studio Test Professional 2010 is optimized for the way that testing is performed today where more than 70% of testing is done manually through ad-hoc tools such as Microsoft Excel and Word. It is an integrated testing toolset that delivers a complete plan-test-track workflow while allowing testers to collaborate with developers effectively and test efficiently.

Integration

 Q: Does this tool integrate with Office Project?

 A:  Yes, via TFS.  Office Project can be used to manage data stored in TFS (such as dependent tasks) to facilitate planning projects, scheduling tasks, assigning resources, and tracking changes.  By using Office Project, you gain access to features such as a project calendar, Gantt charts, and resource views.

    

Q: Does this tool integrate with Office SharePoint?

 A: Yes, via TFS, which enables creation of a team project portal associated with a SharePoint site.  The project portal includes a customizable set of dashboards (SharePoint Web parts) that enable team members to monitor project data in the form of PivotChart reports and lists or counts of work items.  The testing tools are also integrated with SharePoint in the sense that they can be used to test SharePoint sites.

  

Q:  Does this tool have the capacity to integrate with other testing systems such as Rally, QTP, or QC?

 A:  Not directly; but if you can export test cases, bugs, and other artifacts from the other system into Office Excel, you can then import them to TFS as work items.  You can use the Test Case Migrator Tool (available at http://tcmimport.codeplex.com/) to import test case steps and parameterized data along with your test cases.

  

Intellitrace

Q: Does seeing the line of code that failed require a special type of build or deployment?

A: No, no special build or deployment is required.  A tester must however configure to capture intellitrace data on the machine which is running the code while they are testing, this is done via the Test Settings.

 

 Q: Will the developer be able to see the line of code for web-based development where the code sits on the server and the test is executed via client browser?

 A: Yes the developer will be able to see the line of code that failed, so long as the tester configures the webserver to capture intellitrace logs.

 

Internal Use of Test Professional

Q: Were these test tools used during the development of Visual Studio 2010 or any other MSFT products?

A: Absolutely.  The entire Visual Studio team has used this toolset to build Visual Studio 2010.  There are many other teams at Microsoft which are using them as well.

   

Lab Management

Q: Has lab management released with VS2010 on April 12th?

A: Visual Studio Lab Management 2010 is currently available as a release candidate.  You can try it out right now, it has yet to be officially released

 

Q: Are you planning to have host a webinar on lab management?

A: We plan to have a webinar around our Visual Studio Lab Management 2010 product around its release. Please stay tune for announcements from SQE later this Fall (Q3CY2010).

 

Licensing

Q: How much does Test Professional Cost?

A: Visual Studio Test Professional 2010 with a one year MSDN subscription retails for $2169. Your organization may already have volume licensing with us. For details of licensing, please refer to this licensing white paper.

 

Q: What are the licensing details of Test Professional?

A: The ins and outs of licensing for Visual Studio family of products are spelled out in this licensing paper.

 

Load Test

Q: Do you have anything to assist with Load or Performance Testing?

A: Yes, since 2005 we have shipped tools that allow users to load test their application. This blog post is a good overview.

  

Q: Can you load test using Test Professional? 

A: Load testing must be performed through use of Visual Studio 2010 Ultimate.

  

Migration

 Q: Can I migrate existing test cases authored in Excel to Test Manager?

 A: You can use the Test Case Migrator Tool to convert test cases authored in Excel to Team Foundation Server work items that can be used in Test Manager.  The Test Case Migrator Tool can be found at http://tcmimport.codeplex.com/

  

Q: How do I migrate my manual test cases from previous versions of Visual Studio to be run in the current version of Visual Studio?

 A: Please use the Test Case Migrator Tool found at http://tcmimport.codeplex.com/ to migrate old Test Cases to the current version used in Visual Studio 2010.

   

Q: Can I migrate existing test artifacts present in Quality Center to Test Manager?

A: You can migrate your test cases from Quality Center by exporting the test cases in Quality Center to Excel, and then using the Test Case Migrator Tool to import the Test Cases to Team Foundation Server.  The Test Case Migrator Tool can be found at http://tcmimport.codeplex.com/

 

Platform Support

Q: Can you provide some information about the platform support for Microsoft Test Manager? What class of applications can I test with the tool?

A: You can use the tool to manually test almost any platform, though your experience around the creation of rich bugs and support for automation will be better if you are targeting IE7/IE8, Windows Forms, WPF, Win32, MFC or SharePoint. This blog post has an updated enumeration of the automation platform support and will be updated over time as we add additional support.

  

Requirements

 Q: Does Test Professional have requirement tracing built in? 

 A: Requirements are treated as first class citizens in Test Professional.  One can see full traceability of Requirement -> Test -> Code. One can also base a test suite off of a requirement.  This blog post speaks very extensively on the topic.

 

Q: Can a user produce a Requirement traceability report using Test Professional?

 A: Test Professional provides full requirement traceability.  Out of the box, users can see a requirements traceability report on their team dashboard.

  

Test Manager

 Q: Can I email a defect in Visual Studio 2010?

 A: Yes, you can email a link to a defect from Visual Studio. 

  

Q: How can I link Bugs to Test Cases? How can I link bugs to Product Backlog items? 

A: Defects, test cases, backlog items are all work items on Team Foundation Server. Test Professional automatically links the work items together when appropriate (for example, when a test creates a bug, the test and the bug are linked together).  Users can also manually link work items together in Test Manager or inside Visual Studio.

   

Q: Can a tester send a video of their testing to a developer? 

A: Yes a tester can send a video of their testing to a developer.  When a tester creates a bug, with background video recording on, the video of the testing which caused the bug is referenced on the bug form which the testers assigns to the developer.  If you wish, you can save the video which is stored on the test result to a network share as well.  Demo 2 of the webinar demonstrated the rich bug which is accessible to the developer.

  

Q: Is there any way to track testing across a variety of operating systems?

A: By setting up a configuration for each operating system, you can break down your testing for each operating system.  To read more about configurations, please refer to this blog post.

 

 Q: What does Analyze Test Runs do?

 A: Analyze Test Runs allows you to review the results of your automated and manual test runs.  For more details, please refer to this blog post.

  

Q: Can we access Test Manager via the web?

A: Test Manager 2010 installs as a client tool. When it launches, you are presented with the various projects on TFS which you can connect to.

  

Q: We have a large number of testers, can a test manager assign test cases to individuals

A: Test managers can assign test cases to individuals. Testers can also pull from a pool of test cases. Test case assignment is done in the Test Plan Contents, it can be done by clicking on the assign button.

  

Q: Is Test Manager available as a plugin for Visual Studio 2008?

A: No, Test Manager is only available via Visual Studio Test Professional 2010 or Visual Studio Ultimate 2010

  

Q: Does Test Professional support the branching of Test Cases?

A: Test Manager does not support branching of work items.  However you can import test cases from one plan to another so you would be able to use the same test case across code branches.

  

Q: Is it possible to work with components of tests being shared between test cases?

A: Yes, shared steps can be created once and shared between test cases.  They are really powerful.  Please refer to this blog post to learn more about them.

    

Q: Are test steps and expected results which span multiple lines supported?

A: Yes, You can enter multiple lines for the test step and the expected result. The text wraps in both the test runner and while authoring the steps.

  

Q: Is Test Professional compatible with Agile Methods?

A: Yes it is compatible with Agile Methods.  We would suggest that you use 1 test plan for a sprint of work and 1 requirement based suite for each user story. This blog post describes our suggested approach.

  

Q: What information does recommended tests provide?

A: Recommended tests provides a recommendation of the tests to re-run based upon code which a developer has recently checked in.  This blog post describes the feature thoroughly.  This one does as well.

 

Q: When I’m editing a test case in MTM, the steps / parameters section of my work item is changing height, or is very small. This is driving me crazy!

A: This is a known bug, and we’re looking at addressing it in a future release. For the moment, there is a work around: Adjust the height of the window. This should make the issue go away.

 

Q: How can I find all the test cases that I have action recordings for, so I can create recordings for them?

A: Today, there is not an easy way to do this. The recording is stored on the Test Result, and it is not possibly to query between Work Items, and test results. It’s important to note that with Fast Forward for Navigation, the expectation is that people will record, and update, their action recordings as they are performing their normal manual testing.

 

Q: How do I view the results of more than one query?

A: Just click the ‘queries’ link at the top of the application again. This will open a new Queries view, and allow you to have two queries open. You can find the other Queries you have open in the ‘Open Items’ drop down.

 

 Test Impact

 Q: How does Recommended Tests work?

 A: When the Test Impact Data Collector is turned on, Test Runner records the code paths taken by a Manual or Automated Test Case when it is executed.  Recommended Tests recommends test for execution when a developer makes a check in that touches the code covered by the test case. 

  

Q: Do applications need to be Debug builds to track the lines of code hit by the test?

A: No applications do not need to be debug builds to capture Test Impact Data.  The Test Impact Data Collector must just be turned on while testing.

  

Q: What is the difference between Test Impact Analysis and Code Coverage?

A: Code Coverage is a report which can be generated which speaks to the lines of your code under test which have been covered by your automated testing. Test Impact analysis collects the lines of code which your Tests cover to report which Test Cases are suggested for you to re-run based upon code churn.

  

TFS Server

 Q: Can I use Test Professional without a TFS 2010 Server?

 A: No, Test Manager requires a connection to a TFS 2010 Server.

 

 Q: Can I use Test Professional 2010 with a TFS 2008 Server?

 A: No, Test Manager requires a connection to a TFS 2010 Server.

 

 Q: Is there a Microsoft online virtual environment to test the product without having to download a trial version?

 A: Unfortunately, there is not currently an online virtual environment to test the product.

 

 Q: Is TFS 2010 available as a trial?

 A: Yes it is available as a trial.  Please use this link to download it.

  

Training

Q: Who should I contact for a thorough training on Test Professional?

A: We recommend that you contact our partners for hands-on product training. For example, RTTS is a New York based (NYC) partner who specializes on test training as well as services on the Visual Studio platform and tools. Notion Solution and Northwest Cadence, based in the Dallas area and Seattle area respectively, are partners who deliver training as well as services around the country focusing exclusively on Visual Studio Application Lifecycle Management.

  • RTTS’s courses can be found here
  • Notion Solutions’ courses can be found here.
  • Northwest Cadence’ courses can be found here.

  

Webinar Viewing 

Q: Where can I see video of the Webinar entitled Visual Studio Test Professional: The Tool for the Modern Tester?

A: A video recording of the presentation can be viewed at this link.

   


Comments (31)

  1. Dylan says:

    "Q: Are test steps and expected results which span multiple lines supported?"

    I would emphatically say NO with the current version – saying that the text will wrap is not the same thing as allowing multiple lines with hard line-breaks!  You can't copy and paste from another source that has multiple lines and that's exactly what we would like to do in order to transfer our Word-based test plans into Test Manager.  Really the only way to do multiple lines for Expected Result is to add in dummy Steps with no action, then manually copy lines one by one from the source.

    Further our Expected Result sections often have embedded images and font changes which aren't supported either.

  2. Ollie says:

    "Q: Are test steps and expected results which span multiple lines supported?"

    100% disagree.

    Very poor design for creating test cases in test manager, almost makes the process unbearable.

    Test case visibility in TFS is pretty bad also.And worst of all it seems impossible to export a test case with all steps to excel or similar so it can be reviewed/printed etc.I stand open to correction on that one though!  

  3. nitro52 says:

    @Dylan you can copy and paste steps from Word to Test Manager but there is a bit of a trick to it. take a look at

    tinyurl.com/CopyFromWordToTestManager

    @Ollie are you saying you should be able to have multiple lines in one Action or Expected field? In most case i have come across i don't know why you would want that unless you were checking a list of things, like check the valid in field 1, 2 and three. but i just put a comma to separate a value. either way i think there needs to be different Parameter types and also conditional steps. I've logged these as a feature request, you might want to check it out

    connect.microsoft.com/…/iteration-variable-types

    connect.microsoft.com/…/test-manager-conditional-steps

  4. pete says:

    Q: Can several recorded tests be played back as part of a test suite and run via the Test agent? If so, why would I need Coded UI Tests?

  5. Nitro52 says:

    Coded UI Tests allow for more advanced automation such as data driven tests and validations. You can convert a manual test that was recorded into a Coded UI test as a template. This makes it easier to convert manual tests into Coded UI Tests. Once you have the Coded UI Tests they can be run as part of a build or i believe as part of the tests suit.

    The key differences are

    Recording a test in Test Manager only speeds up manual testing and helps keep accuracy when repeating the test. All validation is done manually

    Coded UI Tests allow for a completely autonomous process. It also allows you to compare code that has changes with your Coded UI Tests to show what tests have been impacted.

    Does this answer your question?

  6. Pete says:

    Yes, that's exactly what I needed to know.

  7. Derek says:

    Could you tell me if there is a way to export test results from Test Manager?

  8. Nitro52 says:

    You could try Test Scribe

    visualstudiogallery.msdn.microsoft.com/…/e79e4a0f-f670-47c2-9b8a-3b6f664bf4ae

    It allows you to export some information out of MTM as a word document. At the moment there's not a lot of control over what it exports though. Other wise you could look into TFS Reports, you should be able to make your own that gets the information you need.

  9. Aracknid says:

    Is it possible to create an automated test (I guess you call it a Coded UI test) from scratch, without first recording from a manual test in MTM? I want to start up VS2010 and just write an automated test without using MTM at all. I want to click my buttons and type text into my edit boxes (for example). Can this be done, and if so is there a link to some place that tells you how to start or a topic in OLH about this?

  10. Nitro52 says:

    @Araknid Yes you can, You don't need to use MTM at all, Coded UI Tests can be recorded from within Visual Studio. Take a look at http://www.testingwiz.com/…/how-to-create-and-run-coded-ui-test-in.html On step 5 you see the option to Record or us an existing action recording (from MTM). At this point you can also cancel and start coding manually, though it would recommended to add the controls to the UIMap first, this is how you would do that blogs.microsoft.co.il/…/coded-ui-test-tip-4-add-unused-controls-to-ui-map.aspx

  11. Itsik says:

    Can you customize the "Test Plan" work item? How is this accessible? I plan to have additional fields that are important to the way we manage tests.

    (and please – do not refer to the "Test Case" customization from within VS2010.).

  12. Nitro52 says:

    At the moment plans do not contains any custom fields. they act more like a container. I asked a similar question where i wanted to add notes at a suite level but it is not possible. only the test case fields can be modified i think. What sort of fields did you want to add?

  13. Ron says:

    How do I export defects created in Microsoft Testing Center to Excel?

  14. Nitro52 says:

    @Ron The only way i know how to do this is through Visual Studio, you need to create a query that shows the data you want and then right click on it and select  open in excel or create report in excel.

    here is some more information

    msdn.microsoft.com/…/dd286627.aspx

    msdn.microsoft.com/…/ms181695.aspx

  15. SANZ says:

    Had a few questions regarding reports:

    1. What kind of reports are available?

    2. Is there any visual report showing the number/details of test cases referencing all/selected shared test steps?

    3. Is there any report showing the impacted test cases for all/selected requirements?

  16. Aidan Ryan says:

    I have put together a small utility for editing multi-line actions and expected results in test steps. Please let me know if you find it useful!

    teststepseditor.codeplex.com

    I just completed the first stable release of the tool. It now supports:

    – Adding and deleting steps

    – Editing test cases that include shared steps

    – Editing shared steps themselves

  17. Yokozuna says:

    We have an existing bug database in Salesforce.  Can the VS 2010 bug related functionality be turned off?  Can the bug management functionality in VS 2010 be minimized?  Can it be reconfigured to the content I want to see?  Can Salesforce be invoked from VS 2010?

  18. James P says:

    I have a problem with play back of action recordings.

    When it reaches the part of the test that it has to click on data in a data grid, right click, and then select an option the recording stops.

    I was able to recreate this behavior in several applicaitons.

    With MS Excel, nothing that I did in the spreadsheet would play back.

  19. Phani says:

    Where can I get the Trial Key for evaluation?

  20. IMJack says:

    Is it possible to create a hyperlink for a specific test case or search result, which can be shared via e-mail or on a Sharepoint site?

  21. SLTest says:

    Hi – can you share test case suites between projects? e.g. have a regression project with multiple suites, and copy one suite across to a new project?

  22. Pete M says:

    I have used Fogbugz to record and track bugs.  While I like the integration between Test Manager and bug reporting, Fogbugz provided one thing that appears to be missing from Test Manager bug reporting.

    Fogbugz automatically notified via e-mail a user when he was assigned a bug or the user when a bug was resolved by the developer and reassigned to the user for retesting.

    Does Test Manager also have this capability and i'm just not seeing it?

  23. Scott S says:

    Can .sql files be executed against a database during the execution of tests?  For example, if I record a test case and would like to reset a DB value before running the next test.  I've found how to set a data source for reading values from a DB for test assertions, but can I automate running these separate SQL queries I have somehow?

  24. aan says:

    Test Manager requires a connection to a TFS 2010 Server.

    LAME!!!

  25. Doug B says:

    Is there a way to use Test Professional for unit module testing prior to a UI being available for integration testing

  26. Jhon says:

    I already have team foundation server and i can already build there test cases for manual tests, what's the advantage?, what test manager has to offer aditional of what team foundation server already has?

  27. Kishore says:

    CODED UI <<—>> REPORTS

    How to see Coded UI Automation script execution reports in TFS? please help me ASASP

  28. Previn says:

    I would like to know whether there is a way to convert Issues into Bugs and vice versa, since these can often change during the project lifecycle and it would easier to trace issues when needed.  All is it possible to view Issues and Bugs in one list rather than keeping hem in separate areas ?

    Many thanks

  29. Steve says:

    When running a multi-step manual test case, we add logging info to some of the steps, using the comment field, and pass/fail each of the test steps.  If the test case fails, we end the test, hit Save & Close, and are returned to the suite view.  Please let me know how to re-run the test, without losing all the comments previously inserted.  Thanks!

  30. Savan says:

    If I have recorded my manual steps for 50 steps and now If I want to modify one step or I want to change parameter valu, then do I need to record all 50 steps again or how can I continue with the modification done.

    Please help.

    Thanks.

    Savan D

  31. Marga says:

    How come test professional continuously number even if it is a new program?  is there a way tha it will again start to 1?  this is also true when you are creating a bug.