Web and Load Testing with Visual Studio Team System


Continuing the series on Testing with Visual Studio Team System, I devote this blog to our Web and Load Test functionality.


The web test and load test features were initially introduced in Visual Studio 2005, and enhanced in Visual Studio 2008. These tests are available in the Team Test and Team Suite editions of Visual Studio Team System.


Web tests enable you to generate http requests and responses, test for correctness of the responses, and measure response times and throughput. The primary scenario for web tests is to use them in a load test to generate load against a web application and measure web application performance. In order to efficiently generate load, Web tests work at the http layer, they do not drive the browser.


Load tests enable you to simulate many users hitting using an application at the same time, and then measure server response times, throughput, error rates, and resource utilization on the servers under test. Load tests will help you ensure your applications will stand up under load and deliver acceptable performance. In addition to driving web tests to simulate user activity, load tests can also run “unit” tests under load, which enables you to drive load to any server with a .NET API.


Enterprise Ready


This feature set delivers powerful enterprise-class load and performance testing features to the Team System product line. In the years since it was developed it has become the de facto load testing solution used internally by Microsoft. Nearly every server product team and web property now uses Visual Studio load tests. Server teams using this feature set include SQL, Windows, VSTS, and SharePoint, CRM, Dynamics, Media Center, and many more. Web properties include Hot Mail, Live Earth, Ad Center, Cars, Music, and many more. It is also the de facto tool used by our IT teams, who test everything from accounts payable, to the payroll system, to our polling system using VSTS.


It has also been used by a wide variety of customers and load test consultants to test their mission critical systems, with scale to 100,000s of users, on systems as varied as mission critical customer applications, to huge SharePoint deployments, to delivery of television content over phone lines.


Introduction to Web Tests


As I stated above, the primary use of Web tests is to use them in a load test for performance testing.


In order to efficiently drive load, Web tests work at the http layer. So rather than driving IE or another browser, they generate http requests and responses. A key thing web tests do not do is execute java script. Any http requests sent in java script will be recorded and played back, but again by sending http requests, not by running the java script. Under the covers the Web test engine uses the system.net classes in the .NET framework for generating http traffic.


Web tests can also be used as functional test to validate the application server responses, but since they run at the http layer cannot test anything that happens in the browser: java script execution, page layout, or control behaviors.


You start by creating a test project, then adding a web test to the project:


clip_image002


Which launches the recorder:


clip_image004


Which generates a Web test in Visual Studio:
clip_image006


Which you can then playback:
clip_image008


Playback allows you to view response time, error code status, and response size. For a given request/response, you see a preview of the html that was returned, and you can view the actual request and response data. The context parameters shows what values were in the virtual users context, and you can also see the results of any validation or extraction rules.


clip_image010


Web tests do a lot under the covers to do this effectively. When Web tests are run the Web test engine automatically does many of the same things the browser does:



Handle cookies
The web test engine automatically tracks cookies for a given “virtual user”, storing cookies in a cookie container when received from the server, then sending them back as required on subsequent requests.



Authentication
NTLM, Kerberos, and forms authentication are all supported.



HTTPS Support
The Web test engine supports both http and https requests.



Automatically Download Resources
Most web pages contain images, style sheets, and java script. The Web test engine will automatically parse html responses and fetch the resources on the page (also called “dependent requests”).



Follow Redirects
By default the Web test engine automatically follows page redirects. 



Hidden Field Tracking
The web test recorder automatically generates code to extract hidden field values and bind them to subsequent requests.



Validation Rules
A validation rule is used to pass or fail a test. There are a set of canned rules that ship with web tests, such as the validation text rule. This rule will fail if the specified text is not found on the page.



Extraction Rules
An extraction rule is used to scrape data from a response for use in a later request. A good example of this is the hidden field extraction rule which extracts the values of all hidden fields on a page for use in the subsequent post request.



Dynamic Parameter Handling
Dynamic parameters are any parameters that change each time you run hit a web site. A great example is a session id, which is unique each time you create a new session with the web server. You can’t simply replay the recorded session id, or the test will fail. Dynamic parameters are often stored in cookies and hidden fields, which are automatically handled as outlined above. Some querystring parameters also contain dynamic parameters, and the dynamic parameter detection tool detects these and automatically correlates them  using extraction rules.



Data Driven Tests

For multi-user testing, you typically do not want each user entering the exact same data in the test. Data driven tests provide the ability to bind parameters to data from a data source. This enables you to easily test different data combinations.



Parameterization
You can easily parameterize a web test. A great example of this is using the same test to target different web servers, which you can do using the Parameterize Web Server command. The creates a context parameter with a default value, which can then be overridden from a load test or from an environment variable.



Extensibility
The Web test engine fires events to code before and after the test, and before and after each request, page, and transaction. This provides you with powerful hooks to a wide variety of things in your test. In addition, you can add your own extraction and validation rules.



“Declarative” or Coded Tests
When you create a new web test, a “declarative” style test is generated. Declarative tests are persisted as xml and have a designer. You can also generate a coded web test, which enables you to do looping, conditional execution, and compute parameter values in code. Generally the designer is easier to work with.



Included Test
Web tests can also be broken up into parts, and the parts can be re-used to build other tests. This can make test maintenance easier, since if one part of the app breaks, you only have to fix the web test components that hit that part.


Introduction to Load Tests


Load tests enable to you simulate many users hitting an application at the same time. The primary scenario for load testing is performance testing. How will the server respond to so many users? Will response times be unacceptably slow? Will error rates be acceptable? Another is capacity planning, where you may want to understand how large a server you will need to support the expected user load.


You start out creating a load test by walking through the load test wizard. The wizard will guide you through creating a scenario, which you can think of as a particular group of users. The load test scenario primarily contains:



  • the load pattern, which defines how users will running at any given point in the test

  • the test mix, which defines which test scripts they will be using. A test should roughly be viewed as a single users walk through the app.

clip_image012


In addition to web tests, load tests can drive “unit” tests, which enables you to drive load to any server that can be called from managed code (which is nearly every server or protocol on the planet).


Counter sets allow you to easily control which counters are collected from the load agents and the system under test. It is very easy to configure the counters to collect on a given machine. Individual counters also contain thresholds, which when triggered fire special alerts in the load test monitor.


clip_image014


The run settings allow you to map which machines to collect counters from, as well as various other properties of the test such as how long to run the test for. You can easily configure different run settings for running the same test against different servers.
clip_image016


Load Test Analyzer
You use the load test analyzer both during and after a run to analyzer the load test results. While the test is running, the analyzer displays performance counters collected from the load agents and system under test. You can control which counters are displayed on the graphs, as well as create new graphs. Once the test completes, you can also view a summary report as well as tabular data for the performance measurements collected during the test.
clip_image018


Once the test is complete, you can drill into specific areas of the test:


clip_image020


You can also view tabular data


clip_image022


And drill into details of requests that fail


clip_image024


Reporting
The summary report supports copy/paste to html format for easy sharing via email, web or msword.


clip_image026


The load test data can be published to Team Foundation Server which houses a summary report. All of the data is stored in SQL server, so you can use your favorite tool to report on the data. A powerful set of reports are available on codeplex.com/loadtestreports.


Scale Out Load Generation


With VSTS and VSTT you are able generate a significant amount of load. If you need to generate more load, you can run the load agent product on multiple computers to achieve scale-out load generation to many 1000s of users.


Resources and Community


There are many resources you can leverage to make your effort with these tools more effective. Ed Glas maintains a list of links which includes links to guidance, how to videos, blog posts, and various codeplex projects with complementary tools for load testing, including tools for WCF recording, SQL recording, and load test reports. There is also a very active community on our Web and load test forum.


Summary


As you can see, Web and load tests provide a rich and capable solution for your load testing needs. Give it a try today! You can download the VSTS trial here, which contains the web and load test features.


Comments (56)

  1. The annual testers choice awards are out in STP magazine , and VSTS took top spot in two categories in

  2. [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez

  3. [ Nacsa Sándor , 2009. február 8.] Ennek a processzoronként licencelt szoftvernek a segítségével skálázni

  4. Hi Amit,

    I will look forward to try the load test features of VSTS.

    Thanks,

    Inder P Singh

  5. Amit Chatterjee, General Manager of the VS Test Business, recently started blogging and has a post that

  6. Marc Morrell says:

    I am trying to designate a loadtest that incorporates multiple agents, where one agent runs a specific set of webtests and the other agents run other webtests.  Why can’t the test mix set scripts by agent machine?  This would be easier than trying to create a plug-in to manage this.  Any workarounds that can be done to quickly set this up?

  7. Amita says:

    We are just indroduced to Visual Studio Team Test Load System in our company.

    I want to know what all the counter I can set here. Like if I want see whether Network in slow or Application or may be problem is in my machine.

    How can I setup that.

    Also want to know I can find application’s break point.

    Again how?

    Plaese reply.

  8. Arokia says:

    Folks,

    I am not sure about posting this as comment here. My apologies if this is not the right place.

    I am looking for a Performance tester with VSTS for a contract opportunity for my client in Milwaukee. Contract is till end of ’09.

    Any help is appreciated.

    Tx.

    Arokia

    amariadoss@cube3tech.com

  9. Ed Glas says:

    Mark, in our next release (dev10) we have added the ability to specify a list of agents per scenario. I’m curious, why do you want to control this?

    Ed.

  10. Ed Glas says:

    Amita, probably the best way to find the breaking point is by using a stepping load. Step the load up until you see throughput go down, response times go up. Or until the server starts generating lots of errors.

    By default the standard counters — cpu, memory, disk, and network — are included in each of the counter sets. There are thresholds set for cpu and memory. Network is somewhat trickier, you will need to look at the network counters to understand if you are reaching the capacity of the network card.

    Ed.

  11. Ed Glas says:

    Amita, one more thing, there’s an article on the analyzer you’ll find useful here: http://msdn2.microsoft.com/en-us/library/aa730850(vs.80).aspx

    Ed.

  12. Hema says:

    Hi Amit,

    I recorded my login page and when I played back it showed Object Moved Here. And not able to find out the user name and password parameters. Please help.

  13. Prasad Patel says:
    1. Object moved :- Is this an error or a notification of the response redirection?
      1. How to export load test in .CSV file?. i would be greatful for the answer. It would be great if someone can tell the excat steps.
  14. vsarvotham says:

    In your article you mentioned that reports are saved in SQL server " All of the data is stored in SQL server, so you can use your favorite tool to report on the data ". I have a LoadTest project and how do I look which SQL server it is connecting to and saving the reports. The project was not created by me and I do not know how to create either. In fact the testers in this project, run the load test and from the report they pick only couple of columns for reporting. And my task is to automate the process of picking only couple of columns. I was suggesting of a .NET utility(a console app ) which will pick report details from SQL server and save to excel. Is there any easier way to do it. Please help.

  15. Rangarao says:

    Hi,

    The way its mentioned i have recored the script, this is the first time that i am having using VSTS. When i replay the script without any changes i get an error as "OBJECT MOVED TO HERE" can anyone help me. wht is this error is all about???

    Thanks & Regards

  16. Jonnakanti says:

    Hi Hema,

    Have you got the answers to which you have posted?? because iam also getting the same and iam using this tool for the first time. Can you Help me???

    Regards

  17. Hema, what you are reporting is a redirection and not necessarily an error.  Many times webforms causes this redirection when it does authentication.

    Could you post your question to our forum with any additional details about your error so you can have a broader range of people helping you with any issues

    You can find our forum here http://social.msdn.microsoft.com/Forums/en-US/vstswebtest/threads/#page:1

  18. Prasad, I am not sure what you are asking here. Are you looking to export the data to CSV?  If so you have some better options. Please see the reporting project on codeplex  http://www.codeplex.com/loadtestreports

    Also, see my comment on Hema’s question posted just ahead of this

  19. Sarvotham, the SQL server is specified on the controller that you are running the load test in.  Please see http://msdn.microsoft.com/en-us/library/ms243400(VS.80).aspx on MSDN for instructions.

    For reporting I recommend you look at the reports we have on codeplex as a start http://www.codeplex.com/loadtestreports

  20. Rangarao, is not necessarily and error but it is an indication you have been redirected.  Most browsers handle this automatically so you never see it happen.  If you search our forum http://social.msdn.microsoft.com/Forums/en-US/vstswebtest/threads/#page:1 you can find a variety of reasons this might happen.  

  21. Ritu says:

    Hi, when i did the load test below mentioned is the errors which i got they R of nearly 203 error Count.

    Request failed: Context parameter ‘$HIDDEN1.__RequestVerificationToken’ not found in test context

    Request failed: Context parameter ‘FormPostParam3.MessagePreferencesModel.Preference.FundValuationChange’ not found in test context

    Kindly let me know how to overcome such errors.

    Best Wishes

    Ritu

  22. Gabriel says:

    hello, i have a problem using web tests in load test.

    All the results of the web test are 0. i have global results but when i want to check the results of each test i see all 0.

    please any ideas????

  23. I am very excited proud and proud to introduce the new testing features you’ll find in dev10. In addition

  24. Saurabh says:

    Hi,

    Would anyone be able to shed some light on threshold violations for % Time in RT checks threshold, and how to configure them?

    Does this counter have substantial effect on the load testing?

    Thanks for the help

    S

  25. Wanderson says:

    Hi Amit,

    I’m using de Visual Studio for Tester and had a problem.

    I made a Web Service (WCF) that saves a new entry in data base.

    After, I made a UnitTest that calls the WebService and returns success if the return of WebService is different of null.

    After, I made a Load Test that have five simultaneous users (virtual) and only one interaction.

    I ran the load test with success, but I expected it would be recorded five entries in the database, one per user (virtual).

    But when I ran the load test, only one entry was recorded in data base.

    This behavior is correct? If yes, why?

    I searched for more information of architecture of the Load Test of Visual Studio, but don’t find.

    You have articles or links that explain how works the Load Test?

    Thanks.

  26. Cathy says:

    Hi ,

    Can anyone tell me what  is threshold violation?

    Thanks

    Cathy

  27. raghu says:

    hi amit,

            i have one question that does vsts support java scripts controls ??

  28. Chetandahale says:

    Hi Amit,

    I am a beginer of VSTS Perf testing.

    Can you please tell me how to add counters (which Counter are more necessary) and how to analyse output of counters.

    Also any link to set up rig.

  29. digvijay says:

    hi amit,the blob is really helpful

    but i do need some more info,,

    can we change the browser in which web test is performed,some of the pages are not working in IE but working in firefox,,can u please help with this,,

    help is appriciated..

  30. digvijay88999 says:

    hi amit,the blob is really helpful

    but i do need some more info,,

    can we change the browser in which web test is performed,some of the pages are not working in IE but working in firefox,,can u please help with this,,

    help is appriciated..

  31. Sahil says:

    Hey, I am using Test Load agent for the first time. So might sound a bit silly. I have installed it Test Load agent 2008, but not able to figure from where to start the application. It is not showing any screen or exe to execute. Could anyone help as to how to go about it.

  32. Jaswanth says:

    Hello Amit,

    I am new to software testing, so please apologize for my lack of knowlegde in the same.

    Scenario:

    Step 1:  A new user registers himself by supplying his information on a website and presses a submit button.

    Step 2: From the server side, an email is generated with a link to click for confirmation and sent to the user.

    Step 3: The user clicks the link and logs himself on to the site to confirm.

    Problem: I have to automate the scenario as webtest. Till the 1st step, everything was done well(I hope) by preparing a *.csv file with random usernames and binding it to the webtest so that the username’s formpostparameters access the *.csv file and chose the usernames for registration randomly. Once a username is chosen and registered, an email is generated with a confirmation link so that the user clicks it and logs onto the website to confirm his registration. The link has a 16 characters long randomly generated ID(specific to the username). My problem is: how to integrate this link as a request into the webtest so that the webtest automatically supplies ANY username for registration, clicks the Email link, login the user for confirmation when ever the test runs.

    I have searched for this problem on internet but I could not found any. Please help me with Ideas and suggestions. By the way I am using Visual Studio 2008 professional.

    Thanks in advance for any kind of help.

    Kind Regards

  33. rahul says:

    How can I simulate multiple windows user authentication whikle doing my load test ???

  34. manujaya says:

    I have an application to test the load for 200 users.

    i need test major functionalities as login and registration of their tasks.

    To do the load test for this application i have created the wen test. my question is to do the load for 200 users do i need to create 200 users and provide those ser names to the test ??

  35. manujaya says:

    I have an application to test the load for 200 users.

    i need test major functionalities as login and registration of their tasks.

    To do the load test for this application i have created the web test. my question is to do the load for 200 users do i need to create 200 users and provide those user names to the test ??

  36. deepa says:

    Hi,

    I am using vsts.

    I want to ask….i have recorded an application using vsts.I replayed the script also.

    I want to see the script which was recorded to analyze the same. Where can i find it.

    Thanks

    Deepa.

  37. Ashi says:

    Hi,

    I have to perform a web test for a application which opens up new page for each function. how can I integrate all these new pages to the recorder of the web test.

  38. James R says:

    I have set up about 7-8 web tests to go against one of our servers, but I end up running it against 4 servers in total. It get VERY time consuming having to go through each test and change the parameterized webserver. Is there any way to have some time of global variable that I can change once and all the tests will run against the new URL?

  39. Sravanthi says:

    Hi Amit,

    Does test suite support Mozilla and Chrome. I have seen the list of other browsers that it supports

  40. Pat says:

    Hi,

    I found this tool is useful for managing load test runs and creating test results…. http://www.qualiwebsolutions.com. Anyone used it?

  41. kailash says:

    Hello Amit,

    I used VSTS for load testing, But page response time graph is not showing any data. Even the page response time is showing 0 in test result.’

    How can i check the page response time, will you please help me !

  42. Sravanthi – yes it does support Chrome. For unsupported browsers, you can go to ~Program FilesMicrosoft Visual Studio 10.0Common7IDETemplatesLoadTestBrowsers, and add a new template (use the existing ones as examples)

  43. shailesh says:

    what type of protocol does vsts support for performance testing

  44. Aman Saxena says:

    really i need this information…and provide me a great idea to do the load testing…….I m searching this type of inforamation so that i would be able to perform load testing..to test my website that how many user would access it at a time in a server…thanks amit for sharing such information.

  45. sri says:

    Hi i am testing an application using vSTS. while recording, i dont get any error but while playback we get java script disabled error . Please help me to solve this issue

  46. Satyendra says:

    Hi,

    I am using Visual studio 2008 Team System SP1. I need to perform Load Testing, but when I am going to add a "Load Test" after clicking the Add Tab only "Unit Test" and "Ordered Test" is comming not other options as I saw in the How Do I? videos.

    Can u plz help me out from here..

    Tools–>Options–>Test Tools–>Test Projects

    here also in the right pen I am getting only two options "Unit & and Ordered" test.

    thanks in Advance

  47. Megh Singh says:

    Look and feel is change in 2010 VS and TFS

  48. Raj says:

    Can we do it in the Visual Studio 2010 ultimate.

  49. Preetam says:

    Can we do a web performance test on web applications developed using Java, Informatica, ETL Modules, Microstrategy?

  50. Kranthi says:

    Hi Amit, Good Morning , First of all thank you so much for sharing your knowledge. I have one question. By using MSVS 2010 can we define a script as action block and save it to external library and then call it to different tests? please let me know if you have any information on this. Thanks.

  51. suresh says:

    hi i want information on "How to test the performance of xap file(silver light) in visual studio ultimate.reply me to suresh1.dvm@gmail.com

  52. Imran says:

    I'm not able to load a web test once it is saved . I'm getting error like this load test can not be loaded, The problem is only with our microstrategy application where as I'm able to save and open load test created for normal java applications in the same workspace and project. Please help in troubleshooting this problem

  53. arvnit says:

    hi,

    can anyone tell me how to insert validation  on vsts web test

    thanks

  54. Mahesh says:

    Can i get videos related to performance testing