Integrating Testing into the CI and CD pipelines


 

Tuesday we are doing a DevOps Lab for MSIT and wanted to streamline/tune the walk through for this internal audience.

In this lab, you will learn about the four of the five major DevOps Enablers:

  1. Continuous Integration
  2. Continuous Deployment
  3. Continuous Testing
  4. Application Monitoring

 

Testing in Continuous Integration and Continuous Deployment Workflows

At a high level DevOps is a focus on increasing customer value by reducing cycle times in the development process.  As a result of these faster cycle times testing can be challenging. This section we will look at integrating testing into CI/CD pipelines using to ensure less friction in the testing process and a high quality application.   While there are use cases to do this as part of the CD or CI pipeline this walk through will focus on the integration as part of the CI Build process.

 

Step 1. Creating a machine group.  Machine groups are how the test efforts are scaled across multiple machine and in environments and make up the one of the core building blocks for continuous testing.  To get started, in the Test menu, click on the machines menu.  Click the Action button (Green Plus) to create a new machine group.  If you doing this as part of my lab and running low on time consider reusing the Xlab Machine Group that already exists. Even if you do decide to reuse the the preexisting xlab Machine Group as the corner stone of the Testing CI features please at least open the Test hub and the “Machines” menu.  Select the Test CI machine group and select edit and take a look at the definition.  

image

Step 2.  Adding Machines to a machine group. To enable testers to very easily run the same test against multiple machines we can add several computers to a machine group.  For user name type in your domain credentials in the DOMAIN\NAME syntax.  For The machine name, please enter an IP Address or FQDN i.e. 4TestCI1.redmond.corp.microsoft.com

image

There several caveats for computers to be used with this early version of the Test CI agent and workflow.

  1. We do not have the support for test machines residing in Azure yet. However this is coming soon.
  2. We do not support the Hosted Build controller.
  3. The build agent and test agent machines should be on-premises machines and the build machine should be able to communicate with the target test machine.
  4. The build agent being used for the Test CI tasks must be domain joined
  5. The computer being added the machine group must be domain joined
  6. Need to enable winrm on your test agent machine (run winrm quickconfig to enable this)
  7. We do not support use of the “hosted” build pool for the Test CI tasks
  8. Do not install test agents for visual studio 2015 on a machine with VS2015RC.  There is known issue here that we have fixed for RTM.

Step 3. Edit the Build definition to Add the Test CI Tasks to the CI Pipeline. In this lab we will already have created a CI build.  To save yourself some time right click on this build definition and select “Edit”.

 image

Step 4. Add the tasks to enable Continuous Testing.  For this lab we will be enabling continuous testing with the three tasks: Test Agent Deployment, Copy the Test files, Test using the Test Agent.  To get started click the green action button next to “Add build step….” (see image above) .  Add the three steps listed above.  You will find the the “Windows Machine File Copy” under the “All category.  To save time on subsequent runs you should go into the “Advance” options and deselect “Check for Updates”.

 

image

image

 

Step 5 Configure the Deploy Agent step.  The Agent deploy step requires three pieces of information: Machine Group (either the one you created above use the group already created called: xlab, a username variable and a password variable….no your password is not stored in clear text!!! The syntax to reference these variables is:  $(varname). While the variables in this lab are called “username” and “password”…they could have been called practically anything.

image

Step 6. Creating the username and password variables.

The deployment agent requires the user and password be accessed via a variable.  To create these two variables click the “Variable” menu and press the green “Add variable” action button to add both of these variables and to protect your password click the “lock” icon.

image

Step 6 Copying the Test Files to the Agents.  If the Agent deployment step succeeds we will want to copy the binaries to all the machines in the machine group. This task requires three parameters: the source, the machine group and the path to be copied to.

1. The source references all the build all the assets with the argument: $(Build.Repository.LocalPath)

2. The machine group to be used xlab

3. The path on the agent you want the files copied to…to make it easy for this lab the agent machine has a directory called: c:\drops

image

Step 7 Configure the run with Agent.  Now that the agent is installed into the machines in the machine group and the files have been copied across it is time to run the test.  This Build task only requires two parameters: the machine group (xlab) and the file path (c:\drops).  Also as it is likely some of the unit tests will fail select the option “Continue on Error”.

image

Step 8 Running the build. At this point we have set up everything to run test on an assortment of machines every time a build is started.  To start the build click the menu: “Queue build….” this can be done from the edit menus or right clicking on the build definition.  It is important that you select a locally managed queue as continuous testing is NOT supported on the hosted queue/agent.

image

Step 9 Monitoring the execution. At this point the build will start and you see in real time what steps are executing and their status.

image

If you build succeeds except for the test execution you may have forgot to toggle the option: “Continue on Error:”.  Toggle this and reruning the build should give you a status like the one below.

image

image

Step 11. Viewing the test results.  Opening the build run you will see the results of the run such as the commits, tags, issues and in data we are interested in –Test Results.  Clicking on this Test results will take you the Test>Runs and opens that run ID.

image

 

Step 12: Viewing the Test run Data.  By selecting the “Test results” menu you can drill down into the test data.  In this case it looks like we may have issues with specific environments/browsers.

image


Comments (0)

Skip to main content