Load Test API enhancements in VSTS 2008 SP1 (beta)
The beta for Visual Studio 2008 SP1 is now available. This article describes enhancements to the VSTS Load Test API for use by a LoadTestPlugin that have been added in this update. For complete details on changes and bugs fixes in this release, see Ed Glas's blog post at: https://blogs.msdn.com/edglas/archive/2008/05/27/vs-2008-sp1-beta-is-available.aspx.
Introduction
In VSTS 2008 SP1, the Load Test API has been enhanced to allow a Load Test Plug-in to dynamically change certain settings of a Load Test Scenario during the execution of the LoadTest. Specifically:
· Properties of a particular type of load pattern may be changed by the plug-in during the load test, such as the MinTargetValue and MaxTargetValue of a Goal Based Load Pattern.
· The type of load pattern in use may be changed, for example from a Constant Load Pattern to a Step Load Pattern.
· A custom class implementing a different type of load pattern may be written by a user and assigned to the load test by the Load Test Plug-in.
· The “Think Time Between Test Iterations” setting of one (or more) of the Scenarios in a load test may be changed.
Summary of API Additions
Here is a summary of the new public API elements in the LoadTestFramework namespace to support this:
· A new abstract base class named LoadTestLoadProfile that is the base class for the following classes, and can also be extended by a user written class to implement a custom load profile.
· New public classes for each of the types of load pattern that can be defined in the Load Test editor:
o LoadTestConstantLoadProfile
o LoadTestStepLoadProfile
o LoadTestGoalBasedLoadProfile
(Note that these can also be extended by user written classes.)
· New exception class InvalidLoadProfileException
· Two new read/write properties on the existing LoadTestScenario class:
o public int DelayBetweenIterations
o public LoadTestLoadProfile LoadProfile
Examples:
Note all of the examples below need to include this using statement:
using Microsoft.VisualStudio.TestTools.LoadTesting;
1. Load Test Plugin sets up a GoadBasedLoadProfile object and assigns it to the LoadTestScenario and also assigns a value for DelayBetweenIterations to the LoadTestScenario in the Initialize() method:
[Serializable]
public class LoadTestPluginInitChangeProfile : ILoadTestPlugin
{
public void Initialize(LoadTest loadTest)
{
LoadTestGoalBasedLoadProfile goalLoadProfile = new LoadTestGoalBasedLoadProfile();
goalLoadProfile.MachineName = Environment.MachineName;
goalLoadProfile.CategoryName = "Processor";
goalLoadProfile.CounterName = "% Processor Time";
goalLoadProfile.InstanceName = "_Total";
goalLoadProfile.InitialUserCount = 5;
goalLoadProfile.MinUserCount = 1;
goalLoadProfile.MaxUserCount = 100;
goalLoadProfile.MaxUserCountIncrease = 10;
goalLoadProfile.MaxUserCountDecrease = 5;
goalLoadProfile.MinTargetValue = 20;
goalLoadProfile.MaxTargetValue = 25;
// This example assume there is only one scenario
loadTest.Scenarios[0].LoadProfile = goalLoadProfile;
loadTest.Scenarios[0].DelayBetweenIterations = 5;
}
}
2. LoadTestPlugin that modifies selected properties of a GoalBasedLoadProfile in the HeartbeatEvent handler (note this only works if the load profile specified in the .loadtest file is a Goal Based Load Pattern).
[Serializable]
public class LoadTestPluginChangeGoal : ILoadTestPlugin
{
private LoadTest m_loadTest;
private LoadTestScenario m_scenario1;
private bool m_goalChanged;
public void Initialize(LoadTest loadTest)
{
m_loadTest = loadTest;
// This example assume there is only one scenario
m_scenario1 = loadTest.Scenarios[0];
m_loadTest.Heartbeat += new EventHandler<HeartbeatEventArgs>(m_loadTest_Heartbeat);
}
void m_loadTest_Heartbeat(object sender, HeartbeatEventArgs e)
{
if (e.ElapsedSeconds >= 60 && !m_goalChanged)
{
LoadTestGoalBasedLoadProfile goalLoadProfile =
m_scenario1.LoadProfile.Copy() as LoadTestGoalBasedLoadProfile;
goalLoadProfile.MinTargetValue = 50;
goalLoadProfile.MaxTargetValue = 60;
m_scenario1.LoadProfile = goalLoadProfile;
m_goalChanged = true;
}
}
}
3. A custom LoadProfile implementation:
[Serializable]
public class DecreasingLoadProfile : LoadTestLoadProfile
{
public override int GetLoad(int elapsedSeconds)
{
int userLoad = Math.Max(
MaxUserCount - elapsedSeconds, MinUserCount);
return userLoad;
}
public override void Validate()
{
if (MinUserCount > MaxUserCount)
{
throw new InvalidLoadProfileException(
"MinUserCount is greater than MaxUserCount");
}
}
}
Behavioral notes and warnings:
· Changes to any of the properties of any of the LoadTestLoadProfile classes should all be made before assigning to the LoadTestLoadProfile object to the LoadProfile property of the LoadTestScenario. Once the object has been assigned to the LoadProfile property, it becomes a read-only version, and an exception will be thrown if you attempt to set any of the properties of the LoadProfile object while it is in the read-only state. To get a new writable LoadProfile object, call the Copy() method as shown in example 2 above.
· The LoadProfile and DelayBetweenIterations properties of LoadTestScenario can be assigned to in the Initialize() method or any of the LoadTestPlugin event handlers, except that assigning to either of these in the LoadTestFinished event handler is pointless.
· In a multi-agent test rig, changes to the LoadProfile or DelayBetweenIterations made by a Load Test Plug-in running on any of the agents are propagated to all other agents through the controller. If plug-ins running on different agents both change one of these, the last one to change the value will override the previous value on all agents. Also, there is no synchronization to guarantee that the agents receive and apply the updates at the same time, so especially with a large number of agents, some agents may being using the new settings before others, though in most cases the changes should be applied within seconds on all agents.
· Changes to the DelayBetweenIterations property affect delays between tests that begin after the property change message is received by the agent, but the length of any delay already in progress when the property is set is unaffected.
· Changes to the LoadProfile take affect just before the next HeartbeatEvent (which occur every one second) after the new LoadProfile has been received by the agent from the controller.
· Whenever a Load Test plug-in assigns a new object to the LoadProfile property of LoadTestScenaro, the Validate() method of the class that extends LoadTestLoadProfile is called. If Validate() throws an Exception, the exception is reported in the Load Test Results (UI and database), and the current load profile settings continue to remain in effect.
· Any or all of the performance counter identifier properties (MachineName, Category, Counter, and Instance) for a GoalBasedLoadProfile can be changed when a new GoalBasedLoadProfile is assigned by a plug-in.
· Custom implementations extending any of the LoadTestLoadProfile classes can only be used by assignment by the LoadTestPlugin – there is no support in the Load Test editor for choosing these custom load profile implementations.
· If a custom LoadTestLoadProfile class is used (as in example #3) and the Load Test is run on a rig, the .dll containing the custom class must be manually deployed to the LoadTest directory on the Controller each time the class changes (after stopping and before restarting the controller service).
New Classes:
The public APIs for the new LoadTestLoadProfile base class and the three system provided implementations of the standard load profiles are shown below. The properties shown correspond to the properties on the load patterns in the load test editor. In most cases the meaning of the property is reasonably obvious from the name. If not, you can find the property in the appropriate property sheet in the load test editor and select the property to see the description.
namespace Microsoft.VisualStudio.TestTools.LoadTesting
{
[Serializable()]
public abstract class LoadTestLoadProfile
{
// Public properties
public virtual int MaxUserCount { get {}; set {}; }
public virtual int MinUserCount { get {}; set {}; }
public string ScenarioName { get {} }
// Public abstract and virtual methods
public abstract int GetLoad(int elapsedSeconds);
// Throws InvalidLoadProfileException if the LoadProfile object has
// invalid or inconsistent properties (with an appropriate message)
public abstract void Validate();
public virtual LoadTestLoadProfile Copy() {}
}
[Serializable()]
public class LoadTestConstantLoadProfile : LoadTestLoadProfile
{
public LoadTestConstantLoadProfile() {}
// LoadTestConstantLoadProfile properties
public int UserCount { get {}; set {}; }
// Other properties and methods inherited from LoadTestLoadProfile
}
[Serializable()]
public class LoadTestStepLoadProfile : LoadTestLoadProfile
{
public LoadTestStepLoadProfile() {}
// LoadTestConstantLoadProfile properties
public int InitialUserCount { get {}; set {}; }
public int StepUserCount { get {}; set {}; }
public int StepDuration { get {}; set {}; }
public int StepRampTime { get {}; set {}; }
// Other properties and methods inherited from LoadTestLoadProfile
}
[Serializable()]
public class LoadTestGoalBasedLoadProfile : LoadTestLoadProfile
{
public LoadTestGoalBasedLoadProfile() {}
// Public properties
public int InitialUserCount { get {}; set {}; }
public int MaxUserCountIncrease { get {}; set {}; }
public int MaxUserCountDecrease { get {}; set {}; }
public string MachineName { get {}; set {}; }
public string CategoryName { get {}; set {}; }
public string CounterName { get {}; set {}; }
public string InstanceName { get {}; set {}; }
public float MinTargetValue { get {}; set {}; }
public float MaxTargetValue { get {}; set {}; }
public bool HigherValuesBetter { get {}; set {}; }
public bool StopAdjustingAtGoal { get {}; set {}; }
public override int MinUserCount { get {}; set {}; }
public override int MaxUserCount { get {}; set {}; }
// Other properties and methods inherited from LoadTestLoadProfile
}
}