Explanation of “Off-Box Inclusive Testing” and how it impacts results and troubleshooting


Explanation of “Off-Box Inclusive testing” and how it impacts troubleshooting

An “Off-Box” request is any request that is made by a client to a machine that is not part of the core environment. As an example, if a user opens Internet Explorer and goes to the main Contoso.com website, Internet Explorer will go to the Contoso machines (the core environment). However the browser will also go to a few 3rd party sites on the Internet to retrieve images and possibly data. These “Off-Box” calls are made by Internet Explorer because the core environment told IE to go to this place to get the image or data.

 

Once the core environment tells IE where it should get the image or data, the core environment is no longer involved in any way with completing the final page in IE. The fact that these calls are made to external sources, which are shared by applications from other companies means we have no way of directly influencing, or even knowing what load or issues they might have that can change our numbers.

 

As an example, here is a simple test to the <Contoso> default web site. The request was made five times in a row and timing was measured from each one.

The results show:

·         The performance improvement between runs 3 and 4 was:

o   Total with OFF-BOX: 2.9 times longer (289%)

o   Total of ON-BOX only: 6.4 times longer (642%)

·         The performance improvement between runs 4 and 5 was:

o   Total with OFF-BOX: 1.6 times longer (157%)

o   Total ON-BOX only: 4 times faster (26%)

So test run 4 would give us a *false sense* that degradation was not nearly as bad as it really was.  Likewise, test run 5 makes us believe that our machine was slower than in test run 4, when in fact the machine itself was 4 times faster.

 

Here are the details of the above results, showing that performance improvement and degradation measurements involving substantial “off-Box” calls  can lead to erroneous interpretation of the results:

·         The page I requested was HTTP://www.contoso.com/Default.aspx

·         By making that single request, my browser was instructed to make 34 more requests to sites such as:

o   Images.buy-here.com

o   www.google-analytics.com

o   Origin.buy-here.com

o   Tracking.searchmarketing.com

·         My harness measured the time for every request, including dependent requests.


 

Explanation of “Dependent Request exclusion” testing

Dependent Request Exclusion testing is similar to Off-Box testing, but it involves not making calls to static dependent files which can either be served by cache or return a 304 Not Modified. Even though you can sort requests in the result set and target specific ones, the TEST and PAGE averages reported will be substantially impacted.

When to use these methods

If you are doing tuning and testing, or application specific measurements, you should include the above methodologies. If, however, you want to get a rough idea of client viewed response time, you should not utilize either of these methods.

How to use these methods

If you want to exclude the types of requests listed above, you can use one of the below webtest plugins.

 

    [DisplayName("TSL Toggle Dependent Requests")]
    [Description("Enables or Disables all dependent requests in the web test")]
    public class ToggleDependentRequests : WebTestPlugin
    {
        #region Properties
        [DisplayName("Disable Dependent Requests")]
        [Description("If true, dependent requests will NOT be executed")]
        [DefaultValue(true)]
        public bool bDisableDependentRequests { get; set; }
        #endregion //Properties

 

        public override void PreRequest(object sender, PreRequestEventArgs e)
        {
            if (bDisableDependentRequests)
                e.Request.ParseDependentRequests = false;
        }
    }

 

    [DisplayName("TSL Skip Calls To URL")]
    [Description("Removes any requests that do meet a certain criteria WRT the URL of the request.")]
    public class SkipAllCallsToUrl : WebTestPlugin
    {

 

        [DisplayName("Process Redirect Only")]
        [Description("If true, this plugin will only execute on a redirect request. None of the main requests in the webtest will be affected.")]
        [DefaultValue(false)]
        public bool ProcessOnlyRedirection { get; set; }

 

        [DisplayName("Plugin Enabled")]
        [Description("If set to FALSE, the plugin will not perform any work")]
        [DefaultValue(true)]
        public bool bEnabled { get; set; }

 

        [DisplayName("Prohibited Text")]
        [Description("Excludes any request that contains this string value/or a context param. " +
            "If this value is empty, the rule will remove any dependent requests with a different Base URL than the main request.")]
        [DefaultValue("")]
        public string RequestsURLCannotContain { get; set; }

 

        public override void PreRequest(object sender, PreRequestEventArgs e)
        {
            if (bEnabled)
            {
               
                    // NOTE: This routine is what handles context params passed in as values in the plugin
                    string cleanRequestsURLCannotContain = "";
                    if (RequestsURLCannotContain.Contains("{{"))
                    {
                        RequestsURLCannotContain = RequestsURLCannotContain.Replace("{{", "");
                        RequestsURLCannotContain = RequestsURLCannotContain.Replace("}}", "");
                        cleanRequestsURLCannotContain = e.WebTest.Context[RequestsURLCannotContain].ToString();
                    }
                    else
                        cleanRequestsURLCannotContain = RequestsURLCannotContain;
       
                if (e.Request.Url.ToString().Contains(cleanRequestsURLCannotContain))
                {
                    e.Instruction = WebTestExecutionInstruction.Skip;
                }
            }
        }

    }

 

 

UPDATE: New code from a teammate of mine

Robert George ("Offending the testing gods") shared this code with me which offers much more flexibility than the above code. So if you want to keep it simple, use the above. If you want capability, use the below.

 

//*********************************************************
// Copyright (c) Microsoft. All rights reserved.
// THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
// ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
// IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
// PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
//
//*********************************************************
     
using System;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools;
using System.ComponentModel;
using System.Diagnostics;

namespace MTSL
{
    [Description("Utility plugin that supports various exclusion criteria for requests that you can define.")]
    public class ExcludeRequestsPlugin : WebTestPlugin
    {
        //constructor
        public ExcludeRequestsPlugin()
        {
            this.DependentMustContain = string.Empty;
            this.DependentMustNotContain = string.Empty;
            this.PrimaryMustContain = string.Empty;
            this.PrimaryMustNotContain = string.Empty;
            this.ListDelimiter = ";";
            this.RemoveAllDependentRequests = false;
            this.Enabled = true;
            this.SuppressOutputComments = false;
            this.IgnoreCase = false;
        }
        [Description("Allows you to supress comments output by this plugin by setting true/false")]
        [DefaultValue(false)]
        public bool SuppressOutputComments { get; set; }

        [Description("Allows you to enable/disable this webtest plugin by setting true/false")]
        [DefaultValue(true)]
        public bool Enabled { get; set; }

        [Description("Allows case insensitve comparisons by setting IgnoreCase=true. Note: Context Paramaters are always treated as case sensitive.")]
        [DisplayName("Ignore Case")]
        [DefaultValue(false)]
        public bool IgnoreCase { get; set; }

        [Description("Automatically removes all dependent requests for every request")]
        [DefaultValue(false)]
        public bool RemoveAllDependentRequests { get; set; }

        [Description("Excludes any dependent request that does not contain this string value. Can use context param {{Paramxyz}} syntax also. Example: request -> http://99.1.1.10/images/img.jpg would not be processed if you specify http://10.100.100.10. This can also be a delimited List of values if you have more than one.")]
        [DefaultValue("")]
        [DisplayName("Dependents Must Contain")]
        public string DependentMustContain { get; set; }

        [Description("Excludes any dependent request that contains this string value/or a context param.In case of conflict, this property has higher precedence.  For example: http://99.1.1.10/images/img.jpg would not be requested if the property was set to img.jpg. This is very useful for dealing with requests that are known to cause 404 responses. This can also be a delimited List of values if you have more than one.")]
        [DefaultValue("")]
        [DisplayName("Dependents Must Not Contain")]
        public string DependentMustNotContain { get; set; }

        [Description("Excludes any primary request that does not contain this string value. Can use context param {{Paramxyz}} syntax also. For example: http://99.1.1.10/images/img.jpg would not be processed if you specify http://10.100.100.10. This can also be a delimited List of values if you have more than one.")]
        [DefaultValue("")]
        [DisplayName("Primary Must Contain")]
        public string PrimaryMustContain { get; set; }

        [Description("Excludes any primary request that contains this string value. In case of conflict, this property has higher precedence. You may use context param {{Paramxyz}} syntax also. For example: http://99.1.1.10/images/img.jpg would not be processed if you specify img.jpg as the value for the property. This can also be a delimited List of values if you have more than one.")]
        [DefaultValue("")]
        [DisplayName("Primary Must Not Contain")]
        public string PrimaryMustNotContain { get; set; }

        [Description("Enter the delimiter that you would like to use to seperate list values in the other propeties in the plugin. For example, use a comma or semi-colon to seperate the ExcludeDependentRequests properties.")]
        [DefaultValue(";")]
        [DisplayName("List Delimiter")]
        public string ListDelimiter { get; set; }

 

        public override void PreRequest(object sender, PreRequestEventArgs e)
        {
            //call base class PreRequest
            base.PreRequest(sender, e);

            // only if the plugin is enabled we will continue
            if (this.Enabled)
            {
                //if the request has already been set to skip, by another plugin for example, then there is no reason to
                //execute the work in this plugin
                if (e.Instruction == WebTestExecutionInstruction.Skip)
                {
                    if (!this.SuppressOutputComments)
                    {
                        e.WebTest.AddCommentToResult(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
                        Debug.WriteLine(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
                    }
                    return;
                }

                //if they dont want any dependents we just set the request ParseDependentRequests = false
                if (this.RemoveAllDependentRequests)
                {
                    e.Request.ParseDependentRequests = false;
                    if (!this.SuppressOutputComments)
                    {
                        e.WebTest.AddCommentToResult(this.ToString() + " -> set ParseDependentRequests to FALSE, no dependent requests will be executed.");
                        Debug.WriteLine(this.ToString() + " -> set ParseDependentRequests to FALSE, no dependent requests will be executed.");
                    }
                }

                //our private string vars to work with
                string mustContain = this.PrimaryMustContain;
                string mustNotContain = this.PrimaryMustNotContain;

                //If they are using a context param, we need to fix this up for them.
                //We know the context param format because it contains the "{{" as an example-> {{Parameter1}}.
                if (mustContain.Contains("{{") || mustNotContain.Contains("{{")) //this is just a short circuit if we have no processing needed
                {
                    //make sure the context param exists
                    if (e.WebTest.Context.Keys.Contains(this.PrimaryMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
                    {
                        //assign the current value of the context param into our string
                        mustContain = e.WebTest.Context[this.PrimaryMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
                    }
                    else if (this.PrimaryMustContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
                    {
                        //throw to let them know we do not have their context param available to work with
                        throw new Exception("-- ERROR: ContextParameter Name='" + mustContain + "'  was not found during processing in " + this.ToString());
                    }
                    //make sure context param exists
                    if (e.WebTest.Context.Keys.Contains(this.PrimaryMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
                    {
                        mustNotContain = e.WebTest.Context[this.PrimaryMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
                    }
                    else if (this.PrimaryMustNotContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
                    {
                        //throw to let them know we do not have their context param available to work with
                        throw new Exception("-- ERROR: ContextParameter Name='" + mustNotContain + "'  was not found during processing in " + this.ToString());
                    }
                }

                //create our string array to contain the values we need to check for               
                string[] arrayPrimaryMustContain = mustContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);
                string[] arrayPrimaryMustNotContain = mustNotContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);

                //first we will check the request to see that it contains the required value
                foreach (var item in arrayPrimaryMustContain)
                {
                    if (e.Request.UrlWithQueryString.Contains(item.ToString(),this.IgnoreCase))
                    {
                        // the request is valid
                        e.Instruction = WebTestExecutionInstruction.Execute;
                        break; //exit for loop since we have a match, once valid it cannot be invalid
                    }
                    else
                    {
                        //the request does not contain the value we are searching for, exclude the request
                        e.Instruction = WebTestExecutionInstruction.Skip;
                    }
                }

                //now we will check the request for the case of not containing
                //it should be noted that this check has precedence over the must contain in a conflict
                foreach (var item in arrayPrimaryMustNotContain)
                {
                    if (e.Request.UrlWithQueryString.Contains(item.ToString(),this.IgnoreCase))
                    {
                        // the request is valid
                        e.Instruction = WebTestExecutionInstruction.Skip;
                        break; //once invalid that is it
                    }
                }

                //if we skipped the instruction then add to comments
                if (e.Instruction == WebTestExecutionInstruction.Skip)
                {
                    if (!this.SuppressOutputComments)
                    {
                        e.WebTest.AddCommentToResult(this.ToString() + " forced skip of -> " + e.Request.UrlWithQueryString.ToString());
                        Debug.WriteLine(this.ToString() + " forced skip of -> " + e.Request.UrlWithQueryString.ToString());
                    }
                }

                Array.Clear(arrayPrimaryMustContain, arrayPrimaryMustContain.GetLowerBound(0), arrayPrimaryMustContain.Length);
                arrayPrimaryMustContain = null;
                mustContain = null;
            }
        }

        public override void PostRequest(object sender, PostRequestEventArgs e)
        {
            base.PostRequest(sender, e);

            if (!this.Enabled)
            {
                if (!this.SuppressOutputComments)
                {
                    e.WebTest.AddCommentToResult(this.ToString() + " is currently disabled. No Processing was performed");
                    return;
                }
            }
            //we only need to process if there are dependents
            if (e.Request.DependentRequests.Count > 0)
            {
                //If RemoveAllDependentRequests is true we should never get here, since its wired up in hte PreRequest event
                //to turn off parsing of dependents, so this is just a precaution
                if (this.RemoveAllDependentRequests)
                {
                    e.Request.DependentRequests.Clear();
                    //cleared now, bail out
                    if (!this.SuppressOutputComments)
                    {
                        e.WebTest.AddCommentToResult("Removed All Dependent Requests as specified by plug-in");
                        return;
                    }
                }

                //our private string vars to work with
                string mustContain = this.DependentMustContain;
                string mustNotContain = this.DependentMustNotContain;

                //If they are using a context param, we need to fix this up for them.
                //We know the context param format because it contains the "{{" as an example-> {{Parameter1}}.
                if (mustContain.Contains("{{") || mustNotContain.Contains("{{")) //this is just a short circuit if we have no processing needed
                {
                    //make sure the context param exists
                    if (e.WebTest.Context.Keys.Contains(this.DependentMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
                    {
                        //assign the current value of the context param into our string
                        mustContain = e.WebTest.Context[this.DependentMustContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();
                    }
                    else if (this.DependentMustContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
                    {
                        //throw to let them know we do not have their context param available to work with
                        throw new Exception("-- ERROR: ContextParameter Name='" + mustContain + "'  was not found during processing in " + this.ToString());
                    }
                    //make sure context param exists
                    if (e.WebTest.Context.Keys.Contains(this.DependentMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)))
                    {
                        mustNotContain = e.WebTest.Context[this.DependentMustNotContain.Replace("{{", string.Empty).Replace("}}", string.Empty)].ToString();                      
                    }
                    else if (this.DependentMustNotContain.Contains("{{")) //this will only be evaluated if the context did not already contain the key
                    {
                        //throw to let them know we do not have their context param available to work with
                        throw new Exception("-- ERROR: ContextParameter Name='" + mustNotContain + "'  was not found during processing in " + this.ToString());
                    }
                }

                //create our string array to contain the values we need to check for               
                string[] arrayDependentMustContain = mustContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);
                string[] arrayDependentMustNotContain = mustNotContain.Split(new string[] { this.ListDelimiter }, StringSplitOptions.RemoveEmptyEntries);

                //list to hold the matches, later we will remove those
                List<WebTestRequest> dependentsToRemove = new List<WebTestRequest>();

                //we loop through each dependent request
                foreach (var dependentRequest in e.Request.DependentRequests)
                {
                    //assume the dependent request is not valid
                    bool IsDependentValid = true;

                    //we loop through our must contain array to check for a match
                    foreach (var itemMustContain in arrayDependentMustContain)
                    {
                        //if no match occurs we need to remove the dependent
                        if (dependentRequest.Url.Contains(itemMustContain,this.IgnoreCase))
                        {
                            //valid match - once valid, no more processing needs to occur in
                            //this loop
                            IsDependentValid = true;
                            break;
                        }
                        else
                        {
                            //not valid
                            IsDependentValid = false;
                        }
                    }

                    //we loop through our must not contain array to check for a match
                    //this rule will take precedence over the must contain
                    foreach (var itemMustNotContain in arrayDependentMustNotContain)
                    {
                        //if a match occurs we need to remove the dependent
                        if (dependentRequest.Url.Contains(itemMustNotContain,this.IgnoreCase))
                        {
                            IsDependentValid = false;
                        }
                    }

                    //add the dependent to the list if IsDependentValid false
                    if (!IsDependentValid)
                    {
                        dependentsToRemove.Add(dependentRequest);
                    }

                }

                //remove matches from the DependentRequests collection.
                foreach (var item in dependentsToRemove)
                {
                    e.Request.DependentRequests.Remove(item);
                }

                //if we removed some dependents, let them know we did
                if ((dependentsToRemove.Count > 0) && (!this.SuppressOutputComments))
                {
                    e.WebTest.AddCommentToResult("-- Removed " + dependentsToRemove.Count + " Dependent Requests as specified by " + this.ToString());
                }
            }
        }
    }
    public static class ExtensionMethodsString
    {
        public static bool Contains(this string strValue, string strComparison, bool IgnoreCase)
        {
            if (IgnoreCase)
            {
                return strValue.ToLower().Contains(strComparison.ToLower());
            }
            else
            {
                return strValue.Contains(strComparison);
            }
        }
    }

    [Description("Utility plugin that allows you to skpip an individual request base on exsistence of a Context Paramater that you specify")]
    public class SkipRequestContextParam : WebTestRequestPlugin
    {
        [Description("Allows you to supress comments output by this plugin by setting true/false")]
        [DefaultValue(false)]
        public bool SuppressOutputComments { get; set; }

        [Description("Allows you to enable/disable this webtest plugin by setting true/false")]
        [DefaultValue(true)]
        public bool Enabled { get; set; }

        [Description("Name of the context parameter that is required. If this Context Param does not exist this request will be skipped.")]
        [DefaultValue("")]
        public string ContextParameterName { get; set; }

        [Description("If true, then the string length of the value contained in the Context Param Value must be at least 1 character. If there is no value set then the request will be skipped.")]
        [DefaultValue(true)]
        public bool RequireContextParamValue { get; set; }

        public SkipRequestContextParam()
        {
            this.SuppressOutputComments = false;
            this.Enabled = true;
            this.ContextParameterName = string.Empty;
            this.RequireContextParamValue = true;
        }

        public override void PreRequest(object sender, PreRequestEventArgs e)
        {
            base.PreRequest(sender, e);

            if (!this.Enabled)
            {
                if (!this.SuppressOutputComments)
                {
                    Debug.WriteLine(this.ToString() + " is currenlty disabled, no processing will occur.");
                    e.WebTest.AddCommentToResult(this.ToString() + " is currently disabled, no processing will occur.");
                }
                return; //no processing required              
            }

            //if the request has already been set to skip, by another plugin for example, then there is no reason to
            //execute the work in this plugin
            if (e.Instruction == WebTestExecutionInstruction.Skip)
            {
                if (!this.SuppressOutputComments)
                {
                    e.WebTest.AddCommentToResult(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
                    Debug.WriteLine(this.ToString() + " -> request instruction already set to Skip. No processing by this plugin occured.");
                }
                return;
            }

            //make sure the cotext param value has been set, otherwise thrown an exception
            if (string.IsNullOrEmpty(this.ContextParameterName))
            {
                throw new ArgumentException("The Context Parameter Name Property was Null or Empty", "ContextParameterName");
            }
            //check for existence of the context param and skip if not found
            if (!e.WebTest.Context.Keys.Contains(this.ContextParameterName))
            {
                e.Instruction = WebTestExecutionInstruction.Skip;
            }
            //check for length of string in the context param is not null or empty.
            else if (string.IsNullOrEmpty(e.WebTest.Context[this.ContextParameterName].ToString()))
            {
                e.Instruction = WebTestExecutionInstruction.Skip;
            }
            //output the status of the plugin unless suppress output is true
            if ((!this.SuppressOutputComments) && (e.Instruction == WebTestExecutionInstruction.Skip))
            {
                e.WebTest.AddCommentToResult(this.ToString() + " forced Skip of request->" + e.Request.Url);
                Debug.WriteLine(this.ToString() + " forced Skip of request->" + e.Request.Url);
            }
        }
    }
}

 

Comments (0)

Skip to main content