Fault Injection For Managed Code
Introduction
As part of software testing, we often want to know how the software behaves under failure conditions and to know whether the software system correctly handles error conditions, such as low memory, loss of network connectivity or dependent components unavailable, etc... This actually particularly important for managed code of which big parts are to handle exception. However, this is extremely to test in the normal test environment, it's even harder to do through test automation. This is how fault injection comes into picture. Basically fault injection is to purposely inject fault into software code to simulate failure that occurs in a deterministic and repeatable way. There are many existing tools for fault injection purpose; however, most of them have some significant drawbacks such as statically instrument binaries, not flexible to inject fault or different fault, no API for test automation, etc...
To make fault injection easier, we developed a new Tool/API for fault injection into managed code. It allows you to manually or through test automation to test exception handling, manipulate different code path at run time, and also mocking API calls as well. If this sounds interesting to you, read on....
Under the hood.
Normally, this is how managed code got executed. Developers first writes managed codes and compiles into binaries which is MSIL (Microsoft Intermediate Language). At runtime, the method is invoked and CLR loads binary into memory to do second compilation, i.e Just-In-Time compilation. .Net framework provides a mechanism to monitor this process. Controlled by a set of environment variables, CLR at run time can trigger a set of events such as assembly loaded/unloaded, class loaded/unloaded, etc... One of the most useful events is JITCompilationStarted event which occurs right before the method got JITed. You can subscribe a callback function to this event, so it will call your callback function when the event occurs. One of the parameters to the callback function is FunctionID through which you can call GetILFunctionBody to get function's original IL code, then you can modify it in the memory (inject fault, in this case), as a result the modified IL code passed to the JIT compiler through SetILFunctionBody method. This is in high level how the managed fault injection API works. For .net profiling, please check MSDN (https://msdn.microsoft.com/en-us/library/ms404386(v=VS.80).aspx) for more details.
To use managed fault injection API, you need to simply provide the WHERE, WHEN, and HOW information:
1. WHERE: defines which method to be injected fault, i.e. target method’s signature.
2. WHEN: defines when the method should be faulted, i.e. trigger conditions. A set of build-in conditions like TriggerOnEveryCall, TriggerNthCalls, TriggerCalledBy, etc. are supported. In addition, users can define their own conditions by implementing provided interface Icondition.
3. HOW: defines the behavior of faulted method, which can be either to throw a specified exception or to return a specified value. Users can define their own fault by implementing provided interface Ifault.
I will show you how to use the API step by step in next section.
Let’s get started….
Step1: Prepare Application for testing (AUT)
Assume the following is your application code that you want to test. The logic is simple and self-explanatory.
using System;
using System.IO;
class SimpleExample
{
static int GetNumberOfProc()
{
return Environment.ProcessorCount;
}
static void CreateObjects()
{
if (GetNumberOfProc() == 1)
{
//todo: business logic if the machine is single processor.
Console.WriteLine("Single Processor.");
}
else
{
//todo: business logic if the machine is multiple processors.
Console.WriteLine("Mulitple Processors.");
}
//todo: allocate a large memory to create objects
}
static void Main(string[] args)
{
try
{
CreateObjects();
Console.WriteLine("Sucessfully creates objects.");
}
catch (OutOfMemoryException ex)
{
//todo: business logic if memory is low.
Console.WriteLine("the memory is low: " + ex.Message);
}
Console.WriteLine("Press any key to exit…");
Console.ReadLine();
}
}
Compile the code to aut.exe and run it. The output should be what your expecting.
Step 2: Download TestAPI package.
Go to codeplex website: https://testapi.codeplex.com/Wikipage to download and unpack to your local machine, for example: D:\tmp\TestApi_v0.4. As of now (3/30/2010), the latest version is v0.4.
Step3: Write Test Code
Basically I want to test the business logics when the memory is low.
1. In Visual Studio, create a new console application. Add reference to TestApiCore.dll. I define the rule that every time the method SimpleExample.CreateObjects is called, it will throw outofmemory exception. The following is the test source code.
using System;
using Microsoft.Test.FaultInjection;
namespace auttest
{
class Program
{
static void Main(string[] args)
{
//
// Set up a fault rule to throw outofmemory exception
//
string method = "SimpleExample.CreateObjects()";
ICondition condition = BuiltInConditions.TriggerOnEveryCall;
IFault fault = BuiltInFaults.ThrowExceptionFault (new OutOfMemoryException ("fault inject test"));
FaultRule rule = new FaultRule(method, condition, fault);
//
// Establish a session, injecting the faults defined by the fault rule(s)
//
FaultSession session = new FaultSession(rule);
ProcessStartInfo psi = session.GetProcessStartInfo(@".\aut.exe");
//
// Launch the target application for testing
//
Process p = Process.Start(psi);
p.WaitForExit();
}
}
}
2. Compile it into auttest.exe.
3. Copy FaultInjectionEngine folder from your unpacked folder to auttest.exe folder.
4. Run command line window under administrator privilege
5. Run auttest.exe, you can see aut.exe throw outofmemory exception.
6.
If you run aut.exe again, it will output the original output, which means the binary is not changed.
Congratulations! You got your first example works using managed fault injection API. Next time, I will show you a few more scenarios....
In this blog, I only touch base managed fault injection API, there are other very useful APIs for testing in the TestAPI package, check Ivo's blog (https://blogs.msdn.com/ivo_manolov/archive/2009/10/14/9907447.aspx) for more details.
----------------------------------------
关注我：新浪微博：@billliu_seattle https://www.weibo.com/windowsazure 或twitter: @billliu_seattle