Introduction to TestApi – Part 5: Managed Code Fault Injection APIs


Series Index

+++

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

  • AUT (application under test) – this is the tested application, in which faults are being injected;
  • Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
    • a Method Signature, determining the method where the fault will be injected;
    • a Fault Condition, determining when the specific fault should be triggered (e.g. every Nth call)
    • a Fault — Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
  • Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

//
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

> MyApplication.exe

0) 2 + 3 = 5

1) 2 + 3 = 5

2) 2 + 3 = 5

3) 2 + 3 = 5

4) 2 + 3 = 5

5) 2 + 3 = 5

6) 2 + 3 = 5

7) 2 + 3 = 5

8) 2 + 3 = 5

9) 2 + 3 = 5

 

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

// 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe

0) 2 + 3 = 5

1) 2 + 3 = -1000

2) 2 + 3 = 5

3) 2 + 3 = 5

4) 2 + 3 = 5

5) 2 + 3 = 5

6) 2 + 3 = 5

7) 2 + 3 = 5

8) 2 + 3 = 5

9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

FaultInjectionSample.zip

Comments (30)

  1. Eddie says:

    I am trying to use the FaultInjection libraries to simulate data access failures.

    Is this proper method signature to get the ExecuteNonQuery() method on DbCommand?

           Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()"

    Thanks.

  2. ivom says:

    Yep, this should work.

    Let me know if you have trouble and we will help.

  3. David says:

    I had success when using the fault injection libraries on some sample code I had written, but now that I’m trying to use it against our intended target code it doesn’t seem to be working.

    I want to use the ReturnValueFault on an internal static method which returns an int. The method is within an internal sealed class with the SecuritySafeCritical attribute set.

    I’m creating the fault rule like this:

    FaultRule rule = new FaultRule("static FullNamespace.ClassName.GetDeviceId(uint, IntPtr, IntPtr, IntPtr)", BuiltInConditions.TriggerOnEveryCall, BuiltInFaults.ReturnValueFault(1000));

    I also tried prepending out to IntPtr as these are specified as out parameters in the method signature. I didn’t have success either way. Am I doing something wrong here?

    My other thought is that this class and method are in a library/dll, and not within the exe code itself which is the process I’m starting a fault injection session for.

    Any help is greatly appreciated. So far I’ve been very happy with the low cost of entry for using this API and the good documentation.

  4. bill says:

    Hi David, can you try to replace "IntPtr" with "System.IntPtr" in the parameters list of the method? the data type of the parameter need to be fully qualified name too.

    let me know if it still doesn’t work.

  5. David says:

    I tried System.IntPtr and had no luck here either. Is it an issue if the binary I am trying to inject into is signed?

  6. bill says:

    no, the issue is not related to the sign. i noticed that you mention about ‘out parameter’. please make sure that ‘out’ is also part of the method signature. i.e your method signature should be:

    static FullNamespace.ClassName.GetDeviceId(uint, out System.IntPtr,out System.IntPtr,out System.IntPtr)

    i wrote a dummy app with the same method signature above, and the fault injection api works for me without problem. if this still doesn’t work for you. most likely reason here is that the method signature still not correct somehow, i.e it does not match what clr expecting in the runtime). a few thing you can duble check:

    1. make sure the method got invoked.

    2. make sure the method signature is correct or no typo (i often see people has typo in the method signature by mistakes).

    3. in the %temp% folder, there should have some log, check if there are some error.

  7. David says:

    I have tried with the out parameter and also it did not work. I coulnd’t find any logs in %temp% folder. I realized today that the AUT is a native program that runs the code from the managed assemblies by hosting a CLR runtime. Is this my problem, is there a way to still use TestApi fault injection or will I have to use some native fault injection library instead?

  8. bill says:

    ha. that’s the problem. when i saw the IntPtr parameter at the first place, i suspected that your aut might be using navtive code or p/invoke stuff which is out scope of this tool that targets to managed code only.  in you case, you have to use native fault injection tool or windows API hooker stuff.

    good luck.

  9. David says:

    Darn, this would have been the perfect solution. Thanks for your help!

  10. Eddie says:

    I am just getting around to responding to Ivan’s response to my original comment.

    Ivan, I am unable to get ExecuteNonQuery() to throw an exception for me. Here is all my code from within a (slightly verbose) unit test:

           Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()"

           ‘Dim condition = BuiltInConditions.TriggerOnNthCall(2)

           Dim condition = BuiltInConditions.TriggerOnEveryCall

           Const ExceptionMessage As String = "This is a fault-injected exception."

           Dim fault = BuiltInFaults.ThrowExceptionFault(New DataException(ExceptionMessage))

           Dim faultRule = New FaultRule(method, condition, fault)

           Dim s = New FaultSession(faultRule)

           Dim rp = TestDataFactory.CreateYrtReinsurancePolicy(PolicyNumberFactory.CreateUnique())

           Dim isExceptionThrown As Boolean

           Try

               rp.Save()

           Catch ex As DataException

               If ex.Message.Equals(ExceptionMessage) Then

                   isExceptionThrown = True

               End If

           End Try

           Assert.IsTrue(isExceptionThrown)

    Note that I gave up on the NthCall and went with EveryCall just to see if I could get it to work.

    Any ideas?

  11. bill says:

    if this is your complete version of the code, the fault injection won’t work, because i only see you set FaultSession, but not call Launch method. referring to the document/samples, in order to have fault injection works, you have to provide an executable which execute your target API (ExecuteNonQuery, in your case), and you set the fault injection rule and call launch method to launch your executable.

    I have a private version to better support unit test, but probably take a while to release publicly. for now, this is the high level work flow (suppose you want to inject fault into function Foo()).

    1. write an app to call Foo().

    2. set fault injeciton rule and session, as you did above.

    4. call Launch method to run your app. and when your app call Foo(), the fault will be injected.

    let me know if you have any questions.

  12. Eddie says:

    The problem with that workflow is that I would need to write a process wrapper for each and every unit test that I write. I think my solution file will get out of control quickly.

    Given that, I think that I will wait for your private version.

    Thanks for the clarification. I just didn’t infer that launching an additional process was necessary.

  13. Sanjeev says:

    Hi,

    When i try to add the FaultInjectionEngine.dll as a reference to my code in VisualC#, it is giving an error which says, "Please make sure that the file is accessible and that it is a valid assembly or COM object". Please help me on this.

  14. billliu says:

    In your VS project, you should add reference to TestApiCore.dll, not faultinjectionengine.dll.

  15. Sanjeev says:

    Yah did the same, Still the same error crops in.

  16. billliu says:

    hmm, that’s strange. if you haven’t, can you try to create a brand new project and add testapicore.dll?

    what’s the version of your VS, OS, 32bit or 64 bit?

    also check my blog which has step by step instructions. you can see if you missed any important one.

    http://blogs.msdn.com/billliu/default.aspx

  17. Using FI API's with Application hosted on IIS 7.0 says:

    Hi

    I am planning to use FI API's to introduce faults in to my application hosted on IIS 7.0.

    Can you please guide me on this.. ?

    Thanks

  18. ivom says:

    Re: fault injection in IIS 7.0:

    Check out Bill's recent post demonstrating that:

    blogs.msdn.com/…/using-fault-injection-api-for-web-application-or-windows-service.aspx

  19. Krupa says:

    Hi, I am new to this, I copied the sample solution given here and tried on visual studio 2010 . It simply isn't working .. I mean the fault injection test is not injecting the fault at all.

    Please help.

  20. krupa says:

    Hi , when i try to run the same sample project on VS 2010 (after conversion) on winXP(SP3) the fault injection simply doesnt work. Please help.

    I have tried the sample code on VS 2008 on a windows 7. It does work there though I follow the same steps.

  21. varinder singh grewal says:

    how to create custom fault and custom condition?

    please help

  22. bill says:

    for vs2010 or .net 4.0 application, you need to enable one env variable. see my blog for steps:

    blogs.msdn.com/…/if-it-still-not-working.aspx

  23. bill says:

    the following code snipet to show how to custom the condition. the condition is to trigger fault if the current machine is management server (regkey settings):

       [Serializable]

       class MyCondition : ICondition

       {

           public bool Trigger(IRuntimeContext context)

           {

               RegistryKey masterKey = Registry.LocalMachine.CreateSubKey("SOFTWARE\Microsoft\FaultInjection");

               string reg = "";

               if (masterKey == null)

               {

                   Console.WriteLine("Null Masterkey!");

               }

               else

               {

                   reg = masterKey.GetValue("ServerType").ToString();

                   Console.WriteLine("MyKey = {0}", masterKey.GetValue("ServerType"));

               }

               masterKey.Close();

               return reg.Equals("Management", StringComparison.OrdinalIgnoreCase);

           }

       }

  24. kannan says:

    Wait a minute. I thought you said

    "It uses the .NET Profiler API to dynamically instrument the binaries as they are running so that the binaries are only altered in memory and not on the hard disk. "

    But from the examples it seems you always have launch the application under test itself using the test apis/exe but can't actually target an already launched application and inject code into it, even though that's what you claimed.

  25. Ivo Manolov says:

    @kannan  

    That's correct. Notice that the claim and the code don't contradict each other.

    The API allows you to dynamically inject code in a binary, without having to do any post-build instrumentation steps. At the same time, in order to inject code you have to set up the execution environment appropriately to enable profiling for the process that gets created when the exe gets run. I.e., in the general case, you have to restart the process that you want to inject in.

    I hope this clarifies things,

    Ivo

  26. Performance Injector says:

    Thank you for sharing this very nice post, please keep continue the sharing of this types of information. Here we are waiting for more

  27. Ivo Manolov says:

    @Performance Injector:

    Thanks for the nice words! 🙂

    We are looking into extending the TestApi facilities further.

  28. none of yours says:

    i hate this it didnt give me what i need

  29. Ivo says:

    @none

    What didn't work? Can I help?

  30. zhenyu says:

    what if Class A implement Interface B, with method C. should I injection to specify to inject "A.C" or "B.C"?