Tracing Across Services

How can I put together a stack trace that goes between service operations on different services?

In a normal stack trace, all of the stack frames have a single observer that can view the sequence of calls and recreate them in the proper order. Some stack traces, even though they are located on a single machine, have segments that are separated by boundaries. These boundaries require multiple observers to work together to recreate the sequence. Distributed systems frequently encounter this type of boundary and there is a particular challenge getting the multiple observers to work together.

One technique for coordinating multiple observers is to push identifiers along with the calls that allow the original sequence to be reconstructed at a later time. The identifiers help thread together the call stacks created by the individual observers.

Service activities can be traced both with and without the capability to allow reconstruction across the boundary between two observers. Putting together a meaningful trace that spans services can be done by adding two options to the normal tracing configuration. The first option enables tracing activities. The second option enables propagating those activity traces between services. Here's an example of the configuration that would be written to enable these two options plus the required boilerplate. I've included a default listener but that is just where you'd put in whatever trace listener you're using.

 <configuration>
    <system.diagnostics>
        <sources>
            <source name="System.ServiceModel" switchValue="Warning,ActivityTracing" propagateActivity="true">
                <listeners>
                    <add type="System.Diagnostics.DefaultTraceListener" name="Default">
                        <filter type="" />
                    </add>
                </listeners>
            </source>
        </sources>
    </system.diagnostics>
</configuration>

Next time: Calling Services Without a Contract