Rich Payload Data in EventSource V4.6.

In a previous blog I talked about the first of three interesting features of the new Version of V4.6 .NET EventSource, naming Activity tracking.

In this blog I would like to talk about one of the other two: EventSoruce Rich Data Payloads.

Note that .unlike the Activity support, this feature is fully available in the EventSource Nuget package, however, as always, we strongly encourage people to use the System.Diagnostics.Tracing.EventSource class built into the .NET framework if they can.   If can upgrade to V4.6 you can do this. 

 What are Rich Data Payloads?

As a review EventSource was predicated on the notion that you could pass TYPED payloads to your events for example

      void RequestStart(string url, int size) { WriteEvent(1, url, size); }

Defines a new event 'RequestStart' with a data payload that consists of two data items, the first is a string called 'url' and the second is an integer called 'size'.   EventSource keeps these track of these data item names and types so that processors (either EventListeners or ETW), can access both the data items names and types.   This makes mechanical processing of the event data easier. 

However before Version 4.6, the data items that could be passed were limited, to strings, primitive types, and a handful of special types (e.g. Guid and DateTime).   Now this set of data types covers ALOT ground (probably 95%+ or more of all items), however there are some times where being able to pass arrays (lists) of things are handy.   This is what Rich Data payloads gets you.   Thus there is no new API surface area with this feature, it is just that more things work than did before.   For example before the following EventSource definition would not have worked

      void LogInts(string name, int[] values) { WriteEvent(2, name, values); }

Because 'values' is not of the very few data types that are allowed (e.g. primitives and strings).  With rich data payloads, this call is now allowed. 

Rich Data does NOT mean 'Any Data'

A VERY IMPORTANT point however, is that RIch data does not mean 'anything'.   In almost all uses of EventSource the data you send will find its way OUTSIDE the process that created it.   It will either go to ETW, or to an EventListener that will log it to a file, or a database, or a cloud service of some kind.  In call these cases the data leaves the process that created it and MUST BE SERIALIZED.    As anyone who has dealt with serialization before knows, serialization is a 'hard' problem because frankly not all object can be 'copied' which is what you are trying to do when you serialize something.   Thus serialization schemes very quickly get complex and subtle, which are not good things in general, and especially in something like EventSource that is supposed to be used by 'everything' to do logging.

Thus there have to be rules that keep the data serializable.    The 'sweet spot' that EventSource picked can be summed up as 'Typed JSON'.   JSON can be thought of roughly as a the type closer of strings, structures, and arrays nested in a arbitrary way.    EventSource rich data does the same, but keeps the strong typing.  Thus you can have arrays and structures as well as primitive types, nested in arbitrary way.   Any type that implements IEnumerable<T> can be used as a value for an array, but for structures not just any type can be used.   It must be either

  1. A type marked with the [EventData] attribute.   This makes it clear that this type was intended to be serializable by EventSource.   The rule is that EventSource will serialize EVERY public PROPERTY (it does not do fields, only properties), and the types of those properties much follow the serializable rules.
  2. You can also use anonymous types (e.g  new { X = 3, y = "Hello"} to specify structure values. 

For example it is legal to have a declaration like this

  • void LogComplexValue(string name, object complexValue) { WriteEvent(3, name, complexValue); }

And a call site of

  •  myEventSource.LogComplexValue("hello", new { field1="hi", location = new { x = 3.4, y = 5.0 } }); 

 Which passes an object with a structure as 'complexValue' which has two fields. The first field is named  'field1' of type string and the second is named 'location' which is a structure with an 'x' and 'y' field of type double.  See the EventSoruce Rich Data Payloads spec for more details. 

Receiving Complex Values in EventListeners

Once you have logged a complex value with an EventSource, it is natural to ask exactly how you receive in an EventListener.    An important point is that EventListeners preserve the idea that the data is a COPY OF THE DATA.   Thus you do not get the object itself but a COPY OF THE DATA in your EventListener.    Thus you should not expect object identity, or event the same type.   In fact the rules are

  1. If the data item is an array, all you will know in the Listener is that it implemented IEnumerable<T> for some T
  2. If the data item is a structure, the object passed back in the listener implements IDictionary<string, T> for some T, where the keys are the field names, and the values are the field values.
  3. If the data item is a primitive type or string, a copy is passed through.    

Notice that in the case of a structure, an important transformation has happened, you passed in a object representing a T (often this type is anonymous) but what you get out is a IDictionary<string, T>, which is certainly not a T, however it is MORE CONVINIENT, since you don't need to use reflection to fetch the field values (which is what you would have to do otherwise in the common case that the value was an anonymous type).  See the EventSoruce Rich Data Payloads spec for more details. 

Rich Data Only Supported in Self-Describing ETW

If you are receiving data via EventListeners, you can skip this section as it only pertains to the case when EventSource data flows through the windows ETW system.   Before windows 10, ETW data needed to be declared using a 'manifest' and this manifest had to describe every possible event.   In Windows 10 a new 'self-describing' ETW format was introduced that does need the manifest.   This mechanism was also back-ported into WIndows 8 and 7 (although on those OSes you have to have a recent-enough servicing update).  

The key point, however is that rich data does NOT work on Manifest-based ETW, only on the newer 'self-describing' ETW format.   What his means to you is that if you are using rich data formats AND ETW you need to

  • Indicate to EventSource to use the newer format.   You do that by using the EventSource constructor that takes a EventSourceSettings argument and pass the EvetnSourceSettings.EtwSelfDescrigingEventFormat value.   It would look the line in blue below. 

    class MyEventSource : EventSource
    {
        private MyEventSource() : base(EventSourceSettings.EtwSelfDescribingEventFormat) { }

        public static MyEventSource Logger = new MyEventSource();

        // Definitions of events as normal...
    }

  •  If you use the EventSource(string) overload to create the EventSource (which is designed for use with the Write<T> API) it uses the Self-describing format by default so you don't need to do anything special
  • You need to be on an OS which supports the self-describing format (either Windows 10 or a late update to Windows 8 or 7).

Guidance

It is pretty important to realize that Rich data type ARE TRICKIER, then just using primitives.   They are also SIGNIFNCATNLY MORE EXPENSIVE.    Exactly how much more depends on what you are doing but a good rule of thumb is that if you are generating more than 100 events /second you should be concerned.  If you are over 10 events/sec you should be measuring and seeing just where you stand with respect to performance. 

Also keep in mind that there are other ways of dumping data into a payload.   In particular you could always serialize the data you want to send yourself as JSON and then just send that string as a field in EventSouce payload.   This makes ALOT of sense if you ALREADY had to generate the JSON payload anyway, because you can pass it through easily.  

I compare the Rich datatype support in EventSource much like reflection in the .NET Runtime.  It is a powerful feature, and when you REALLY need it is INDISPENSIBLE.   However like reflection it is expensive, and should be used with some care, with an understanding of its performance penalty.    If you are using it everywhere, and you log events frequently you are probably using it poorly. 

Summary

EventSource has a new ability to pass rich JSON-like payloads from EventSources to EventListeners (and ETW, on appropriate OSes).   It is available on V4.6 runtimes as well as the EventSource Nuget Package.    It is a powerful feature (like reflection), but also has a non-trivial cost (like reflection), so should be used with some care (like reflection), and performance measurements should be done if it is used on hot paths.   For more information see the EventSoruce Rich Data Payloads spec attached to this blog entry.

EventSourceRichPayloads.docx