CAPTURING AND ANALYZING AN ETW TRACE (event TRACING FOR WINDOWS)

  • Article

Continuing my conversations on using tools, today I want to explain how to capture an ETW trace and parse it. Event tracing for Windows (ETW) is a very powerful, tracing mechanism built into the Windows operating system that allows you to view messages from various subsystems. This is very helpful in troubleshooting problems on the server side. We use it a lot in the IIS support group to troubleshoot various customer issues. In an earlier post, I discussed and explained the various Providers in Windows that spit information. We will use some of these providers today.  To view a list of providers available on your system, run the following command in a command prompt window.

Logman query providers

Stage 1: Capture an ETW trace in Windows Server 2003 with Service Pack 1 or upwards.

NOTE: You must be logged on as a computer administrator to perform these steps:

  1. Create a Trace text file that contains a list of the providers that you want to trace.
  2. Create a folder called ETW in C drive root
  3. Open a command prompt.
  4. Switch to C:\ETW folder.
  5. Type: Logman Query Providers > Providers.txt <enter> [This will output the results into Providers.txt file]
  6. Open Providers.txt in Notepad. you will have 2 columns – Provider & GUID.
  7. Remove the lines of those providers you do not wish to use. For IIS it will be something like this

IIS: WWW Global {d55d3bc9-cba9-44df-827e-132d3a4596c2}
IIS: SSL Filter {1fbecc45-c060-4e7c-8a0e-0dbd6116181b}
IIS: Request Monitor {3b7b0b4b-4b01-44b4-a95e-3c755719aebf}
IIS: WWW Server {3a2a4e84-4c21-4981-ae10-3fda0d9b0f83}
IIS: Active Server Pages (ASP) {06b94d9a-b15e-456e-a4ef-37c984a2cb4b}
IIS: IISADMIN Global {DC1271C2-A0AF-400f-850C-4E42FE16BE1C}
IIS: WWW Isapi Extension {a1c2040e-8840-4c31-ba11-9871031a19ea}
HTTP Service Trace {dd5ef90a-6398-47a4-ad34-4dcecdef795f}

Now delete the values from the first column. We get:

{d55d3bc9-cba9-44df-827e-132d3a4596c2}
{1fbecc45-c060-4e7c-8a0e-0dbd6116181b}
{3b7b0b4b-4b01-44b4-a95e-3c755719aebf}
{3a2a4e84-4c21-4981-ae10-3fda0d9b0f83}
{06b94d9a-b15e-456e-a4ef-37c984a2cb4b}
{DC1271C2-A0AF-400f-850C-4E42FE16BE1C}
{a1c2040e-8840-4c31-ba11-9871031a19ea}
{dd5ef90a-6398-47a4-ad34-4dcecdef795f}

Now we will add flags to each provider. Flags indicate the areas to trace and the verbosity levels. Each flag will be separated by a TAB. After each entry, press Tab and type 0xFFFFFFFF, press Tab and type 0x5. For IIS: WWW Server only, this will be 0xFFFFFFFE 0x5. The flags 0xFFFFFFFF & 0xFFFFFFFE indicate all areas and 0x5 indicates full verbose mode.

Thus, we get:

{d55d3bc9-cba9-44df-827e-132d3a4596c2} 0xFFFFFFFF 0x5
{1fbecc45-c060-4e7c-8a0e-0dbd6116181b} 0xFFFFFFFF 0x5
{3b7b0b4b-4b01-44b4-a95e-3c755719aebf} 0xFFFFFFFF 0x5
{3a2a4e84-4c21-4981-ae10-3fda0d9b0f83} 0xFFFFFFFE 0x5
{06b94d9a-b15e-456e-a4ef-37c984a2cb4b} 0xFFFFFFFF 0x5
{DC1271C2-A0AF-400f-850C-4E42FE16BE1C} 0xFFFFFFFF 0x5
{a1c2040e-8840-4c31-ba11-9871031a19ea} 0xFFFFFFFF 0x5
{dd5ef90a-6398-47a4-ad34-4dcecdef795f} 0xFFFFFFFF 0x5

  1. Save the Providers.txt file.
  2. Go back to command prompt. We are now ready to start ETW Tracing.
  3. At the command prompt, type: Logman -start MyTrace -pf providers.txt –ets <enter>
  4. MyTrace is a name for the trace and is a mandatory parameter.

Now, we can reproduce the problem. Execute the HTTP request from any HTTP capable client or make the request via browser. Once the problem has been reproduced, we can stop the trace as follows:

At the command prompt, type: Logman –stop MyTrace –ets <enter>

We should now have a file called MyTrace.ETL in C:\ETW folder. This file is not readable using any editor. It first needs to be parsed.

Stage 2: Parsing an ETW Trace.

OK, so we now have a trace file that cannot be read by any editors because it is a binary file. To parse an ETW trace file, we need another tool – LogParser. I am sure many have heard great stories about this. This is another powerful tool and provided by Microsoft as a free download. Download LogParser v2.2. Then install it to C:\LogParser folder.

Important: You need to parse an ETW trace on the same version of the OS from where it was captured. This is because the ETW providers will be different for different versions of Windows.

Before you can use LogParser, there are a couple of things you need to do - Register LogParser.dll in system registry and make a slight modification to the parsing Windows script so that it doesn’t prompt you.

  1. Open another command prompt and switch to C:\LogParser folder.
  2. Type: Regsvr32 Logparser.dll <Enter>. Click OK after the message.
  3. Copy the DumpTraceReqs.js file from C:\LogParser\Samples\Scripts to C:\Logparser [this is just for convenience]
  4. Open DumpTraceReqs.js in Notepad.
  5. Search for the text: “//Prompt the user to press a key”
  6. Comment out the lines WScript.Echo("[Hit ENTER...]"); and WScript.StdIn.ReadLine(); It should be as
    • //WScript.Echo("[Hit ENTER...]");
    • //WScript.StdIn.ReadLine();
  7. Save the file & close Notepad.

We are now ready to parse the ETW trace file. At the second command prompt, type the following command and press ENTER

Cscript DumpTraceReqs.js C:\ETW\MyTrace.etl > Output.txt <enter>

Wait for the command to finish and then open the file, Output.txt in Notepad. You should now have a file that contains a very organized collection of information that shows the activities in each stage of the request processing pipeline. Any kind of permissions problems, compression failure etc should show up here. It is self explanatory. However, if you do have questions on what you see in the ETW trace file, please share it or post it here and I’ll help.

LogParser is a very powerful tool and you can use SQL like commands for a variety of purposes including reading event logs, IIS logs, Pure text files, Network trace files and many others. It also includes many functions which you can use to transform data. Take a look at this post that contains a lot of examples.

I hope you found this post useful in debugging problems. Please feel free to leave your feedback.