A SQL Profiler trace Swiss Army Knife
Have you ever needed to find your most expensive queries and quickly grew weary of writing T-SQL against trace tables to try to ferret them out? Have you ever had to wade through gigabytes of trace data just to find one ill-behaving query? Have you ever struggled to decide what performance metrics really matter when analyzing Profiler traces: duration, reads, writes, etc? Today’s post is about a new tool Bart and I wrote to help you do all this and more.
Origins
The original concept behind Retrace was to create a simple tool to load a SQL Server Profiler trace file into a database ala the Relog tool that ships with Windows (and provides similar functionality for Perfmon logs). Retrace would “re-trace” a session of SQL Server activity as a server-side trace table. I often needed to query trace files using T-SQL and didn’t like resorting to fn_trace_gettable every time, so Retrace was born to automate that process for me. That was the original idea.
Of course, these kinds of things tend to take on a life of their own, and Retrace soon began to demand that I enhance it in all sorts of interesting ways. Around this same time, my friends in CSS got underway with getting the SQL Nexus performance analysis platform that Bart and I built released via CodePlex. (Great news: it’s out there now. More on that later.) One of the cooler features we built for Nexus was a facility we called “TraceBuster.” Nexus supports this notion of generic diagnostic data loaders that implement a common interface. Nexus interacts with these assemblies via this generic interface, and users can build their own loader assemblies so long as they adhere to the interface we expose. TraceBuster is a Nexus loader we built to process SQL Profiler trace files and load them into a data warehouse. It automatically parameterizes the queries it reads from each trace such that the performance of a query executed with different parameter sets can be tracked over time. Unlike SQL Profiler itself, TraceBuster is able to identify and aggregate performance statistics such as CPU use, duration, reads, writes, etc., for these query “templates” rather than treating multiple instances of the same query with different parameters as different queries. Lightweight, fast, and extremely simple from a coding standpoint, it does exactly what we wanted it to do and no more.
For a number of reasons, it was decided that TraceBuster would not be included in the CodePlex release of Nexus (though you can still get it here in binary form). That got Bart and I to thinking. How cool would it be if Retrace used TraceBuster and basically provided a nice command line tool to load SQL Profiler traces into a data warehouse? What if Retrace was merely a thin console app shell over the TraceBuster assembly originally designed for Nexus? Since Nexus interacted with TraceBuster and its other diagnostic loaders generically, there was no reason a small console app couldn’t do the same thing without requiring Nexus itself to be on the box. Rather than merely load the trace into a trace table on the server, what if we let TraceBuster do all of its cool aggregation and analysis for us, thus saving all kinds of work manually querying the trace files? And what if on top of that we used the client-side Reporting Services reports that Bart developed for TraceBuster’s data warehouse and had Retrace display them? Then we’d have a single tool that could blaze through our trace data, load it into a data warehouse for further perusal, and finally display useful reports over that data. How cool would that be?!
So, we spent a couple late nights and a weekend here or there and morphed Retrace into a “Swiss Army Knife” for Profiler traces. One of the things we found ourselves often doing in our tuning work within SQL Server development was running a SQLDiag collection immediately before doing something we wanted to measure on the server, then loading the Profiler traces it collected via Retrace. Given Retrace’s one-stop-shopping nature, we thought it made sense to build that functionality into Retrace, so we added a command line option to Retrace to allow it to call SQLDiag automatically before a load operation.
When it was all said and done, Retrace knew how to collect Profiler traces, to load those traces into a performance warehouse on the server, and to run reports over that warehouse. We had a simple little console app that automated many of the tasks we found ourselves doing over and over as we tuned this or that component in our day jobs.
Summary
Retrace knows how to do three basic things:
Collect a SQL Profiler trace (using the SQLDiag utility that ships with SQL Server 2005 and later)
Load Profiler traces into a data warehouse that resides in a SQL Server database. These can be traces it collects via SQLDiag or traces from other sources (e.g., SQL Profiler)
Show reports listing the top N most expensive queries from the trace files loaded into the warehouse
Examples
To load a trace file into a data warehouse on the default SQL Server instance using Windows authentication:
retrace -i mytrace.trc
To load a trace file into a data warehouse and display the GUI (Reporting Services-based) reports:
retrace -i mytrace.trc -r
To load a trace file into a data warehouse and display the top N queries report in HTML format:
retrace -i mytrace.trc -f html -r
To start a Profiler trace collection, then load the collected trace file(s) into a data warehouse:
retrace -c TRACE
To start a Profiler trace collection, load the collected trace file(s) into a data warehouse, and then display the GUI reports:
retrace -c TRACE -r
To load all trace files in a given folder into a data warehouse and display the GUI reports:
retrace -i myfolder\*.trc -r
To load a trace file into a data warehouse and write the top N queries report to an XML file:
retrace -i mytrace.trc -f xml_full -o myreport.xml
To display the GUI reports (following a previous load of trace data into the warehouse):
retrace -r
To load a trace file into a database other than [retrace] or to a non-default SQL Server instance:
retrace -d mydatabase -S myserver\myinstance -i mytrace.trc
As you can see, you can load one or more trace files from the command line. These can be ones you instructed the tool to collect via SQLDiag or ones you’ve collected through other means (e.g., SQL Profiler). You can then display reports showing the top N most expensive queries in a variety of formats (HTML, XML, PDF, Excel, JPEG, etc.) The default report format is a simple GUI report that makes use of client-side Reporting Services report files (you don’t need to have Reporting Services itself installed).
Customizing Retrace
You can customize Retrace’s output in a variety of ways. You can customize the RDLC files that make up the reports shown in the Retrace GUI if you like. And you can also modify the XML stylesheet (XSL file) that Retrace uses to translate its XML output into HTML when HTML output format is selected. This stylesheet is named retrace.xsl and resides in the XML subfolder. Here’s what the HTML output looks like by default:
You can customize the T-SQL script Retrace uses to create its data warehouse and reporting stored procedures by modifying the TraceAnalysis.sql script file in the Scripts subfolder. Be careful here, however, as modifying the objects in this script may prevent Retrace from being able to display meaningful analysis reports.
You can also customize the Retrace application itself. Retrace is a ScriptIt application. For those who don’t know what ScriptIt is, it’s a technology developed internally within Microsoft that allows you much of the flexibility you have with scripting tools such as Perl and VBScript with managed code applications. It combines the ease of deployment and quick modification you see in traditional scripting tools with being able to develop in your favorite managed code language. It takes advantage of the fact that every machine that has the .NET Framework 2.0 or later installed also has the full MSBuild environment (the toolset used by Visual Studio to compile and build application binaries). Using ScriptIt, you can modify the source of a managed code application, rebuild it, and redeploy using only the executable—you don’t need to store the source in separate files or keep track of solutions and projects in Visual Studio. You can edit your apps in Notepad if you like, then rebuild them on the spot without need of any other tools. The current plan is for ScriptIt to be released to the public next year, and you can contact Vance Morrison, its author, if you have any questions about it. Once ScriptIt is released, you’ll be able to edit Retrace’s source code and make whatever changes you need without resorting to full-blown application development.
SQL Nexus
I mentioned earlier that SQL Nexus has been released on CodePlex. This means that you can download the current binaries and source code today and extend/use Nexus in your own work. Read the license agreement over there for details. We will continue to keep a binary version of TraceBuster available here in case you’d like to use it with Nexus. If you get Nexus, you have a much more graphical, full-featured performance analysis tool than Retrace was intended to be. That said, if you like simple, fast command-line tools, you may find Retrace useful. I wrote a good chunk of both tools, and I think each has its place.
Conclusion
They say necessity is the mother of invention, and I guess necessity is just as responsible for Retrace as are Bart and I. Retrace was a tool we needed to do our jobs. It evolved into its current form through the necessity of our both needing a Swiss Army Knife-type Profiler trace tool and our belief that TraceBuster provided useful functionality that we should continue to leverage in our work on the product. Download it, use it to carve up a few of your own perf problems, and let us know what you think.