One of the types of performance problems that can occur with IIS web applications is when the process hosting the web application (in IIS6 in Worker Process Isolation Mode it is w3wp.exe) spins up the CPU utilization of the server. Usually this is associated with end users experiencing a long period of waiting for the web content to be displayed in their web browsers. This scenario is generally referred to as a “high CPU hang”.
Troubleshooting high CPU hang problems can be done with two tools: Performance Monitor, and a debugger. We use Performance Monitor to get a ‘perfmon’ log of the high cpu behavior, and we use the debugger to get a hang dump of the process that is spiking the cpu. Then we analyze the info in the perfmon log and associate it with the info in the debug dump file. Debugging user-mode dumps is a whole topic in and of itself which I won’t cover in depth here. Instead the emphasis here is on getting the right data at the right time to actually analyse. In Part 2 of this topic I’ll go through a sample analysis of a high CPU scenario.
HOW DO I SET UP PERFMON TO GET THE RIGHT DATA?
Following these steps should get you the right data (and not much useless data that would confuse things) in perfmon:
1. In Performance Monitor, expand “Performance Logs and Alerts”
2. Right Click on “Counter Logs”
3. Choose “New Log Settings…”
4. Enter a descriptive name
5. Note the log file location for later (or go to the “Log Files” tab and change the location)
6. Click the “Add Counters” button
7. Click the “All Counters” radio button. Click the “Select instances from list” radio button, and select the process hosting the IIS or ASP.net application. On Windows 2000 and Windows XP, this will be inetinfo, dllhost, or aspnet_wp . On Windows 2003 this will be w3wp. NOTE: It is possible you will see multiple instances of dllhost on IIS5/5.1 or w3wp on IIS6 in the list of available processes. If you do, or if you are unsure of which process to monitor, please cltrl-click to multiselect all of the available IIS/ASP.net choices.
8. If the application is not an ASP.net application, select the following from the “Performance Object” dropdown, being sure to “Add” each one as you select it:
If the application is an ASP.net application, select the following from the “Performance Object” dropdown, being sure to “Add” each one as you select it:
• .NET CLR Data
• .NET CLR Exceptions
• .NET CLR Interop
• .NET CLR Jit
• .NET CLR Loading
• .NET CLR LocksAndThreads
• .NET CLR Memory
• .NET CLR Networking
• .NET CLR Remoting
• .NET CLR Security
• ASP.NET Applications
9. Click “Close”
10. Click “OK”
Note: For more information on Performance monitor, see http://
WHEN TO START PERFMON
Due to the nature of high cpu hangs, we don’t need any perfmon data collected when the CPU spike isn’t occurring. Logically, why would we need data of when everything is running fine? This is a common misconception of many people when troubleshooting high CPU issues; they let perfmon run and run collecting data when the problem isn’t occurring, and when the problem does occur and they save the perfmon log file, the resulting .blg file is so massive that it’s impossible to analyze.
So the first thing to remember in this data collection is, don’t start the perfmon log until after the problem starts occurring! In a perfect world, a high cpu hang will manifest itself in such a manner that the cpu usage of the process spikes up to 100% or so and stays spiked until the process is restarted. In these cases, getting the right perfmon data is easy. Of course, things get a bit more tricky if the high cpu behavior comes and goes on its own.
WHEN TO GET THE MEMORY DUMPS
Now that the high cpu hang is occurring and the perfmon log is running and collecting data, it’s time to capture the memory dump information. The two debuggers that I recommend using are the Microsoft Debug Diagnostics Toolkit and ADPlus (which is part of the suite of tools commonly referred to as Debugging Tools for Windows). Both of these tools can be downloaded via http://download.microsoft.com.
The important thing to note here is that the memory dumps must be captured while the high CPU hang is occurring and while the perfmon is collecting data. Otherwise we’ll have memory dumps of a process working correctly, or memory dumps with no associated perfmon data, respectively.
Another important thing to remember is, get the memory dump of the right process. If you’re not sure which process to capture the dumps of, look at Task Manager to see which process is spiking the cpu….that’s the one you want. If you’re still not sure which process to get the memory dump of, both DebugDiag and ADPlus.vbs contain functionality to automatically capture dumps of all of the currently running IIS processes.
In Part 2, I’ll give a walkthrough of troubleshooting a high CPU hang using perfmon data and memory dumps.