Troubleshooting High Processor Utilisation usage by IIS Processes

  • Article

As a New Year's Resolution I'm planning to blog at least once a week. Hopefully I will maintain this rate but only time will tell.

I've had a lot of issues in the last couple of months so I wanted to talk about the approach I take to analysing these issues.

But first of all let me tell you a little story.

A couple of years ago I realised that I was very busy at work but when I looked back on my day I never seemed to achieve a lot.

So I took some time out and analysed how I was working and made some changes and found that I was spending time on other areas while waiting for another task to be completed.

E.G

I was checking e-mail while waiting for a long running Debugger command to finish or I was working on another cases while copying down some dump files.

So I started to focus on the one task in hand rather than multi-tasking and I found that I got some more done. (I'll do another post later on about some really good online resources about this area.)

So in computer terms a process is consuming a lot of the Processor Time when the Threads are taking a long time to get something done.

So when I get an issue where IIS is consuming a lot of the Processor Time normally start at the very beginning and ask questions like.

1) Is this something that started to happen recently or is this the normal operation of this server?
2) What is IIS doing and have any changes been made recently to the Applications hosted on this server?
3) What versions of IIS, Asp.Net and ASP are you using?

Once we have an idea of what the server is doing then we can get to work on analysing what the issue is.

The first place I like to start is establishing a baseline of what good performance looks like.

If the processor utilisation issue is intermittent then we normally have a good idea of what the baseline is for good performance.

If the processor utilisation issue is constant then we need to establish the baseline ourselves. This is where the results from your test environment will help enormously.
If you've got a good idea of how your Web Server is performing in Test under your expected load then this is probably the baseline.

So now that we've established the baseline the next step would be to gather the data for when the Processor Utilisation is high.

This is where we start by establishing the pattern of events.

I normally ask for a Performance Monitor Counter log covering the period when the processor utilisation is high and then 2 or 3 IIS Memory Dump files from the same period.

It's very important that the Dump Files and the Performance Monitor log files are from the same time period.

This is because we need the Performance Monitor Counter Logs to establish what Process and Threads are using the Processor Time and then the dump files will help us establish what is happening within the Process and those threads.

The Performance Monitor Counter log should contain at least the Processor, Process and Threads Performance Monitor Objects.

The theory goes like this.

1) The Processor %Processor Time counter should be showing between 90 to 100% CPU and this would normally be spread equally across all of the Processors if this is a a multi Processor machine.
2) The Process % Processor Time counter would normally show that one Process is taking this processor time. This is normally W3WP.exe on IIS 6 servers.
3) The Thread % Processor Time counter would normally show that one or more Threads in the W3WP.exe process are consuming the processor time.

Then I go to the dump files and look at those threads to understand what they are doing.

This normally helps us understand what is causing the issue and how we can solve it.

I also think it's worth mentioning that a dump file is a snapshot of a moment in time and that multiple dump files is always very important to understanding what is happening.

The issue with analysing one dump file for a performance issue is that you never now what happens next. That thread that looks like a suspect could have finished processing 2 milliseconds after the dump file is taken.

Next week I think I might run through a sample issue of high processor utilisation but until then you can take a look at this Tess On Performance Issues to see some examples of Asp.Net issues resulting in high processor utilisation.

(Edit on the 18th of January)

I've also come across this link to a similar article on the Ask Performance Team Blog.

https://blogs.technet.com/askperf/archive/2008/01/15/an-overview-of-processor-bottlenecks.aspx