Observing processor queue lengths inside of virtual machines

One of the more esoteric performance metrics that users can access when trying to determine bottlenecks in the performance of their systems is that of the processor queue length.  Simply speaking the processor queue length metric reports how many threads are currently queued up and waiting to be executed on the computers processor.

As discussed here: http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/en-us/core/fnef_mul_kfhm.asp – when a computer is idle this should pretty much remain at 0.  Further more if this number is constantly higher than 0 it is probably a fair sign that you need more processing power in your computer.

However this metric is not at all accurate inside of a virtual machine. 

Most people who have checked this metric inside of a virtual machine have noted that it is usually a 2 or higher and never actually seems to go down to 0.  The reason for this is actually relatively simple. 

A processor with 0 tasks in its processing queue is referred to as being idle (meaning that it has nothing to do).  If we detect that the processor inside of the virtual machine is idle – we will stop executing the virtual machine and return the processing resources back to the host operating system until such a time as when we need to process something again (for example – when the next interrupt comes in for the virtual machine).  However – the processes inside of the virtual machine are completely unaware of the fact that we are doing this – as, by definition, for the processor inside of the virtual machine to be idle – all of the processes have to be doing nothing at that particular point in time. 

The result of all of this is that it is impossible for the Performance monitoring tool under Windows to ever see that the processor inside of the virtual machine has a queue length of 0.  It will only actually be running and able to check the processor queue length if there is something in the queue.