Why can’t I start my VM when there is plenty of free memory?

Issue #1: A number of customers has reported than VM's fail to start when there is plently of memory.  The error reported from the Hyper-V manager is displayed below.  Here I tried to start a 4GB VM when there was 7+GB free.   What's going on?

Hyper-V Memory Error

The most common cause has to do with a bug in Windows Server 2008 NUMA memory allocations.  This means you'll only hit this bug on NUMA machines (all multi-proc AMD machines are NUMA + many high end Intel based servers).  You can verify you have this bug by looking at the Task Manager reported "Cached" and "Free" memory.

Hyper-V Free Memory Task Manager

Here free memory is 9MB but really much of the "Cached" memory can be made free also.  Standby Pages, Modified Pages, and System File Cache make up the "Cached" memory.  Most (all but a couple 100MB) of the cached memory can be converted to "free",

Fix #1: To resolve the issue will need to contact Microsoft Product Support http://support.microsoft.com/contactus/cu_sc_more_master#tab1 and request the following hotfix - KB953585

Alternate Fix #1: The second way to avoid this bug is to not do any work in the root parition directly.  Create a "management" VM and do your work there.

Background #1:  The most common way of hitting this issue is by running large file copies in the root (aka host) parition.  Doing lots of file copies causes the System File Cache to bloat in size.  When Hyper-V goes to start it will flush the System File Cache if it needs memory.  The System File Cache pages (aka file pages from the copies) move to the Standy list and from there will get zeroed and freed.  The bug is WS08 prevents NUMA allocations from the Standyby list.

Issue #2: The second most common cause to the can't start the VM issue when it appears there is plenty of memory is due to configuring a NUMA machine to use fewer processors than the system actually has.  Typically this is done using "bcdedit /set numproc X". 

Fix #2: If you set NUMA machine in this mode for testing memory attached to the "hidden" processors will be inaccessible to Hyper-V.  Some machines allow you to reassign the memory (via the BIOS or front end controller) to the "active" processors.

  - Tony Voellm

Comments (8)

  1. I have the same problem. i try to install the fix that you require, but i can’t install a this. I receive a error that the installation is not confirm my OS. I have Server 2008 Enterprise 64x

    [Tony’s Reply]  Hotfixes get rolled up into service packs and that is why you can not install the hotfix.  It is resolved in SP2.  The other root cause of this issue is incorrectly built drivers / services that set a min commit for memory.  msinfo32.exe can sometimes help find this offending software.  I suggest calling MSFT support or posting more details on your issue.

  2. Murat Berk says:

    I started to hit this issue after almost 2 years of usage. The only change I did is install 2 2GB memory replacing 2 1G memory.

    So I have total 6GB memory now.

    Is it possible that this somehow causes this problem?

    I am using Windows 2008 with latest updates/patches as of today…

    [Tony’s reply]  Adding memory is unlikely to have caused any issue.  I’d need more info from task manager / msinfo32.exe to help triage the issue.  More than likely a driver or service is camping on memory.

  3. Steve Frank says:

    We’re seeing this on a 2k8 R2 box with Hyper-V role installed–we’re not doing anything new in the root partition. Over time it just gobbles Cache.  

    Any insight? I think the KB above doesn’t apply to R2.  

     [Tony’s Reply]

    I would recommend calling Microsoft Product Support.  There are no known issues with R2 in this area.  What is likely is either a bad driver, service, etc that is slowly leaking memory.  I’ve seen non-MSFT drivers and services lock memory and cause a similar issue.  msinfo32.exe, performance counters, and resource monitor are useful tools to tracking down issues like this.

  4. Robin says:

    Same issue in R2 on a server core installation. Cache is about 2 GB… for a server core ?

  5. jos says:

    Is there anyway to solve this temporarily without restarting the server? Any way to have it "release" the cached memory?

  6. jos says:

    Is the hotfix included in Service Pack 2 for Windows Server 2008? I'd rather install the real Service Pack than a single hotfix, if possible.

    Thanks for the info, hard one to catch!

  7. Hi there,

    we've the error described above on our local server. It's a CPU with 8 cores and 16GB RAM Enterprise Server 2008 R2.

    All system tools (resource monitor, task-manager and msinfo32) tells that there is free Memory of 10 GB but if I try to start a virtual computer with 4GB RAM, V-Server tells that the avaiable memory is 0 MB.

    Any ideas?

    Greetings Matthias

Skip to main content