Out of memory? Easy ways to increase the memory available to your program


When you run your VB or C# application, you might get an OutOfMemoryException thrown, even if your  machine has lots of  memory.

 

Every 32 bit process has a 2^32 bit (4 Gig) address space. That means every pointer has a size of 32 bits (4 bytes) and thus is limited to 4 Billion.

 

That’s the equivalent of saying a vehicle license plate number consists of 6 digits and thus there are 1 million possible numbers.

 

That 4 Gigs is divided into half: the user application gets the lower half and the OS gets the upper. (This boundary can be changed: see below).

 

Start VS 2010. File->New->Project->VB or C# Windows WPF Application.

Paste the VB or C# code below. It creates a heap then allocates 100Meg of memory in a loop continuously until an exception is thrown.

 

On my 64 bit Windows 7 machine with 8 Gigs of RAM (your digital camera or phone might have  more memory!), I get about 1.4Gig allocated before it dies.

 

Iter #10 1,048,576,000

Iter #11 1,153,433,600

Iter #12 1,258,291,200

Iter #13 1,363,148,800

Exception Exception of type ‘System.OutOfMemoryException’ was thrown.

 

 

Now choose Project->Properties->Compile->Build Events->PostBuildEvent Command and  added these 2 lines

call “$(DevEnvDir)..\..\vc\vcvarsall.bat” x86

“$(DevEnvDir)..\..\vc\bin\EditBin.exe” “$(TargetPath)”  /LARGEADDRESSAWARE

 

Note: the positions of the quotes are critical

The first line calls a BAT file that makes various tools available on the path.

The second runs EditBin on the target binary, using the LARGEADDRESSAWARE flag (that’s almost all left hand keys on the keyboard!)

Also uncheck the option: Project->Properties->Debug->Enable the Visual Studio Hosting Process

 

The only effect of these 2 lines is to call EditBin to toggle a bit in the EXE. When the EXE starts a process, that entire process is flagged as able to work with pointers above 2G.

With such pointers, the high bit is a 1, which, in 2’s complement notation, is a negative number, and some applications may not be designed to work with “negative” pointers.

 

Now when I run the code I get 3.5 Gigs: More than twice as much memory!

 

Iter #30 3,145,728,000

Iter #31 3,250,585,600

Iter #32 3,355,443,200

Iter #33 3,460,300,800

Exception Exception of type ‘System.OutOfMemoryException’ was thrown.

 

 

This Editbin “trick” works fine on a 64bit OS. For a 32bit OS it works too, but you need to “bcdedit /set IncreaseUserVA 3072” (reboot) and you won’t get as much extra memory.

 

Want even more memory? If you’re on a 64 bit OS, try compiling to 64 bit:

VB: Project->Properties->Compile->Advanced->Target CPU->Any CPU (or x64)

C#: Project->Properies->Build->Platform Target->Any CPU (or x64)

 

Iter #110 11,534,336,000

Iter #111 11,639,193,600

Iter #112 11,744,051,200

Iter #113 11,848,908,800

Iter #114 11,953,766,400

Exception Exception of type ‘System.OutOfMemoryException’ was thrown.

 

Yes, that really is almost 12 gigs on my 8 gig machine!  You can also verify via Task Manager.

 

A 64 bit process has a pointer of size 64 bits (8 bytes). 2 ^ 64= 18,000,000,000,000,000,000. (1.8 x 10^19)

That’s about 18 exabytes, or 900,000 years of DVD quality video.

 

Still that’s nowhere near the number of possible chess games or even the waaaay smaller number of electrons that will fit in the universe J(Hubble’s law, big bang physics)

 

When I first worked for Microsoft, I developed a thunking mechanism to allow 32 bit applications to call 16 bit DLLs. (http://support.microsoft.com/kb/139070 )

We thought there’s no way anybody could run out of 32 bit address space!

 

If a license plate number were to double its length from 6 to 12 digits, that would go from 1 Million (population of a medium sized city) to 1Trillion (1x 10^12) , 200 times the world population.

 

 

See also

BCDEdit Command-Line Options

Using multiple heaps efficiently

What is your computer doing with all that memory? Write your own memory browser

LARGEADDRESSAWARE

Create your own Test Host using XAML to run your unit tests

Undocumented APIs and 16 bit DLLs

 

<C# Code>

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Windows;

using System.Windows.Controls;

using System.Windows.Data;

using System.Windows.Documents;

using System.Windows.Input;

using System.Windows.Media;

using System.Windows.Media.Imaging;

using System.Windows.Navigation;

using System.Windows.Shapes;

using System.Runtime.InteropServices;

using System.Diagnostics;

 

namespace Heapcs

{

    /// <summary>

    /// Interaction logic for MainWindow.xaml

    /// </summary>

    public partial class MainWindow : Window

    {

        public MainWindow()

        {

            InitializeComponent();

        }

        private void Window_Loaded(object sender, RoutedEventArgs e)

        {

            var hHeap = Heap.HeapCreate(Heap.HeapFlags.HEAP_GENERATE_EXCEPTIONS, 0, 0);

            // if the FriendlyName is “heap.vshost.exe” then it’s using the VS Hosting Process and not “Heap.Exe”

            Trace.WriteLine(AppDomain.CurrentDomain.FriendlyName + ” heap created”);

            uint nSize = 100 * 1024 * 1024;

            ulong nTot = 0;

            try

            {

                for (int i = 0; i < 1000; i++)

                {

                    var ptr = Heap.HeapAlloc(hHeap, 0, nSize);

                    nTot += nSize;

                    Trace.WriteLine(String.Format(“Iter #{0} {1:n0} “, i, nTot));

                }

            }

            catch (Exception ex)

            {

                Trace.WriteLine(“Exception “ + ex.Message);

            }

 

 

            Heap.HeapDestroy(hHeap);

            Trace.WriteLine(“destroyed”);

            Application.Current.Shutdown();

        }

    }

 

    public class Heap

    {

        [DllImport(“kernel32.dll”, SetLastError = true)]

        public static extern IntPtr HeapCreate(HeapFlags flOptions, uint dwInitialsize, uint dwMaximumSize);

 

        [DllImport(“kernel32.dll”, SetLastError = true)]

        public static extern IntPtr HeapAlloc(IntPtr hHeap, HeapFlags dwFlags, uint dwSize);

 

        [DllImport(“kernel32.dll”, SetLastError = true)]

        public static extern bool HeapFree(IntPtr hHeap, HeapFlags dwFlags, IntPtr lpMem);

 

        [DllImport(“kernel32.dll”, SetLastError = true)]

        public static extern bool HeapDestroy(IntPtr hHeap);

 

        [DllImport(“kernel32.dll”, SetLastError = true)]

        public static extern IntPtr GetProcessHeap();

 

        [Flags()]

        public enum HeapFlags

        {

            HEAP_NO_SERIALIZE = 0x1,

            HEAP_GENERATE_EXCEPTIONS = 0x4,

            HEAP_ZERO_MEMORY = 0x8

        }

 

    }

 

}

 

 

</C# Code>

<VB Code>

Option Strict On

Imports System.Runtime.InteropServices

 

 

Class MainWindow

    Private Sub Window_Loaded(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs) Handles MyBase.Loaded

        Dim hHeap = Heap.HeapCreate(Heap.HeapFlags.HEAP_GENERATE_EXCEPTIONS, 0, 0)

        ‘ if the FriendlyName is “heap.vshost.exe” then it’s using the VS Hosting Process and not “Heap.Exe”

        Trace.WriteLine(AppDomain.CurrentDomain.FriendlyName + ” heap created”)

        Dim nSize As UInteger = 100 * 1024 * 1024

        Dim nTot As ULong = 0

        Try

            For i = 1 To 1000

 

                Dim ptr = Heap.HeapAlloc(hHeap, 0, nSize)

                nTot += nSize

                Trace.WriteLine(String.Format(“Iter #{0} {1:n0} “, i, nTot))

            Next

        Catch ex As Exception

            Trace.WriteLine(“Exception “ + ex.Message)

        End Try

 

        Heap.HeapDestroy(hHeap)

        Trace.WriteLine(“destroyed”)

        End

    End Sub

End Class

 

Public Class Heap

 

    <DllImport(“kernel32.dll”, SetLastError:=True)> _

    Public Shared Function HeapCreate(

               ByVal flOptions As HeapFlags,

               ByVal dwInitialSize As UInteger,

               ByVal dwMaximumSize As UInteger

         ) As IntPtr

    End Function

 

    <DllImport(“kernel32.dll”, SetLastError:=True)>

    Public Shared Function HeapAlloc(

               ByVal hHeap As IntPtr,

               ByVal dwFlags As HeapFlags,

               ByVal dwSize As UInteger

         ) As IntPtr

    End Function

    <DllImport(“kernel32.dll”, SetLastError:=True)>

    Public Shared Function HeapFree(

               ByVal hHeap As IntPtr,

               ByVal dwFlags As HeapFlags,

               ByVal lpMem As IntPtr

         ) As Boolean

    End Function

 

    <DllImport(“kernel32.dll”, SetLastError:=True)>

    Public Shared Function HeapDestroy(

               ByVal hHeap As IntPtr

         ) As Boolean

    End Function

 

    <DllImport(“kernel32.dll”, SetLastError:=True)> _

    Public Shared Function GetProcessHeap(

         ) As IntPtr

    End Function

    <Flags()>

    Public Enum HeapFlags

        HEAP_NO_SERIALIZE = &H1

        HEAP_GENERATE_EXCEPTIONS = &H4

        HEAP_ZERO_MEMORY = &H8

    End Enum

End Class

 

 

</VB Code>

Comments (18)

  1. h says:

    Why the 64bit process OOMs at 12 GB?

  2. Crescens2k says:

    The reason why the 64bit process runs out at 12GB is down to the page file commit limit of Windows, 12GB isn't a hard limit though. In it's default settings, Windows sizes the page file depending on how much physical RAM is in your system, the more RAM, the more it will put into the page file. But you can also change the page file settings manually.

  3. Greg says:

    This should be built into the .NET base classes and not require interop to kernel32.dll.

  4. Crescens2k says:

    Just because this post uses the heap functions doesn't mean you have to use interop to access it. It is easier to show this kind of thing at work using the Windows heap functions rather than using the framework.

    With the heap functions, you can use one variable, allocate and then forget about it leaking memory until you reach the limit. With the framework you have to keep everything referenced otherwise the gc will start reclaiming memory or do other things to make the gc not clean up, but either way, it will not be as simple as the above code.

  5. Calvin_Hsia says:

    To verify if an EXE file is Large Address Aware or not:

    Open a Visual Studio Command Prompt (Start->All Programs->Microsoft Visual Studio 2010->Visual Studio Tools->Visual Studio Command Prompt

    Link /dump /headers <filename>

    The output will include the line “Application can handle large (>2GB) addresses”

    Here’s a sample output:

    Microsoft (R) COFF/PE Dumper Version 10.00.30319.01

    Copyright (C) Microsoft Corporation.  All rights reserved.

    Dump of file MemSpect.exe

    PE signature found

    File Type: EXECUTABLE IMAGE

    FILE HEADER VALUES

                14C machine (x86)

                  4 number of sections

           4CEA07E2 time date stamp Sun Nov 21 22:04:18 2010

                  0 file pointer to symbol table

                  0 number of symbols

                 E0 size of optional header

                122 characteristics

                      Executable

                      Application can handle large (>2GB) addresses

                      32 bit word machine

  6. pb says:

    I am using visual basic 2010 express. I can't find

    Project->Properties->Compile->Build Events

    or

    Project->Properties->Compile->Advanced->Target CPU (target framework but not target cpu)

    Any hints please on what I should do?

  7. Clado says:

    Right click the project name in the solution explorer toolbar and you'll see at the bottom the 'properties' menu.

  8. When using multiple class libraries in a solution with one exe, do all the project need these lines added?

    call "$(DevEnvDir)….vcvcvarsall.bat" x86

    "$(DevEnvDir)….vcbinEditBin.exe" "$(TargetPath)"  /LARGEADDRESSAWARE

    I am assuming, yes.

    thanks

  9. Mias Van Den Berg says:

    Being able to store information for users and for the application in server memory is really awesome when you are developing asp.net web applications. I have in the past abused it a little and at on point I had a web application which would actually throw a OutOfMemory exception every now and then. I started searching for tool which would help me analyze session objects and application state objects. After an extensive search, I came accross the Asp.Net Web Profiler, which allows for viewing session variable values which has been declared in different sessions and their sizes to be particular. It displays variables and objects which has been declared in the application state, the current session state as well as in all user sessions and the asp.net cache. You can download it from <a href="http://www.aspwebprofiler.com">www.aspwebprofiler.com</a&gt;

    It also alows for drill down into object properties in order to detect large objects and optimizing them for memory usage. This way I could detect large sessions and realized that I had the same large object which runs in every user session. I moved it to the application state and my problem was solved.

  10. EricLaw says:

    Does the fact that IntPtr is a signed value impact the ability to use this type (e.g. for PInvoke) when LargeAddressAware is enabled? Or does it still marshal cleanly to other native code?

  11. Shubham says:

    Let Me Know where is the problem during compiling……

    Error 4 The build was aborted because of an internal failure.

    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

    at Microsoft.Build.BuildEngine.BuildItem..ctor(String itemName, ITaskItem taskItem)

    at Microsoft.Build.BuildEngine.TaskEngine.GatherGeneratedTaskOutputs(TaskOutput ……………….

  12. Tomek says:

    HI, what about windows service?

    Is there any way to use this solution there? I am trying to apply this solution to service but i still got nothing…

    :( pease help me

  13. Ishant says:

    Hi,

    I tried this  call "$(DevEnvDir)….vcvcvarsall.bat" x86

    "$(DevEnvDir)….vcbinEditBin.exe" "$(TargetPath)"  /LARGEADDRE

    but when I build solution it shows error

    Error 4 The command "call "C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDE….vcvcvarsall.bat" x86

    "C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDE….vcbinEditBin.exe" "C:UsersikrishnaDocumentsRectool With Versionversion-8-22-2013ReconciliationToolReconciliationToolbinReleaseReconciliationConsole.exe"  /LARGEADDRE

    " exited with code -1073741515. ReconciliationConsole

    Please help me.

  14. NK says:

    Hi, I am having an out of memory issue with a simulation model I am building in vb.net.

    I'm using visual studio 2008 and I have added the two lines stated above to the Post BuildEvent Command.

    I have created a new Heap class using your vb.net code and then I added your 'main window' code to my frmMDI so that it's called when the model loads.

    This doesn't seem to have made any effect on my model and I still hit an out of memory error. Have I made a mistake in the way I've implemented your code?

    Would appreciate any help – thanks :)

  15. Kapil Manandhar says:

    Thank you this helped a lot on memory exception.

  16. Lalit kumar says:

    Hey this code not working in Windows phone I getting

    DllImport cannot be used on user-defined methods.

  17. Robert Parker says:

    Thank you.  Our 32bit application would crash often very close to the 1.4G mentioned.  The task manager would report that the process was using between 1.1G and 1.8G when it crashed.  Presumably it was crossing the 1.4G boundary which caused the crash. This problem has now gone.

    Our application was in visual basic, so I discovered that C++ needs to have been installed for EditBin.exe to exist in the visual studio 2010 path.

  18. Rupam Das says:

    I am posting this comment to let you know that you have literally saved my startup with this single blog. Our's is an Animation heavy smart class for schools. We developed it on 64 bit. Turned out the schools are still struck with 2GB dual core ( luckily W7). We are unable to get big simulations working untill I found your post.

    Thanks mate.