Recently I worked with a developer who was seeing an Unexpected Out of Memory Exception from his .Net 4.5 Application that was using custom stack reserve and commit sizes for the application. He was setting the values in Visual Studio as shown below
This particular application had some code that create new AppDomain using AppDomain.CreateDomain() method. It was working without any issues on .Net 4.0 but started throwing Out of Memory Exception on invoking AppDomain.CreateDomain() ever since he upgraded this to .Net 4.5 framework.
The exception was thrown despite the process had plenty of system resources available for its execution. Digging into this, we found the issue was happening due to an optimization introduced in .Net 4.5, to reduce the size of the stack requested for various internal helper threads that the CLR creates. This leads to a scenario where the stack reserve for a thread becomes less than stack commit size when we set a value > 253952 for stack commit size. Microsoft has identified this as a bug and are working on it, though we don’t have an ETA for a public fix at this moment. Time being one can use a value < 253952 for stack commit size as a workaround