When running a Windows Azure project in Microsoft Visual Studio, you may encounter the following Windows Azure Compute Emulator: Unexpected Failure dialog:
Windows Azure Tools for Microsoft Visuals Studio prior to SDK 1.7 would display this error dialog:
The seemingly random nature of this issue made reproducing thus troubleshooting exceptionally difficult. A retry loop was added to SDK 1.7 which alleviated the issue for some users.
A bug in legacy 64-bit operating systems caused the kernel address-conversion routines to incorrectly sign-extended the 32-bit address when converting to a 64-bit address. When the sign-extended memory address was accessed, it caused an Invalid Access to memory location with NativeErrorCode 998 error.
The best solution is to install Windows 8.1; although it was first fixed in Windows 8. If you are using one of the following 64-bit operating systems, there is a hot fix documented in KB 2588507 to correct the address-conversion routines.
- Windows Vista
- Windows Server 2008
- Windows 7
- Windows Server 2008 R2
How did we find the solution?
May thanks to Ryan Johnson and the developers at Wellero for suffering through this nasty bug and for their patience working with us to get this issue resolved. In hindsight, it is easy to see that the issue only reproduces in high memory conditions when running a legacy OS and that the KB applied because Visual Studio was linked to use addresses greater than 2 GB. As Scott Adams recently wrote “failure is where success likes to hide in plain sight.”
For those curious about how we found the solution, start by examining the six errors displayed in the Unexpected Error dialog: System.Net.NetworkInformation.NetworkInformationException Invalid access to memory location with NativeErrorCode 998.
The Windows Azure Authoring Tools running inside of the Visual Studio process calls the .NET method GetActiveTcpListeners to obtain the list of ports currently in use so it doesn’t try to deploy the project to a port already in use. By debugging the kernel, we can see that a kernel function tries to allocate heap memory to process the list. The allocation fails on an address like this: A3BA2E38 with the return value STATUS_ACCESS_VIOLATION (0xc0000005). This status gets translated to ERROR_NOACCESS 998 (0x3e6) “Invalid access to memory location” in a NetworkInformationException by GetActiveTcpListeners.
Now that we understand the error message, we need to understand why we get the error. Visual Studio is a 32bit application so the Windows architecture would normally allocate a total virtual address space of 4GB of which the process can use the lower half x00000000 to x7FFFFFFFF. Luckily, Visual Studio uses the linker option /LARGEADDRESSAWARE in order to handle large addresses. You can verify this by using the dumpbin utility like this:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE>..\..\vc\bin\dumpbin /headers devenv.exe Microsoft (R) COFF/PE Dumper Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file devenv.exe PE signature found File Type: EXECUTABLE IMAGE FILE HEADER VALUES 14C machine (x86) 6 number of sections 524FCB34 time date stamp Sat Oct 05 04:17:56 2013 0 file pointer to symbol table 0 number of symbols E0 size of optional header 122 characteristics Executable Application can handle large (>2GB) addresses 32 bit word machine
As you can see in the dumpbin output this x86 application can handle addresses larger than 2 gigabytes when running on a 64-bit version of the operating system. Since most of us are now running 64-bit OS versions, when Visual Studio runs out of help heap addresses in the lower 2 GB it will return a 32-bit addresses that normally would belong to the kernel like A3BA2E38. You know this by looking at the top bit of the address A which is greater than 7.
A bug in the kernel address-conversion routines incorrectly sign-extended the 32-bit address to a 64-bit address. The incorrectly sign-extended address looked like this: 0xffffffff`a3ba2e38 instead of 0x00000000`a3ba2e38. The sign-extended address is reserved for the kernel so it is not part of the user-mode address pool. Correcting the address-conversion routines eliminates the bad address and everything works as expected.