Introduction
This post is about Intel® Memory Protection Extensions (Intel® MPX) support in Microsoft Visual Studio* 2015; content provided by Gautham Beeraka, George Kuan, and Juan Rodriguez from Intel Corporation.
Overview
Update 1 for Visual Studio 2015 was announced on November 30, 2015. This update includes experimental compiler and debugger support for Intel MPX. Intel MPX can check all pointer reads and writes to ensure they remain within their declared memory bounds. This technology can detect buffer overflows and stop program execution at runtime, averting possible system compromises. It enables C/C++ code to make use of the new MPX instruction set and registers introduced in the 6th Generation Intel® Core™ Processors (“MPX-enabled platform”).
The Microsoft Visual C++ Compiler* and linker can now generate checks automatically, a capability enabled by specifying a command line option.
This blog explains how you can use automatic MPX code generation and debug MPX-enabled binaries. For more details on Intel MPX, please see the Intel MPX Technology web page.
How to enable automatic MPX code generation
Visual Studio 2015 Update 1 introduces a new compiler option: /d2MPX.
/d2MPX currently supports:
- Checking memory writes for potential buffer overflows. This provides protection for local and global pointers and arrays.
- Extensions to the calling conventions to automatically propagate bounds associated with pointer arguments.
To enable automatic MPX code generation for your project:
In Visual Studio, add the /d2MPX option in the Additional Options box (Project|Properties|Configuration Properties|C/C++|Command Line|Additional Options), Figure 1.
Figure 1. Add the /d2MPX compiler option for each desired configuration.
Usage Example
The following example is a program that contains an illustrative buffer overflow
Figure 2. Code with buffer overflow that will be detected with /d2MPX.
In Figure 2, the statement inside of the for loop would have overflowed the out array when it attempts to write past the end of the array since out is smaller than string str. Just before the program would have performed the out-of-bounds store, the MPX hardware will generate a #BR (bound range exceeded) exception, which is manifested as a structured exception handling (SEH) exception “Array bounds exceeded”. The default behavior in absence of an exception handler for the array bounds exceeded exception is immediate termination of the program. Alternatively, one can add an exception handler as shown in the example code to log the exception or to perform some context dependent recovery such as tearing down the process all the while having avoided the out-of-bounds store.
Steps to build and run the example:
-
Check that the Intel® MPX Runtime Driver is installed on your Microsoft® Windows® 10 November 2015 Update or greater system by verifying its presence in Device Manager under System devices (Figure 3). If it is absent, please download and install the driver from the Intel® Memory Protection Extensions Enabling Guide.
-
Install Visual Studio 2015 Update 1. Note, if Visual Studio is installed with the phone emulators, Hyper-V will have to be disabled (bcdedit /set hypervisorlaunchtype off and reboot) because this version of Windows does not expose MPX instructions to the guest.
-
Create a Win32 Console Application named “MPXExample” and use the code in Figure 2 for the driver code.
-
As noted above, please, double check that the /d2MPX option is enabled for the current Configuration.
-
Build the project for the X64 platform from within Visual Studio. This should produce an MPXExample.exe binary.
-
Execute the binary MPXExample.exe on an MPX-enabled platform with Windows 10 – which has the OS support for MPX.
- To have the Visual Studio Debugger break on the array bounds exceeded exception, please enable the option for “Array bounds exceeded” in Exception Settings (Debug|Windows|ExceptionSettings) as shown in Figure 4. Executing MPXExample.exe in the debugger should now break on the exception (Figure 5). In this example, the #BR exception is thrown when MPX detects that we are about to write beyond the upper bound of the out array (Figure 6).
Figure 3. Verify that the Intel MPX Runtime Driver is installed via Device Manager.
Figure 4. Enable break on the array bounds exceeded exception in the Exception Settings window to have the Visual Studio Debugger break on the exception.
Figure 5. The Visual Studio Debugger breaks on the array bounds exceeded exception.
Figure 6. The exception is thrown when checking the upper bound as shown in this snapshot of the Disassembly window.
Visual Studio 2015 Update 1 supports the display and manipulation of the MPX registers via both the Register (Figure 7) and Watch windows (Figure 8) when running on an MPX-enabled platform.
Figure 7. To observe the contexts of the MPX bounds registers, enable MPX in the Debugger Register window.
Figure 8. Adding a bounds register to the Debugger Watch window is also simple. BND0.UB and BND0.LB in the Watch window refer to the upper and lower bounds in the BND0
register respectively. Note that the upper bound of a bounds register is displayed in 2’s complement form.
How to tell if a binary is MPX-enabled
Run dumpbin /headers MPXExample.exe. The MPX debug directory entry should be similar to what is shown in Figure 9.
Figure 9. To tell if a binary is MPX-enabled, check whether a binary includes the mpx debug directory using dumpbin. The mpx debug directory should be listed in the Debug Directories section in the dumpbin output.
Must I compile everything with MPX?
You don’t have to compile all of your code with MPX enabled. A mixture of MPX and non-MPX enabled code will execute correctly. However, code compiled without MPX support will not have any MPX checks.
What hardware and version of Windows do I need?
To gain the benefits of MPX, MPX-enabled code should be executed on an MPX-enabled platform running a version of the Windows Operating System that is MPX aware. As of today, MPX is supported on the following:
What if I execute MPX-enabled code on a platform or on a version of Windows
that does not support MPX?
The MPX-enabled code will execute correctly, but it will not benefit from MPX. You need to execute the code on an MPX-enabled platform with an MPX-aware operating system. The MPX instructions will be treated as NOPs, so you might experience a performance decrease in these scenarios.
Performance Impact
MPX technology provides a powerful safeguard against buffer overflow. Inserting checks for every write to memory may incur some execution time and memory footprint overhead. The amount of overhead is tolerable during testing. However, when enabled for production code, the developer must balance whether the improved memory safety outweighs their customers’ performance needs. We plan to improve performance based on feedback.
Known Issues
There is a known issue with x86 debug build where debug instrumentation interferes MPX operation.
More Information and Feedback
For more information on how Intel MPX works, details on MPX intrinsic functions, calling convention extensions, and runtime behavior of MPX, please refer to the Intel® Memory Protection Extensions Enabling Guide.
Please try out automatic MPX code generation in Visual Studio 2015 Update 1. We are eager to hear about your experiences, especially in terms of usability, code size and runtime performance impact, and your suggestions for how to improve this feature. Please leave feedback in the comment box below or at the Intel ISA Extensions Forum on Intel® Developer Zone.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at Intel MPX Technology web page. Intel, the Intel logo, 6th Generation Intel® Core are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2016 Intel Corporation
Sounds great!
Out of curiosity, have you observed similar overheads as the ASan team?
Concretely: "time difference is 2.5x, which may be caused by the naive compiler implementation, but there is also a 4x RAM usage increase."
("Performance Impact" doesn't seem to include any numbers at the moment.)
Here are the full results: github.com/…/AddressSanitizerIntelMemoryProtectionExtensions
Hi MattPD
This is Juan from Intel. Thanks for noticing today's VC++ blog post. We have not done any specific comparisons vis a vis ASan. We'll take a look at the example on the link you provided.
One of the reasons why the feature is being rolled out in VC++ as experimental is to get the functionality out there and, based on feedback from users, develop actionable improvements in future VC++ updates.
Each compiler has implemented support for MPX given its design, deployment, and servicing constraints and priorities. What we wanted to focus on was on getting the functionality out to developers. Please take a look at the Updated Intel® Memory Protection Extensions Enabling Guide, which has been updated to cover VC++ 2015 Update 1.
We are looking for developer feedback on the feature: best way is to have developers out there give it a try, as your "mileage may vary" :)
Thanks
-Juan
What parts of memory are protected? For example, can it protect against an overrun of a heap-allocated array?
Is the intent for this feature to be only enabled in debug builds? or in other words should this be enabled in release builds running at customer environment?
So, for now we should only test/enable it in x64, because of the known issue?
thx,
Vertex
Just gave it a try – every time I start the test program, the system bluescreens with KMODE_EXCEPTION_NOT_HANDLED in MpxRuntime.sys. (Windows 10, Xeon ES-2667)
There seems to be a robustness failure in the fallback situation.
I don't know if this is a Microsoft OS issue, a compiler issue (checking CPU capabilities on entry to the application) or a driver issue.
Last night I saw this post, and being unsure if my new(ish) July 2015 Lenovo Z50 (i5 CPU) (running Win10 Pro x64,with all patches, including the Nov 2015 update) supported MPX, I manually installed the MPX driver as described. I have VS 2015 Update 1 installed also.
I compiled up the code etc as described above, having verified the driver (dated Aug 2015) was installed, and began working through it in the debugger.
The code then BSODed my system with a Kernel Mode Exception Handler not found BSOD. The problem was repeatable.
I got the impression from the post above that the system was fail safe (e.g. the comments about NOPs etc.), yet my experience suggests otherwise….
Comments please?
My experience is the same as Michael's – the BSOD is in the MPXRuntime.sys driver.
For info – the MPXRuntime.sys driver was version 1.0.0.8, dated sometime in Aug 2015.
Good stuff. Is there a chance that the Haswell MOVBE instruction could be added to intrinsics ?
Sounds very interesting technology but could you please describe your graphs using plain text? My screen reader can't read graphs and especially the platforms that this feature supports is important for me to know.
@Blind User
Would it help to share the source code to the example (figure 2)?
The last figure is supported hardware/software. MPX is currently supported on 6th generation Intel core processors running Windows 10 November 2015 update or higher with the MPX driver.
Thanks for the feedback.
Eric (ebattali@microsoft.com)
The known issue is really vague. I'm afraid to experiment with this because I don't know how exactly it's going to interfere.
Running an application with MPX without the VS debugger attacher causes the application to crash when a buffer overflow occurs. Trying to attach VS to it when this happens causes the application to exit with code 0xc0000409
@Michael – MPX is not supported in Intel(R) Xeon(R) ES-2667. That product line is not based on the 6th Generation Intel® Core™ (code named "Skylake"). Please, follow up with me via email to juan.a.rodriguez at intel.com.
@Mike Diak – MPX is not supported on the Lenovo Z50 – per the specs, it is based on 4th Generation Intel® Core™, not the 6th Generation Intel® Core™ (code named "Skylake"). Please, follow up with me via email to juan.a.rodriguez at intel.com.
High level comments:
Jose – I will follow up with you, but this seems as it stands terribly flakey and fragile.
Microsoft – If only the very very very latest CPU's support this ability, is there really much point in supporting it yet especially in this immature format? Clearly it will be 1-2 years before there are significant numbers of PCs in the field with this hardware, particularly with the generally slower purchase rate of PCs and longer lifecycles before they are replaced.
Comments to both Microsoft and Intel.
1) Most users will not know whether their CPU supports this. Thus as a minimum – the driver installer needs to be a wrapped (e.g. Installshield installer) that can interrogate the CPU to check if the support is available BEFORE installing the driver.
2) The driver needs to fail safe. If it's running on an unsupported CPU, it needs to do something more sensible than just throw a BSOD, e.g. unload itself? This seems a terrible omision.
3) Surely the compiler/runtime (which supposedly treats this as a NOP) should do something sensible at application startup. If things are this risky stability wise, it would be a very brave developer who currently shipped software compiled with this option etc, due to the risk of BSODs.
@Blind User
Thanks for your feedback. I have pasted the Figure 2 sample code below..
// mpxexample.cpp
// compile with: /d2MPX
#include "stdafx.h"
#include <windows.h>
const int OUTBUFSIZE = 42;
wchar_t out[OUTBUFSIZE];
void copyUpper(wchar_t* str, size_t size) {
__try {
for (unsigned int i = 0; i < size; i++) {
// buffer overflow when attempting to write the 43rd wchar
out[i] = towupper(str[i]);
}
}
__except (GetExceptionCode() == STATUS_ARRAY_BOUNDS_EXCEEDED) {
wprintf(L"Caught array bounds exceeded exceptionn");
}
}
int main(int argc, char* argv[]) {
wchar_t str[] = L"the quick brown fox jumps over the lazy dog";
memset(out, 0, OUTBUFSIZE);
copyUpper(str, wcsnlen_s(str, 255));
wprintf(L"%sn", out);
return 0;
}
To take advantage of MPX you need a system that support the MPX instruction set. Our recently introduced 6th Generation Intel® Core™ Processors include MPX. In addition, you will need Microsoft® Windows® 10 November 2015 Update ("November Update") and the Intel® MPX Runtime Driver installed. Please refer to the Intel® Memory Protection Extensions Enabling Guide landing page link for further details.
Hope this helps.
I've emailed Jose. I can't help thinking that for most developers using the existing hardware techniques for finding writes/reads of unallocated memory via various libraries such as heapcheck (one I contributed to, based on Bruce Perens' famous electric fence) and various other techniques to do similar things in hardware, are preferable.
While these are not just a matter of recompiling, they do have the ability of running on todays and yesterdays hardware, and use hardware not software, so don't necessarily occur huge performance overheads….
e.g. users.softlab.ntua.gr/…/HeapCheck.html
http://www.codeproject.com/…/Toggle-hardware-data-read-execute-breakpoints-prog
@Mike: HeapCheck doesn't work on stack memory though.
Why SEH and not language exceptions? Because of C?
Looks good to speed-up debug builds, at least it should be faster than traditional buffer bound checks.
I do not like so much the NOPs on unsupported CPUs, is there a way to check manually if the MPX is supported or not by the system?
Alessio is right. We need to be able to check robustly (and failsafely) for MPX support
I created empty project with /d2MPX option and got warning "LNK4075 ignoring /EDITANDCONTINUE due to /OPT:LBR specification".
@Myria
MPX technology is designed to protect static as well as dynamically allocated buffers, such as those coming from a heap. For the dynamically allocated case, the allocators/libraries need to be provided which are MPX aware. The upcoming version of GCC 6 will enable MPX library support by default. However, at this point in time we do not have a similar set of library support for UCRT VC++ libraries.
Custom code can be written to “wrap” heap allocators like malloc with MPX intrinsics in lieu of the default version until a suitable MPX enabled set of libraries or wrappers can be made available. Please see the MPX Enabling Guide for further details.
Below is an example of an MPX wrapped malloc version.
void *__wrap_malloc (size_t n)
{
void *p = (void *) malloc (n); // call original malloc
if (p)
{
return __bnd_set_ptr_bounds (p, n); //bnd: [p, p+n-1]
}
return p;
}
Your feedback on support for MPX enabled libraries is something we are interested in capturing.
@Vertex
Vertex
The feature can be enabled for both debug and release builds. The design of MPX technology is flexible and as such you, the customer/developer, needs to determine its suitability.
VC++ already provides features such as /GS, /analyze, /guard, secure CRT, and other capabilities. MPX is one additional capability available to help mitigate potential buffer overflow vulnerabilities.
As for whether you should only test/enable the functionality on x64, the answer is no. The issue is documented as issue 1 in the list of known issues in the Section 8.5.1 in the MPX Enabling Guide, which I am including below — this only impacts x86 debug builds.
1. Issue: The bound registers may be set to init [0, -1] in debug builds built with /RTC1 or /RTCs (Runtime checks flag). This is because of calls to RTC-related legacy functions _RTC_CheckESP and _RTC_CheckStackVars in the function epilog.
Workaround: Disable runtime checks. In the project set Configuration Properties -> C/C++ -> Code Generation -> Basic Runtime Checks to Default.
@Cleroth
Please see Section 8.5.1 in the MPX Enabling Guide for the known issues and workarounds that exist. Good news is that we have fixes already implemented and are currently under internal review for them to be incorporated in a future Visual C++ update.
@Richard Nutman
Richard
Visual C++ already supports the following MOVBE intrinsics.
unsigned short _load_be_u16(void *);
unsigned int _load_be_u32(void *);
unsigned __int64 _load_be_u64(void *);
void _store_be_u16(void *, unsigned short);
void _store_be_u32(void *, unsigned int);
void _store_be_u64(void *, unsigned __int64);
@Cleroth
A process terminated with 0xc0000409 exit code typically means that there was a stack overflow detected.
msdn.microsoft.com/…/cc704588.aspx
@Juan Rod. Aha! Excellent thanks!
@ Juan Rod Are you sure they have been added ? They're not listed on MSDN anywhere and don't show in any VS2015 headers. Are you confusing with _byteswap_ulong ? Specifically I'm referring to the new Haswell instructions that are faster.
@Richard Nutman
The MOVBE intrinsic declarations are defined in the immintrin.h include header file that ships with Visual Studio 205 Update 2.
@Michael
We can confirm that we have been able to reproduce the Blue Screen you and others experienced with the downloadable MPX driver package on systems that do not support MPX. The fix has already been identified, implemented, and is undergoing final validation. We plan to release the updated driver shortly for download and will get back here and on the Intel® Memory Protection Extensions Enabling Guide web site, software.intel.com/…/intel-memory-protection-extensions-enabling-guide, with an update.
If you do not have a 6th Generation Intel® Core based platform, please do not install the posted Intel® MPX Runtime Driver at this time.
@Mike Diack
Thanks for the feedback.
We can confirm that we have been able to reproduce the Blue Screen you and others experienced with the downloadable MPX driver package on systems that do not support MPX. The fix has already been identified, implemented, and is undergoing final validation. We plan to release the updated driver shortly for download and will get back here and on the Intel® Memory Protection Extensions Enabling Guide web site, software.intel.com/…/intel-memory-protection-extensions-enabling-guide, with an update.
If you do not have a 6th Generation Intel® Core based platform, please do not install the posted Intel® MPX Runtime Driver at this time.
Does this work on older Intel processors with the help of SDE? I guess not, because of the driver component.
All
We have uploaded a new MPX driver package which resolves the blue screens previously experienced by @Mike and @Mike Diack on systems that do not support Intel MPX Technology for download at the Intel® Memory Protection Extensions Enabling Guide web site: https://software.intel.com/en-us/articles/intel-memory-protection-extensions-enabling-guide/
Driver version is 1.0.0.11, Dated 1/28/2016
Thank you
This feature is not working on the new visual studio update 2?
@Richard Nutman
The MOVBE intrinsic declarations are defined in the immintrin.h include header file that ships with Visual Studio 205 Update 2.
@Chaos
Could you please elaborate on what is not working? Several fixes where made in Visual Studio Update 2 for some MPX related bugs.
Adding /d2MPX options is not working, no MPX instruction is generated. The example is working on Update 1, but the same code and the same solution on Update 2 cannot generate any MPX related instructions.
I finally got a Skylake CPU to test this. Release mode works great with Update 3 RC and performance isn’t too bad. I would love to enable this feature for our unit tests and/or for debug mode, but unfortunately the incompatibility with the other runtime debug checks is a blocker. I really don’t want to disable them and I don’t think we’ll upgrade the CPUs in our build system, where unit tests are run in release mode, anytime soon.
@Marcel Raad
Thank you for the feedback. The incompatibility with runtime debug checks (RTC) has been fixed in Visual Studio 2015 Update 2. The Intel® Memory Protection Extensions Enabling Guide referred above has been updated to reflect this.
Great, thanks!
Hmm, I always get the “array bounds exceeded” exeption in debug mode when using boost::call_once, both 32-bit and 64-bit mode. Release mode works fine.
@Marcel Raad
Thank you. We are able to repro the false bounds violation you reported and I understand that a VC++ compiler fix is currently under validation.
Just curious: are you using the available mechanism, e.g. https://connect.microsoft.com/VisualStudio, for submitting bugs?
Sorry, 9 months late and the “email me on reply” didn’t work…
Thanks, that’s great to hear! Then I’ll give it another try now :-)
I’m normally using MS Connect for bug reports, but not for MPX as the blog post said “leave feedback in the comment box below.” :-)
When library support for UCRT VC++ libraries is planned?
@Eliyahu
A few folks from Intel monitor this blog on an ongoing basis, so please forgive our delayed responses.
Generally speaking: external customer input and requests are always taken into consideration…sharing your specific scenarios in your input is helpful as well in order to help build support for your use case. The more feedback/interest we will get collectively is appreciated. Please us this and other available mechanisms like User Voice: https://visualstudio.uservoice.com/forums/121579-visual-studio-2015
I can’t speak for Microsoft on any timelines for support.