App compat shims are a terrific tool for enabling you to modernize your operating system. By resolving app compat issues, this tool has enabled many, many customers to update to the latest version of the operating system, and sometimes even remediate issues with patches. And I would assert that the security value of this modernization has been huge.
But, like any code, the shim engine is surface area. And an area of recent concern has been leveraging the shim engine as a persistence mechanism. There have been a few researchers who have explored this avenue. The risk is this: the shim engine injects a new DLL into another application's process space and intercepts specific APIs to run additional code.
Now, it's not as if it's been no-man's-land before. A critical mitigation is that you need to be a local administrator in order to customize the shim engine. (Pre-emptive comment: there have been researchers who have found ways to get around this constraint, and this has led to patches to resolve known techniques - reference the earlier points on the criticality of patching.) As you'll hear me say at conference after conference, event after event: if you are trying to secure endpoint devices, do not pass go, do not collect $200 until you have removed local administrator rights.
A less effective mitigation has been security through obscurity. The design intent of the shim engine has always been that the shim database (which maps a binary to the shims to apply) was extensible, but the shims themselves weren't an extension point. However, the controls which would block the use of this as an extensibility point this weren't terribly strong. The tool we provide for creating shim databases wouldn't allow you to pull in an arbitrary DLL, but a skilled reverser could discover the compilation mechanism which generates the database, and modify the input prior to generating the database. The second weak control was that we never documented the file format (and, I'll be honest, even with access to all of the internal INCLUDE files, it's still not the world's most straightforward design). However, again, a skilled reverser could determine a way to lay out a shim DLL and construct their own. It's a lot of work, but it wasn't technically impossible.
This of course leads to risk as a persistence mechanism. Yes, we'd love to keep you from being admin and installing them in the first place, but the best persistence mechanisms are ones where you don't even know you should be looking there. Be honest - are your detective controls today looking at the contents of the shim engine? (There are registry entries you could inspect to spot the addition or removal of them, but most people don't scan for this.) Since this is a way that an adversary could potentially inject some code that runs inside of your apps and isn't typically monitored, that's a potential persistence mechanism.
As such, we've been investing in making it harder to inject arbitrary code this way. So, while you can still install your own custom shim databases, what we are blocking is the loading of arbitrary DLLs as shim DLLs. The addition of this check in the shim engine has broken some "legitimate" scenarios, though I put that in scare quotes because if you are reversing multiple OS components in order to build your solution, it should probably occur to you somewhere along the way that this isn't a supported extensibility point. However, the important thing is to start to block the illegitimate scenarios that could allow an adversary to persist undetected.
Of course, if you have local administrator rights, we can't completely block you from removing these checks and undoing the blocking, because you have complete control over the device, but this does still move the bar forward in terms of providing defenses to reduce the number of holes where an adversary could persist undetected.