On installing a custom unhandled exception filter and intentionally raising an exception to get its attention


A customer reported that they were seeing inconsistent behavior when they intentionally raised a Win32 exception and tried to catch it in a custom unhandled exception filter.

#define CUSTOM_EXCEPTION 0xABCDEF01

LONG WINAPI CustomFilter(EXCEPTION_POINTERS* exceptionPointers)
{
 if (exceptionPointers->ExceptionRecord
                      ->ExceptionCode == CUSTOM_EXCEPTION) {
  return EXCEPTION_CONTINUE_EXECUTION;
 }
 return EXCEPTION_EXECUTE_HANDLER;
}

void Test()
{
 auto previousFilter = SetUnhandledExceptionFilter(CustomFilter);
 RaiseException(CUSTOM_EXCEPTION, 0, 0, nullptr);

 try {
  RaiseException(CUSTOM_EXCEPTION, 0, 0, nullptr);
  throw (int)0;
  Log("Returned from throw");
 } catch (int) {
  Log("Caught");
 }
 SetUnhandledExceptionFilter(previousFilter);
}

The customer observed a few things.

First, if the Test function was called from inside a window procedure, then the behavior changed depending on the execution environment, as documented here.

Second, the custom filter was not called at all if the program was running under the debugger. The documentation for Set­Unhandled­Exception­Filter says that if the program is not being debugged, then the custom unhandled exception filter is called, but it doesn't say what happens if the program is being debugged.

The customer's question was "What is the expected behavior if the program is being debugged?"

First, let's answer the question: The expected behavior if the program is being debugged is that the custom unhandled exception filter is ignored.¹

But let's step back and look at the bigger picture here.

This program is violating one of the cardinal rules of Win32 exceptions: Exceptions must not cross foreign stack frames. If you are going to raise an exception in one place and handle it in another, then every stack frame that the exception travels through must be aware of your little game. After all, if they aren't aware of your game, you don't know what they will do when they see your custom exception!

The unhandled exception filter runs as the very last exception filter, which means that before control reaches the unhandled exception filter, it must go through every single active stack on your thread, including the stack frames outside your control, like the ones that set up the call to the window procedure. So you've already left the world of predictable behavior.

¹The intended purpose of the custom unhandled exception filter is to capture additional program state for post-mortem debugging purposes. If a debugger is connected to the process, the thinking is that you connected the debugger because you want it to be informed of the exception and freeze the program so you can, y'know, debug it.

Comments (7)
  1. kantos says:

    I would think that expecting defined behavior for any exception not raised with `throw` would be rather foolish... as `throw` is the only one carrying an ISO standard behind it. In fact I would expect that `RaiseExeception` would explicitly be documented to NOT clean up stack frames and support RAII

    1. Darran Rowe says:

      The only wrinkle in that idea is that Microsoft themselves implement their C++ exceptions using SEH.

      1. kantos says:

        True, but throwing an exception through C code is undefined behavior anyway; though most runtime libraries "support" it. A better way is to assume that all OS or third party library callbacks must be noexcept unless explicitly documented otherwise. This fits with the whole "Exceptions on WIN32 are supposed to be Exceptional and therefore Fatal" thing.

        1. ipoverscsi says:

          Amazingly, you have managed to rephrase "exceptions must not cross foreign stack frames" with eight times as many words.

  2. DWalker says:

    It may seem obvious that "if the program is running under the debugger, the custom exception handler is not called", but that's not always the case.

    Lots of program and API documentation says things like "If A, then B" while omitting to say "If not A, then C", especially when C is not obvious. Or, it says "If A, then B" while leaving out "If C, then D" and "If E, then F". I know that writing documentation is hard.

    My other pet peeve about documentation is that every place where the syntax for SETTING some value or option should also tell how to query the setting.

    Anyway, that was a slight digression, but thanks for the info.

  3. Why are there two RaiseException calls?

    1. IInspectable says:

      The customer apparently was also trying to verify, whether mixing C++ exception handling and SEH is a smart thing to do. (Plot spoiling ahead: It ain't.)

Comments are closed.

Skip to main content