Activate-As-Activator activates as activator

This particular problem burned my team somewhat the other day, so I figured writing a blog post about the issue was a good thing.

When you activate a COM object (with CoCreateInstance), one of the things that COM has to do is to figure out the activation model for the newly instantiated COM object.

Most of the time, the COM object is instantiated as an Activate-As-Activator object, however you can override this setting using the APPID registry value in your CLSID registration.

In general there are four forms of activation - activate-as-activator, activate-as-user, activate-as-interactive-user and activate-as-service (you can see these in the dcomcnfg tool).

These are controlled by various keys under the appid section.  If you don't have an appid, or your COM object is an InProcServer or LocalServer, you're typically going to be using activate-as-activator.

There's a clever trick that we use (I don't know if it's intentional, we sort-of stumbled into it by accident).  For Vista, the actual audio engine runs out-of-proc from the audio service.  We do this primarily to isolate 3rd party code - if the 3rd party code crashes, we'll silently restart the audio engine - you'll get a horrendous glitch but that should be about the extent of the damage.

When you're running a COM server, there's a call - CoRegisterClassObject that's used to register the class factory for all of your COM objects.  When COM tries to activate a COM object, before it looks in the registry, it looks to see if there's a server already registered for that class.  This is actually pretty cool - as long as we guaranteed that the engine was running, we could simply call CoCreateInstance and instantiate our internal COM objects.  There was an unexpected bonus that came along with that - since our classes didn't specify an APPID, COM instantiated the objects as Activate-As-Activator.  

There was also an unexpected bonus - when you're activating an activate-as-activator  in turn checked to make sure that the only person activating the internal interfaces was running as our service (thus adding an extra level of defense-in-depth).

Things were going great; we didn't need a class registry for our COM objects, things just worked fine.  Until Friday, when we were preparing for a routine RI.  All of a sudden, our calls to CoCreateInstance started failing with REGDB_E_CLASSNOTREG (Class not registered).   We sat there and scratched our heads for a while and gave up - we called up the COM wizards for help.

After the COM developer debugged the problem for a bit, he figured out what had gone wrong.

One of my changes had leaked an impersonation - I impersonated the client, and then forgot to revert to self.  I had run through all my unit tests and hadn't caught the problem.  The problem only showed up because the call to CoCreateInstance attempted to activate the COM object while impersonating, and the token of the user I was impersonating didn't match the token of the service, so the CoCreateInstance call failed.

You could argue that the REGDB_E_CLASSNOTREG error is incorrect in this case, but...

One slightly clever trick I picked up while trying to track down the impersonation:

#if DBG
void AssertNotImpersonating()
{
    HANDLE threadHandle;
    if (OpenThreadToken(GetCurrentThread(), TOKEN_QUERY, TRUE, &threadHandle))
    {
        OutputDebugString(L"Called while impersonating, this will cause problems.\n");
        DebugBreak();
        CloseHandle(threadHandle);
    }
}
#endif
 

I just sprinkled these throughout my code and was able to find the offending change quickly.

Btw, please note that this only works if you're checking to ensure that you are NOT impersonating. If you invert the check (to assert that you ARE impersonating because the call to OpenThreadToken succeeded)), then you may have a problem: If you're impersonating at the Anonymous level of impersonation, then the call to OpenThreadToken will fail with ERROR_CANT_OPEN_ANONYMOUS.