Http SSL service failed to start with error

Customer is running a Windows 2003 SBS server, he found some function of exchange doesn't work and the reason is HTTP SSL service was not started. He got follow error when tried to start this service.

Based on the error message, looks this was due to some file corruption/missing. Then, we ran Process Monitor tool, however we didn't find any file not found issue.

Then, we enabled the IIS debug log and found below 2 errors:

00000000 0.00000000 [488] 9384 strmfilt!STREAM_FILTER::Initialize [\streamfilter.cxx @ 459]:Error initializing SSL_STREAM_CONTEXT globals. hr = 80070002  

00000001 0.00018007 [488] 9384 ??!StreamFilterInitialize [\streamfilter.cxx @ 741]:Failed to initialize global STREAM_FILTER object. hr = 80070002

To further identify the problem, we collected iDNA trace and found the error was due to an error when initialize the HTTP API. This API call communicates with HTTP.SYS and system returned 0xc000000e. This sounds the HTTP driver was not started.

Error code: (NTSTATUS) 0xc000000e (3221225486) - A device which does not exist was specified.

Then, we decided to re-start the HTTP driver. However, we got follow errors when try to start/stop the HTTP driver.

C:\Windows\system32>net start http

The requested service has already been started.

More help is available by typing NET HELPMSG 2182.

C:\Documents and Settings\Administrator>net stop http /y

The service could not be controlled in its present state.

More help is available by typing NET HELPMSG 2189.

The problem was resolved by reboot the machine since HTTP driver was in un-know state. However, customer said the problem happen again after he ran an ISA configure wizard. Based on this, we found the problem can be reproduced. During the configuration of ISA, it tries to stop and start the HTTP driver. However, HTTP driver was failed to stop with "could not be stopped" error. After that, if we try to stop the HTTP driver again, it reports "the service could not be controlled in present state".

To find out the cause, we captured a kernel mode dump. And found HTTP driver was in pending unload status. And the driver object reference count was not zero. This is why the unload was in pending status. Investigate the dump further, we found there were w3wp.exe(s) still not released. All the w3wp.exe(s) should be terminated when shutdown the WAS service. So, it is abnormal to see the w3wp.exe(s) were still alive. However, looks all the resources related to those processes had been released already.

5: kd> !drvobj 87d78030

Driver object (87d78030) is for:

\Driver\HTTP

Driver Extension List: (id , addr)

Device Object list:

878678e0 87873e48 87867b50 

5: kd> !devobj 878678e0

Device object (878678e0) is for:

AppPool \Driver\HTTP DriverObject 87d78030

Current Irp 00000000 RefCount 1 Type 00000012 Flags 00000040

Dacl d7cba584 DevExt 00000000 DevObjExt 87867998

ExtensionFlags (0x00000001) DOE_UNLOAD_PENDING

5: kd> !process 0 0 w3wp.exe

PROCESS 859f0bc8 SessionId: 0 Cid: 14d4 Peb: 7ffde000 ParentCid: 1278

    DirBase: 9fbd18a0 ObjectTable: 00000000 HandleCount: 0.

    Image: w3wp.exe

PROCESS ffadd8c8 SessionId: 0 Cid: 1f94 Peb: 7ffdf000 ParentCid: 1278

    DirBase: 9fbd1880 ObjectTable: 00000000 HandleCount: 0.

    Image: w3wp.exe

This used to be related to the anti-virus drivers. In addition, we found there was a 3rd party anti-virus driver monitor the process creating notification.

808ad280 nt!PspCreateProcessNotifyRoutineCount = 1

         808ad260 nt!PspCreateProcessNotifyRoutine[0] = d7a0d360 -> b9dd822c = ***

ERROR: Symbol file could not be found. Defaulted to export symbols for 3rdparty.sys - 3rdparty+0x522c

1 Process Notify intrusions

The problem resolved after un-install the 3rd party anti-virus driver.

See you next time,

Wei Zhao from APGC DSI Team