SSPI failures due to stack size.

Article
04/04/2007

I was recently engaged by a SQL dev in order to help out on a tough nut they were working on. A customer had written a service to connect to SQL 2005 it failed with an error:

SQL message [[Microsoft][ODBC SQL Server Driver][DBMSLPCN]ConnectionOpen (SECDoClientHandshake()).]

The SQL folks already had a trace ( I assume via these methods - https://msdn2.microsoft.com/en-us/library/aa964124.aspx ) they called it a BID trace? In the article it looks like BID == Built-in Diagnostics

Anyway - the BID trace showed:

DBNETLIB 0x000016EC 3, "<ClientHandshakeLoop|ERR> InitializeSecurityContext 0x80090304{WINERR}", 0, 0

The SSPI dance is outlined here - https://www.microsoft.com/technet/prodtechnol/windows2000serv/maintain/security/sspi2k.mspx

1. Call AcquireCredentialsHandle - get CredHandle
2. Call InitializeSecurityContext with this handle to receive a security token
3. Send this token to the server.
4. Server processes the token and sends a new token back.
5. Call InitializeSecurityContext again with the token provided by the server.

The service was a non-interactive one so we hooked a debugger to it via a remote debug session - else the debugger is not in an interactive desktop.

The following reg key will use image file execution to do so:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\MySvc.exe]
"Debugger"="c:\\debuggers\\cdb.exe -server tcp:port=5155"

Then we fire up the service and attach to it ASAP

windbg -remote tcp:server=localhost,port=5155

First off I noticed they were using the SSL package - "Microsoft Unified Security Protocol Provider". I also noticed that the same service connected to SQL 2000 ok, but it never connected to the SSL package. The sql devs let me know that the SQL Server 2000 never hit SSL code because SQL Server 2000 does not have a SSL certificate. SQL Server 2005 generates a self-signed certificate, which triggers the Schannel code path on the client.

https://msdn2.microsoft.com/en-us/library/ms189067.aspx

"Credentials (in the login packet) that are transmitted when a client application connects to SQL Server 2005 are always encrypted. SQL Server will use a certificate from a trusted certification authority if available. If a trusted certificate is not installed, SQL Server will generate a self-signed certificate when the instance is started, and use the self-signed certificate to encrypt the credentials. This self-signed certificate helps increase security but it does not provide authentication or nonrepudiation. If the self-signed certificate is used, and the value of the ForceEncryption option is set to Yes, all data transmitted across a network between SQL Server and the client application will be encrypted using the self-signed certificate"

I also noticed wen I fired up a debugger on it, the calls to ISC ( InitializeSecurityContext ) always seemed to succeed. How very frustrating.

So - in order to get a closer look ( ok, honestly I wasnt sure if the SQL traces were accurate ) so I added some logging to secur32.dll in order to give me data straight from the horses mouth.

I ended up seeing this:

Process 4052 called InitializeSecurityContextW returns status 90312
Process 4052 called InitializeSecurityContextW returns status 90312
Process 4052 called InitializeSecurityContextW returns status 80090304

OK - looks like it all lined up - so why was secur32 returning this? 90312 = SEC_I_CONTINUE_NEEDED and is typical and we loop until the context is done or we err. In this case it errors with 80090304 =SEC_E_INTERNAL_ERROR. Nothing like an excellent error :)

Each package has a specific handler - in this case it is SSL and schannel.dll - which called to the Microsoft Enhanced Cryptographic Provider Library - and it turns out that this does some checks for the current stack commit and reservce stack size. If the reserve size does not allow for a guard page ( see https://msdn2.microsoft.com/en-us/library/ms686774.aspx ) it will fail the calls with an error - 8009001d ( NTE_PROVIDER_DLL_FAIL). In their case the service commit and reserve was the same:

<snip>

OPTIONAL HEADER VALUES
10B magic # (PE32)
6.00 linker version
26000 size of code
F000 size of initialized data
0 size of uninitialized data
1E308 entry point (0041E308)
1000 base of code
27000 base of data
400000 image base (00400000 to 00435FFF)
1000 section alignment
1000 file alignment
4.00 operating system version
0.00 image version
4.00 subsystem version
0 Win32 version
36000 size of image
1000 size of headers
0 checksum
3 subsystem (Windows CUI)
0 DLL characteristics
400000 size of stack reserve
400000 size of stack commit
100000 size of heap reserve
1000 size of heap commit
0 loader flags
<snip>

As a test I chnaged this via editbin /STACK: and it worked ok.

This was rejected as a bug as we expect behaved apps to have a guard page.. dont shoot the messenger.

PS - when I hooked into the system via a kernel debugger you could see the errors and debug it.

spatdsg

SSPI failures due to stack size.

Additional resources