Couple of days ago I was on one of my regular customer engagements and as always, there was some proper fun with SQL Server that I had to get my hands dirty with.
So the situation is the following: a highly consolidated environment of SQL Server 2012, 1014 and 2016 standalone instances with Always On Availability Groups, syncing to a secondary site. So far so good. Everything was in production, working without any issues until one day, there was a requirement to install another pair of instances for a new application. The requirement was for a SQL Server 2012 with latest SP (in our case – SQL Server 2012 SP3). The customer starts with the installation of RTM and everything goes well. He then proceeds with applying SP3 and all of a sudden – the patch process hangs and ultimately fails, leaving the instance in an inconsistent state. The fun thing was, that this entire installation and patching went perfectly fine on the second machine.
During our investigation, we noticed the following symptoms:
- During the patch, the SQL Server instance was not able to start and the Windows log was filled with exceptions like below
- The SQL Server error log contained the following error
So as it seems, due to the patching process, there was something preventing the instance from starting up. And that something was causing a stack dump. So…I basically had three options:
- Open a case to Microsoft Support so they can analyze the stack dump
- Try to analyze the dump myself (that is fun, but also time consuming and not something that is fun while onsite with a customer)
- Scan whatever knowledge base/internet article I can find
I tried to start with the third option first. Naturally I found number of articles with the exception code, pointing to a stack dump generation and here and there some third party tools, messing the SQL Server memory stack. My customer had only antivirus there, so naturally I went for stopping it immediately, but that did not help.
One of the many good things when you work for Microsoft is, that you get to access our internal database with all the customer support cases, so that was my next attempt. Besides all the cases with some third-party tools, preventing the startup, I noticed couple of situations, where the SqlServerSpatial110.dll causing some issues due to wrong versions. WRONG VERSIONS?!?
Yep, I immediately went for checking this DLL. As it turns, there is one DLL for every SQL server version, installed on the machine, which is stored at C:\Windows\system32 folder:
As it turned out, the SqlServerSpatial110.dll for some reason was the version, coming from SQL Server 2014. Great! So, I found a SQL 2012 SP3 version of that DLL (2011.110.6020.0), replaced it and everything was going well.
I am not sure at this point if this would help anyone else, having the same exception, however I guess if you ever hit it, it is worth trying!