Farm Patching Explained – High Availability SharePoint

SharePoint patching can be a non-trivial task and patching any SharePoint farm from one version to a newer version of the same product involves taking the entire farm offline at a certain point by necessity - this post explains when and why. Later we’ll look at how to avoid outages during patching but for now we’re focussing on when we have to down the entire farm and/or specific services.

So let’s start with the basic steps of how to patch a SharePoint 2013 farm:

  1. Update binaries - install patch on each server, either all at once or one-by-one. As each server is patched IIS & the timer-service are stopped and depending on the role, the server being offline will cause stability problems or not as there’s now one less server for whatever roles/farm-services it was running.
  2. Upgrade databases - when all servers are patched, the farm databases often needs to be upgraded by running the configuration wizard/psconfig one of the servers. This isn’t always the case – it depends on what version you’re upgrading to and from but in this post I’ll assume you’ll need to do so.
  3. Complete upgrade - when the databases are patched, all remaining servers need to complete the patch process by running psconfig on each after the farm is upgraded.

This process seems straightforward enough but there are some caveats that basically mean you cannot patch a farm without taking the entire farm offline at a certain point. These caveats are:

  • For each server, when the patch is applied IIS will stop and won’t resume again until the patch is installed & the configuration wizard (psconfig) is run afterwards.
  • The wizard/psconfig won’t run on a patched server after patch install until all servers in the farm. You can force an upgrade anyway but this really isn’t recommended.

SharePoint Binary/Database Version Compatibility

SharePoint binaries are hard-coded to only touch certain databases schema versions, with only one schema version for any type of database (content, user-profile, search, etc) having a fully-supported schema version. That’s to say for example SharePoint ​15.0.4535.1000 will open databases created by binary version​15.0.4505.1002 –> 15.0.4535.1000 with databases created by binaries 15.0.4535.1000 being the only fully supported schema version*. Databases prior to 4535 won’t even be attempted to open by binaries 15.0.4535.1000.

New binaries often will work with old databases but again, we don’t support this so if we see this happening the first thing we’ll ask you to do is upgrade them. That said, version upgrades don’t always upgrade the schema version so the amount of downtime patching will cause can vary and may even be entirely avoidable if done intelligently and the version-difference isn’t large.

To see what versions your databases have had, check the “versions” table on each one (but please don’t modify any databases or data owned by SharePoint by hand, ever) – the highest value is the current version:

image

* This I gave as an example of how the compatibility ranges work- I can’t confirm these compatibility ranges are actually the true or not – it’s just to give you the idea.

Process Breakdown

In real terms then let’s see what happens when we patch a simple 2-server farm (servers S1 and S2) with each server handling service-apps and web-apps; a rare and not-exactly-optimal configuration, but it saves us having to worry about what service-apps will become unavailable because that particular server is being patched.

Begin Patching – Run Setup

 

Patching Stage

Availability (green = usable)

1

Prepare for patches. All servers and services online.

image

2

Start patching server 1 binaries.

Server 1 goes offline taking with it any services the farm might’ve needed. Server 2 will quite happily continue servicing requests.

image

While patching; SharePoint services stop, giving us an outage on that machine.

clip_image006

3

Server 1 will finish patching and may even need a reboot.

Again, until the patch is initiated on either Server 1 to upgrade the farm DBs or Server 2 to patch the binaries, Server 2 will have no problems continuing to serve pages or requests.

image

Upgrade Databases - Total Outage!

We now have to take the farm offline completely to proceed with the patching process. Once all servers are patched run the configuration wizard on the 1st server you want to bring back online and following instructions (or run psconfig for advanced users). If you haven’t patched all your servers you’ll see something like the following:

clip_image010

4

We now have to start with Server 2 because PSConfig/The Configuration Wizard won’t let us continue.

You could force the upgrade on S1 without S2 being updated but that’s potentially asking for a world of hurt so we won’t force skip this in this example. Continue at your own risk.

image

5

Now we need to upgrade the databases – this is done by finishing the patch process on any of the servers – the wizard/psconfig will realise there’s half-baked databases that need attention and will upgrade those too. This is where you’ll have a total farm outage while everything is upgraded offline.

Pro-tip: you can disconnect content & service databases before patching anything so this stage completes much faster, although a word of warning: Microsoft doesn’t support half-upgraded farm so this’ll need to be completed once the farm is back online – this option is for emergencies where a DB or other is holding back the upgrade from completing on the 1st pass.

image

clip_image016

Upgrading databases happens here, done in the background on a timer-job.

Complete Upgrade - Restore Service

The hard part’s over and we should now have some kind of service again once the wizard’s done on one server – we just need to repeat the same on all the other servers now. Once done, the wizard will look like this:

clip_image018

6

Once Server 1 is done configuring post-patch you’ll have a farm something like this…

Server 1 will be able to process requests now it & the databases are upgraded.

image

7

All set – all servers & databases now upgraded.

image

Verify Completion - Check Patching Status

You’ll know if you’ve not completed the patch on a server by running the configuration wizard, if you’ve not done it yet you’ll see this screen.

clip_image024

Farm Availability Timeline

From the above steps, even with the most optimistic time-line we’re looking at a complete and unavoidable outage from the moment you begin patch install on the last server in the farm right up until the configuration wizard (or psconfig) finishes with the 1st server the post-install configuration.

image

Remember, the stage in red represents a total outage which at a given point in the patching process is unavoidable. Before the point of no return (DB patching) you should have availability but if you start patching a server which services a specific service-app and there are no more available then you’ll lose that service-application even if the farm is technically online. A service-app outage is sometimes as bad as a total outage so this has to be taken into account.

Generally if you want absolutely guarantees of all services being available then you’ll need to switch to a backup farm while the target farm is upgraded – more on that in a later post.

Notes

As mentioned already, depending on the patch in question, databases may or may not be changed - if you want to be sure what’s going to happen then run a test install on a test farm. Each SharePoint database comes with a “versions” table which contains the version of that particular database (there’ll quite possibly be more than one – the highest is considered the current version with the others being version history) and a SharePoint server will know for which version-range it can touch.

Important Disclaimer: Microsoft only supports SharePoint farms that are fully upgraded, even if the software will mount the database ok. Make sure you complete the update fully once you have serviceability again!

Patching Without Downtime

This of course is possible but it basically involves having another farm on standby to cover for when your main farm drops off the internet. Read more about this @ https://blogs.msdn.com/b/sambetts/archive/2013/11/15/patching-sharepoint-farms-with-no-downtime-high-availability-sharepoint.aspx

 

Cheers!

// Sam Betts