Optimization of SharePoint 2013 Patching

As SharePoint 2013 continues to mature, the size of the update packages (both the security updates and the cumulative updates) continue to get larger. In working my customers who are SharePoint Administrators, I am starting to get more and more questions regarding ways to optimize the SharePoint patching process and looking for opportunities to reduce the duration of the maintenance window required to apply SharePoint patches. The guidance on this topic continues to evolve over time and what I present here is an approach that I have seen used by many customers to optimize their SharePoint patching operations.

Before we get too far into the meat of the guidance, let’s first establish a bit of common knowledge. Setting a knowledge foundation and set of common terms will not only help you to understand the guidance, but it will also make it possible to tweak and adjust this guidance based upon your own requirements and constraints. Let’s start by taking a look at the types of updates that are available.

Type of Updates

Updates to SharePoint are made available to the public in three varieties. The first one is the security update (also known as a PU or Public Update), the second one is the cumulative update (also known as a CU), and the third one is the Service Pack (SP).

Public Updates

Public updates are released via the Microsoft Download site as well as the windows update process. Released as frequently as once a month, public updates typically contain security updates or minor fixes to a specific SharePoint component (such as a specific .dll) and will only include those updated components that are affected by the specific change. Public updates are cumulative for the components that are included (i.e they will have the latest versions of each component in the update package) but they are not inclusive. “Not inclusive” means the other components of the system that were not updated by the specific change will NOT be included in the update package. 

Cumulative Updates

Cumulative updates are also released via the Microsoft download site but are not published to windows update. They are released as frequently as once a month, and come in two flavors. One flavor includes all components (in all languages) in a given “grouping” of components. A “grouping” typically includes all the components for a given “subsystem” with SharePoint such as search or OfficeFile formats, etc. and are usually contained within a single .msi package. The other flavor, called “server” or “uber” packages, includes updates to all components (in all languages) for all “groupings” of components. A “server package” is truly cumulative and all inclusive. Installing the latest “Cumulative Update Server Package” will ensure that all components of the entire SharePoint products is current for all installed languages. It is this cumulative and all-inclusive aspect of server packages that accounts for their large size (the latest server package CU tops 1.4 GB in size). An important thing to note about CU updates is that they include all fixes that have been released to the point in time in which they are compiled and built. This means that the May 2016 Cumulative update will include everything that was in the April cumulative updates as well as any security updates that may been released prior to compilation and packaging of the update release.

Service Packs

Service Packs are yet another update release mechanism that are made available only via the Microsoft download site. Like cumulative updates, service packs include the latest version of all components available at the time of compilation and packaging. However, unlike, cumulative updates, service packs contain only the components for a single language. This means that if you have installed multiple languages packs, you will need to download and install the service pack release for each required language pack. (This is not true for CUs – as stated above, each CU contains all components in all languages). 

For more information about SharePoint patches, see https://blogs.technet.microsoft.com/stefan_gossner/2014/08/18/sharepoint-patching-demystified/

Overview of the Update Deployment Process

Now that we have a good background on the type of patches, let’s take a closer look at the patch deployment process. The deployment process is comprised of two keys phases: Patching phase and “Build-To-Build” (B2B) upgrade phase.

Patch Phase

The patch phase is comprised of two steps: the patch deployment step and the binary deployment step.  During the patch deployment step, the windows installer patches are extracted from the downloaded cab files and placed in temporary storage in preparation for deployment. The windows installer patches are then merged with the existing installed images to produce the target components that actually get deployed. Once step one is complete, step two begins. During step two (“binary deployment”), the SharePoint services that are being patched are temporarily stopped so that the new binaries (executables and dynamic link libraries) can be copied to the appropriate locations (Remember that for Cumulative Updates, all components are upgraded so therefore all processes get stopped). Running services are stopped before the components are replaced in order to prevent a restart of the server. However, in some circumstances a restart will still be required.

The patch phase MUST be completed on every server in the farm and failure to do will render the farm in an inconsistent state. Once the patch phase has been completed and before the “upgrade” phase has started, the farm is in a state called “compatibility mode”. The farm will remain in this state until the upgrade phase is successfully completed. You can safely leave your farm in this state for a short period of time but should try to complete the upgrade phase as soon as possible. Furthermore, when you are in “compatibility mode” it is suggested that you not apply any additional updates until you complete the “upgrade” phase.

Optimizing the Patch Phase

As we know, SharePoint is a complex system and is comprised of many, many services. As a result, the process of shutting down services and updating dependencies can take a long time.  There are a number of recommendations in the forms of blogs and other online publications that provide ways to optimize the patching process. The most common one is found at https://blogs.msdn.com/b/russmax/archive/2013/04/01/why-sharepoint-2013-cumulative-update-takes-5-hours-to-install.aspx. I have worked with many customers that have successfully used the provided scripts to significantly reduce the time it takes to complete the patching phase.

“Build-to-Build” Upgrade Phase

Once the patching phase is complete, the “build-to-build” upgrade phase must be started. The “build-to-build” upgrade (also known an b2b) is typically completed by running the SharePoint Configuration Wizard (psconfigui.exe) but can also be initiated by running the command-line utility psconfig.exe. During this phase, the SharePoint objects in the configuration database are updated, the SharePoint services and their databases are upgraded, and finally the SharePoint content databases are enumerated and upgraded. Depending on the type of schema changes present and the size of the content database, the content upgrade tasks can take a considerable amount of time to execute. Furthermore, the SharePoint Configuration Wizard will upgrade content databases in a sequential manner. Since the databases are shared across all servers in the farm, this database upgrade process takes the most amount of time on the first server. However, the upgrade process must be completed on all servers in the farm in order for the farm to remain in a predictable and supported state.

More information about the update deployment process can be found in the TechNet Article entitled “Software updates overview for SharePoint 2013“. Please go to https://technet.microsoft.com/EN-US/library/ff806329.aspx for more information.

Upgrade Strategies

The various TechNet articles (as well as general recommended best practices) tend to recommend one of two upgrade strategies: in-place and database attach. Let’s look at each of these in a bit more detail.

In-place upgrade

For the in-place upgrade, the patch phase is executed on each server in the farm (typically starting with the Central Admin server) and then when that is complete, the SharePoint Configuration Wizard is then executed on each server (again typically this is done first on Central Admin server). 

In simplified form the process is:

1. Install patch on first server in Farm (typically CA)
2. If installing service pack, install additional SP language packs
3. Install patch on all other servers in the farm. (This can be done in parallel with Step #1)
4. If installing service pack, install additional SP language packs on all other servers
5. Run the SharePoint Configuration Wizard on first server in the farm (Typically CA) (This could take a very long time if content databases are large and patch includes schema changes)
6. Run the SharePoint Configuration Wizard on all other servers in the farm (This should go faster on subsequent servers)

Some of the advantages of this approach is that it is the simplest approach and requires the least administrative effort. Some of the disadvantages are that the upgrade operations are performed sequentially and there is no way to determine or change the order in which items are upgraded. As a result of the sequential nature of this approach, it typically results in the longest maintenance window.

Database attach

For the database attach upgrade, all of the SharePoint content databases are detached from the farm before the patching and upgrade process is started. Then the typical “in-place” process is executed. Once that is complete on all servers in the farm, the SharePoint content databases are reattached to the farm. 

In simplified form the process is:

1. Install patch on first server in Farm (typically CA)
2. If installing service pack, install additional SP language packs
3. Install patch on all other servers in the farm. (This can be done in parallel with Step #1)
4. If installing service pack, install additional SP language packs on all other servers
5. Detach all content databases from the farm (Sometime this is performed as Step #1)
6. Run the SharePoint Configuration Wizard on first server in the farm (Typically CA)
7. Run the SharePoint Configuration Wizard on all other servers in the farm.
8. Attach all content databases to the farm via Central Admin or Attach-SPContentDatabase. (The attach process can be done in parallel (multiple at once) and will automatically upgrade content databases to the current schema version) 

The primary advantages of this approach are that multiple database attach operations can be run at once and the order of reattachment can be controlled by the administrator (upgrade most critical or largest content databases first). The primary disadvantage of this approach is that while detached, content is unavailable. Furthermore, there is some risk in detaching content databases as there are some timer based processes that expect the content to be available and could result in unexpected results if those databases are detached while the timer processed run.

These strategies (and others) are documented in the TechNet Article at https://technet.microsoft.com/en-us/library/ff806338.aspx

In-place with Manual Database Upgrades

Over the years, working with many, many different customers, I have learned that neither of these approaches are ideal for all customers. As a result, experience has taught us that alternative approaches should also be considered. One such approach is to complete the patch phase on all servers then utilize the SharePoint Management Console and PowerShell to execute Upgrade-SPContentDatabase on all SharePoint content databases. Depending on the capability of the SQL servers it is quite likely that multiple database upgrade sessions can be run in parallel. Once all content databases have been upgraded, the upgrade process can be completed by running the SharePoint Configuration Wizard to complete whatever upgrade tasks remain. 

1. Install patch on first server in Farm (typically CA)
2. If installing service pack, install additional SP language packs
3. Install patch on all other servers in the farm. (This can be done in parallel with Step #1)
4. If installing service pack, install additional SP language packs on all other servers 
5. Upgrade all content databases in the farm using Upgrade-SPContentDatabase. (The upgrade process can be done in parallel (multiple at once) and will upgrade content databases to the current schema version)
6. Run the SharePoint Configuration Wizard on first server in the farm (Typically CA) (Since the content databases have already been upgraded, this should run relatively quick since just SPservice DBs need to be upgraded)
7. Run the SharePoint Configuration Wizard on all other servers in the farm.

One disadvantage of this approach is that additional administrative involvement is required and it could put more strain on the database infrastructure. However, one big advantage is that the content remains available through most of the upgrade process (Sites in a given content DB may be unavailable or slow while Upgrade-SPContentDatabase is being run against that database). This is also likely to result in the shortest maintenance window as multiple content database(s) can likely be upgraded simultaneously.

Hope you find this information useful!