Site provisioning techniques and remote provisioning in SharePoint 2013

Since eventually SharePoint is all about creating sites and functionalities for the end users as efficiently as possible, I’ve been always extremely interested on fine tuning the site provisioning techniques to ensure efficient and easily maintainable SharePoint deployments. These techniques have been pretty much stayed the same since SharePoint 2007, except the web template functionality introduced in 2010. If you are not familiar of the different models and options, check the short description from one of my most popular posts

With the SharePoint App patterns and change of approach for the SharePoint customizations, we can actually be more efficient compared to previous approaches. Since SharePoint will be also concentrating more and more to cloud based deployments, we’ll need to design a pattern, which is valid or easily doable both on-premises and in Office365 or with other cloud providers.

This post is an introduction to remote provisioning techniques and we’ll be releasing more detailed guidance and examples using official channels. I’ve also included a demo video on the remote provisioning, which can be watched to see thing in action.

[Update on 9th of Sep 2013 – separate video recording of the presentation covering these topics available on separate blog post , including slides and pictures from this post. Added also link to download the video for offline viewing]

Provisioning techniques in SharePoint 2013

Before getting familiar on the remote provisioning, let’s quickly cover some of the more classic options from SharePoint 2013 perspective and move to remote provisioning with video demonstration after that. I’ll raise some key pointers on the each approach.

One of the classic classic challenges of all of these approaches is that at least most of them, require full trust (farm) solutions to be deployed to SharePoint farm. This then has immediate operational and maintenance impact on the farm from availability, DR and from upgradability perspective. Some levels of this challenge was tried to be addressed with sandbox solutions in SharePoint 2010, but that technique was not as flexible as needed for real enterprise projects.

Feature stapling

Feature stapling is technique to modify the provisioning model for existing site definitions. As part of the site creation, stapled features will be activated, which can modify the site anyway needed. It’s important to notice that feature stapling can only be used for extending or changing behaviour of the out of site definition, but you can’t create new options and you can’t associate features to web templates. Stapled features are also activated BEFORE any modules or lists which are in onet.xml have been provisioned, which means that you can’t use feature stapling technique for example to modify welcome page.

image

Let’s open up the numbers one by one.

  1. One feature to create association between existing site definition and second feature using FeatureSiteTemplateAssociation –element
  2. Users access site creation options and different installed templates are shown
  3. Site creation process starts based on selected site definition
  4. Actual site provisioning is done based on associated onet.xml file
  5. Stapled features are activated in “random” order. Activation of site scoped stapled features will happen immeidately after site scoped features from onet.xml are activated and web scoped stapled features are activated after web scoped features in the onet.xml, but before module or list instances
  6. Actual stapled features are activated and modifications can be applied using xml or code

If you are upgrading from SharePoint 2010, there’s really no considerations for the feature stapling technique as such. Obviously there might be some changes required to actual features, but model is completely the same in SharePoint 2013 than in SharePoint 2010.

Site templates

Difference between site templates and web templates is terminology thing, since essentially both site templates and web templates are based on same technology, by using WerbTemplate xml-element. I do tend to differentiate them to avoid mixing the discussion. For me, the site templates are the one’s which are created by using the “Save site as template” functionality from the site settings.

Notice that like with previous versions, saving sites as template is not supported for publishing sites (even if you’d know the URL), since they can cause un-expected issues. Natively also site templates are only available in the Site scope, meaning that you can’t directly create new site collections based on site template, since template is not available in farm or tenant scope. You can however first provision empty site collection and then apply site template afterwards to this newly created site collection, but there’s no out of the box APIs for doing this programmatically, so automation would be difficult (but not impossible, with code you can do anyway).

Since anyway the site provisioning model is identical as for web templates, you can check the details from below.

If you are upgrading from SharePoint 2010, your existing site templates have to be all re-created to SharePoint 2013. You should first create site in SharePoint 2010, upgrade that to 2013 and save it again as a template, if needed.

Web templates

Web template is technique based on WebTemplate feature framework element, which was introduced in SharePoint 2010. Technique provides us a way to use alternative onet.xml during provisioning time of the sites without any dependencies on that onet.xml file when provisioning has been completed. This gives us flexibility to use alternate templates during initial site creation, but to ensure that sites will still fall into same upgrade and maintenance processes like out of the box sites. You can read more detailed description of the differences and models from SharePoint 2010 and web templates blog post. Web templates can be deployed either using full trust (farm) or sandbox solutions and they can be scoped to site or farm level, depending on their usage pattern.

One of the big challenges of the web templates is that we do not process any stapled features from the so called base template, to which sites are associated after creation. This means that if you take for example out of the box team site as your starting point and you copy that onet.xml file to your web template, there will be huge differences on created sites. This is due the fact that especially in SharePoint 2013, there’s considerable amount of new stapled features to out of the box sites, which enable new capabilities.

We can fix this challenge  by associating the stapled features directly to the onet.xml file of the web template, which will work for initial updates, but we might face additional concerns in future. Like it has been communicated by SharePoint product team, we will start having more frequent releases on SharePoint in upcoming years, even thought this refers more to Office365, it actually applies partly also to on-premises in format of cumulative updates (CU), which will introduce also new capabilities. Since some of the these capabilities will be introduced to out of the box site definitions by using feature stapling technique, our web templates would have to be also updated during each CU, based on updates. This is not obviously optimal from the cost perspective, which means that web templates might not always be the right answer.

 

image

Let’s open up the numbers one by one.

  1. Show site options based on activated features with WebTemplate element either from farm or site scope
  2. When template is selected, onet.xml from the corresponding feature is executed and site provisioning is started based on that
  3. Features mentioned in the onet.xml are activated in the order of their presence and each feature has their own feature.xml with element.xml files which is processed
  4. Dependent features are activated before features in the onet.xml are activated… features can also activate other features programmatically from feature receiver
  5. After all features have been activated, possible list instances and modules are provisioned from the onet.xml and site is ready to be used by end users

If you are upgrading from SharePoint 2010 to SharePoint 2013, you’ll need to carefully re-create your onet.xml files for web templates to ensure that you have all the new features which are stapled to out of the box sites, also included to onet.xml of the particular web template.

Site definitions

Site definitions really haven’t changed that much after SharePoint 2007, when feature framework was associated to them. Site definitions are primarily for out of the box site structures and should be avoided in all costs within deployment specific customizations, since they will have significant long term cost impact on the deployments. One of the key downside is also the fact that site definitions will require full trust or farm solutions, which means that they are only supported in on-premises deployment and all deployments which have been using them, will have significant challenges with possible migration either to out of the box sites or to cloud services.

We do have however many deployments for example from 2007 version which have been using site definitions and these are tricky to get fixed, since there’s really no easy way to dot that, which means separate migration project to ensure that content is properly transferred and secured also in future. Key challenge with custom site definitions is also the impact on upgrade to newer versions of the SharePoint. This impact slightly depends on how the site definitions have been created, but we should anyway avoid usage of them. We should consider site definitions as a technology only aimed for SharePoint product, not as extension point.

image

Let’s open up the numbers one by one.

  1. Show site options based on web temp files in the 15 template folder – notice that each language has it’s own web temp folder, which is created when language pack is installed
  2. When template is selected, onet.xml from the corresponding folder is executed and site provisioning is started based on that
  3. Features mentioned in the onet.xml are activated in the order of their presence and each feature has their own feature.xml with element.xml files which is processed
  4. Dependent features are activated before features in the onet.xml are activated… features can also activate other features programmatically from feature receiver
  5. After site scope feature activation and web scope feature activation corresponding stapled features are activated for each scope, which have their own xml configuration
  6. These features can also have dependencies and other activations
  7. After all features have been activated, possible list instances and modules are provisioned from the onet.xml and site is ready to be used by end users 

 

Provisioning providers

Provisioning provider pattern is actually really interesting, but it’s considered as is legacy technique and is approach is deprecated. Model is based on extension point in the webtemp*.xml file, which provides us capability to fully customize what happens when end user select template from UI and move forward. Essensially provisioning provider is replacing whole onet.xml requirement, but does how ever have to utilize out of the site definitions to actually create the site.

Model is based on SPWebProvisioningProvider class or actual implementation does have to inherit from it, which is then called to take full responsibility of the site provisioning process. Classically implementations have been based on models where we first create new site based on some out of the box site definition and then extend based on customer requirements. Advantage of the model is however that you can provide multiple different variations of the templates to be available, like separate templates for standard publishing content site and news or organizational site, which would be all based on out of the box publishing, but sites are pre-configured based on their usage model.

As mentioned, provisioning providers are based on extensions point in the webtemp*.xml file, which means that to be able to get them available for end users, you’ll need to deploy full trust solutions, which then means additional costs for operations and maintenance for example due required downtime for all services in the farm.

image

Let’s open up the numbers one by one.

  1. End users access the site creation from
  2. All templates are shown to end users based on installed site definitions and templates.
  3. If template with provisioning provider entry is selection, external provisioning class as configured is executed and optional information from ProvisionData is provided as parameter. This provides us method to customize the provisioning model based on parameterization in WebTemp*.xml file
  4. Actual site creation will be on the responsibility of the provisioning provider with given attributes, like URL and other information from form.
  5. Provisioning provider classically will create new site based on some out of the box site definition
  6. Actual sites are based on site definition onet.xml file
  7. After creation of the site provisioning provider can modify the site anyway as needed by uploading new files, content or performing additional configurations on the just created site, before context is shown for end users

 

Full server side custom solutions

Obviously full server side customizations can be whatever, but in general they are based on model where we introduce new request form or web part, which can be used to request new sites based on the given metadata. This is pretty common pattern in scenarios where customers would like to collect more precise metadata on created sites or where additional approval of the sites is required.

Classically these implementations have been based on timer jobs and can be pretty complex, based on business requirements. In SharePoint 2013, we can however achieve pretty similar end results also using remote provisioning, which would mean that there would not be a need for full trust solutions, which would have impact on operational and maintenance perspective directly to the SharePoint farm.

 

image

Let’s open up the numbers one by one. This is one classic pattern, which has multiple adaptations, so this is just one example.

  1. End users can access actual site request form using either application page or simply a web part
  2. Requests are stored to separate list, which works as site directory and metadata list. By storing the request to list for async processing, we don’t force end users to watch gif animation while the site is being created and we can also associate workflows to the process
  3. If workflow is required, it can associated to this specific list, so that when new requests are created, there’s separate approval workflow started, which has to be finished before moving forward
  4. Approved requests are being processed by a custom timer job, which orchestrates the process
  5. Actual sites are created using site definitions or web templates, based on requirements… Best option would be to always apply first the out of the box site definitions and then modify the site based on needs, to avoid the versioning issues on web templates with cumulative updates. This could be easily done using code.
  6. After site(s) have been created, an email notification is sent to requestor and site administrators

Similar approaches would be the use of workflow activities, which would also require server side code to be deployed to SharePoint farm. These approaches can be pretty difficult to maintain, especially if they contain timer jobs and other complex elements, which brings challenges from customization upgrade perspective (classic challenge of updating timer jobs with wsp’s).

 

 

Quick introduction to remote provisioning

Remote provisioning is model where we actually do the template system outside of the SharePoint by using the new SharePoint app patterns. In this case we would be using provider hosted pattern to provide customizable sub site creation experience for the end users of the SharePoint.

This model is suitable for both on-premises and Office365, which is clearly beneficial from long term maintenance and operation perspective. One solution fits all model is obviously better than thinking on-prem or cloud as an exception to story. One of the clear benefits of the remote provisioning model is that we don’t need to have any feature framework elements or other customizations deployed to SharePoint farm, we rather control the customizations outside of the SharePoint, which gives possibility to update and change the provisioning engine without any impact on SharePoint side from availability perspective. Second key advantage is actually the impact to DR models, since when we push all of our changes to site collections from remote locations, we don’t have any dependency on any solution packages in our content databases. This means that as long as we have the content database, we can setup the farm back online.

 

image

Let’s open up the numbers one by one.

  1. Override create sub site link with hook to provider hosted app (if we target sub site creation)
  2. Actual template selection is coming from provider hosted app side based on configuration – abstraction layer to provider different variations of the oob sites
  3. Different templates and capabilities are handled in the provider hosted app side based on requirements, no configurations in the SharePoint side
  4. Sites are provisioned based on configuration using CSOM/REST. All sites are created based on oob site definitions, but are then modified based on configurations
  5. All features and functionalities associated to oob sites are activated and additional changes are applied, like master pages or themes uploaded and applied to sites

 

Notice that we can also use remote provisioning techniques to replace the out of the box my site creation experience to provide branded and modified personal sites (SkyDrive Pro) sites for end users. This is slightly different approach than for sub sites, but is also fully supported and valid approach. We’ll be releasing more details on this one as well.

With SharePoint 2013 client side object model has been significantly improved to enable us new remote scenarios. There are still some limitations in the exposed capabilities, which are however being addressed. We move gradually towards the cloud application model (CAM) and client side object model will start exposing more and more capabilities to ensure that customer objectives can be achieved also using more agile patterns.

Key benefit of the remote provisioning model is simply the fact that we really don’t need to have ANY customizations deployed to SharePoint farm using classic full trust or sandbox solutions, we rather imitate the actions from end users to manually upload or configure the settings using browser. We can also automate this process in our remote provisioning tool by introducing the required configuration options with our own xml based configuration. This way we can adjust the behaviour and model based on ever changing requirements in the particular deployment.

 

 

Example usage of the remote provisioning to replace sub site experience

In the below video, I’ll be demonstrating usage of remote provisioning from the sub site creation perspective. We’ll be looking into how to create customer specific templates and the engine works by using remote provisioning pattern also for sub sites. Be sure to also check on Richard diZerega's Blog blog post related on site collection provisioning with custom self-service site collection creation form. Notice that currently Richard’s example code only works in Office365 due certain limitations in the client side object model, but these issues are already being addressed and you’ll be seeing additional capabilities pretty soon.

Video is available for download from my SkyDrive if needed for offline purposes.

 

Summary

Choosing the correct site provisioning pattern will have long term impact on deployment costs and how deployment can be maintained. By using the modern approaches, we can take advantage of the all out of the box capabilities and still provide nice user experience for site selection. We are in the move to more cloudy deployments of SharePoint and remote provisioning is technique to reduce the complexity and operation impact of the solution packages and feature framework. This transition is gradual and won’t happen overnight, but it’s evident taking into account the roadmap of the SharePoint.

Evaluate your technical and business requirements compared to long term cost impact. This will help to choose the right pattern for your customer and deployment. As mentioned, there will be more guidance provided on this topic using official channels, so that remote provisioning can be more easily taken into use.

Notice that showed code package is available for download from the following location. I also want to thank my colleague Koen Vosters for helping on the coding part of the this reference solution, which was initially an idea during SP2013 alpha stage, but has been now successfully taken into use with multiple customers as the replacement model from old full trust solutions towards the cloud application model.

What do you think about this option as an example on transitioning away from full trust code to cloud application pattern? Please leave a comment.

 

References

Few useful links and resources.