Asynchronous site collection provisioning for Office 365


One of the highly requested samples related on the Office 365 and Azure usage is the asynchronous self-service site collection provisioning. This is pretty much a follow up for the similar blog post concentrating for the on-premises and Office 365 Dedicated side, but this one concentrates on the normal Office 365 with the provider hosted apps hosted in Microsoft Azure platform. There are small differences between the on-prem and the cloud, but the logical design is pretty much the same as in following picture, but for this one I wanted to indicate where the individual elements are actually hosted and using which technique.

image

  1. End users are exposed self service site collection creation UI
  2. Request are saved to SharePoint list, which could be optionally associated with a workflow for approval purposes. You could also store the requests to some other centralized location, like to Azure Storage Queue, which is then accessible by the remote timer job. Details on this slightly depends on the exactly business requirements.
  3. Remote timer job is schedule as a web job or as a worker process, which is checking if there’s any site collections to be created in the request queue. If there are and the request have been approved in the case of approval workflow, remote timer jobs actually starts the creation of the site collection
  4. Needed assets like images and themes are stored in some location where the remote timer job can provision them to the newly created sites
  5. Actual creation starts by creating out of the box site and then site can be specialized based on end user selection, like for example by providing multiple different adaptations of the team sites, which can differentiate for example with the created document library or content type perspective. Whatever is needed for the particular deployment.

 

How do I handle branding for the sites? Do I use web templates?

This is pretty common question and the recommendation is to use directly CSOM as part of the remote timer job to deploy the needed assets to the just created site collection. The reason why we don’t recommend web templates is that they are not automatically updated whenever the introduce any new capabilities ot the out of the box site definitions. This means that after a while, your web templates are falling behind of the updates which are applied to the out of the box sites.

Ok – so how should I approach this then?

You should always start by provisioning out of the box site, which means basically either team site or publishing site. Next step is then to apply the needed changes to the provisioned sites based on the selected template. This means branding, content types, libraries and whatever is needed, which we used to provision to the sites using the web template technique. That does not mean that you are limited to just few templates, you can still have many templates like with web templates side, but you just start with by creating the site using out of the box site.

image

  1. Provisioning engine which could be hosted in Azure or in on-premises. Could be as simple as set of PowerShell scripts which are provisioning the site collection and uploading the needed elements.
  2. We always start with the out of the box site provisioning, so that you get all the latest improvements for the sites.
  3. Actual specialization based on the user selection. Is it collaboration, project or organizational site. This is pretty much the stage which can be compared to the web templates. Main difference is however that we apply the needed settings and configurations to the sites using CSOM/REST. Uploading of the assets or creating the needed content types. This way the site has no dependencies on any external elements, which will help with the future updates on the service.

 

How is it developed?

You can download the code from the Office 365 Patterns and Practices guidance (PnP) and for detailed explanation, please check this video from Channel 9 by Andrew Connell where he explains the code structure and how things work based on the code we provided in the PnP project.

 

How would I deploy this to Azure and Office 365?

Like explained in AC’s video, this sample consists from two different part. One part is the UI part which exposes the self service site collection request form, which does not actually provision anything. Second part is then the remote timer job part. UI part will be deployed to the Microsoft Azure as a web site and the remote timer part will be added there as a web job. This way you can easily schedule how often your code is executed.

Following video is showing deployment to the Office 365 and Microsoft Azure in practice.

 

Like mentioned in the video, the normal timeout for the web job execution in the Microsoft Azure is 121 seconds. Since the site collection creation and applying the customizations will take longer than that, you’ll need to increase this timeout. This can be done by adding SCM_COMMAND_IDLE_TIMEOUT key to the application settings using the Azure console or you could also add this to the app.config of the console application when you deploy that to Azure.

image

Office 365 Developer Patterns and Practices

Office365PnPLogoRed_thumb1Techniques showed in this blog post are part of the Provisioning.Cloud.Async sample in the Office 365 Developer Patterns and Practices guidance, which contains more than 50 projects demonstrating different patterns and practices related on the app model development. Check the details directly from the GitHub project at https://aka.ms/OfficeDevPnP. Please join us on sharing patterns and practices for the community.

“From the community for the community” – “Sharing is caring”


Comments (25)

  1. Thorsten Hans says:

    I've built a similar App a few weeks ago. But instead of storing the requests in SharePoint I've used Azure Storage Queue. Using a Queue for this situation seems a little bit more natural to me.

    I'll try to sumarize it also in a blog post 🙂

  2. john lowe says:

    this…   is…    AWESOME!

  3. Vesa Juvonen says:

    Hi Thorsten.

    that would be indeed more reliable and more sophisticated process. With this particular example we intentially wanted to keep things as simple as possible around the asycn option, but having the queue in the Azure Storage Queue would be indeed good option for storing the data in SP list.

    That would be indeed great thing to blog about with the details. Looking forward on seeing it.

  4. Sven says:

    Great post, Vesa! I still have a question though…

    One of my customers currently uses a web template (save site as template) to setup new site collections in the O365 admin center (select a template later). They now want to automate this process because not all items are included in the web template (things like security, property bag items, …). I will have to create a provider hosted app to set this up. If I understand your post correctly it's NOT recommended to perform a SPWeb.ApplyWebTemplate after site collection provisioning  to provision branding, lists, CT & fields elements? It would break possible updates to the existing site definitions? In SP2007/2010 you could create a blank site and apply a webtemplate programmatically. In SP2013 I don't know if this even possible because a blank site template isn't available anymore?

    On the other hand it would be nice to have the flexibility of a web template. The customer is then able to build the 'basic' web template himself and the provider hosted app then only have to provision the web template and do the 'harder' work. This way, when the customer wants to add an additional list or some other content in the newly created site collections, he just have to create a new web template – and don't need to hire a developer to modify the provisioning engine (provider hosted app).

    It would be nice to read your thoughts about his.

    Kind regards

  5. Vesa Juvonen says:

    Hi Sven,

    We do not indeed recommend usage of the web template as such no longer, due the constant updates in the service. Problem is that web template is a snapshot of the site at the time it was taken and you don't get any improvements automatically to your sites, which means that you will have to update the templates constantly. Also deactivation of the web template which is sandbox solution could be problematic. You also mentioned that customer is looking to apply also additional settings which are not available using the web template approach, which means that you anyway need to fall back to use code.

    Code example referenced with this blog post from the Patterns & Practices shows the model for starting with creating the site using out of the box site, but then you modify that using CSOM based on the other requirements. With few lines of code you can apply content types, create libraries, assign permissions, so this is not that hard and when you use code, you can also debug the code, which is impossible with the xml modifications.

    Site templates or web templates are still completely supported, but they do not work well with the constantly evolving platform with by-weekly updates, like what we do with Office 365. On the other side you have the familiarity of the web templates and on the other side aligning to long term roadmap of the service. We recommend usage of direct CSOM from the provider hosted app side, but again things are never black and white.

  6. Sven says:

    Thx Vesa, clear for me… appreciate your feedback.

  7. Jonas says:

    Great post!

    <rant>

    BUT I hope that the team will soon deliver a solution that doesn't require me to provision a customized site using CODE and a provider hosted app.

    It is mind-boggling that we don't have a "simple" declarative solution for this scenario… And I hope that this "upcoming" solution will also cover Publishing sites. I'm tired of having to open up Visual Studio just to package a publishing site, the product has been around for 12 years now 🙂

    </rant>

    Thanks,

    Jonas

  8. Vesa Juvonen says:

    Hi Jonas,

    if you are referring to web template or feature framework based provisioning of any sites using remote provisioning patterns, this is not something what will be coming out from the product or from service. We recommend usage of code for any customizations which in the case of publishing site would mean that you first would provision out of the box publishing site as the starting point and then you'd apply the needed customizations to it, like needed page layouts and other.

    This is however awesome opportunity for any partners or ISV to build that solution which would not require VS to  be opened and in fact you can find all the needed building blocks for that from the Office 365 Patterns and Practices guidance at – github.com/…/PnP

    … trust me, I understand the feedback (rant section), just pointing out where we are heading…  

  9. Ad van den Berg says:

    Hi Vesa,

    Great article! I'm working for a Microsoft Solution Partner called CTB in the Netherlands.

    We're using the web templates for years now in many projects at many customers, and didn't have any major issues with it yet. The main reason we're using them is because a functional consultant (or even the customer itself) can create or update such templates by a few clicks in the SharePoint UI and then use 'save site as template'. We then provision the web or site collection and apply the web template and other customizations which are not included in the web template afterwards. We've built an engine to support all those functionality, which is fully customizable without coding, by just using XML mappings and functions.

    We're currently porting our On-Premises full-trust provisioning solutions to an online variant. The Office 365 Patterns and Practices are a great help for our development team!

    Can we expect more issues on the SharePoint Online platform in combination with web templates because of the constant updates? Do you have some samples of 'Improvements' which are not pushed to the sites based on those custom web templates (which are in fact based on the Team Site template).

    Regards,

    Ad van den Berg

    Technical SharePoint Consultant at CTB xRM

    http://www.linkedin.com/…/advandenberg

  10. Vesa Juvonen says:

    Hi Ad van den Berg,

    issue is not actually on the sites which have been provisioned, issue is the evolving on the saved web template. All sites created based on web template will indicate that they are based on oob site definitions, so any updates targeted to those, will be there for sites which have been already provisioned.

    Challenge is however the web template it self (wsp), which is a snapshot of one specific time and is not updated automatically with for example newly stapled features. This means that someone would have to check every now and then what have been changed in oob sites to ensure that those same changes are also included to the web template file.

    This is not necessarily a massive deal, but after a while can cause you some challenges. If you start by provisioning new sites using out of the box site definition, you always get the latest changes for the sites as a starting point. Then following step is to apply the needed changes or differences based on end user selection.

    We do not also unfortunately have as detailed change log in the service to track down what changes would have been applied to the sites with the latest changes. This would be great information, but overhead for such change log in feature definition level would be too big compared for the value.

  11. Ad van den Berg - CTB xRM says:

    Hi Vesa,

    Thanks for your reply.

    In most cases we do keep a 'Template site' available in the customer's web application which we use to create new versions of the web template every now and then. So this site is getting the updates targeted to the OOB site templates. I think we should stick to that procedure to prevent most of the possible challenges. In most cases this will fit for our customers/solutions. But again thanks for your explanation!

  12. Ilya says:

    Hi Vesa,

    Good article. It would be great if Microsoft provide normal way to provision third party apps to the sites. because now you need to activate Side Loading feature to do it but it's not best way because of this feature is not recommended for production..

  13. Vesa Juvonen says:

    Hi Ilya,

    technically we do support provisioning of third party app to the sites, so this is something which is normally available. We do not however recommend it, since it would bypass the normal app catalog process and it would mean that from centrally perspective there's no way seeing which of the apps are used and where. This does not however mean that you could not use this technique. It's simple matter of an architecture design decision based on available options.

    Second consideration for side loading to be fully automatic is that you'd have to grant high permissions for the app beforehand and not while it's provisioned for the app. In some sense this could be considered as a security challenge, if it would be wrongly used, but you don't really compromise the security, since you'd be still granting specifically permissions for the app.

    We are not really expecting to do additional work on this particular area right now, but PG is always interested on the feedback for the future development areas.

  14. Ad van den Berg - CTB xRM says:

    Hi Vesa,

    Are you familiar with this bug with provisioning site template in SP2013? social.technet.microsoft.com/…/custom-web-template-fails-list-does-not-exist We're having this issue since the April 2014 CU. Still no fix available! This update makes the save site as template with content feature completely useless.

  15. Vesa Juvonen says:

    Hi Ad van den Berg,

    I have not actually seen that, but looks something which should be reported as a bug to Office 365 Support or to customer support if it's on-prem environment. Key point is to get that reported using official channels, so that there's tracking on getting it fixed. Personally I have not seen that, since with remote provisioning you'd be not using that capability in first place and have not really used it earlier either.

  16. Jane says:

    Hi Vesa,

    We're trying to do a provider hosted app for provisioning sites to SP online (async).  Whenever we run the console solution to provision the sites, sometimes it works, sometimes we get the 'request timed out error'.  It errors out on the ExecuteQuery after the web.applywebtemplate.  Is there a different configuration or approach we should be taking for provisioning sites with custom templates?

    Thanks  

  17. Vesa Juvonen says:

    Hi Jane.

    that is caused by time out in the CSOM execution. You can control that by using RequestTimeOut property of the client context. Something like follows:

    <ClientContext>.RequestTimeout = Timeout.Infinite;

  18. Jane says:

    Thanks Vesa.  Even with Timeout.Infinite, I still get the timeout error.

  19. Martin says:

    Hi Jane,

    I also faced that problem, but I never experienced it again after setting RequestTimeout = -1

    I mentioned that I had two open client contexts that were wrapped into each other (one for checking new jobs, and one on the tenant-administration, for actually creating the site. I had to set the requesttimeout on both to infinite – maybe you are facing a similar issue?

    Regards

    Martin

  20. Vesa Juvonen says:

    Thanks Martin,

    that's what we are seeing as well in our implementations for customers. You do need to assing the time out for each specific instances like you mentioned. Otherwise default value will be used and the calls will timeout if they take took long.

  21. Jane says:

    Thanks Martin.  I did miss setting the time out for the other client context.  I do not get a timeout error anymore when I'm running the code on my machine.  

    However, for some reason, I still get the time out error when I try to run the console solution as a webjob in Azure.  I've set the SCM_COMMAND_IDLE_TIMEOUT key as noted in the article above.  Is there a different setting that I've to do to run it in Azure?  Thanks again.

  22. K Solanki says:

    Hello Vesku,

    We need your help in fixing one issue. We are provisioning site collections using CSOM for 1) Provider hosted App and 2) Console application(for initial setup of multiple site collection for Intra).

    for 1) Getting error sometimes like "429: too many requests error"

    2) Getting error like "request uses too many resources error"

    We found below article which says about CSOM limitations msdn.microsoft.com/…/jj163082%28v=office.15%29.aspx

    So need  your suggestion what other way or how to overcome this error? also atleast for on-prem we can increase request size. but what about the office 365?

    Thanks in advance!

  23. Martin says:

    Hi Jane,

    I'm sorry, I don't have experience with the Azure web jobs until now. but since I planned to do the same like you, I'd appreciate if you post it somewhere if you find out.

    Regards

    Martin

  24. Jan says:

    Hi,

    The solution was working fine last week.  Now I am getting the error below when the code tries the run ctx.ExecuteQuery in the code snippet below from the console app part of the solution.

                   ctx.Load(listItems);

                   ctx.ExecuteQuery();

    =======================================================================

    Exception:Thrown: "Access denied. You do not have permission to perform this action or access this resource." (Microsoft.SharePoint.Client.ServerUnauthorizedAccessException)

    A Microsoft.SharePoint.Client.ServerUnauthorizedAccessException was thrown: "Access denied. You do not have permission to perform this action or access this resource."

    =======================================================================

    Any advise will be much appreciated.

    Thanks,

    Jane

  25. Vesa Juvonen says:

    Sorry for the delayed response, completely missed the comments. As a rule of a thumb would suggest to post any questions or comments to Office 365 Developer Patterns and Practices Yammer group at the below address. We have 1500 members in this group with app model interest and it's really active group to assist with any questions.

    aka.ms/officedevpnpyammer

    Jane, Martin – We are not facing that time out issue with production deployment after setting the timeouts propertly and using web jobs.

    K – 429 is throttling warning, you should be looking following PnP sample for assistance –

    Jan – Working previous week, not now usually means expiration of tokens. Easiest way to force re-trusting is to update the version number of the solution to 1.0.0.1 and you should be good to go.