Real-life Team-Based Development for Web Content Management with SharePoint 2007

  • Article

It took me a while to come up with everything I needed for this post as I want to make it as accurate as possible with the experience I had on the field.  The goal of this post is to provide information on the different ways to work as a team for a WCM web site with MOSS 2007.

 

I believe that MOSS is large enough that a "one size / fit all" approach is unlikely.  There are simply too many variables.  Fortunately, I believe that what myself and awesome people I've worked with have overcome most issues and we have automated most deployment procedures.  I also believe that most of this knowledge will likely apply to other MOSS scenarios such as Document Management.

 

This post applies to : Microsoft Office SharePoint 2007 (based on real-life deployments pre-SP1), Microsoft Office SharePoint Designer 2007, Microsoft Visual Studio 2005 Team System, Microsoft Team Foundation Server (not required but nice to have).

 

Contents

 

 

Overview

What this post will contain is directions and guidance on how to achieve a working multi-environment team-based development lifecycle for WCM with MOSS 2007.  Unfortunately, for legal purposes, I cannot offer all STSADM extensions, Event Handlers, Features, and other code artifacts we developed for specific entities.  While I may eventually create a more "blank code version", this isn't available at the moment.  However, the how-to knowledge should be relatively easy for developers.

 

In my experience, there are 2 primary types of development topologies in a MOSS WCM project:

  1. IT oriented lifecycle with multiple environments such as :
    1. Individual development (while team-based)
    2. Integrated testing
    3. Staging
    4. Production
  2. HTML oriented lifecycle with 1 (production only) or 2 environments (staging + production)

 

The main differences between the 2 is where your "base environment" is and where you allow "SharePoint Designer users" to act.  In an IT oriented lifecycle, every deployment follows a strict cycle of testing, staging, and scheduled deployment to production for all artifacts.  In an HTML oriented lifecycle, that may be true for "code artifacts" but you may want to have web integrators use SharePoint Designer directly in the staging environment and/or production.

 

Obviously, there are advantages and disadvantages for both sides :)  I'll try to describe them to the best of my knowledge, but first, I have to explain what Content Deployment is and why it's not meant for development environments:

 

IT oriented lifecycle with multiple environments

What I call an IT oriented lifecycle is when you already have a development lifecycle in place.  Often case, this is for secure or critical environments where downtime is not acceptable.  A secure close-to-production Staging environment is usually available in order to test deployments before doing them in Production.  You will want minimal steps to deploy as the deployment may be done by a 3rd party who may not have as much SharePoint experience and who's goal is to maintain the environment, not debug it.  This lifecycle is more typical for Intranet web sites.

 

Developers

When you have this environment in place, are you are likely to have separate developer environments as well as an integrated environment.  First of all, let's talk about the separate developer environments.  You can take a look at this general Team-Based Development document on MSDN : https://msdn2.microsoft.com/en-us/library/bb428899.aspx for more information.  Basically, in order to develop MOSS artifacts such as:

  • ASP.NET Web Parts
  • MOSS library Event Handlers
  • Custom Workflows
  • ASP.NET User or custom controls to use in MOSS
  • STSADM extensions
  • Expression Builders
  • BDC related code
  • Feature or SharePoint solutions (WSP) event handlers

... a developer requires to have MOSS installed on the same machine as Visual Studio.  That will allow him to develop and debug his artifacts without impacting anyone else.  Note that, often case, in order for a developer to be effective, he will require the same page layouts, master pages, and other "graphical artifacts" to completely test his development.  The deployment process has to allow a developer to receive graphical updates without scraping his complete site collection.

 

IT_oriented_lifecycle_simple

 

Web integrators

For the web integrators doing graphical work and layouts, they do not require a separate development environment.  The best solution is for them to have SharePoint Designer (and their graphical design application) on their workstation and all of them should access the same MOSS server.  If working on the same Site Collection, SharePoint Designer may be challenging at times and requires some quirks (at least, pre-SP1).

 

Sometimes, for web integrators who already had their tools prior to developing for SharePoint, I found it easier for them to design their layouts outside first with the tools they know best.  Obviously, knowledge of ASP.NET, Master Pages, and SharePoint will help having an easier transition to SharePoint after.  Once their layout is as desired in their respective tools, they can then move that content in SharePoint Designer and apply the required updates to work in a SharePoint world.

 

The MOSS server they work on should also receive development updates when they are at least unit-tested.  This update should occurs outside of their working hours if possible and possibly be automated.

 

Staging

The Staging environment is where you want your customers to first see the end results of their web site.  This environment has to be stable and is probably not updated frequently.  This is also where you test your complete deployment process.  It's not unusual that non-SharePoint resources execute the deployment or it may even be managed by a 3rd party.  The process has to be as transparent and automated as possible.  This will allow to validate how the deployment to production will occur.

 

This environment is also often where you have all components similar to the production and hopefully a multi-server MOSS farm.  That allows other type of testing (i.e.: deploying WSP across multiple servers, deployment of MOSS hotfixes in a farm, etc.).  The network may also be more constrained/secured.

 

Production

This is the environment where everything has to run smoothly.  The deployment process should be the same as the Staging environment.  Authoring occurs here, either on the same site collection or a different one (the site collection could even be in a different farm if desired).

 

Summary

It's clear that, with that many environment, an automated solution to deploy to all of these environments has to be elaborated.  Unfortunately, the only one available with the product is not meant for these deployments.  SharePoint Features and SharePoint Solution projects are a big key to the solution but may be incomplete depending on your scenarios (as they have been in my experience).  I will cover Features, Solutions, and alternatives further.

 

The main observation on what I call IT focused lifecycle is that the "SharePoint Designer" role is not a heavy role and updates to page layouts are streamlined through tested IT deployments.  Designers aren't able to use Designer directly in Staging or Production.  Let's see the HTML oriented lifecycle.

 

 

HTML oriented lifecycle

What I call an HTML oriented lifecycle is where your web site requires frequent layout updates.  Your designers are likely to be close to the authors rather than the developers.  This can be typical for highly frequented Internet web sites.  Essentially, the same roles seen with the IT lifecycle has to be fulfilled, however, since developers are higher up in the chain, the "bubbling down" of their work may not be as obvious and less controlled.

 

HTML_oriented_lifecycle_simple

 

Developers

Developers essentially do the same thing as in the IT lifecycle.  Integrated testing is done similarly as before on a centralized server that receives more frequent updates from the designers.  The integrated testing environment acts as a staging for the development artifacts.

 

Web integrators

The main difference of this lifecycle reside here.  Web integrators are doing daily updates to layouts on an Authoring Site Collection.  They might even be "power authors" that also manage content.  They often work directly in the production environment albeit a different Site Collection that may or not be in a different MOSS farm than the Publishing Site Collection.

 

This allows them to easily deploy layout updates without requiring an IT deployment that is usually more complicated or constrained.

 

Staging

The real staging environment is really the Authoring Site Collection since this is the origin for all layouts.

 

Production

This environment will be the same as before with the exception that SharePoint Designer is allowed on the Authoring Site Collection.

 

Summary

As you can see, there are similarities in the 2 lifecycles, however, from an architecture perspective, there are differences when you want to extract designer updates.  For example, in the IT lifecycle, we can link the integrated environment with a Source Control (Team Foundation Server for example) and generate builds.  The same does not easily apply to the HTML lifecycle as your source control environment is unlikely to be in the same network environment and you will rather want to schedule extracts of the graphical artifacts, copy them to the development environment, and then create a build for developers.

 

Also, while I've seen both topologies in the field, one could debate why would designers need to update layouts so frequently.  In my opinion, if you design your layouts correctly and provide the correct level of customization, you should probably not need updates frequently and you might be able to work out with the IT oriented lifecycle.  I wanted to discuss about the 2 main scenarios I've experienced and provide solutions for both.

 

Now that we have seen the 2 lifecycles, I want to discuss Content deployment quickly to assert why it isn't a viable solution for deploying on all environments.

 

A primer on SharePoint's Content Deployment

Content Deployment is a SharePoint feature that allows a complete site collection to be deployed from a source to a destination.  These 2 environments can be in the same farm or a different farm.  This is done through the Operations tab of the Central Administration web site and requires simple configuration.  Basically, you create a path between 1 or 2 Central Administration servers (i.e.: 1 or 2 farms), and then you can create jobs to copy content on a scheduled or manual basis.

 

The primary requirement is that the target or destination has to be a site collection with the Blank Site template (of the same language of the source) and that all activated features to be installed but not activated at the destination.  When the first content deployment runs, it will copy all libraries and items to the destination and then activate the same features as the source. 

 

This is very important and to explain it easily, take a look at the Publishing feature.  If you activate it, it creates the out of the box master pages, page layouts, and styles for a Publishing Web Site.  If you activate it before doing the first content deployment, it will create a "Style Library" with a different GUID than the source.  If you were to use content deployment, it would fail as there would be an item with the same name at the source with a different GUID and it won't accept to deploy it.  This is the same if you create custom features.

 

So in essence, since Content Deployment is used to have 2 sites that will be exactly be the same, it's primary use is for having a "staging authoring environment" where you author and test everything.  Then Content Deployment is used to update the production environment according to a schedule or a manual request.  Note that there are 3 variants of Content Deployment:

  1. Through the Central Administration web site
  2. Through the STSADM -o import and STSADM -o export commands
  3. Through the APIs (Stephan Gossner wrote a great article about the APIs on his blog : https://blogs.technet.com/stefan_gossner/archive/2007/08/30/deep-dive-into-the-sharepoint-content-deployment-and-migration-api-part-1.aspx.

 

A deployment job consists of 3 primary steps :

  1. Export (at the source).  This operation may be computer intensive.
  2. Transport (between the 2).  This is simply the file copy, however, there's a related article that can affect performance and I had to apply the fix for a customer.  It made the transport much faster.
  3. Import (at the destination).  This operation is not so computer intensive, yet very long to process.  I have cases where it takes less than 5 minutes to export and transport, and then 1h30min to import.

 

The Central Administration web site's Content Deployment interface allows you to schedule deployment jobs easily.  Unfortunately, you cannot separate the 3 steps so it's good if you only need to deploy to one environment.

 

Unfortunately, this is usually not an acceptable scenario to update all environments such as Development or even Integrated Testing for the following reasons:

  • You do not want multiple exports;
  • The import process is long;
  • It's not meant for destinations where you are also authoring on it (i.e.: developers testing their new web parts);
  • It's not easy to "label" (as in Source Control labelling) a deployment.

 

Because of these concerns, other scenarios such as SharePoint Features, SharePoint Solution (WSP) projects, and the SharePoint APIs have to be explored and the following sections will elaborate on my findings for these technologies.

 

Note, Chris O'brien also have great posts on Content Deployment and Features.

 

 

Solutions for deploying SharePoint artifacts

Now that we have a better understanding of what we are up against, let's look at what solutions are available to us.  Here are the possible solutions :

  1. Content Deployment
  2. Backup/Restore
  3. Export/Import through SharePoint Designer (i.e.: Personal Web Package)
  4. Publishing through SharePoint Designer
  5. SharePoint Solution Generator
  6. SharePoint Features
  7. SharePoint Solution Packages (WSP)
  8. SharePoint APIs

 

There may be other possibilities that I'm unaware of but these are the ones I've tested with.  Note that this was done pre-SP1.  I'll work by elimination (and reduce the list dramatically :)):

  • Content Deployment : As stated previously, it's good only for equal environments so it's not practical for our scenarios;
  • Backup/Restore : essentially the same as Content Deployment albeit much faster but less user friendly, not "scheduled", but usually with less errors/issues;
  • Export/Import through SharePoint & Personal Web Package : My experience with it was ... very buggy.  I simply could not make it work correctly; for example, it created a 2nd Master Page Gallery (after renaming the original) at the destination when I tried it.  It didn't seem like a reliable solution, at least, before SP1;
  • Publishing through SharePoint Designer is similar to Content Deployment.  I'm not sure if it uses the same Web Services or not, but the end result is the same with a confusing user experience;
  • SharePoint Solution Generator : This tool is available through the WSS 3.0 Tools.  It's great to create site definitions, however, it doesn't work with the Publishing templates and isn't meant for WCM.

 

So we are left with 3 plausible solutions:

  1. SharePoint Features
  2. SharePoint Solution Packages (WSP)
  3. SharePoint APIs

 

Let me start by summarizing each of them:

 

A primer on SharePoint Features

SharePoint Features are basically what SharePoint uses to deploy functionality to a SharePoint site.  For example, the Publishing feature is activated on a Blank Site to create a Publishing Web Site.  This will add the Style Libraries and the Master Pages gallery along with modifying the look & feel of your web site.  For example, the Site Actions will contain different actions than for a Collaboration web site.

 

A SharePoint Feature consists of a manifest Xml file that is deployed in the 12 hive.  That manifest in turn specify what actions need to be done to the web site.  These actions can be from simply copying files (i.e.: page layouts, master pages, CSS, images, etc.) for creating the baseline of your web site, it can create site columns and content types, as well as even executing any .NET code you want through Event Handlers.  On paper, it's the solution and SharePoint itself uses it.

 

Unfortunately, creating features can be challenging as it requires knowing CAML.  CAML is the language definition for configuring any features.  While MSDN provides documentation for lots of the Xml elements available, MSDN is meant more as a reference, not for training.  Also, while I still create SharePoint Features today, I do not use is exclusively anymore as I found some bugs & quirks that I couldn't workaround well.  Particularly, using it to provision the baseline files.  For some other scenarios (Timer Jobs or Site Actions for example), Features are still the best solution.

 

A primer on SharePoint Solution Packages (WSP)

WSP files are really CAB files that SharePoint knows what to do with it.  The goal of SharePoint Solution Packages are to deploy a package once to the MOSS Farm and then it will deploy and execute what's required for each server of the farm.  It includes even more features such as an interface in the Central Administration where you can schedule when you want a solution to be deployed or even retracted!  And to sugarcoat it, if you add servers to a farm, it will apply the solution packages automatically.

 

** Update ** : A colleague pointed me to the fact that solution will even automatically deploy after a server crash. His scenario was that a solution was deployed in a farm and one of the servers had crashed. When they fixed the servers, they simply had to bring it back online and the solution was automatically deployed to that server without any manual operation (no wizard execution). ** /end Update**

 

Common scenarios include copying file to the GAC, modifying the web.config (using SPWebConfigModification class), copying resources files, and copying physical files to the web site.  SharePoint Solution Packages are a must to most SharePoint deployments.

 

A primer on SharePoint APIs

SharePoint APIs might seem to be a curious addition to this list as it's not really a deployment mechanism, or at least, not yet a user-friendly one.  However, while there may be bugs with SharePoint Features and Content Deployment in some cases, the SharePoint APIs is very robust and complete.  More so, strong developers can find more code samples as well as using tools such as Intellisense to guess possible solutions.

 

In your deployments, you are likely to have to use APIs for Event Handlers or web.config modifications, at the least, but my reason for mentioning it as a deployment alternative is that we ended up replacing some "Feature" scenarios that we had issue with for a complete SharePoint API solution.  Here's some operations that we automated, by creating custom STSADM extensions, for deploying complete Publishing web sites:

  • Create Site Columns including Lookup columns
  • Create and update Content Types
  • Create and update Master Pages, Page Layouts, CSS, Javascript, images, and other "baseline files" for a web site
  • Create Publishing Sites and Pages with content (great for creating content-less (or near content-less) landing pages as well.  We also create an input file to create complete files with content so that, at a click, we can create a fully working development environment with content)
  • Create SharePoint lists and referenced by Lookup columns
  • Update Available Page Layouts for a publishing site
  • Update Default page for a publishing site
  • Create SharePoint Permission set and groups
  • Update Versioning information for publishing site (i.e.: number of major & minor versions, activate/deactivate workflow requirement, etc.)
  • Update a site's master page
  • Update Publishing navigation settings (hiding sites/pages, inheriting from parent (or not) navigation, etc.)
  • and more!

 

Since lots of these cannot or should not be done by Features, we had to drill the APIs extensively.  When strong (showstopper) issues came up for some Features scenarios, we shifted our focus to the APIs.  Let me go into more details on the pros & cons of Features versus APIs.

 

 

Features versus SharePoint APIs

Features can be a great reusable way of deploying new functionality to a SharePoint site.  A SharePoint Feature is a set of functions, settings, and files to be added to a site (there are different levels of features but I'll keep it simple for now) when activated.  It's a great reusable mechanism for lots of scenarios such as adding a Custom Site Action, adding a Timer Job, and to some level, adding columns and content types.  You can take a look at this MSDN article for starting to work with Features : https://msdn2.microsoft.com/en-us/library/ms460318.aspx. 

 

On one of my projects following the IT Lifecycle, we tried to use Features as much as possible since it was, for us, how reusable deployment artifacts should have been.  Unfortunately, we ran into some severe issues when using them for Columns/Content Types and files such as Page Layouts (ASPX), Master Pages, CSS, JavaScripts, and images.  While we have successfully used features for other artifacts as mentioned in the previous paragraph, our feature that creates the "web site's baseline" with all the right master pages using custom CSS, images, and scripts;  is being phased out.  Here are the issues that we ran into:

 

  1. Sometimes, the BlobCache may corrupt some files that are deployed through a feature.  This requires some explanation as the first goal of the BlobCache is to physically cache a file on the Web Front-End instead of fetching it from the database all the time.  Since the feature already brings the file on the disk (in the 12 hive), BlobCache wouldn't be necessary.  Unfortunately, the BlobCache is also the mechanism by which the Cache-Control Http header is set and that header is the one that let a browser (or proxy) cache a file instead of making it expire right now.  We experienced 2 types of corruptions, one where the files were half-correct, and half-blacked-out; one where JS and CSS were simply cropped and were missing content.  Note that all files were correct in the feature but simply not in the BlobCache folder.  Luckily, we use Content Deployment from Authoring to Publishing and that makes it so that the files are now in the database and there is no corruption then.  While MOSS itself uses files in features (i.e.: _layouts or styles library), there is a special Virtual Directory for them and the BlobCache doesn't treat them the same way.
  2. If you want to export (i.e.: content deployment) a site collection (the production/publishing environment) and import them in a test environment, a feature update will not work since the Import process will put the files in the database (similar to unghosted files, but really, it's not reading the feature at that point).
  3. If you update a Content Type and add a new column to it, the column will not be propagated to all libraries (this is not related to the checkbox to force inherit settings on child content types).  Let me explain a bit more, when you create a Site Column or a Content Type at the site collection level, and then you create a page that uses that content type in a sub-site's Pages library, the content type and columns are copied to the library.  When you add a column to a content type through the GUI or the APIs, it will automatically add the column in all libraries where the content type is used.  Unfortunately, a Feature simply does not do it.

 

While number 1 was annoying for the authoring environment, and number 3 wasn't happening often enough;  number 2 is the real issue where we hadn't found a workaround yet.  Plus we knew that the APIs work great for all of them.  What we ended up creating was an STSADM extension that reads the same CAML files that this feature used and:

  1. Creates the site columns
  2. Create the content types
  3. Add page layouts and master pages
  4. Add files (CSS, GIF, JPG, PNG, JS) in any libraries

 

We still package a WSP that runs the commands, actually, we could still use a Feature that adds and removes the files when activated/deactivated (through code) instead of using the CAML definition of provisioned files.  In my experience with Publishing Web Site, it's much safer in the database than in physical files.  Also, learning how to create columns and content types in .NET, or to add files in the database, is very easy and you'll have access to Intellisense.  Learning CAML and try all the scenarios (updating an environment, updating an environment that uses Content Deployment, versioning of documents deployed through features, etc.) is likely to be more challenging and time consuming.

 

The next section is for detail's sake and because I have lots of questions on Virtual environments for developing in SharePoint.

 

 

A note on using Virtual Machines with SharePoint

While SharePoint can work correctly in a virtual environment, you have to be aware that the solution is not officially supported by Microsoft unless it's on Virtual Server.  What Microsoft Support will offer is what they call "Best effort support" where they'll help until they cannot reproduce the error and may ask to reproduce the error on a non-virtual environment.

 

Also, in my experience, I wouldn't recommend virtual environment for production web sites.

 

For development and up to staging however, I have seen both successful and very sluggish environments.  Personally, if going VM, I recommend the following parameters:

  • Have at least 2 GB of RAM
  • You can use a single SQL for all developers
  • SQL will offer better performance if not in a VM
  • SQL's disks should be physical if possible as I've seen saturated SAN scenarios for having a few MOSS developers

 

 

 

In a coming article, I'll detail how we used Team Foundation Server to synchronize SharePoint Designer users with the main source database which allowed us to easily create WSP packages.

 

Maxime