A Few Critical Success Factors for TFS 2008 to TFS 2010 Migrations

  • Article

With the considerable uptake of Team Foundation Server 2010, it's time to get some blogs out there that reflect what I have learned over the past few months performing installs and migrations for customers. In general, my first suggestion is that you follow the Team Foundation Installation Guide for Visual Studio 2010 fully and to the letter. One false step can make for a bad installation (which you may not find out until after the install), but fortunately TFS 2010 is far more forgiving than TFS 2008 when it comes to configuration and modifying an existing configuration. This post will specifically speak to migration upgrade scenarios, where you would basically take a copy of your TFS databases, restore them to a new machine or instance of SQL, and perform an upgrade with the TFS configuration wizard. Additionally, it is imperative that you leverage the TFS 2010 Supplemental Upgrade Guidance by the Visual Studio ALM Rangers, which will provide excellent planning advice and will also answer questions about an upgrade that didn't quite make it into the published upgrade procedures. This blog post will augment this with a few hard-learned lessons that will help make your TFS upgrade more successful.

Migration Upgrade Alternatives

There are two alternatives available in lieu of peforming a migration upgrade as follows:

1) You can perform a fresh install and leverage the TFS Integration Platform provided by the Visual Studio ALM Rangers on Codeplex.com, which can be found here. This tool is designed to enable TFS to integrate with other version control and work-item/bug tracking systems, but may also be used to migrate data from TFS 2008 to TFS 2010. You can find a number of scenarios here with pros and cons for various scenarios of consolidation, migration, or synchronization, each of which has a path where you might consider the TFS Integration Platform. There are however, some known limitations of the TFS Integration Platform as follows per the documentation:

  1. Only Version Control items, Work Items, and the links between them will be migrated. Other portions of Team Projects are not migrated by this tool, including Reports, Team Build data, and SharePoint content.
  2. Work Item IDs and Changeset numbers are not preserved during migration and new IDs are assigned sequentially as items are migrated. This implies that any references to these IDs will be invalid after migration; however, the links between work items and changesets will be migrated correctly, despite the ID changes.
  3. Timestamps on work item revisions and changesets will be updated to the time of migration. Any work item fields that store date time information will have their values correctly migrated, but the time of the revision will not be migrated. The most significant impact this has on a system is in reports that rely on a time dimension (i.e. Bug Trends, Code Churn). Because the migration typically occurs over a much shorter time span than the original operations, the time axis in such reports will effectively be compressed.

2) You can perform a fresh install and use the TFS import feature to migrate existing TFS 2008 projects, following the instructions on Will Lennon's blog post here. However, TFS Import is not a generally recommended course of action because it does not upgrade the SharePoint dashboard portals or Reporting data from TFS2008. After the import you must manually hook up your SharePoint sites and reports. Each time the import is run, it produces a new project collection, so we encourage you to perform a migration upgrade for each TFS 2008 server and then consolidate them in TFS 2010 by running collection detach and attach based on the "Move a Team Project Collection" procedure found here. If you are not currently using SharePoint portals and don't wish to leverage "legacy" TFS 2008 reports (which I discuss later), then this may be the best scenario for you.

So What Must Actually Be Backed Up to Be Migrated?

In your TFS 2008 installation, there are 11 total databases plus the warehouse cube housed in Analysis Services. Here is a list of the tables slightly modified from the TFS installation guidance that must be backed up and restored for a successful TFS 2010 migration upgrade:

Table

Notes

ReportServer

 If you did not configure Team Foundation Server with Reporting Services, you will not have the ReportServer databases.

ReportServerTempDB

 If you did not configure Team Foundation Server with Reporting Services, you will not have the ReportServerTempDB database.

STS_Content_TFS or WSS_Content

The SharePoint content database. The name may vary depending on your particular installation.

TfsBuild

TfsIntegration

TfsVersionControl

TfsWorkItemTracking

TfsWorkItemTrackingAttachments

TfsActivityLogging

Even though backup is optional for this database, it still must be present during a TFS 2010 upgrade, even though TFS 2010 doesn't actually use it.

Notice that the TFSWarehouse database is not included in the list. This database is ignored when upgrading to TFS 2010. If you do restore it, it will still be there after upgrading to TFS2010 and you can delete it then. TFS 2010 will create a new dabase named Tfs_Warehouse in its place. You also don't need to restore STS_Config_TFS/WSS_Config, which are the likely names for the SharePoint configuration database. The new configuration database iscreated on the new server when you configure SharePoint. Once configured, you can then restore the WSS_Content DB which is the only database from the old server that TFS 2010 cares about. That is, presuming that you are using the same version of SharePoint, which is more than likely Windows SharePoint Services v3. If you are planning on migrating your portal to SharePoint 2010, please see the SharePoint Portal section. (Note: Special thanks goes out to Will Lennon on the TFS team for clarifying some of the nuances in this section)

 The next section will discuss the appropriate and recommended backup strategy if you are not already pursuing this, and is very important to a successful migration upgrade.

Migration Backup Strategy

For more than one reason, I implore you to not neglect the following guidance regarding backup of your TFS databases from the TFS installation guidance .chm file:

To successfully back up Team Foundation Server, you must not only back up all databases that are used by the deployment, you must also synchronize the backups to the same point in time. The easiest way to ensure this is by using marked transactions. Routinely marking related transactions in every Team Foundation database establishes a series of common recovery points in the databases.

If your databases are not synchronized, which can happen even if you are taking scheduled backups late at night only a few minutes apart, you could have trouble with your upgrade installation. Specifically, you are likely to get a "TF24016: Cannot find team project" error when you create your first team project after what appears to be a successful install. Technically, there are several possible causes of this condition, including restoration of backups from the original TFS 2008 databases that were not taken as part of the same file group at the same time; or cloning of the TFS 2008 server where the instanceID was not updated, causing cross-talk between the original TFS database and the clone. The following blog posting discusses the possible implications of incorrectly cloning a TFS server instance: https://blogs.msdn.com/b/buckh/archive/2006/10/17/creating-a-new-server-from-an-old-one-beware-of-the-instanceid.aspx.

There may also be some secondary issues as well. So bottom line, don't make this mistake, and be certain to take your backup strategy seriously! The good news is that I have, in the field, run into this scenario, and have communicated the need for a verification check to the TFS product team. They have acknowledged the need for more verification checks prior to allowing an upgrade and will be addressing this in a future service pack or release of TFS 2010.

The Short Route: No Marked Transactions (Use This For Migration Only, and Not For Subsequent DR)

With respect to doing a migration, I have some guidance that may ease a bit of the pain if you're not using marked transactions, as marked transactions may be a bit of overkill for a migration scenario and may introduce its own set of issues if you do not have experience with them (Microsoft is working on a power tool to simplify this process, which is great news). The simple route is to take IIS and the TFS services (scheduler, build) offline before doing the backups, which will ensure the databases are in sync. This is the best current guidance, but I will provide an update if I find out more. This does not apply to your backup/recovery strategy moving forward once your migration is complete, as if there is any corruption in your database or your database crashes, you will need the synchronized backups to bring yourself to a known, integral state. So follow the instructions below for using marked transactions once your migration is complete, or feel free to go ahead and do so now.

Using Marked Transactions (The Long Route During Migration, But Still the Proper Route for DR After Migration)

If you decide to go the long route, after a bit of digging I found the appropriate link for setting up Marked Transactions in Full Recovery Mode for SQL Server 2005 (typically, folks on SQL Server 2005 will be upgrading their database to SQL Server 2008, so you will want to perform this before the upgrade so that your data is in its proper state prior to the upgrade). The interesting thing is the TFS 2010 documentation discusses backing up Team Foundation Server 2008 with the presumption that your database is SQL Server 2008, but for backing up Team Foundation Server 2005 it presumes you're using SQL Server 2005, and doesn't mention a thing about marked transactions. Specifically, the following link explains the steps for setting up marked transactions within the context of TFS: https://msdn.microsoft.com/en-us/library/ms253070.aspx. Below are general links for setting up marked transactions in varying versions of SQL Server.

The actual step-by-step instructions for marking transactions is essentially the same across all versions, so following the link in the checklist page of the TFS 2010 documentation titled How to: Back Up a Team Foundation Server (Visual Studio Team System 2008 Team Foundation Server) should be all you need, with database table names being the only real difference.

Reporting 

Another thing to be aware of on a migration upgrade is the status of your reports post-upgrade, and the post-upgrade steps you must follow for your reports to work properly. The Ranger supplemental guidance addresses making the TFS 2008 reports work against the TFS 2010 installation through two pointers to blog posts by John Socha Leialoha, the first of which you can find here. It is a bit dated, but has some great info. However, you may think this scenario applies to you when it actually may not. The TFS 2010 migration does move the data of TFS 2008 to to a new warehouse in TFS 2010, while retaining the data in the TFS 2008 warehouse so you can pull the old reports. I question the value of pulling old reports since updates to work items moving forward will be reflected in the new warehouse, and not the old. Secondly, the only reason I can think of that you would want to modify the "legacy" reports to work with TFS 2010 would be if you have customized the reports. So if you just want the out of box reports, then you can skip futzing around with the migrated reports and just delete the team project folders generated during the migration. Instead, you can follow this link here, which provides instructions for generating the updated TFS 2010 folder hierarchy and reports required by your newly migrated project through a script. Additionally, you can provision a project portal site with dashboards and Excel reports for your upgraded team projects if you didn't migrate SharePoint, but there is a problem with provisioning portals thatI I will describe in the SharePoint Portal section below.

Warehouse Notes (with special thanks to Doug Mortensen)

I have often been asked how quickly updated work items are shuttled off to the data warehouse. The job service regularly schedules a series of jobs to copy data from each Team Project Collection into the warehouse. The out of box default is two minutes, but is configurable. Hence when a work item is updated, the change should appear in the relational warehouse quite quickly. The job service schedule for cube updates is defaulted to every two hours, and is configurable. If you want to force a manual cube refresh in order to test the shuttling of work items, then see the following link. Out of box reports by default use the Reporting Services caching feature so once executed the report results are cached for 30 minutes.  Hence this can add another layer of latency in some cases where even though the warehouse/cube has newer data, the report will appear to show older data – because the report you see is coming from the report cache.  Report caching is configurable on a per report basis via standard RS web site.

Another question I get is with what reports come from the relational store versus reports that come from the cube. The answer is some out of box reports pull their data exclusively from the relational store (e.g. build success over time, status on all iteration, stories overview), but others mostly pull from the cube (bug trends, bug status, build quality indicators, test plan progress etc).  You can tell for sure where a report pulls its data by looking at the Reporting Services web page for the report on the data sources tab. If it only has one data source and that points at the relational warehouse, then you know 100% of the data comes from that source.  However if you see two data sources (one for relational and one for the OLAP cube) then it is relatively likely that most of the data comes from the cube and only some config/latency type data is being pulled from the relational warehouse.

SharePoint Portal

The TFS Installation .chm file states the following about preparing the portal server:  "You cannot install Windows SharePoint Services 3.0 when you upgrade Team Foundation Server. You must use your existing portal or point to an existing SharePoint Products site that meets requirements." Thus, you can leverage your existing SharePoint v3 install by migrating it to new hardware (so we won't bother the old instance in case you need to go back to it), or you can generate a new portal leveraging SharePoint Foundation 2010 if you wish (which is fully supported by TFS 2010, though it won't install it). If you have made extensive use in TFS 2008 of the portal for project documents and have links to documents from your work items, then I would strongly suggest migrating the portal for the purpose of continuity on migrated projects, and immediately upgrading to SharePoint Foundation 2010 before running the upgrade wizard.

Provisioning Team Project Portals If You Didn't Migrate SharePoint (Out of Box Templates Only)

In the reporting section above, I shared a link that allows you to provision project dashboards (i.e., project portals) and reports, but there may be a problem in this approach to generating a project portal in TFS 2010 for a team project migrated from TFS 2008. You may also run into the same problem with the approach from the article titled Adding Dashboards and Reports To Upgraded Team Projects, which uses a batch file to accomplish the same thing. The process templates are wholly different in TFS 2010, which should be expected since TFS 2010 has been completely revamped to provide for hierarchical work items. Thus, there is no migration path, per se, for the process templates, even though existing migrated process templates will work. You will get the following exception produced in the log if you try to generate a project portal using the 5.0 process templates:

---begin Exception entry---
Time: 2010-07-01T10:39:30
Module: Engine
Event Description: TF30162: Task "SharePointPortal" from Group "Portal" failed
Exception Type: Microsoft.TeamFoundation.Client.PcwException
Exception Message: TF255357: The following query was not found: Product Planning. Download the process template, open it, and verify that the query exists.
Stack Trace:
   at Microsoft.VisualStudio.TeamFoundation.WssSiteCreator.SetWorkItemQuery(WssCreationContextWrapper contextWrapper, String filePath, String serverInstanceId, String queryName)
   at Microsoft.VisualStudio.TeamFoundation.WssSiteCreator.UploadFile(WssCreationContextWrapper contextWrapper, String sourceFile, String siteUrl, String target, DocumentLibraryInfo docLibInfo, String currituckQuery)
   at Microsoft.VisualStudio.TeamFoundation.WssSiteCreator.HandleFileCreation(WssCreationContextWrapper contextWrapper, XmlNode taskNode)
   at Microsoft.VisualStudio.TeamFoundation.WssSiteCreator.Execute(ProjectCreationContext context, XmlNode taskXml)
   at Microsoft.VisualStudio.TeamFoundation.ProjectCreationEngine.TaskExecutor.PerformTask(IProjectComponentCreator componentCreator, ProjectCreationContext context, XmlNode taskXml)
   at Microsoft.VisualStudio.TeamFoundation.ProjectCreationEngine.RunTask(Object taskObj)
--- end Exception entry ---

The problem above is there is no "Product Planning" query in the migrated template (as well as others), so the generation of the portal may appear to succeed, but the portal will not have the expected document folders and Process Guidance. The reason for this is that the work item types for the 5.0 process templates don't match up with the 4.2 process templates. If you follow some of the guidance out there that explains how to add a dashboard portal "after the fact," I haven't found any that will work as there is too much of an impedance mismatch to make this work properly. My best advice to you is to simply migrate your WSS v3 portal from TFS08. If you don't, then you can follow the procedure above, but you will just have to accept that your portal won't be perfect, and you will have to create the shared document folders yourself. I have alerted the product group and they will be working to resolve this specific migratio scenario where you didn't migrate WSS v3 from TFS08. I wish the story was better, but unfortunately it isn't for now.

Conclusion

I know that between the TFS installation documentation, the supplemental guidance, and now my notes, it may seem overwhelming to perform a migration upgrade. It is not as hard as it may seem, but my strong suggestion is to conduct the appropriate upgrade planning by carefully perusing all of the documentation, following the documentation to the letter, and performing a "dry run" migration in a test environment before performing the production migration. When you do perform the production migration, I would suggest doing so when it won't interrupt productive developer work (such as over a weekend). This post is not exhaustive in terms of the various issues you may encounter and I will update as appropriate. Regardless, I will continue to provide posts that I hope will be of value to those performing fresh installations of TFS 2010 as well as migrations from TFS 2008 to TFS 2010.