Migrating documents into SharePoint - tools and tips to improve performance

  • Article

An important part of any large SharePoint deployment project is the migration of content across from other data stores. Recently I've seen a few customers experience problems or poor performance when migrating large numbers of documents into SharePoint (note that I'm referring to SPS 2003 and WSS). Typically the upload starts transferring well, but the connection speed gradually drops and the connection eventually fails.

 

There are a number of steps that you can take to improve document upload performance. A great resource to help prepare for a migration is the Migrating and Upgrading Web Sites section of the Windows SharePoint Services Administrator's Guide. Here's a quick overview of the steps most likely to help:

  • Check whether a quota is defined on the site collection, and if the quota has been exceeded. If so, increase the quota limit.
  • Check the maximum file upload size limit. The default maximum file size for uploads to SharePoint is set at 50MB. Try increasing this value if you are uploading large files.
  • Check that you are not attempting to upload file types which are listed on the blocked file extensions list (such as .mdb, .exe). If this is required, you can remove the file types from the blocked extensions list to enable an upload.
  • Increase the paging file size for the server to at least 1 gigabyte (GB).
  • Increase the IIS timeout settings to 65,000

Note that the amount of memory available is critical; all uploads are buffered in memory, and sometimes twice. Therefore, you need to have enough memory overhead available on each server in the chain to allocate 2x the upload size and not peg the available RAM.

 

 

 

Upload tools

 

Out of the box Windows SharePoint Services has a “multiple file upload” dialog and an “explorer view” available through the browser, which both allow file transfer from NTFS file shares. There is a default maximum file size allowed for this type of transfer of 50MB per file, but this is a configurable setting at the virtual server level (see above). This article explains more.

 

The SharePoint Document Library Migration Tools (AKA “Spin” and “Spout”) handle migrating documents from SPS2001 or SPS2003 “backwards compatible” document libraries to SPS2003 libraries. (Note that these tools do *not* handle migrating from NTFS file shares).

 

Also see: https://workspaces.gotdotnet.com/spimport for some recently released upload tools announced at Tech Ed 2004:

  • Windows File System Extractor
  • Lotus Quickplace Extractor
  • Exchange Public Folder Extractor
  • STS to WSS migration tools
  • SharePoint Products and Technologies Custom Importer (can choose to create either WSS sites or portal areas for content imported)
  • Toolkit to enable developers to write their own custom extractor utilities

Note that these tools are not supported by Microsoft, but have a “gotdotnet community” supporting them; so over time any problems should be resolved. Judging by the demonstrations given at Tech Ed 2004, they seem pretty robust already.

 

 

Third-party tools  are also available.