Records Management Feature — Declaring Records in SharePoint Sites



Finally! After much patience on the part of this community, we can now talk about the next big area of our records management capabilities in the 2007 release – record declaration, classification, and the “Records Center” site.

But before moving forward, it’s worth spending a few minutes recapping what we’ve talked about so far, and how it fits into the “big picture” that we introduced in some of our earliest posts.

A quick recap


In our posts about “RM in the Information Age”, we proposed a framework that categorized DM/RM systems into “collaborative spaces” and “records spaces” as being appropriate for many organizations.

The first set of features we introduced, Content Types and the Document Information Panel, are especially relevant when dealing with “collaborative spaces” –they help ensure that employees will be classifying content and filling in appropriate metadata for it from the earliest stages of its lifecycle – even before that information becomes a record.

Information Management Policies also play a significant role in collaboration spaces, since they give records managers and compliance officers a way to introduce appropriate policies and controls into active content without adding additional burden on employees – such as expiring unimportant content to reduce unnecessary retention and auditing user activities on content where appropriate. (Of course, these capabilities are also vital for enforcing disposition schedules in a records space, but more on that in the next few posts.)

Now that we’ve laid out that toolkit for records managers trying to cope with collaborative spaces, let’s talk about what happens when content in those spaces reach a point where it needs to be declared as a record.

When does content become a record?


Before discussing any technology, the first question for record declaration is “when does content become a record”? For some organizations and types of information, content must be considered a record from the moment of its creation. (Think of a bill of materials created for a customer delivery, for example.)

However many types of information should not be considered records until they reach a certain lifecycle stage. For example, think of a professional services firm publishing a research report: early drafts of the reports may not need to be retained as records, but the final version delivered to its client is “evidence of a business transaction” and must be preserved accordingly.

Ensuring that records falling into the latter category are retained and declared is especially challenging – given that the employees who are collaborating on that content and performing the business transactions aren’t always highly motivated to actively retain those records.

Fortunately, the 2007 release of the Office system provides the ability to reduce the employee burden on record declaration and classification to a minimum. Content types provide the way for content to be classified, and workflows provide the way to automate business processes like content publishing and finalization… now let’s talk about SharePoint’s record declaration features.

Connecting collaborative sites to a “Records Center”


A key component of our original framework for “collaboration” and “records” spaces is a mechanism by which records can flow from collaboration spaces to a records space.

To address this need, we’ve defined an “open” set of interfaces for how collaborative sites can communicate with records sites, and enabled all of our products in the Office 2007 system to use those interfaces. (For the more technology inclined: the interfaces are defined using Web Services and e-mail protocols. And when we say “open” we mean that any Records Management application, not only the 2007 release of Office SharePoint Server, can be enabled to work with the declaration capabilities discussed below.)

So, a Windows SharePoint Services collaboration site can be connected to a Records Center site for use as its “records space”. (And when we start talking about e-mail records management, we’ll see that Exchange 2007 can also use a Records Center site for that purpose.)

The key concept of this interface, which you’ll learn more about in our next posting where we explore the inner working of a Records Center site, is that it provides a very simple way for collaborative spaces to interface with records spaces, without the collaborative space needing to know the inner workings (or complete file plan) of the records space. This is the main reason why record declaration from Windows SharePoint Services or Office SharePoint Server 2007 is a minimal end-user burden.

What happens when a record is declared from a SharePoint site?


Now that we’ve talked through the concepts, let’s look at what it means for a record to be declared from a SharePoint site.

Record declaration in SharePoint can be part of an automated process (for example, a workflow) or a manually initiated process via a “Send To” command (see image below).

Menu for a document in a SharePoint team site, showing the "Send to" command for the Records Center

Caption: Menu for a document in a SharePoint team site, showing the “Send to” command for the Records Center. (Note: The name displayed for the Records Center is customizable. The term “Northwind Records Center” is just an example.)

Whether it occurs as part of an automated process or manually, when a record is declared, the latest version of the record, all of its metadata, its audit history, and its content type information are sent to the Records Center site to which the SharePoint site is connected. The user doesn’t need to specify which Records Center site to use, nor do they need to specify a record series at declaration time – this information is inferred from the content type by the Records Center. (We’ll explore exactly how this inference is made in our next post.)

Once the Records Center has determined the appropriate record series, it will either accept the submission as is or, if additional metadata is required for the record in the Records Center beyond what was provided while the record was “active,” it can prompt the user to provide that information.

But in either case, the burden on the user is minimal. In fact, if the record is declared automatically via a workflow and no additional metadata for the record is required, then no end-user interaction is needed at all.

What happens to content in a SharePoint site after it is declared as a record?


The last question to address in this post is what happens after active content in a SharePoint site is declared as a record. For physical records, once a record is declared (i.e. sent to a records storage location) then the original owners of the record no longer have easy access to it –the physical object is out of their hands.

However, with digital records that need not be the case: once the active content has been declared as a record, an “official” record copy can be retained in the Records Center site, but content owners can still have a reference/working copy in their collaborative spaces (and thus have the same level of access to the content even after it has been declared for an appropriate period of time).

This is the approach we’ve enabled in SharePoint 2007. When a record is declared (and a copy sent to the Records Center site), we leave a working copy of the record in the SharePoint collaboration site, which can be expired appropriately using expiration policies. Workflows or other automated solutions can certainly choose not to leave a working copy in the original location, but by default the working copy is left in place – to minimize the end-user disruption of working with content that has been declared.

Hopefully this post starts to answer some of the big questions that we haven’t addressed thus far though it probably raises other questions. In our next post, we’ll explore in more depth the “Records Center” site provided by Office SharePoint Server 2007, including how it stores records declared from collaborative spaces like SharePoint sites and Exchange 2007 e-mail servers.

Thanks for reading!

– Ethan Gur-esh, Program Manager.


Comments (22)

  1. IAT313 says:

    Very interesting; I have just finished catching up with the BLOG and I am looking forward to future updates. On this particular feature, declaring something a record and then sending it to a "Records Center" I find to be a potential point of massive failure in the systematic management of an organizations information management. If this server (record center) is to be the point of discovery, how do we ensure it contains the information that it should? In other words, what control points do we have in place to ensure the record gets filed? Is this to be part of the auditing system? Or are we depending on training that will inform the user of their responsibilities to file the documents? Will the system potentially allow a record to be created in a workspace, only to be deleted and never sent to the records center? Given this, how would an organization fair in court with the failure to produce information but can’t because of the same?

    Thanks for the opportunity to contribute,

    Cliff

  2. sam_n99 says:

    nice, but if I want to use sharepoint as end to end solution, so, how can I integrate the document management and the record management?, whats the benefits of document center in portal? in fact, I can’t understand how MOSS 2007 will be help me to manging my documents life cycle?

    any suggestions?

  3. munzerts says:

    After a document has been declared a record and moved to the Records Center site, is the record status of the remaining default "working copy" in the Collaborative Space clearly indicated?

    Thanks,

    Tom in DC

  4. Russell says:

    Sorry but not so sure that I see this is a great step forward for records management.  Are you saying that SPS 2007 is actually supporting multiple information silos and that only when a user actively contributes/declares a business object to a ‘Records Center’ that it then becomes governed by records management policies etc.?  

    If I have understood this correctly then there appears to be a fatal flaw in this approach as it is actively undermining a single records repository model.  

    Collaboration is very powerful functionality but basing it around fragmentary storage in information silos is going to be very risky for many organisations.

    From a Records Continuum philosphical approach I’d argue that all business documents should be treated as records from the moment of their creation until disposal.  The premise you are using seems to be defining ‘records’ solely as the final iterations of siginificant documents.

    This overlooks the requirements in much of the world to support transparency of processes, authenticity, accountability, development of policies etc.

    Don’t get me wrong, SPS 2007 looks like a big step forward for SharePoint but this feels like Records Management Lite so far.

    Cheers

  5. @ IAT313:

    Cliff, you’re absolutely right that ensuring that the right set of content in any collaborative space is declared as a record is a key challenges for electronic records management (and even for physical records management). Tools like search can facilitate discovery outside the records center, but the "right" way to achieve this is by using several different control points available in the 2007 release of the Office System.

    The first control point (and likely the optimal one for many organizations) will be the new workflow capabilities – by having record declaration as an automatic aspect of workflows covering business processes like sign-off & finalization, records will be declared without end-users specifically having to think about records management. (Which in an ideal scenario would be how the overwhelming majority of records get declared —  since as you imply, relying on trained/informed users to voluntary declare records can lead to insufficient retention.)

    And auditing is also an important component of ensuring records are being declared/filed appropriately, as the audit trail can be used by the organization to demonstrate (to any internal or external inquiry) that the right workflows were followed, the right content was declared, that content was deleted only in accordance with its retention schedule, etc.

    @sam_n99:

    Perhaps the use of the word “SharePoint” in the title of this post was a bit unclear. When thinking of MOSS 2007 as an overall “Enterprise Content Management” product, there are several different aspects of the product that interrelate. Features like Content Types, workflows, and information management policies are all valuable for managing documents across their entire lifecycles – whether you think of those as managing documents or records.

    The “Document Center” in particular is a new type of site template in MOSS 2007 that’s optimized for team’s that want to have a site specifically focused on storing & sharing documents (as opposed to, for example, having a team calendar). But the features mentioned above certainly aren’t limited only to sites using the “Document Center” template.

    For more information about the “Document Center” site template, check out Microsoft’s ECM team blog @ http://blogs.msdn.com/ecm/ .

    @ munzertz:

    By default when records are declared we don’t modify the working copy in any way to indicate that it’s been declared as a record. (The reason being that not changing the appearance of the working copy in the collaborative space provides the minimal level of end-user disruption.)

    However, workflows that include a record declaration step could certainly update the working copy to indicate that it has been declared as a record, in whatever way is appropriate for your organization’s needs (such as preventing users from modifying the working copy, modifying its metadata, etc.)

    Hope this helps,

    – Ethan Gur-esh, Program Manager.

  6. @ Russell:

    Firstly, thanks for the honest feedback on the features we’ve blogged about so far.

    You raised the question of how collaborative repositories are linked to records centers – and whether there should be a single repository for all content, since “documents should be treated as records from the moment of their creation until their disposal”.

    I agree that having a single repository where all users work on all content from creation to disposition would in principle be the easiest for a records manager to administer. The problem is that those types of systems aren’t workable for most organizations in practice, as we mentioned in some of our early posts – the requirements end-users have for edit-ability of work-in-progress content, flexibility in how they organize their workspaces, etc. don’t line up well with how records managers need to work with their RM systems. (This is true for both documents and for e-mail systems, which we’ll be talking about soon on this blog). What we’ve seen at many organizations who try to take that approach is that users start circumventing the system entirely, for example by using their e-mail inboxes or file shares to store important content.

    So a lot of the functionality we’ve talked about so far has been about the tools we’ve provided to help Records Managers get an appropriate level of control over collaborative spaces, so that they can be managed via information management policies in coordination with their Records Management applications. This does introduce an additional burden on records managers, but I’d also claim that it greatly increases the likelihood of success of their records programs.

    That said there are still a bunch of features we’ve yet to blog about that enable better transparency & accountability and which may address some of your concerns. These include reporting & monitoring features to ensure that records (and non-records) are being managed appropriately via policies, and our Records Center site – which conceptually maps to the traditional definition of a “Records Management application”, and we’ll be examining in depth in the next few posts.

    The model you describe (where there’s a single repository and all document should be treated as records) is certainly appropriate for some organizations, esp. those that in highly regulated industries or jurisdictions. In our next posts, we’ll also cover how such an organization could use only the Records Center to manage items in the way that you’re suggesting.

    Thanks,

    – Ethan Gur-esh, Program Manager.

  7. As I “teased” in my last post , this week at the ARMA Conference Microsoft and Iron Mountain unveiled

  8. Garethm says:

    Hi, i have been working with leading ECM vendor for many years where we had integrated document and records repository. With correct user UI configuration you could control the amount of input users actually had to provide to the RM side of life. The advantage here is that the users only see a single repository and are fully unaware of any records features(apart from locked records)that dont affect them.

    I am concerned with your policy here of separate areas for record and non record content, ecm frameworks should provide contextural views on content including records. By using a ‘copy to’ paradigm it will increase the risk of uncontroled non locked copies of what is essentially a declared item to flow throughout and organisation and in due coarse generate confusion. If the content is still active in a collaborative sense for consumption after being declared surely it is better for a pointer to be made in the originating items place refering to the declared item in the records centre. This in turn removes any ambiguity about the object and prevents the duplication of the disk space. Anyway theres my two pence worth, will be watching with keen intrest on the development of this area.

  9. mkelly says:

    When a document is sent to the Records Center, why is it assigned a new version? We would rather see it keep the version assigned in it’s original document library.

  10. @ MKelly:

    (Firstly, sorry for the delayed response to your comment.)

    When a document is sent to the Records Center, we keep its version number as assigned in its original document library as metadata that is submitted along with the document to the Records Center (in the “Properties” sub-folder of the document library in the Records Center).

    Automatically synchronizing that version number directly into the versioning system in the Records Center is something we should probably consider for a future version, but even in the current version we definitely don’t lose that information.

    Hope this helps clarify,

    – Ethan Gur-esh, Program Manager.  

  11. @ GarethM:

    Firstly, thanks for the feedback! Part of the reason for this blog was to engage in exactly this kind of dialog.

    You’re absolutely right that the framework described here isn’t necessarily the right approach for all users/organizations.  (We actually said so ourselves in a few posts — see http://blogs.msdn.com/recman/archive/2006/05/01/587675.aspx   and http://blogs.msdn.com/recman/archive/2006/09/19/762561.aspx ).

    As you mention, having separate areas for records & non-records does introduce some temporary duplication of content (which does have an impact on storage usage.) But that limitation is part of a trade-off, which brings with it the benefit of greater usability for end-users, and simpler management burden for both IT (which has less “records” areas which need a higher quality of service) and for Records Managers (who can have more autonomy implementing their RM programs without adding new restrictions to end-users).

    Different organizations will want to make that trade-off differently: some may want to have only a single repository that all users work against (as we mentioned in our “Records Center Q&A” post). Others will choose the model that we’ve laid out here.

    But please do keep the conversation going! It’s great for us to hear (and helps us think about possible future directions for the product as well. 🙂 )

    Thanks,

    – Ethan Gur-esh, Program Manager.

  12. mic.dan says:

    Greetings for the great blog!

    I’m building a DRM solution for a giant "heavy regulated" organisation which darefully decided not to "sink" into a gigantic ECM solution but to exploit the future abilities of the MOSS.

    As blogged here, there are several disadvantages in MOSS-DRM solution, some of them critical to this kind of organisations – I’ve checked a complementary product named "Meridio" which seems to fit in as a practical MOSS extension to DRM.

    It solves the problems of unique repository, classifications (besides the usual permissions), Fileplan "tree" etc.etc. and all of that built-in through MOSS UI and menus.

    What’s your opinion?

  13. budpr says:

    How will RM handle vital records? I can’t seem to find anything on this. Or, will the individual company be responsible for setting up a vital records category and workflow?

  14. mic.dan –

    You are absolutely correct that our partner ecosystem is really one of the strengths of the MOSS 2007 release.  Meridio is a great example of how partners can add great value to MOSS 2007.

    In the future, we plan to feature partner solutions on this blog – so be sure to check back in the coming months for a deeper understanding of what’s out there.

    budpr –

    Most of the pieces are already in place for a customer to implement a system to review records on a periodic basis.  I’d definitely suggest our workflow foundation for designing such a process.

    However, Vital Record Review in particular will be one of the features that we add in the Records Center Add-on Pack. (http://blogs.msdn.com/recman/archive/2006/11/08/dod-5015-2-certification-for-office-sharepoint-server-2007.aspx).  This will provide a workflow based solution for periodically reviewing records as well as auditing and history logging to evaluate how well the review process is going.

  15. mic.dan says:

    Thanks! I’m more relaxed now…

    Meanwhile I’v noticed that Meridio released a new version of its product (Meridio 5.0) which claims to have important "leverages" above the MOSS 2007 togeather with full integration to the MOSS – Let’s wait and see:

    http://www.meridio.com/news/press_releases/?id=99

  16. budpr says:

    Can you tell me whether the "copy" copied to the Records Center is considered a "copy" or an "original". Is there any way to make the copy of the document in the Records Center the original?

  17. Document and Records Management Definition Document Management According to Wikipedia : "A document

  18. steveglo says:

    Is there a feature in SharePoint 2007 for calculating when a physical record that has been circulated to a user has been kept by the user too long (is "overdue")? Also, is there a tickler capability for pending requests for physical records that are needed by one person but have already been checked out to another person?

  19. Audio says:

    Finally! After much patience on the part of this community, we can now talk about the next big area of our records management capabilities in the 2007 release – record declaration, classification, and the “Records Center” site. But before moving forward

  20. Arno Nel 2.0 says:

    Document and Records Management Definition Document Management According to Wikipedia : "A document