Regular Expressions Support in SharePoint 2010 Crawling

Search admins often need to omit from a crawl files that match a certain pattern. E.g.:

· In a bank, file names starting with SSN

· In a business site, files names with credit card number

· URLs having specific value of a certain parameter of an aspx file

· etc..

The usual solution is to allow admins to create “crawl rules” that restrict crawlers from following specific links. The most basic crawl rule specifies a complete URL for the file to be crawled, which requires the admin to create as many rules as there are files in their repository. A more practical solution often implemented involves the use of the wildcard character: “*”. This character matches everything, so admins can create a rule using the wildcard to include (or omit) all files under a particular folder or path:

\\myfileserver\myclientsfolder\*

This works if all the files are located neatly in one folder, but what if they are spread across the repository (or Web site)? This is the problem that is solved by using regular expression (RegEx) syntax.

The SharePoint Solution

In SharePoint 2007, the wild card operator “*” is the only operator supported in crawl rules for matching characters. As mentioned, it is a brute force operator that matches everything. Wildcard-only rules do not provide the admin the flexibility to, for example, recognize and omit URLs that contain Social Security Numbers, or that have an aspx parameter with a specific value.

SharePoint 2010 includes some new capabilities in this area. The default behavior of crawl rules in SharePoint 2010 is the same as it was in SharePoint 2007, but with SharePoint Search 2010, administrators can create crawl rules to include or exclude URLs that match regular expressions. To enable regular expressions, the admin need only select the check box on the Crawl Rules creation UI as shown in the image below.

Regular Expression Operators

The table below lists and describes the regular expression operators that are supported for crawl rules in SharePoint 2010:

Using RegEx Operators in Crawl Rules

Once you understand the RegEx operators above and how to enable them in the crawler, there are only a couple other things you need to keep in mind:

Protocol part

Regular expression operators cannot be used in the protocol part of the URL. This means, for example, the following RegEx rule cannot be created:

.*//www.microsoft.com/.*

If you try to create a rule like this, the system will add https:// in the beginning and thus make “.*” as the second part of the URL. The resulting rule in this case will be:

https:// .*//www.microsoft.com/.*

which may not be what you intended.

Case sensitive comparison

RegEx rules are case insensitive by default. In order to allow a rule to do case sensitive matching of a URL, the administrator should select the “Match case” check box in the rule creation UI as shown below:

If the “Match case” checkbox is selected, the crawler will preserve the case of matching URLs during the crawl. In the example above, the rule will match: https://test/AbC123.html and WILL NOT match to https://test/Abc123.html.

This feature comes in handy when SharePoint is used to crawl web sites hosted on Unix based web servers, which are case sensitive.

Examples

Here are some interesting examples demonstrating the usefulness of Regular Expression in crawl rules:

 Syed Anas Hashmi | SDET | Microsoft Enterprise Search Group