SharePoint 2013: How to Rebalance Links across Crawl Stores ?


 

Issue

When using Central Administration to view content sources for the Search Service Application the content source history and content source pages takes a very long time to load. Crawls might hang with WCF related messages in the ULS logs or will try to run Rebalance Crawl store partition job.

Similar issue has been claimed to have been resolved in April 2011 CU for SharePoint 2010 via this Knowledge Base article

Symptoms:

As the content in the content source increases, the time it takes for the content sources page will increase. This is a good litmus test.

We can also query the Crawl Store database against the MSSCRAWLURL for the number of links against the content sources and Host ID

The ULS Logs will indicate the following sort of message in the Database category with event ID fa43

 

04/16/2015 08:25:14.47 mssearch.exe (0x3CAC) 0x1678 SharePoint Server Search Crawler:Common ad9s8 Assert location: search\libs\utill\hangrecoverer.cxx(189) condition: !"Crawl hangs" StackTrace: at Microsoft.Office.Server.Native.dll: (sig=c3cf111a-d6fc-4ec5-ba1c-c9b8ebb47e41|2|microsoft.office.server.native.pdb, offset=1F074) at MSSrch.dll: (sig=bcc79b69-8dc0-4f7b-b42b-469030237946|2|mssrch.pdb, offset=22759) at MSSrch.dll: (offset=226FD) at MSSrch.dll: (offset=1DE54) at ntdll.dll: (sig=9d04eb0a-a387-494f-bd81-ed062072b99c|2|ntdll.pdb, offset=16CEB) at ntdll.dll: (offset=16405) at ntdll.dll: (offset=16BF2) at kernel32.dll: (sig=c4d1d906-5632-4196-99a8-b2f25b62381d|2|kernel32.pdb, offset=1652D) at ntdll.dll: (sig=9d04eb0a-a387-494f-bd81-ed062072b99c|2|ntdll.pdb, offset=2C541)

One will also see the rebalance job running against owstimer for the crawl partitions.

04/16/2015 09:07:12.28 OWSTIMER.EXE (0x390C) 0x2164 SharePoint Foundation Monitoring nasq Medium Entering monitored scope (Timer Job Rebalance crawl store partitions for bf01da64-8d00-441e-add5-79ab3741a479). Parent No 4a85179d-420b-e0df-4af4-51bc9809ff48

Solution

In SharePoint 2013 however we need to balance the SharePoint Crawl Store links content every now and then. Adding a new Crawl store DB doesn’t help the situation as well. Crawl stores maintain an indicative threshold called Imbalance threshold and a method that replies with a Boolean as to whether or not the Crawl Store is Imbalanced.

 

$SSA = Get-SPEnterpriseSearchServiceApplication

$CrawlStorePartitionManager = new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)

$CrawlStoreParititonManager.CrawlStoresAreUnbalanced()

True

The Search Service Application has a Method called CrawlStoresAreUnbalanced that gives us a boolean indication of whether or not the Crawl Store Links are full enough to do this rebalancing at all.

$CrawlStoreParititonManager.CrawlStoresAreUnbalanced()

False

Note: 1 Million is the default value for CrawlStoreImbalanceThreshold. In case they are unbalanced, we would know immediately.

It’s Property bag also has a Property called CrawlStoreImbalanceThreshold which can be obtained and modified using the GetProperty and SetProperty methods.

$SSA.GetProperty("CrawlStoreImbalanceThreshold")

$SSA.SetProperty("CrawlStoreImbalanceThreshold",10000)

Increasing Crawl stored for various hosts also helps to improve performance

Just increasing the number of crawl stores does not automatically balance the Links in the MSSCrawlURL across the hosts. Fortunately for us, Sharepoint gives us a Property to manage this as well. There is a job called “Rebalance crawl store partitions” that runs to perform this for us.

This runs automatically when some thresholds are attained called CrawlPartitionSplitThreshold. The default value for rebalancing to start is 10 Million items but it can be done earlier based on our needs.

When we make the above change, we can test this value against the registry key under CatalogNames with the same name CrawlPartitionSplitThreshold

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0\Search\Applications\<SSAGUID>-crawl-0\CatalogNames

image

 

We can call a method called BeginCrawlStoreRebalancing from the CrawlStorePartitionManager Object we created some time back

 

$SSA = Get-SPEnterpriseSearchServiceApplication

$CrawlStorePartitionManager = new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)

$SSA.GetProperty("CrawlPartitionSplitThreshold")

10000000

$SSA.SetProperty("CrawlPartitionSplitThreshold",50000)

$CrawlStorePartitionManager.BeginCrawlStoreRebalancing()

Note: The Balance job doesn’t run unless the threshold is crossed. Even if we force the rebalance to run, the actual rebalancing happens only if the threshold is met. So setting the value for CrawlPartitionSplitThreshold is critical.

We can achieve the same effect by forcing the Rebalance crawl store partitions job to run by following powershell commands.

 

image


(Get-SPTimerJob | where {$_.Name -match "Rebalance"} ).RunNow()

POST BY : Ramanathan Rajamani [MSFT]


Comments (3)

  1. SharePoint Online Training says:

    It is in point of fact a great and helpful piece of info. I am happy that you shared this useful info with us. Please stay us up to date like this. Thanks for sharing.

    http://staygreenacademy.com/

  2. Mr. T says:

    Very good article, thanks for sharing. When a Rebalance crawl timer job starts, what is the time expected to reach the end of the job on average? I guess it's no normal spends 1 day on it… Any way to see if it is working or it is stuck, apart from ULS and CA timer jobs status?

    Regards.

  3. Thanks for sharing the PowerShell codesnippet.

    But I noticed a typo in the script below.

    The third line has an i in the wrong place ($CrawlStoreParititonManager):
    $SSA = Get-SPEnterpriseSearchServiceApplication
    $CrawlStorePartitionManager = new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)
    $CrawlStoreParititonManager.CrawlStoresAreUnbalanced()

Skip to main content