How to configure a Dedicated Web Front End for crawling in SharePoint 2010


In SharePoint 2007 (MOSS v3) we have an option in the Office Server Search User Interface to configure a specific Web Front End as the server to be used for crawling. This functionality is very useful as it allows us to have a dedicated WFE for crawling as that ensures the crawling load is not distributed across the entire farm. Usually the indexer is configured with the Web Application services and is dedicated for crawling and then, this server is removed from the NLB .

Screen shot from SharePoint 2007

image

This option in SharePoint 2010 has been removed from the UI and is part of Powershell functionality. So how do you go about doing this? Here goes,

########## Start of Script to Configure a Dedicated Web Frontend in SharePoint 2010  ##########

$listOfUri = new-object System.Collections.Generic.List[System.Uri](1)

$zoneUrl = [Microsoft.SharePoint.Administration.SPUrlZone]’Default’

$webAppUrl = “Your Default Zone FQDN URL”

$webApp = Get-SPWebApplication -Identity $webAppUrl

$webApp.SiteDataServers.Remove($zoneUrl)          ## By default this has no items to remove

$listOfUri.Add(“Your Dedicated WFE Server URL”);

$webApp.SiteDataServers.Add($zoneUrl, $listOfUri);

$WebApp.Update()

########## End of Script to Configure a Dedicated Web Frontend in SharePoint 2010  ##########

Now you might have a requirement to roll back this change, which means change the setting back to “Use all web front end computers for crawling”. The script for doing that is as follows,

########## Start of Script to reset Dedicated Web Front end crawling settings to default in SharePoint 2010  ##########

$zoneUrl = [Microsoft.SharePoint.Administration.SPUrlZone]’Default’

$webAppUrl = “Your Default Zone FQDN URL

$webApp = Get-SPWebApplication -Identity $webAppUrl

$webApp.SiteDataServers.Remove($zoneUrl);

$WebApp.Update()

########## End of Script to reset Dedicated Web Front end crawling settings to default in SharePoint 2010  ##########

Comments (11)

  1. vijgang says:

    Regarding – $webAppUrl = "Your Default Zone FQDN URL"

    Should that be the load balanced URL?

    Regarding – $listOfUri.Add("Your Dedicated WFE Server URL");

    what additional configuration do we need in AAM for this?

  2. Vijay - MSFT says:

    $webAppUrl = "Your Default Zone FQDN URL" – This is your load balanced URL.

    $listOfUri.Add("Your Dedicated WFE Server URL");  – This is the URL of the WFE that you would like to dedicate for crawling.

  3. Hello Vijay,

    Let me explain my environment:

    There are 4 servers. 2 acting as WFE and 2 acting as Application servers (excel services, Web Analytics, search index).

    SPSAPPSRV1 – APP Server

    SPSAPPSRV2 – APP Server

    SPSWFESRV1 – WFE Server

    SPSWFESRV2 – WFE Server

    For my applications there are only one URL for load balance, as following:

    http://portal.domain.local

    http://intranet.domain.local

    http://blog.domain.local

    http://epm.domain.local/pwa

    Currently, when the crawling is started, it will be served for one of two WFE, then, the WFE that is serving for crawling the content, will rise the CPU usage to 100% (W3WP process) and if an user try to access one of these above URLs, will face a slow response.

    Because of this situation, I would like know in my scenario, how to add a third WFE and set it to be the dedicated WFE to crawling.

    Could you give me the sample in this script to perform the appropriated configuration in my environment?

    Thanks in advance!

  4. Vijay - MSFT says:

    Hi Marco,

    What you are saying is that your crawler is currently hitting the load balanced URL and you would like to configure it for a single WFE (new one say WFE3). For this, you will first need to introduce the new WFE into the environment. I guess you know how to do this by running PSConfig and joining an existing farm. Once this happens, and the new WFE starts functioning do not add it to the load balancer. Now from the above script change it to include the new WFE name & FQDN name.

    ########## Start of Script to Configure a Dedicated Web Frontend in SharePoint 2010  ##########

    $listOfUri = new-object System.Collections.Generic.List[System.Uri](1)

    $zoneUrl = [Microsoft.SharePoint.Administration.SPUrlZone]'Default'

    $webAppUrl = "http://portal.domain.local&quot;    <- Change here

    $webApp = Get-SPWebApplication -Identity $webAppUrl

    $webApp.SiteDataServers.Remove($zoneUrl)          ## By default this has no items to remove

    $listOfUri.Add("http://WFE3&quot;);    <- Change here

    $webApp.SiteDataServers.Add($zoneUrl, $listOfUri);

    $WebApp.Update()

    ########## End of Script to Configure a Dedicated Web Frontend in SharePoint 2010  ##########

  5. Nate says:

    Hi Vijay,

    Thanks for the info.  How would I modify the script if I would like to add more than one dedicated WFE for crawling?

  6. Vijay - MSFT says:

    Hi Nate,

    This should take care of it,

    $listOfUri.Add("http://WFEFORCRAWL1&quot;);    

    $listOfUri.Add("http://WFEFORCRAWL2&quot;);    

    $listOfUri.Add("http://WFEFORCRAWL3&quot;);    

    Regards,

    Vijay

  7. Hi Vijay,

    How above configuration is different from crawl component settings for a single server.Is this a alternative way of setting search topology (crawl component).You mentioned as "This option in SharePoint 2010 has been removed from the UI and is part of Powershell functionality. "

    If crawl component and above settings are different which will be treated as on priority if i give different server name under crawl component.

    Please help.

    Thanks

  8. Vijay - MSFT says:

    Hi Ashar2k11,

    I am guessing there is some confusion here. In SharePoint 2007, we had the following functionality in the UI,

    On the Configure Office SharePoint Server Search Service Settings on server page, in the Web Front End And Crawling section, we could select a computer under "Use a dedicated web front end computer for crawling". This would ensure that the crawler would target this WFE server only for crawling purpose. This option is not present in SharePoint 2010 and this article is to show how to do that in SharePoint 2010.

    Regards,

    Vijay

  9. Hasan says:

    Hi Vijay,

    Though we set CT servers via these scripts , but I have noticed still crawl goes to WFE servers if we dont have any host file entries made in crawl componant servers. How this happens is if i try to access the web application by default (without any host files) the host name gets resolved with load balancer VIP and redirects to any one of WFE's.

    To over come this i have made crawl component servers to have the IP of CT server in host file. Is there any alternative approach there? Please suggest. Thanks

  10. Vijay - MSFT says:

    What I recollect is that once we set a dedicated crawler WFE, SharePoint itself adds a hosts entry wherein it will use the URL with the IP of the dedicated WFE in there. So I expect this to happen the minute you run the scripts above.

  11. BKK says:

    You need to extend the web application with the dedicated Server URL and then use this. Otherwise it still redirects the request to WFE and not to dedicated server. Check this

    blogs.msdn.com/…/set-the-crawl-component-to-use-a-dedicated-front-end-for-indexing.aspx

Skip to main content