FAST​ 2010 Query Completion/Suggestions lifecycle- An Insight

Introduction 

Query suggestions, also known as search suggestions, appear as a list of suggested words below the search box. The list suggests search terms while the user is typing a search query.

The query suggestions are automatically created over time for any query that has a specified number of result click-throughs (default is six queries with click-through). The suggestions are compiled by the "Prepare query suggestions" job in Central Administration, which is scheduled to run once every day.

The query suggestions functionality is enabled by default. You can disable it or change the default settings by editing the Search Box web part. 

Search Service Applications (SSA)

 SharePoint 2010 performs its search using service application called “Search Service Application (SSA) ”. You can find more info on Service Applications here

Each service applications will have an associated database in SQL Server. SharePoint 2010 follows a naming convention for naming a corresponding service application database. The naming convention is like <ServiceApplicationName>_<Guid>. For example if a PerformancePoint service application name is PPS_SA then SharePoint uses PPS SA_3b75130f8fd44495b79e238cb95f5787 as database name.

To configure FAST Search with SharePoint 2010, you need to create 2 Search Service Applications. One is for FAST Content and other is for FAST Query. For more information on how to create FAST Content and Query Search Service applications, refer here (FAST Content, FAST Query).

Let’s say I have created 2 both Search Service Applications by naming “FASTContent” and “FASTQuery”. It will have following corresponding databases.

Custom Query Suggestions​

 In Out of box scenario query suggestions are automatically created over time for any query that has a specified number of result click-through. Although, it may be required to facilitate users with a prepopulated set of suggestions, so suggestions would be available to user right from the start. 

Following are the steps to add custom Query suggestions.

1.      Start PowerShell via the SharePoint 2010 Management Shell and run the following commands:

New-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication <FAST Query SSA NAme> -Language En-Us -Type QuerySuggestionAlwaysSuggest -Name <Suggestion key word>

2.      Run the timer job by using command.

$job = Get-SPTimerJob -Identity "Prepare Query Suggestions"

$job.RunNow()

 Sometimes it may be required to add n numbers of query suggestions with a single command. In order to achieve the same following approach could be used.

1.      Prepare a QuerySuggestion.txt file with all the suggestions separated by a line.

2.      Start PowerShell via the SharepointSharePoint 2010 Management Shell and run the following commands:

 #make sure to change the Identity to the name of your Query SSA

$ssa1 = Get-SPEnterpriseSearchServiceApplication -Identity "FAST Query SSA"

Get-Content <file path>\QuerySuggestions.txt | Foreach-Object {

    $q = New-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication $ssa1 -Language En-Us -Type QuerySuggestionAlwaysSuggest -Name $_

} 

Query Suggestion - Lifecycle

When we add custome phrases for query suggestions, phrase (Custom suggestion) will get added to MSSLanguageResources table in FAST Query DB with type 2. The type value 2 is for all user entered phrases for query suggestions.

The same could be seen in screen shot below, where we have added “SharePoint 2007” and “SharePoint 2010” as suggestions.

Once phrases are added to the table, a timer job has to prepare query suggestions out of the values in table MSSLanguageResources. This timer job can be run manually using command

$job = Get-SPTimerJob -Identity "Prepare Query Suggestions"

$job.RunNow()

Usually this timer job runs once in a day, and is responsible for creating Query Suggestions out of the phrases added in table MSSLanguageResources.

This job updates the MSSQLLogSearchCounts table. The job picks phrases from MSSLanguageResources and adds them to the column Query String of the MSSQLLogSearchCounts table. Query string is the field which is used to be shown as a query suggestion.

 Job also updates MSSQLLogQueryTerms table. Each phrase stored in table MSSLanguageResources gets split in multiple words. Split character for a phrase could be a space or ‘@’ or ‘.’ Etc. After a phrase is split into multiple words, those words are stored in MSSQLLogQueryTerms table. Each word is stored as a single Query Term.

E.g. “SharePoint 2007” will be split into 2 different possible query terms. The same way “SharePoint 2010” again would be split into two different possible query terms.

Also when a phrase is added via PowerShell script default Query count is given as 100000000 to term as well as to Query String in both the tables (MSSQLLogQueryTerms, MSSQLLogSearchCounts).

Query Count is the field which is used by GetQuerySuggestion method to filter suggestion. If value of count is less than 6 Query string will not be shown as a suggestion.

The screenshot below shows that term “msengage” will also be shown in query suggestion since value of query count is greater than 6. This term has come from click through of links.

On the search site, we have Search text box and when we have query suggestion enabled for a search text box, it shows suggestion for queries as user types in. Basically there is an asynchronous call which is made to a web service after each key press.

The web service which is being called is https://<site>/_vti_bin/Search.asmx. This service exposes a method called GetQuerySuggestions(string xmlQuery), and this method returns an array of strings, which is basically collection of all the suggestions.

In order to get a list of all the suggestion, GetQuerySuggestions method calls a stored procedure “proc_MSS_GetQuerySuggestions” in FAST Query DB. Following screenshot shows it running

This stored procedure picks values of query suggestions from table MSSQLLogQueryTerms and MSSQLLogSearchCounts. Stored procedure will check for the query terms in the table MSSQLLogQueryTerms, which are matching to the user input. Based on the Query IDs of matched query terms, it will return Query Strings from MSSQLLogSearchCounts table. These Query Strings will be shown as query suggestions.

For example mentioned in the section 3, when user types “sh” it matches two query terms in table MSSQLLogQueryTerms and hence two suggestions are returned. And “SharePoint 2007” and “SharePoint 2010” are shown as suggestions.

 Get Available Query Suggestions

 You can get all query suggestions using the SharePoint Point 2010 powershell cmdlets.

Following are the steps to get Query suggestions.

Start PowerShell via the SharePoint 2010 Management Shell and run the following commands:

 $ssa = Get-SPEnterpriseSearchServiceapplication -Identity "FASTQuery"

Get-SPEnterpriseSearchQuerySuggestionCandidates -SearchApplication $ssa

Following screenshot shows an example

Block Unwanted Query Suggestions

You can also block suggestion terms that you don’t want them as query suggestions for search users. This is a common requirement for the administrators to block some terms (eg: confidential terms) from being suggested. The same cmdlet “New-SPEnterpriseSearchLanguageResourcePhrase” for blocking selected terms.

Following are the steps to block unwanted Query suggestions.

1.      Start PowerShell via the SharePoint 2010 Management Shell and run the following commands:

$searchapp = Get-SPEnterpriseSearchServiceApplication "FAST Query SSA"

New-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication $searchapp -Language en-us -Type QuerySuggestionBlockList –Name “SharePoint 2001”

2.      Run the timer job by using command.

$job = Get-SPTimerJob -Identity "Prepare Query Suggestions"

$job.RunNow() 

Remove Query Suggestions

You can remove query suggestions using the SharePoint Point 2010 powershell cmdlets.

Following are the steps to remove Query suggestions.

Start PowerShell via the SharePoint 2010 Management Shell and run the following commands:

 $ssa = Get-SPEnterpriseSearchServiceapplication -Identity "FASTQuery"

Remove-SPEnterpriseSearchLanguageResourcePhrase -SearchApplication $ssa -Language En-Us -Type QuerySuggestionAlwaysSuggest -Identity "<input query suggestion here>"