Fixing incorrect mapping/aliases for Search Index in Team Foundation Server


Please refer to the parent post here for context: https://blogs.msdn.microsoft.com/tapas_sahoos_blog/2017/09/11/analyzing-search-in-microsoft-team-foundation-server/

In certain scenarios, especially if you have done some manual updates around the Search index data folder, deleted the data, tried to re-index through scripts and/or done any of the above post an upgrade, chances are if the sequence of steps aren't accurate (I will cover the re-indexing scenarios in another post soon), you could possibly end up with some incorrect index state. This could lead to Search query failure, even though the indexing silently succeeds. Some of the resilience factors have been fixed in TFS 2017 Update 2. So this issue discussed in the post would be more applicable for TFS 2017 Update 1. But at the same time, it can be applicable based on the symptoms as explained below.

Issue Symptoms:

Overall, the scenario is, "indexing is done, but why are Search results till not showing up?"

There are 3 steps to this guidance:

  • Verify the index status
  • Run a sequence of steps to recreate the index
  • Verify the index status again, monitor the indexing progress and verify search results.

 

Verify the index status

  • Verify that index do exists with correct settings

http://<SearchUrl>:9200/_cat/indices?v

health status index                                          pri rep docs.count docs.deleted store.size pri.store.size
green  open   projectsearchshared_0_0      12   0          0            0                   1.8kb          1.8kb
green  open   workitemsearchshared_0_1  12   0          0            0                   1.8kb          1.8kb
green  open   projectsearchshared_1_0       12   0          0            0                   1.8kb          1.8kb
green  open   workitemsearchshared_1_1   12   0          0            0                   1.8kb          1.8kb
green  open   codesearchshared_0_0          12   0          4            0                    23kb           23kb
green  open   codesearchshared_1_0           12   0          0            0                   1.8kb          1.8kb

Check the number of shards (i.e. pri column above) It should be 12. If you see some other value such as 5, then the index is not created correctly in the pipeline.

  • Get the mappings for the indices (you can first pick the one where the docs.count is high (see above table) since that’s where docs are getting indexed)

http://<SearchUrl>:9200/codesearchshared_0_0/_mapping

You should observe some mappings like -

"codesearchshared_0_0": {
        "mappings": {
                 "SourceNoDedupeFileContractV2": { … }}}

With some properties such as

"content": {
    "type": "string",
    "norms": {
        "enabled": false
    },
    "index_options": "offsets",
    "analyzer": "codesearch",
    "include_in_all": false

If the mapping is not as expected above, it implies the index is created incorrectly and queries will not work.

  • Check the aliases (this applies only for {TFS 2017 Update 1 and earlier}; or if the TFS instance has been upgraded from {Update 1 or earlier} to {Update 2 and later} without recreating the index post upgrade)

http://<SearchUrl>:9200/_cat/aliases?v

alias                                                                                         index                                         filter routing.index routing.search
activesharedidx                                                                    codesearchshared_1_0           -       -                         -             
activesharedidx                                                                    codesearchshared_0_0          -       -                         -             
codeindex_90fad0ef-8e10-4c8e-8da6-2d7cd5106ef5     codesearchshared_0_0         -       -                         -             
codequery_90fad0ef-8e10-4c8e-8da6-2d7cd5106ef5     codesearchshared_0_0         *       -                        7             
workitem_activesharedidx_1                                            workitemsearchshared_0_1 -       -                         -             
workitem_activesharedidx_1                                            workitemsearchshared_1_1  -       -                         - 

Both the codeindex_{collectionId} and codequery_{collectionId} aliases need to be present.

codeindex_{collectionId} alias needs to be existing at all time for indexing to progress.

codequery_{collectionId} alias is created once indexing is over for the entire collection (see last section which explains how to detect when indexing is done for a collection). If it’s not there, code will not be searchable.

If you see inconsistent data in any of the above, we need to reset the index and re-index the collections again.

 

Run a sequence of steps to recreate the index (be extremely careful on the sequence of steps!)

  • Pause Indexing for all collections. Run the following script on TFS Configuration DB

https://github.com/Microsoft/Code-Search/blob/master/PauseIndexing.ps1

  • Login to the machine where the Elastic Search (ES) is running

Stop the ES service.

Delete the entire Search Index folder (something like, C:\TfsData\Search\IndexStore, or wherever you had configured it to be)

  • Restart the TFS Job Agent service(s)
  • Delete the following tables from each of the collection DBs.

DELETE FROM [Search].[tbl_IndexingUnit]
DELETE FROM [Search].[tbl_IndexingUnitChangeEvent]
DELETE FROM [Search].[tbl_IndexingUnitChangeEventArchive]
DELETE FROM [Search].[tbl_JobYield]
DELETE FROM [Search].[tbl_TreeStore]
DELETE FROM [Search].[tbl_DisabledFiles]
DELETE FROM [Search].[tbl_ResourceLockTable]

Try the last script on a smaller collection first (which has less number of repositories) so that you can verify that indexing happened correctly and the results are queryable. (See the next section for verification)

 

Verify the index status again, monitor the indexing progress and verify search results

  • Run this query on the configuration DB:

SELECT TOP 100 [JobId],[QueueTime],[StartTime],[EndTime],[Result],[ResultMessage]
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
WHERE JobSource = '<CollectionId>'
and (ResultMessage like '%completed with status%' or JobId = '02F271F3-0D40-4FA0-9328-C77EBCA59B6F')
ORDER BY StartTime desc

Verify that the Result = 0 (i.e. success) for the recent jobs and the specifically a job with JobID '02F271F3-0D40-4FA0-9328-C77EBCA59B6F' was successful. This is the initial index trigger job.

  • Run the initial 3 index verification steps (in the initial section above) to ensure they are consistent

Note: the codequery_{collectionId} alias might not have been created by now if the collection is not completely indexed. (See next step on how to check if indexing is complete for all repositories for the collection)

  • Once the entire collection is indexed (it will take time depending the number of repositories and code size in all those repositories), you will observe a Job success with message “Successfully completed finalize index for Collection Id {collectionId}”

You can keep querying for job success,  -

SELECT [JobId],[QueueTime],[StartTime],[EndTime],[Result],[ResultMessage]
FROM [Tfs_Configuration].[dbo].[tbl_JobHistory]
WHERE JobSource = '<CollectionId>'
and ResultMessage like '%completed finalize index for Collection Id%'
ORDER BY StartTime desc

  • Once indexing is complete for the collection, you can verify from the portal that search results are showing up for that collection

 

Comments (0)

Skip to main content