CodeLens Team Indicators – Processing Pipeline

Mathew Aniyan MSFT

Introduction

CodeLens is a heads-up display in your Visual Studio Editor where you can find information about your code in-context. See this MSDN article for an introduction to CodeLens – Get history and other info about your code.

You can use the CodeLens Indicators to quickly identify the changes that have happened on the code and the authors of these changes. You can also see the code reviews, work-items and bugs associated with the changes. In addition, CodeLens will also show you incoming changes from other branches. These CodeLens Indicators (Changes, Authors, Code Reviews, WorkItems, Bugs & Incoming Changes – collectively called Team Indicators) are computed on the Team Foundation Server. [This feature requires at least Team Foundation Server 2013 version control. If you are using Git version control, please see CodeLens for Git] When you open a file in Visual Studio 2013 Ultimate, it contacts Team Foundation Server, retrieves this data and displays them as CodeLens Indicators.

NOTE: References and Test CodeLens Indicators are computed locally and do not depend on data from Team Foundation Server.

In this blog post, I will describe the processing done to compute CodeLens data and discuss some tips to manage the CodeLens jobs and data.

 

CodeLens Jobs

Step 1 – Keepup

When the user checks in a file into Team Foundation Server Version Control, it triggers a notification which is consumed by CodeLens. CodeLens initiates a “Keepup” job to process this changeset. This job will pick each file in the changeset and using Roslyn identify the code elements which have changed in this checkin. It will also identify the workitems & bugs which have been associated with this checkin and associate them to the changed code elements. This information is serialized into a JSON format.

Step 2 – Aggregator

Another job “Aggregator” is then initiated which picks up the information created by the Keepup job, summarizes it, merges it with past data and creates the final CodeLens data file. This file contains all the information required to show the CodeLens Team Indicators for the corresponding code file. Thus there is a matching CodeLens file for each code file stored in Team Foundation Server Version Control.

Step 3 – Cleanup

Yet another job “Cleanup” runs at periodic intervals and cleans up the temporary data created by Keepup and Aggregator jobs. In order to improve performance, this job does not actually remove the files but marks them for deletion. Another Team Foundation Server job comes in and deletes the file after a specific retention period.

Step 0 – Catchup

Since CodeLens was introduced only in Visual Studio Ultimate 2013, there may be lots of changesets already existing in the system. There is another job “Catchup” which processes all changesets that are existing in the system before upgrading to Team Foundation Server 2013. This will pick each changeset in descending order and process it exactly like Keepup. This job will be disabled after all historical processing is complete.

Job Run Frequency

Job Characteristics

• Keepup & Catchup jobs pick each file and tries to process it as quickly as possible.
• An instance of each of these jobs is created for each Team Project Collection.
• Each job processes one changeset at a time.
• CodeLens jobs uses the existing Team Foundation Server job agent infrastructure.

Administration

The administration of CodeLens jobs is done using the TFSConfig command line utility. The capabilities are described in this MSDN article – CodeIndex Command.

Using this tool, it is possible to

• Understand the current status of all jobs.
• Turn off indexing for specific team project collections
• Ignore a set of files or folders in CodeLens processing
• Reindex all the CodeLens data
• Destroy CodeLens data

Team Foundation Server Upgrades

When Team Foundation Server is upgraded to a new version, in some cases, CodeLens will re-initiate Catchup. This generally happens when there is a change in the CodeLens data schema. We attempt to minimize such cases, but were required to make a schema change when we introduced the Incoming Changes Indicator in Team Foundation Server 2013 Update 2 RC.

Thus you will see that catchup job is restarted after an upgrade to a Team Foundation Server 2013 Update 2 RC from an earlier version. Since catchup job is resource intensive, you may see a spike in CPU usage on the Team Foundation Server Job Agent(s) when it is running. You can use TFSConfig CodeIndex /IndexingStatus to understand progress of the catchup job.

Another side-effect of the catchup job restart during the upgrade is the buildup of temporary data. Since the catchup job is processing a large number of changesets (potentially), an equivalently high volume of temporary data is created. As I described above, the cleanup job marks all temporary data for deletion. But the data is deleted only after a retention period, which by default is 7 days. During this period of 7 days, the database size will be significantly larger than normal. If you want the temporary data to be deleted sooner, you can set the retention period to a lower value. (See Tip#2 below).

Tips

1. We have seen some cases where CodeLens files grow very large. The examples we have seen are all for code files generated by some tool (with a large number of code elements) and updated frequently. Since these code files and their corresponding CodeLens files have to be serialized/de-serialized, CPU utilization will be high when processing them. If you have such files, you can ignore them from CodeLens processing using TFSConfig CodeIndex /IgnoreList:add

2. The temporary data marked for deletion is deleted after a retention period which is by default 7 days. The following PowerShell snippet sets the retention period to 1 day.    [Caveat – This is intended to be run by experienced TFS administrators]

#Load Team Foundation Server Client library

[Reflection.Assembly]::Load(“Microsoft.TeamFoundation.Client,Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a”)

#Connect to Team Foundation Server – Replace with your Team Project Collection URI

$tpc = [Microsoft.TeamFoundation.Client.TfsTeamProjectCollectionFactory]::GetTeamProjectCollection(http://localhost:8080/tfs)

#Get the Team Foundation Registry Service

$registry = $tpc.GetService([Microsoft.TeamFoundation.Framework.Client.ITeamFoundationRegistry])

#Set the Retention Period to 1day

$registry.SetValue(“/Service/FileService/SqlRetentionPeriod“, 1)

NOTE: Setting the value of retention period to 1 (as shown in the script above) does not have any side effects and it is safe to do so.

Finally

In this blog post, I described the various jobs which compute CodeLens Team Indicators data. I also described how these CodeLens jobs can be managed. In addition, I also explained the behavior of CodeLens jobs during a Team Foundation Server upgrade. As always, looking forward to your feedback and questions on this topic and CodeLens in general.

Mathew Aniyan MSFT

Follow