Troubleshooting issues with Visual Studio Team Services and TFS Notifications / Alerts

Article
08/23/2016

TFS and Visual Studio Team Services provide a way to Set alerts and get notified when changes occur.
One of the common issues that occur with these alerts is that users who are expecting to get a certain form of alert / notification do not get it, this guide is meant to help troubleshoot these issues.

Problem 1: I am expecting to get an alert but I am not getting one

If you are using TFS 2015 Update2 and having issues with Code Review events, you need to update to TFS 2015 Update2.1
First thing to check is to make sure a notification is actually expected. If you use the existing subscription templates and you are the one making the changes then it's most likely expected that you will not get a notification, this is by far the most common reason alerts are being discarded
- For example, when you create a "My work Items are changed by others" subscription, by default there will be a clause in the subscription that says "And Authorized As" <> [Me], this translates to (And the person making the change is not me), this ensures that the user making a change to a work item isn’t then notified about the change. Most users don’t want an email telling them they just did something
- For testing purposes, try removing this clause and seeing if things are working for you now

If you still see issues, check if there are any failures in the JobHistory by running this query against your config DB

 SELECT TOP 100
 JobSource,
 JobId,
 QueueTime,
 StartTime,
 EndTime,
 Result,
 ResultMessage
FROM tbl_JobHistory
WHERE JobId IN
(
'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7', -- event processing
'A96D6177-BEEF-477A-A2EE-2C31433214D0', -- email, soap, etc. delivery
'A0C22F34-652E-4B9C-A06B-3F4AFE4BE458' -- service hooks delivery
)
AND Result <> 0
ORDER BY StartTime DESC

Check the Message column, this should give you more details about the reasons for the errors.
- If it has a security/permissions error then check that your SMTP user/password are correct.
- If it has SOAP errors or ServiceUnavailable exceptions then your SOAP endpoint is misconfigured.
Are you using VS 2013 client to make Work Item / code review changes, and your subscription has some "Identity" fields, e.g. Work Item Assigned to = [Me], Reviewers Contains [Me], etc? If so then unfortunately this might be caused by a bug with VS2013, the bug has been fixed in Update 3.
When all else fails
- Starting TFS Update 2.1 you can enable detailed subscription tracing in TFS - Please Note that this is for advanced troubleshooting only, so it is not a supported method for troubleshooting , it might be easier at this point to file a connect bug through https://connect.microsoft.com/, or you can email me to help analyze the traces (rihams @ microsoft dot com)
  - Get the Subscription Id either by double clicking on the subscription or from any previous email from this same subscription
    
    1- Run this on the configuration database (mine is called tfs_Configuration)
    
               EXEC prc_CreateTrace @traceid = '5c3da495-1ded-4916-bc48-4d58fc52d3d4' , @userDefined='Subscription79' --Replace 79 with the id of the subscription that is not working, you can get it by double clicking on the subscription in the UI
    
    2-     Reproduce the issue by generating an event that you expect to see an email from
    3-     Disable the trace to avoid generating too much noise, On Configuration database run:
             EXEC prc_DeleteTrace '5c3da495-1ded-4916-bc48-4d58fc52d3d4'
    4 -     On the Job Agent machine, open event viewer -> Applications and Service Logs --> Microsoft-Team Foundation Server --> Debug
    5-     Filter the log to show only errors and verbose
  - The log will show you step by step details for how the steps that the code went through to evaluate your subscription, e.g. in the below trace you will find that it was evaluating the expression
```
  "CoreFields/StringFields/Field[Name='Authorized As']/NewValue" <>'@@MyDisplayName@@' 
```
    Which translated to
```
  'a0aa2f1f-8a03-463f-9655-62563228acb9' <> ' a0aa2f1f-8a03-463f-9655-62563228acb9' 
```
    So the evaluation failed
  - ```
   Calling matches for subscription 3 and Event 40
     EvaluateCondition: String to evaluate is "PortfolioProject" = 'test123' AND "CoreFields/StringFields/Field[Name='Created By']/NewValue" = '_@@MyDisplayName@@'         AND "CoreFields/StringFields/Field[Name='Authorized As']/NewValue" <> '_@@MyDisplayName@@'
     EvaluateCondition: Subscriber display name is a0aa2f1f-8a03-463f-9655-62563228acb9, Subscriber unique name is a0aa2f1f-8a03-463f-9655-62563228acb9
     EvaluateCondition: Condition references project with id: 63ffdcfc-cba4-47fd-be10-63c7efdc99a2 and name : test123
     StringFieldCondition.Evaluate: Begin evaluate Subscription 3, Event 40, Expression "CoreFields/StringFields/Field[Name='Created By']/NewValue" = '@@MyDisplayName@@'
     StringFieldCondition.Evaluate: Target Value = @@MyDisplayName@@
     FieldCondition.Evaluate: EvaluatedFieldName = CoreFields/StringFields/Field[Name='Created By']/NewValue
     StringFieldCondition.EvaluateOneValue: Value = a0aa2f1f-8a03-463f-9655-62563228acb9, Operator = 12, EvaluatedTarget = a0aa2f1f-8a03-463f-9655-62563228acb9
     StringFieldCondition.Evaluate: Begin evaluate Subscription 3, Event 40, Expression  "CoreFields/StringFields/Field[Name='Authorized As']/NewValue" <>'@@MyDisplayName@@' 
     StringFieldCondition.Evaluate: Target Value = @@MyDisplayName@@
     FieldCondition.Evaluate: EvaluatedFieldName = CoreFields/StringFields/Field[Name='Authorized As']/NewValue
     StringFieldCondition.EvaluateOneValue: Value = a0aa2f1f-8a03-463f-9655-62563228acb9, Operator = 13, EvaluatedTarget = a0aa2f1f-8a03-463f-9655-62563228acb9
     StringFieldCondition.Evaluate: Condition match was unsuccessful
     Condition match was False for event with id40 of type WorkItemChangedEvent
```
  Problem 2: Only a specific user is not receiving alerts
  
  - Again check the job results of any errors related to this user's email
  - Make sure the user is a member of the team - you can check by looking at the team membership page, the URL should look something like this:
  https://<server>:8080/tfs/DefaultCollection/<Project>/_admin/_security?_a=members
  If the user is not a member of the team they will not be receiving alerts
  
  Problem 3: There is a huge delay in Notifications
  - If you are not on Update 2.1 or Update 3 yet we highly recommend upgrading as a first step, lots of performance fixes were made in these two releases
  - Check out the job history to see where the time is being spent - run this on config DB
```
   SELECT TOP 50
   DATEDIFF(ss, QueueTime, EndTime) TotalTime
  ,DATEDIFF(ss, QueueTime, StartTime) TimeSpentInQueue
  ,DATEDIFF(ss, StartTime, EndTime) ProcessingTime
  ,[JobSource]
  ,[JobId]
  ,[QueueTime]
  ,[StartTime]
  ,[EndTime]
  ,[AgentId]
  ,[Result]
  ,REPLACE(REPLACE([ResultMessage], CHAR(13),  ), CHAR(10),  ) 
  ,[QueuedReasons]
  ,[QueueFlags]
  ,[Priority]
  FROM tbl_JobHistory WHERE JobId IN ( 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7', -- event processing 'A96D6177-BEEF-477A-A2EE-2C31433214D0', -- email, soap, etc. delivery 'A0C22F34-652E-4B9C-A06B-3F4AFE4BE458' -- service hooks delivery ) ORDER BY 1 DESC
```
  - If too much time is spent in the Queue you might want to add more JobAgents or look at other jobs that are taking too long.
  - If too much time is spent in SOAP delivery you might want to fix your endpoint
  - If too much time is spent sending emails you might want to check the throttling settings on your mail server, as it can slow down the email delivery drastically.

Troubleshooting issues with Visual Studio Team Services and TFS Notifications / Alerts

Additional resources