Troubleshooting issues with Visual Studio Team Services and TFS Notifications / Alerts


TFS and Visual Studio Team Services provide a way to Set alerts and get notified when changes occur.
One of the common issues that occur with these alerts is that users who are expecting to get a certain form of alert / notification do not get it, this guide is meant to help troubleshoot these issues.

Problem 1: I am expecting to get an alert but I am not getting one

  • If you are using TFS 2015 Update2 and having issues with Code Review events, you need to update to TFS 2015 Update2.1
  • First thing to check is to make sure a notification is actually expected. If you use the existing subscription templates and you are the one making the changes then it's most likely expected that you will not get a notification, this is by far the most common reason alerts are being discarded
    • For example, when you create a "My work Items are changed by others" subscription, by default there will be a clause in the subscription that says "And Authorized As" <> [Me], this translates to (And the person making the change is not me), this ensures that the user making a change to a work item isn’t then notified about the change. Most users don’t want an email telling them they just did something
    • For testing purposes, try removing this clause and seeing if things are working for you now

wit changed by others          wit_assigned to me

 

  • If you still see issues, check if there are any failures in the JobHistory by running this query against your config DB
  • SELECT TOP 100
     JobSource,
     JobId,
     QueueTime,
     StartTime,
     EndTime,
     Result,
     ResultMessage
    FROM tbl_JobHistory
    WHERE JobId IN
    (
    'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7', -- event processing
    'A96D6177-BEEF-477A-A2EE-2C31433214D0', -- email, soap, etc. delivery
    'A0C22F34-652E-4B9C-A06B-3F4AFE4BE458' -- service hooks delivery
    )
    AND Result <> 0
    ORDER BY StartTime DESC
  • Check the Message column, this should give you more details about the reasons for the errors.
    • If it has a security/permissions error then check that your SMTP user/password are correct.
    • If it has SOAP errors or ServiceUnavailable exceptions then your SOAP endpoint is misconfigured.
  • Are you using VS 2013 client to make Work Item / code review changes, and your subscription has some "Identity" fields, e.g. Work Item Assigned to = [Me], Reviewers Contains [Me], etc? If so then unfortunately this might be caused by a bug with VS2013, the bug has been fixed in Update 3.
  • When all else fails
    • Starting TFS Update 2.1 you can enable detailed subscription tracing in TFS - Please Note that this is for advanced troubleshooting only, so it is not a supported method for troubleshooting , it might be easier at this point to file a connect bug through https://connect.microsoft.com/, or you can email me to help analyze the traces (rihams @ microsoft dot com)
      • Get the Subscription Id either by double clicking on the subscription or from any previous email from this same subscription

        1-    Run this on the configuration database (mine is called tfs_Configuration)

                   EXEC prc_CreateTrace @traceid = '5c3da495-1ded-4916-bc48-4d58fc52d3d4' , @userDefined='Subscription79'   --Replace 79 with the id of the subscription that is not working, you can get it by double clicking on the subscription in the UI

        2-     Reproduce the issue by generating an event that you expect to see an email from
        3-     Disable the trace to avoid generating too much noise, On Configuration database run:
                 EXEC prc_DeleteTrace '5c3da495-1ded-4916-bc48-4d58fc52d3d4'
        4 -     On the Job Agent machine, open event viewer -> Applications and Service Logs --> Microsoft-Team Foundation Server --> Debug
        5-
             Filter the log to show only errors and verbose

      • The log will show you step by step details for how the steps that the code went through to evaluate your subscription, e.g. in the below trace you will find that it was evaluating the expression
        "CoreFields/StringFields/Field[Name='Authorized As']/NewValue" <>'@@MyDisplayName@@'

        Which translated to

        'a0aa2f1f-8a03-463f-9655-62563228acb9' <> 'a0aa2f1f-8a03-463f-9655-62563228acb9'

        So the evaluation failed

      • Calling matches for subscription 3 and Event 40
           EvaluateCondition: String to evaluate is "PortfolioProject" = 'test123' AND "CoreFields/StringFields/Field[Name='Created By']/NewValue" = '_@@MyDisplayName@@'         AND "CoreFields/StringFields/Field[Name='Authorized As']/NewValue" <> '_@@MyDisplayName@@'
           EvaluateCondition: Subscriber display name is a0aa2f1f-8a03-463f-9655-62563228acb9, Subscriber unique name is a0aa2f1f-8a03-463f-9655-62563228acb9
           EvaluateCondition: Condition references project with id: 63ffdcfc-cba4-47fd-be10-63c7efdc99a2 and name : test123
           StringFieldCondition.Evaluate: Begin evaluate Subscription 3, Event 40, Expression "CoreFields/StringFields/Field[Name='Created By']/NewValue" = '@@MyDisplayName@@'
           StringFieldCondition.Evaluate: Target Value = @@MyDisplayName@@
           FieldCondition.Evaluate: EvaluatedFieldName = CoreFields/StringFields/Field[Name='Created By']/NewValue
           StringFieldCondition.EvaluateOneValue: Value = a0aa2f1f-8a03-463f-9655-62563228acb9, Operator = 12, EvaluatedTarget = a0aa2f1f-8a03-463f-9655-62563228acb9
           StringFieldCondition.Evaluate: Begin evaluate Subscription 3, Event 40, Expression "CoreFields/StringFields/Field[Name='Authorized As']/NewValue" <>'@@MyDisplayName@@'
           StringFieldCondition.Evaluate: Target Value = @@MyDisplayName@@
           FieldCondition.Evaluate: EvaluatedFieldName = CoreFields/StringFields/Field[Name='Authorized As']/NewValue
           StringFieldCondition.EvaluateOneValue: Value = a0aa2f1f-8a03-463f-9655-62563228acb9, Operator = 13, EvaluatedTarget = a0aa2f1f-8a03-463f-9655-62563228acb9
           StringFieldCondition.Evaluate: Condition match was unsuccessful
           Condition match was False for event with id40 of type WorkItemChangedEvent
        
        

        Problem 2: Only a specific user is not receiving alerts

        - Again check the job results of any errors related to this user's email
        - Make sure the user is a member of the team - you can check by looking at the team membership page, the URL should look something like this:
        http://<server>:8080/tfs/DefaultCollection/<Project>/_admin/_security?_a=members
        If the user is not a member of the team they will not be receiving alerts

        Problem 3: There is a huge delay in Notifications
        - If you are not on Update 2.1 or Update 3 yet we highly recommend upgrading as a first step, lots of performance fixes were made in these two releases
        - Check out the job history to see where the time is being spent - run this on config DB

        SELECT TOP 50
         DATEDIFF(ss, QueueTime, EndTime) TotalTime
        ,DATEDIFF(ss, QueueTime, StartTime) TimeSpentInQueue
        ,DATEDIFF(ss, StartTime, EndTime) ProcessingTime
        ,[JobSource]
        ,[JobId]
        ,[QueueTime]
        ,[StartTime]
        ,[EndTime]
        ,[AgentId]
        ,[Result]
        ,REPLACE(REPLACE([ResultMessage], CHAR(13), ), CHAR(10), ) 
        ,[QueuedReasons]
        ,[QueueFlags]
        ,[Priority]
        FROM tbl_JobHistory WHERE JobId IN ( 'A4804DCF-4BB6-4109-B61C-E59C2E8A9FF7', -- event processing 'A96D6177-BEEF-477A-A2EE-2C31433214D0', -- email, soap, etc. delivery 'A0C22F34-652E-4B9C-A06B-3F4AFE4BE458' -- service hooks delivery ) ORDER BY 1 DESC

        - If too much time is spent in the Queue you might want to add more JobAgents or look at other jobs that are taking too long.
        - If too much time is spent in SOAP delivery you might want to fix your endpoint
        - If too much time is spent sending emails you might want to check the throttling settings on your mail server, as it can slow down the email delivery drastically.

    Comments (0)

    Skip to main content