Notifications overview for VSTS

Core Objects

There are three fundamental objects in the notification system:

Workflow

The actual flow can be expressed simply as well:

• Some part of the product fires an event (WIT, Git, Gallery, etc.)
• The notification system writes a record to tbl_ClientEvent with the event payload and starts the appropriate event processing job for the event type.
• The event processing job reads the pending events from tbl_ClientEvent and loads subscriptions for the matching event types from tbl_EventSubscription as well as builtin and contributed subscriptions.
• Each event is matched against each subscription. If there is a match, a notification is created. After each batch of events (up to 1000 at a time by default), the notifications are flushed to tbl_EventNotification and the appropriate notification delivery job(s) are started.
• The delivery job reads pending notifications in batches (up to 100 by default) and delivers them.
  • Email messages are delivered in parallel (up to 8 at a time by default)
  • Service Hooks notifications are batched up and sent to the Service Hooks service for actual delivery unless...
  • ...the subscription is for Azure Service Bus for specific hosts/subscriptions - these are delivered directly from the host service without calling to the Service Hooks service at all

Additional Tables

Notification Diagnostic Logs

Beginning with M131 and TFS 2018 Update 2

Extended diagnostic logs are now stored in tbl_NotificationLog. These logs can get large so they are cleaned up every few days. However, they are easily accessible via a REST navigate in a web browser. Logs are identified by LogSource and LogId. LogSource is typically a well known ID and LogId is a particular instance of that LogSource. If you happen to know the LogSource and LogId, you can refer to the log directly. Otherwise, you can specify a date range. The format of the REST API URL is:

<your host>/_apis/notification/DiagnosticLogs/LogSource/entries/LogId

or

<your host>/_apis/notification/DiagnosticLogs/LogSource/entries?startTime=time1&endTime=time2

Job Logging

Looking at the result message in Job History can be a big help when troubleshooting what's happening with event processing and notification delivery. However, it is limited to 8k and really isn't structured in any way - it is essentially a sequence of trace messages. Extended diagnostics are recorded for every single job run. The JobId is used as the LogSource and each run of the job gets a unique LogId - it's just a GUID with no further meaning. The result message in JobHistory contains the URL for that particular run.

To get the diagnostic log for the Work Item Processing job (915F48F2-1B64-40D9-A43F-FE2528B4F296) with a LogID of 68DE62E0-AA65-47BB-9369-FCD534B9A69A:

https://mseng.visualstudio.com/_apis/notification/DiagnosticLogs/915F48F2-1B64-40D9-A43F-FE2528B4F296/entries/68DE62E0-AA65-47BB-9369-FCD534B9A69A

If you don't know the LogId, you can ask for entries using a date range like this:

https://mseng.visualstudio.com/_apis/notification/DiagnosticLogs/915F48F2-1B64-40D9-A43F-FE2528B4F296/entries?startTime=2018-03-01T01:05:21.2253819Z&endTime=2018-03-01T12:05:21.2253819Z

This will return an array of logs. There can be a lot of logs and they can be on the larger size of things so the smaller you can make the date range the better.

Subscription Diagnostics Logging

We can enable even more diagnostic logging for a particular subscription. It can be enabled for even processing and/or notification delivery. The format for the LogSource is the subscription ID encoded into a well known GUID.

For event processing the LogSource is 08675309-EEEE-0000-0000-nnnnnnnnnnnn and for notification delivery the format is 08675309-DDDD-0000-0000-nnnnnnnnnnnn where nnnnnnnnnnnn is a leading zero padded subscription Id in decimal. The Job History result message and the diagnostic log will have a list of URLs for all of the subscription diagnostic logs that it generated. Once enabled, subscriptions diagnostics for 25 events for up to an hour and for 25 deliveries for an hour.

You can enable access to subscription diagnostics in the browser by appending diagnostics=true to the notifications page you are viewing. For example, on your personal subscriptions:

<your host>/_notifications?diagnostics=true

This will add an option to the "..." menu to allow you to enable/disable diagnostics for a particular subscription.

These settings are exposed via a REST API as well.

To enable processing and delivery diagnostics for a subscription, make an HTTP PUT call with the following body:

 {
  "evaluationTracing": { "enabled": true },
  "deliveryTracing": { "enabled": true }
}

The URL format is:< yourhost>/_apis/notification/subscriptions/SubscriptionId/diagnostics?api-version=4.1-preview.1

If you issue an HTTP GET, you will receive the current status. The results will look like this:

 {
  "evaluationTracing": {
    "enabled": true,
    "endDate": "2018-03-01T20:03:59.929866Z",
    "startDate": "2018-03-01T19:03:59.929866Z",
    "maxTracedEntries": 25
  },
  "deliveryTracing": {
    "enabled": true,
    "endDate": "2018-03-01T20:03:59.929866Z",
    "startDate": "2018-03-01T19:03:59.929866Z",
    "maxTracedEntries": 25
  }
}

Notifications Jobs

Job IDs (as of TFS 2018)

SQL JobHistory Filters

Processing Job

JobId in ('915f48f2-1b64-40d9-a43f-fe2528b4f296', '9a688110-9e33-4cdc-affd-75d16303e7f1', 'a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7')

Email Delivery Jobs

JobId in ('631f49b3-46e1-42ec-8fff-081bd176c18a', '8833fc71-42ca-441b-ab12-25314877772d', 'a96d6177-beef-477a-a2ee-2c31433214d0')

Service Hooks Delivery Jobs

Non Service Bus

JobId in ('b7494ac7-dee6-4342-ad43-5d6944576645', 'd745068c-7dab-4135-8665-6a7e1c20f29c', 'a0c22f34-652e-4b9c-a06b-3f4afe4be458')

Service Bus

JobId in ('ce76d8b3-3e04-4758-8430-4e9266d38938', '1214f948-151c-4f98-9ae2-1c9816db1eea', 'c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126')

Groups

JobId in ('d6be0cce-5320-45de-9297-596a9be68a37')

Notifications Legacy Delivery

JobId in ('007b33d4-c189-4576-96dd-36178e2f6803')

Event Type groupings

WIT Email and Service hooks

JobId in ('915f48f2-1b64-40d9-a43f-fe2528b4f296', '631f49b3-46e1-42ec-8fff-081bd176c18a', 'b7494ac7-dee6-4342-ad43-5d6944576645', 'ce76d8b3-3e04-4758-8430-4e9266d38938')

Git Email and Service hooks

JobId in ('9a688110-9e33-4cdc-affd-75d16303e7f1', '8833fc71-42ca-441b-ab12-25314877772d', 'd745068c-7dab-4135-8665-6a7e1c20f29c', '1214f948-151c-4f98-9ae2-1c9816db1eea')

Any other events Email and Service hooks

JobId in ('a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7', 'a96d6177-beef-477a-a2ee-2c31433214d0', 'a0c22f34-652e-4b9c-a06b-3f4afe4be458', 'c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126')