Microsoft Dynamics 365 (online) – Asynchronous Service Quotas

Applies to: Microsoft Dynamics 365 (online)

 

The Microsoft Dynamics 365 Asynchronous Processing Service (called the async service) executes long-running operations independent of the main Microsoft Dynamics 365 (online & on-premises) core operation.  This results in improved overall system performance and improved scalability.
 

How does it work?

The asynchronous service features a managed queue for the execution of asynchronous plug-ins, workflows, and long-running operations such as bulk mail, bulk import, and campaign activity propagation.  These operations are registered with the asynchronous service and are executed periodically when the service processes its queue. Since these operations are queued up and must be executed in time globally, there are some well-defined resource quotas that enable resources to be distributed amongst all online customers equally.

In Microsoft Dynamics 365 online environment, there are certain quota settings applied for every organization, which are mentioned in the list below.
 

Settings

  1. AsyncMaxExceptionCountInTimeUnit
  2. OrgMaxAsyncThroughput
  3. OrgMaxThreadCount
  4. AsyncMaxExceptionCounterTimeUnitInMinutes
  5. AsyncThroughputCapResetTimeIntervalInMinutes

 
Some important points to note here:

  • These quotas are applied to a priority queue (plug-ins, SLA operations) and a regular queue (workflow, rollup, bulk delete, import, etc.) only and don’t apply to email jobs.
  • Since plug-ins and workflows are executed in different queues, each has an independent quota within each organization.
  • The values for these quota settings will differ between customer organizations. These settings are to be adjusted by Microsoft as per the performance needs of the platform, and are not configurable by customers.
  • These settings are for Microsoft Dynamics 365 Asynchronous Processing Service and not Microsoft Dynamics 365 Asynchronous Processing Service (maintenance), which is for system maintenance like executing the deletion and re-indexing services.

 

Setting details

 

AsyncMaxExceptionCountInTimeUnit

This setting helps Microsoft throttle failing async operations and prevent them from consuming valuable resources. If a workflow job execution fails A times in D minutes, it gets postponed. If an operation is postponed for the first time, it would be postponed for D minutes. If it’s postponed for the nth time, it would be postponed for D times n (D x n) minutes. There is a max limit for the number of times it can be postponed, after which it would be postponed forever. A, D, and n being values that are set at the organization level to ensure optimum throughput and performance.
 

OrgMaxAsyncThroughput

OrgMaxAsyncThroughput is the number of jobs that the async service will process in E minutes.

OrgMaxAsyncThroughput is the actual throughput which is the net sum of various other operations like perceived throughput, system internal operations, and operations with async auto-delete. It should not be confused with the number of operations marked completed in the UI (perceived throughput). The values for these throughputs (i.e. perceived throughput and actual throughput) will be different depending on the load on the system, at any given time.
 

OrgMaxThreadCount

OrgMaxThreadCount helps to control the total number of parallel executions, for any given organization, at a given time.
 

Key takeaway

Since async operations have to operate within the quota limits mentioned above, all operations which directly or indirectly impact async operations should be created judiciously. Here are some tips to consider, while working with asynchronous operations.

  1. Since there is no limit on the total number of async operations that can be created, the total number of operations can grow fast. With the limits mentioned above, it will further delay the async operations from getting picked up for execution. Some of the scenarios where this could happen are bulk import or migration from an on-premises to an online environment. If there are asynchronous workflows created to do backend updates to system or other entities for import of every single record, then these workflows or plugins should be disabled before starting the bulk import or migration.
  2. Review scenarios where async plugin or workflows throw exceptions. Exceptions should not be thrown indefinitely without any handler to stop after a certain number of retries or timeouts. Such indefinite execution could lead to exception limits, mentioned in the settings above- = (AsyncMaxExceptionCountInTimeUnit, being hit often and async operations becoming throttled).

 

Review health of asynchronous operations

These quotas hold good for a regular workload, however customers can view the health of asynchronous operations and know when they are being throttled, due to quota limits, by visiting the Organization Insights dashboard.

 

Organization Insights dashboard showing health of asynchronous jobs

 

The chart above depicts a scenario when the organization’s async queue is not getting impacted by significant load, which is beyond quota limits. During times of usual load, when quota limits are not exceeded, there should be no jobs showing up as backlogged. 

If you’d like to provide feedback, or offer suggestions, please go to Microsoft Dynamics Ideas. This website provides a collaboration platform for gathering actionable feedback to build and improve products and services.

 
 
Thanks,

Srikumar Nair