Managing Your Azure Data Lake Analytics Compute Resources (Account-level Policy)

In Managing Your Azure Data Lake Analytics Compute Resources (Overview), we introduced why customers ask for ways to manage their ADLA compute capacity and what capabilities we have provided to help them achieve their goals. In this blog, let's go in to more details about account-level policies.

ADLA supports three types of account-level policies:

• Maximum AUs  -- Controls the maximum number of AUs that can be used by running jobs
• Maximum Number of Running Jobs  -- Controls the maximum number of concurrently running jobs.
• Days to Retain Job Queries  -- Controls how long detailed information about jobs are retained in the users ADLS account.

This setting controls the total number of AUs that can be assigned across all the U-SQL jobs running concurrently in your ADLA account. If the AUs requested for a newly submitted job plus the AUs consumed by the running jobs is more than the available AUs, it will be placed in a queue until the enough AUs are released when the running jobs complete.

By default, this setting is configured to 250 AUs (the system limit). To control costs, ADLA account administrators can set a lower limit by using the slider as shown in the figure below. In the example below, the administrator has set the maximum AUs available in this account to 200.

If you need more AUs than the configured system limit, please open a support ticket using the "Help+Support"? option in the Azure Portal. After the maximum AUs limit is increased, you may need to adjust the slider in order for your U-SQL jobs to take advantage of the increased limits.

This setting controls the maximum number of jobs allowed to run concurrently. By default, the system limit for this setting is 20. If there are AUs available, new jobs will be scheduled for immediate execution until the total number of running jobs reaches the value of this setting. After this point, any subsequent jobs that are submitted will be placed in a queue in priority order until one or more running jobs complete.

In the above example, the administrator has used the slider to reduce the maximum number of running jobs to 10.

By default, each account can have 200 jobs waiting in the queue. If the queue is full, any subsequently submitted jobs will receive an error message indicating that the job queue is full.

If you need to run more than 20 jobs concurrently or need to have more than 200 jobs placed in the queue, please go ahead and open a support ticket using the "Help+Support" option in the Azure Portal.

When your users run U-SQL jobs, the ADLA service keeps all related files like the U-SQL script, the DLL referenced in the U-SQL script, compilation and runtime output files etc. in folders under your ADLS account. These files can be used to debug and do performance tuning of the jobs in the future. By default, the files are retained for 30 days.

Please note that your subscription is billed for storage of these files as per ADLS pricing policies (see here for details). You can choose to optimize your cost by reducing how long the per-job files are to be retained. You can also choose to extend the retention period and allow the jobs to be analyzed or debugged beyond 30 days.

In the above example, the account administrator has reduced the retention period of the job files to 15 days. All the job files will be set to expire after 15 days and will be deleted automatically thereafter.

In next blog, we will introduce the job level policies.