Do you use Data Lake Analytics and wonder how many Analytics Units your jobs should have been assigned? Do you want to see if your job could consume a little less time or money? The recently-announced AU Analyzer tool can help you today! See our recent announcement of the AU Analyzer, available in both Visual…
Keeping Data Lake Costs Under Control: Creating Alerts for AUs Usage Thresholds.
Have you ever been surprised by a larger-than-expected monthly Azure Data Lake Analytics bill? Creating alerts using Log Analytics will help you know when the bill is growing more than it should. In this post, I will show you how to create an alert that emails a message whenever the total AUs assigned to jobs…
Simple Trick to Stay on top of your Azure Data Lake: Create Alerts using Log Analytics
If you manage one or more Azure Data Lake accounts, do you ever find it hard to stay on top of everything that is happening? Ever feel the need to know more about them? Are you regularly asking yourself any or all of these questions: What are our most expensive jobs ? When was a…
Using the first job run to optimize subsequent runs with Azure Data Lake job AU analyzer
Customers have been telling us it’s hard to find the right balance between Analytics Units (AUs) and job execution time. For too many users, optimizing a job means running the job in the cloud using trial and error to find the best AU allocation that balances cost and performance. Today we are happy to introduce…
Delivering Consistency and Accuracy: Improvements for Azure Data Lake Store Reporting
Have you ever uploaded a 70GB file from Windows into your Azure Data Lake Store and noticed that the total shown in the data explorer showed as 75.2 GB? Today, we are fixing that! Previously when reporting the size, we aggregated the total using the technically correct term (1GB = 1E+9 bytes). Whenever calculating the…
Azure Data Lake Command-line Quickstart using Azure Cloud Shell
Have you been using Azure Data Lake Analytics mainly from the Azure Portal or Visual Studio interfaces? Do you want to use Data Lake Analytics programmatically without installing anything? Do you want to get started with Azure PowerShell, but you’re on a Mac or Linux machine? Here’s how to get started with Azure PowerShell or…
Struggling to get insights for your Azure Data Lake Store? Azure Log Analytics can help!
Customers love to use Azure Data Lake across their organizations by enabling their Data Lake Analytics accounts to use multiple Data Lake Store accounts as data sources. Business can have flexibility with resource usage and allocation, but this creates challenges for the Data Lake account administrators who struggle with understanding what is being used, how…
From unstructured data to dashboard with Azure Data Factory and Azure Data Lake
When I joined the Big Data team at Microsoft, sifting through all the technologies and products left me with more questions than answers – as it does for many customers taking their first steps on the cloud. I’ve always learned best by doing, so I decided to roll up my sleeves and build something in…
Debugging Azure Data Lake Job Failures Made Easy (part 2) – Efficiently troubleshoot anomalies in recurring jobs
Azure Data Lake Analytics recently announced advanced job tracking and management features that make it easy to work with pipelines and recurring jobs. Releases after version 2.3.0000.1 of Azure Data Lake Tools for Visual Studio, introduce new features that simplify analyzing and troubleshooting such jobs. Troubleshooting issues in recurring job is a time-consuming task. It…
How to Save Money and Control Costs with Azure Data Lake Analytics
Where there is great power there is great responsibility -Winston Churchill Azure Data Lake Analytics with its serverless approach and scale-up U-SQL query execution puts a lot of power in developers hands. As you start exploring what is possible with Azure Data Lake and your datasets start growing, it becomes ever more important for developers…