Enabling U-SQL Advanced Analytics for Local Execution

After we announced the ability for U-SQL to massively distributed Python code in the Azure Data Lake Analytics service, a lot of developers have been asking us when the the Python support will work using U-SQL Local Execution. In this post, we'll describe how to make that happen.

To set expectations, what we will describe below is not officially supported - we do intend to enable Python extensions in Local Execution as part of the product - but until then these instructions are meant for those brave souls who are willing to try something at the cutting edge.

STEP 1: Install the U-SQL Advanced Analytics extensions in your Data Lake Analytics Account

• Launch the Azure Portal
• Navigate to your Data Lake Analytics Account
• Click Sample Scripts
• Click More and select Install U-SQL Extensions
• Wait until the extensions have finished installing - this should take about 5 minutes

STEP 2: Examine the output of the installation

• When Step 1 is finished, it will have created a folder called /usqlext in the default Data Lake Store account
• Go to your Data Lake Analytics Account in the Azure Portal
• Click Data Explorer
• Navigate to your default Data Lake Store account
• Verify that you see a folder called /usqlext

STEP 3: Setting up your ADLTools4VS local "DataRoot"

• Launch Visual Studio
• Select Tools > Options > Azure Data Lake > General
• Under U-SQL Local Run, find and copy the value for DataRoot
  • The value will look like this: C:\Users\username\AppData\Local\USQLDataRoot
• Open Explorer and browse to that folder
• Create a "usqlext" folder in that root.
  • The full path should look like this: C:\Users\username\AppData\Local\USQLDataRoot\usqlext
• Launch Azure PowerShell and login to your subscription with the Login-AzureRmAccount cmdlet
• Run the cmdlet below to copy the usqlext folder from your Data Lake Analytics account to your local DataRoot
  • Export-AdlStoreItem -Account <youradlsaccount> -Path /usqlext -Destination C:\Users\username\AppData\Local\USQLDataRoot\usqlext -Recurse
• This cmdlet will download about 4GB of data from the Data Lake Store and take a few minutes.

STEP 4: Install the U-QL Advanced Analytics assemblies into the Local Execution U-SQL Catalog

• Launch Visual Studio
• Start a new U-SQL project
• Copy the text from C:\Users\username\AppData\Local\USQLDataRoot\usqlext\RegisterAll.usql into your U-SQL script
• Run your U-SQL script in Local Execution mode
• Once the script finishes, you will be able to use the U-SQL Advanced Analytics features -(Python, R, Cognitive) on your own machine.

• Anonymous
  November 22, 2017
  Hello,when I do STEP 3, I see the progress bar of downloading but no file in my local folder. Do you have any idea of my mistake ?