Enabling U-SQL Advanced Analytics for Local Execution

  • Article

After we announced the ability for U-SQL to massively distributed Python code in the Azure Data Lake Analytics service, a lot of developers have been asking us when the the Python support will work using U-SQL Local Execution. In this post, we'll describe how to make that happen.

To set expectations, what we will describe below is not officially supported - we do intend to enable Python extensions in Local Execution as part of the product - but until then these instructions are meant for those brave souls who are willing to try something at the cutting edge.

STEP 1: Install the U-SQL Advanced Analytics extensions in your Data Lake Analytics Account

  • Launch the Azure Portal
  • Navigate to your Data Lake Analytics Account
  • Click Sample Scripts
  • Click More and select Install U-SQL Extensions
  • Wait until the extensions have finished installing - this should take about 5 minutes

 

STEP 2: Examine the output of the installation

  • When Step 1 is finished, it will have created a folder called /usqlext in the default Data Lake Store account
  • Go to your Data Lake Analytics Account in the Azure Portal
  • Click Data Explorer
  • Navigate to your default Data Lake Store account
  • Verify that you see a folder called /usqlext

STEP 3: Setting up your ADLTools4VS local "DataRoot"

  • Launch Visual Studio
  • Select Tools > Options > Azure Data Lake > General
  • Under U-SQL Local Run, find and copy the value for DataRoot
    • The value will look like this: C:\Users\username\AppData\Local\USQLDataRoot
  • Open Explorer and browse to that folder
  • Create a "usqlext" folder in that root.
    • The full path should look like this: C:\Users\username\AppData\Local\USQLDataRoot\usqlext
  • Launch Azure PowerShell and login to your subscription with the Login-AzureRmAccount cmdlet
  • Run the cmdlet below to copy the usqlext folder from your Data Lake Analytics account to your local DataRoot
    • Export-AdlStoreItem -Account <youradlsaccount> -Path /usqlext -Destination C:\Users\username\AppData\Local\USQLDataRoot\usqlext -Recurse
  • This cmdlet will download about 4GB of data from the Data Lake Store and take a few minutes.

 

STEP 4: Install the U-QL Advanced Analytics assemblies into the Local Execution U-SQL Catalog

  • Launch Visual Studio
  • Start a new U-SQL project
  • Copy the text from C:\Users\username\AppData\Local\USQLDataRoot\usqlext\RegisterAll.usql into your U-SQL script
  • Run your U-SQL script in Local Execution mode
  • Once the script finishes, you will be able to use the U-SQL Advanced Analytics features -(Python, R, Cognitive) on your own machine.