Using Custom Python Libraries with U-SQL

The Python extensions for Azure Data Lake Analytics ships with the standard Python libraries and includes pandas and numpy. We've been getting a lot of questions about how to use custom libraries with the Python extensions. The good news is that this is simple.

Introducing zipimport

The adoption of  PEP 273 (zipimport) gave Python's import statement the ability to import modules from ZIP files since Python 2.3. Please review the zipimport documentation now.

To review the basics:

  • You create a module (a .py file, etc.)
  • ZIP up the module into a .zip file
  • Add the full path of the zip.file to sys.path
  • Finally, import the module

Build and test a simple zipped package

Before you try to use a custom module with U-SQL, master the mechanics of zipimport on your own box.

Create a file called with the following contents:

# demo module
hello_world = "Hello World! This is code from a custom module"

This simple code define a single variable called hello_world.

Create a zip file called that contains the at the root .

  • In Windows you can create right-click on and select Send to compressed folder
    • This will create a file called
  • Rename to
    • This renaming step isn't mandatory, but will help highlight how the process will work

Create a Python file in the same folder as

import sys
import mymodule

Your folder should contain:


Now run the program


The output should look like this:

Hello World! This is code from a custom module

Deploying Custom Python Modules with U-SQL

First upload the file to your ADLS store - in this case we will upload it to the root of the default ADLS account for the ADLA account we are using - so its path is "\"

Then run this U-SQL script


// is inside the file

DECLARE @myScript = @"
import sys
sys.path.insert(0, '')
import mymodule

def usqlml_main(df):
 del df['number']
 df['hello_world'] = str(mymodule.hello_world)
 return df

@rows = 
 SELECT * FROM (VALUES (1)) AS D(number);

@rows =
 REDUCE @rows ON number
 PRODUCE hello_world string
 USING new Extension.Python.Reducer(pyScript:@myScript);

OUTPUT @rows
 TO "/demo_python_custom_module.csv"
 USING Outputters.Csv(outputHeader: true);

It will produce a simple CSV file with "Hello World! This is code from a custom module" as a row.

Comments (2)

  1. Erika Menezes says:

    This is very helpful!

  2. Erika Menezes says:

    Great article! A follow up question: If I wanted to use a custom library such as tensorflow that uses different versions of numpy than what is pre-installed with the U-sql python extension, what would be the best way to do this ?

Skip to main content