Static Compilation of IronPython scripts


The ability to compile IronPython scripts into .NET IL and to save them to disk existed in IronPython 1.0 but has been missing in 2.0 so far. With IronPython 2.0 Beta4 this has been added back.

Why would I compile dynamic language scripts?

There are a lot of reasons to compile scripts into a binary form. Shri talks about some of them here. For folks who don’t want to distribute source code in plain text this provides one level of obfuscation. To that end, a function called CompileModules has been added to the clr module to compile scripts into executable IL. The signature of the function is:

CompileModules(str assemblyName, dict kwArgs, Array[str] filenames)

So to compile a file foo.py into foo.dll you would do this:

import clr
clr.CompileModules("foo.dll", "foo.py")

This can now be brought in using the regular clr.AddReference. When clr.AddReference sees a compiled assembly, it publishes the module as well. So one can simply import the module into the code.

clr.AddReference("foo.dll")
import foo

Multiple files and main

The function can take multiple python files and compile them into one dll. What if you want it to be a standalone executable? There are two things to be done. First, a stub exe is needed that can load the dll. Second, a way to distinguish the main module is needed. The keyword args that CompileModules can take comes in handy here

import clr
clr.CompileModules("foo.dll", "foo.py", "bar.py", mainModule="main.py")

Now a stub exe can be written that loads up this compiled dll. The IronPython sample pyc.py has code that does shows how to generate a stub exe.

Wait, what is -X:SaveAssemblies mode then?

When IronPython is started with -X:SaveAssemblies, it generates a dll containing IL corresponding to the code it executed. Sounds an awful lot like compilation doesn’t it? The difference is one is executable IL and the other isn’t.

To understand the difference, one needs to understand that IronPython under normal course of its execution generates IL anyway. Every statement is converted to the DLR AST and IL gets spit out for the ASTs which is then executed. The SaveAssemblies mode simply dumps the generated IL into a dll. It is meant as a debugging device. So what is missing from this IL that prevents it from being re-executable code? The short answer is Dynamic Sites. The sites that are generated during the execution are not persisted. The compilation feature does exactly this – it persists the dynamic sites as well. Lets look at an example here and compare the generated IL in reflector. (Only the relevant code is copied over from reflector). This python code:

print 2 + 5
print 2 * 5
print 3 / 5

when run with -X:SaveAssemblies mode produces this code:

public static CallSite<DynamicSiteTarget<int, int, object>> #Constant207;
public static CallSite<DynamicSiteTarget<int, int, object>> #Constant208;
public static CallSite<DynamicSiteTarget<int, int, object>> #Constant209;
$lineNo = 1;
PythonOps.Print(__global_context, #Constant207.Target(#Constant207, 2, 5));
$lineNo = 2;
PythonOps.Print(__global_context, #Constant208.Target(#Constant208, 2, 5));
$lineNo = 3;
PythonOps.Print(__global_context, #Constant209.Target(#Constant209, 3, 5));

Notice that the Constants defined here are actually defined as fields on the generated type and this type doesn’t get instantiated anywhere and therefore the sites don’t get assigned anywhere. The same python code when compiled with clr.CompiledModules produces this code:

object[] objArray = new object[] { 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Add")), 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Multiply")), 
CallSite<DynamicSiteTarget<int, int, object>>.Create(PythonOps.MakeOperationAction(context, "Divide")) 
};
line = 1;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[0]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[0], 2, 5));
line = 2;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[1]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[1], 2, 5));
line = 3;
PythonOps.Print(context, ((CallSite<DynamicSiteTarget<int, int, object>>)objArray[2]).Target(
    (CallSite<DynamicSiteTarget<int, int, object>>)objArray[2], 3, 5));

You can see that all the dynamic call sites are being created here and their targets are being invoked. This then is perfectly executable code – maybe not as succinct as the python code but it does the same thing 🙂

Comments (18)

  1. Juba17 says:

    Hi.

    I made a small tool in Ironpython. Im new to it and when testing everything is ok.

    But I’ve been trying to compile both scripts (program script and form class script without sucess)

    I just can’t get it to work has a standalone exe. I followed the example above:

    import clr

    clr.CompileModules("form.dll", "foo.py", "form.py", mainModule="program.py")

    And I get the form.dll but then I want to get the executable and I get lost. Saw the pyc example and tried to compile with the pyc but get an error about the CompilerSink.

    Could you please point more info on how to compile to exe the applications?

    My version is Ironpython v. 2 beta 5

    Greetings

  2. srivatsn says:

    For creating a exe you have to use the pyc sample like this:

    ipy.exe pyc.py /main:program.py foo.py form.py /target:winexe

    I suppose this is throwing an error for you. Why don’t you send a mail to me via the contact link with the exact error?

  3. Juba17 says:

    Sry but can’t see contact link. I will post the error:

    line:

    ipy.exe "D:IronPython 2.0SamplesPycpyc.py" /main:"D:IronPython 2.0proyeto cod4warning_system.py" "D:IronPython 2.0proyeto cod4cod4form.py" /target:winexe

    error:

    Traceback (most recent call last):

     File "D:IronPython 2.0SamplesPycpyc.py", line 35, in D:IronPython 2.0Sam

    plesPycpyc.py

    AttributeError: ‘namespace#’ object has no attribute ‘CompilerSink’

  4. Juba17 says:

    Yes I checked and pyc can’t be used with 2.0 version since the only attributes available in Hosting call are:

    [‘ErrorCodes’, ‘Python’, ‘PythonCommandLine’, ‘PythonConsoleOptions’, ‘PythonOptionsParser’]

    So how is the independent executable made after all?

  5. srivatsn says:

    Are you using the 1.0 pyc sample by any chance? The sample was updated for 2.0 and is here – http://www.codeplex.com/IronPython/Release/ProjectReleases.aspx?ReleaseId=14353

  6. Juba17 says:

    Thanks for your help 😉

    I managed to compile it with pyc but I must’ve misunderstand the topic since when I compiled it still requires python lib and Ironpython dlls.

    am I doing something wrong or it’s suppose to be that way? I mean to a simple app to distribute I have to included the all of python lib? It’s not pratical, I must be wrong.

  7. srivatsn says:

    You will require the four IronPython DLLs for the exe to run but you can compile the python libraries along with your py files. It doesn’t have to be the entire python library as well- only files that are required by your app.

  8. Juba17 says:

    So i need to include in the compile all the regular modules? os, sys, socket, so on? But sometimes they need other modules to run. For instance. When i compiled the last time it required me some modules and other modules those modules run. Isn’t there a easy way to know wich one are the sub-modules?

  9. srivatsn says:

    There might be some static dependency analysis tools for python out there. I’m not aware of any though.

  10. Guy Rozendorn says:

    Hi,

    I have written a file called myProg.py, and compiled it to myProg.dll and myProg.exe using the pyc.py example file (using IronPython-2.0).

    The thing is that the presence of IronPython.dll and myProg.dll is required in the directory from which I am executing pyProg.exe.

    What’s the magic I need to make so I could execute pyProg.exe from anywhere (i.e it would search the dlls in the same directory pyProg.exe is in, and not in the directory I’m executing from)?

    For example,

    pyProg.exe, pyProg.dll and IronPython.dll are in C:Temp, and I’m doing:

    C:WINDOWS> C:TemppyProg.exe

    This doesn’t work, because the DLLs are in C:Temp, and not in C:WINDOWS.

    I want to run pyProg.exe from C:WINDOWS (or any other directory), but I don’t want to copy the DLLs to C:WINDOWS (or any other directory).

    Is this possible?

    Is it possible to make compile my ironpython script as a standalone exe? meaning, compiling it to an exe that contains all the code,

  11. srivatsn says:

    C:TemppyProg.exe should work since the dependencies are picked up from the directory of the executable as well. If you want to copy pyprog.exe to some other place, you can create a pyprog.exe.config file and use the assemblybinding tag – http://msdn.microsoft.com/en-us/library/twy1dw1e(VS.80).aspx

  12. Dinesh Varadharajan says:

    Hi,

    we are migrating an existing cpython app to ironpython to work with silverlight. We have so many packages and subpackages in cpython and to work with silverlight we tried to convert them all into a dll.

    The dll conversion is fine and all files are part of dll, but all the packages can be referred only from the main module. The modules that are present inside packages are not able to refer any other packages. Are we missing something basic.

  13. Estuve investigando el supuesto camino de pre-compilar m&oacute;dulos para ver si lograba resolver los problemas de performance que mencion&eacute; hace unos d&iacute;as. Resulta que la punta del iceberg estaba escondida en el pack de ejemplos de …

  14. srivatsn says:

    Dinesh, packages importing other packages from within the dll should work without any extra work. If you can send me a minimal reproduction of the problem, I can probably be of more assitance. Feel free to use the contact link to e-mail me – http://blogs.msdn.com/srivatsn/contact.aspx

  15. Dominik Kuropka says:

    Can you please help me? I have a simple problem which I don’t manage to solve with IronPython.

    I have modules ‘a.py’ and ‘s.py’. The module ‘a.py’ imports ‘s.py’. Both moduls import ‘StringIO’ which is a standard .NET module available in IronPython.

    Now a generate an ‘a.dll’ with this command:

    ipy pyc.py a.py s.py

    From this I would expect that I simply need to bind ‘a.dll’ to access its library functions. But this doesn’t work:

    import clr

    clr.AddRefernce("a.dll")

    import a

    Traceback (most recent call last):

     File "<stdin>", line 1, in <module>

     File "alfabet", line 19, in alfabet

     File "SimpleSqlClient", line 8, in SimpleSqlClient

    ImportError: No module named StringIO

    So what do I need to do to make ‘a.dll’ importable?

  16. srivatsn says:

    StringIO is part of the standard library. So you should include stringio.py as well (and its dependencies) while compiling a.dll

  17. Dominik Kuropka says:

    Thank you, this helped me a bit further. Now I am running in to a new error which has to do with code access security [ADO.NET]:

    SystemError: Fehler bei der Anforderung des Berechtigungstyps System.Data.SqlClient.SqlClientPermission, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.

    Is there any easy way to grant the "a.dll" all the rights which IronPython has?