How to embed IronPython script support in your existing app in 10 easy steps
Previously, I added IronPython scripting support to a real existing application, MDbg (a managed debugger written in C#). In this entry, I'll go through step-by-step how I did that. I'll call specific attention to the goofy issues so that it should be very easy for other end users to get right to the point. I estimate you can modify your existing managed app to have decent IronPython embedded scripting within an hour.
I described how this functionality is exposed to the end user in my previous blog entry. If you haven't read that, you may find it useful. I'll refer to it throughout here.
I also need to repeat that I had never used python before this and am not on any python forums. I recommend https://diveintopython.org/toc/index.html as a basic python reference and https://docs.python.org/tut as a more comprehensive reference. Also consider playing around with the IronPython console in the IronPython download to get a better feel for primitives in the language. You can also debug the IronPython console itself to see the concepts below in action. The code snippets below refer to the source for my Mdbg python extension, which can be found here. (Update: This is updated for a breaking change in IP 0.9.3 based off feedback here) You can also browse through that source to get context on the steps below. My test python script is here, which will provide non-python users with some basic examples.
I saw there are some toy examples of embedding in the python download. Sometimes toys don't scale to real things. Mdbg is a real app and reasonably large and so I believe it's a valuable demonstration to show that you can reasonably embed IronPython into Mdbg. I also thought it would be valuable to have a non-python guru do this as further proof of how viable it is.
Here are the steps you can follow to add IronPython into your own apps:
1. Add a reference to IronPython.dll to your project. You can either refer to pre-built binaries, or you could add the whole IronPython project to your existing VS solution.
2. Declare the scripting engine. It's of type IronPython.Hosting.PythonEngine. The rest of your python support will need access to this instance. I recommend declaring as a field like:
static IronPython.Hosting.PythonEngine m_python;
3. Allocate the engine and set initial properties.
Just allocate an instance. The default ctor is good enough:
m_python = new IronPython.Hosting.PythonEngine();
There's also an IronPython.AST.Options structure with static fields for different options. These have intelligent defaults, so you can ignore it for now.
4. Set the library search path . This will be used when you try to load pre-existing python scripts (which quickly becomes much easier than typing in all the python commands yourself). This path will be used by python “import” statements and by the PythonEngine::ExecuteFile() method. I suggest at least adding the current directory:
m_python.AddToPath(Environment.CurrentDirectory);
If your app has other search paths (eg, in MDbg, I could use the symbol path), you may want to consider adding that to the python search path too. This gives your end users an easy way to add to the path. It turns out they can also modify it via the "sys.path" python variable.
5. Set default variables.
Python uses reflection to access the hosting application (MDbg in my case) (see Joel's blog about under the hood of dynamic languages for more details). You can call PythonEngine::SetVariable to expose your application objects to python. Python will use reflection over these variables, so it helps to give it things that have good public members. In the MDbg case, you can access everything from the an instance of Microsoft.Samples.Tools.Mdbg.MDbgShell, so we give it one of those:
m_python.SetVariable("Shell", Shell); // let python scripts use the string "Shell" to access the object Shell.
After that statement, I could execute the following python statement:
Shell.IO.WriteOutput("STDOUT", x)
Python would map "Shell" to the Shell object I passed in, and then use reflection to find the "IO" property, reflect again to find the WriteOutput method, and then invoke that.
So even if you don't know any python (like me before I started writing this), you can still get some immediate results by evaluating basic properties on the objects you passed in.
Some methods are not very easy to access from reflection. You could consider creating a utility class that exposes such methods to python in an easy-to-use fashion. Then use SetVariable() to give Python an instance of the class. In my example, I had trouble with calling static methods or methods that took out parameters, so I created a "Util" class to convert cover up out params and use return values.
m_python.SetVariable("MDbgUtil", new Util());
Another tip is that you can create Python function to alias things for you. For example, I could create a python function CurrentFrame() to easily get at the current frame from the Shell object.
def CurrentFrame():
return Shell.Debugger.Processes.Active.Threads.Active.CurrentFrame
6. Issue calls to PythonEngine::Execute() to execute python statements. Now onto the good part. Once you have a PythonEngine instance, you can immediately use it to execute python statements via the Execute() method.
m_python.Execute(args)
For my Mdbg-python extension, I added an Mdbg command, "python", which just passes its arguments straight through to the Execute() method. Execute() takes in a string which can be either a python expression (which gets implicitly wrapped in a python Print statement) or a python statements like “def” (to define a function) and “import” (which loads a python script). You can have multiple statements grouped together separated by a semicolon such as "print 1+2;print 3+4".
Execute() returns void, making it difficult to get the result of an expression. The last expression result is stored as the global variable “_”, so you can retrieve it like so:
object o = m_python.GetVariable("_");
However, if you really need to get the result, considering calling Evaluate instead of Execute as I describe next.
7. Use PythonEngine::Evaluate() for places where you explicitly want a python expression. Note that PythonEngine::Execute() is different than PythonEngine::Evaluate(). Evaluate() takes in a string representing an expression and returns a System.Object for the result of that expression. Evaluate() will give a syntax error if you pass in a statement.
This can be great for filtering things or predicates. In the Mdbg case, I added a new breakpoint subclass ("pyb") which would evaluate a python expression to determine whether the breakpoint should stop. In other words, I created a conditional breakpoint where the condition was an arbitrary python expression. And since python expressions can call functions, and functions can be arbitrarily complex and even access global state, you can now have some pretty powerful conditional breakpoints. I suspect your own apps have similar opportunities for python expressions.
Note that if you pass multiple expressions via a semicolon (eg: "1+2;3+4"), Evaluate() will only evaluate and return the first one.
8. Loading python modules.
Typing everything in from the console doesn't scale, so it's critical that you can load prewritten python scripts. Since python lets you define functions and classes, it's nice to be able to build up libraries and start getting some very cool stuff. There are two ways you can load python files. Both mechanisms deal with short names and use the python library search path specified via PythonEngine::AddToPath().
I recommend to use PythonEngine::ExecuteFile(filename) to load python scripts(aka modules) . This will execute the given file in the "<eval>" scope, which means any functions defined in the file can still access objects provided by SetVariable(). If you do this, then everything just works. Note that filename here is short name with extension (eg: "test.py") and the library search path is used to find the full directory. Your host can load predefined scripts when you first initialize your PythonEngine. You can (and should) also expose some UI to let users load arbitrary python scripts. In my mdbg case, I added a "pimport" command which turns around and just calls ExecuteFile() to let users load scripts.
What about Python's builtin "import" command? Modules loaded via python's "import" command get put into their own module and get their own scope. This means functions defined in those modules can't access variables set via SetVariable() (which go into the "<eval>" scope). I talked to Martin about this and he said he and Jim Huginin were working on a solution.
Also, IronPython's implementation of "import" calls LoadFrom when IronPython.AST.Options.DoNotSaveBinaries = false (which is the default value). LoadFrom is evil (see Suzanne Cook's blog for plenty more reasons). Basically, "import" will compile the python script ("test.py") into its own CLR module ("test.exe") which then has a dependency back to IronPython.dll. LoadFrom may or may not be able to resolve that dependency (even though IronPython.dll is already loaded) (yes, LoadFrom is stupid) and so it means you'll need to hook the AssemblyResolve event to help it resolve the dependency. It's all very annoying. ExecuteFile() avoids this problem because it compiles everything into the same module (called "snippets.dll"), which already has a resolved reference to IronPython.dll. This is also consistent with what I mentioned above that ExecuteFile() uses the same scope as the normal evaluation prompt.
Here's trivia that's significant when you're debugging your host: You can actually debug the python source files, along with the rest of your host, when they're loaded via "import" (IronPython is just another .Net language, so it gets free debugging support). Empirically, you can't debug files loaded via ExecuteFile() - I don't know if this is a bug or not.
9. Entering Python Interactive mode: Hookup the IConsole interface.
So far, the embedded app owns the console / GUI and the input loop, and it forwards commands onto the python engine at its discretion. In the MDbg case, the app already owns the console and already has its own input loop; thus I added a "python" command which can forward input to the python engine via PythonEngine::Execute(). However, this is cumbersome if you want to do a bunch of Python operations in a row. It may also prevents you from pasting multiline text directly into the console. It's also bad if you don't have a console in the first place. For example, suppose you have a game and you want to open a shell that lets the user input python commands.
The way you solve this is by implementing the IronPython.Hosting.IConsole interface. In my case, I declared my own class, MyMdbgConsole, to do that and then I hook it up to the python engine as follows:
m_python.MyConsole = new MyMdbgConsole();
The IConsole interface is very small and just has a ReadLine(), Write(), and WriteLine() method. In my case, the MyMdbgConsole is a bridge to redirect both input and output through MDbg's IMdbgIO interface on the Shell.IO property. Thus if another mdbg extension (such as the mdbg winforms gui) redirects the console to a winform, my python extension gets redirected for free.
You can then call:
m_python.RunInteractive();
to tell give Python control. The PythonEngine will keep reading and executing commands from ReadLine() until that returns Null. It will use Write() and WriteLine() to write the python prompt.
This just redirects the python prompt and input, but it does not redirect the output of actual python commands (eg "Print"). For that, see the next section.
10. Redirecting Python output (sys.stdout) for Gui apps.
If your application is a gui, you'll want to redirect python's output (sys.stdout, sys.stderr, sys.stdin) to the appropriate gui element, such as a tool window. IronPython will let you redirect IO to a System.IO.Stream class. You can supply your own derived implementation of Stream and then direct the IO to anywhere you wish.
Create your Stream derived class. (VS's IDE can automatically provide stubs for all the abstract methods). It turns out you can reasonably make a write-only stream. Make CanRead() and CanSeek() return false, make CanWrite() return true, and the only real implementation you'll need is for Write(byte[] buffer, int offset, int count). See the "MyStream" class I wrote in my sample for an example of what I mean.
Then you can wrap a Stream instance in a IronPython.Objects.PythonFile object and assign that to the sys.std* properties when you initialize your PythonEngine() instance. In other words, include this line as part of your initialization code after instantiate your PythonEngine() instance:
IronPython.Modules.sys.stdin = IronPython.Modules.sys.stderr = IronPython.Modules.sys.stdout = new IronPython.Objects.PythonFile(new MyStream(), "w", false);
I'm told the 0.9.3 release added an extra "name" parameter (shown in blue). So the that would look like this instead:
IronPython.Objects.Ops.sys.stdin = new IronPython.Objects.PythonFile(new MyStream(), "stdin" , "r", false);
IronPython.Objects.Ops.sys.stderr = new IronPython.Objects.PythonFile(new MyStream(), "stderr" , "w", false);
IronPython.Objects.Ops.sys.stdout = new IronPython.Objects.PythonFile(new MyStream(), "stdout" , "w", false);
Now all output from python commands will go through your Stream class and get directed to wherever you put it.
Conclusion:
Overall, this was pretty easy, and the results are great. With very little effort, you can add a fully functional dynamic scripting engine to your existing managed app. There's a whole community around the IronPython engine, Microsoft is also doing active development on it, and all the source is available. That's much better than trying to write your own scripting engine.
I encourage you to check out the IronPython website (https://workspaces.gotdotnet.com/ironpython) and download IronPython 0.9.1 and explore for yourself.