Embedding Python in a C++ project with Visual Studio

Steve Dower

Watch the video version of this post on VS Toolbox

Let’s get started

In this post, we’re going to walk through a sample project that demonstrates scripting a C++ application with Python using CPython, PyBind11 and Visual Studio 2017. The sample code is available at github.com/zooba/ogre3d-python-embed and setup instructions are below.

Ogre3d is an open-source game engine written in C++ that has been used in games such as Hob and Torchlight 2. Both the engine and its source code are freely available from their website. For this sample, we have taken one of their character animation demos and extended it with Python. Rather than using the keyboard to move the character around, we can use Python code to call into the C++ functions that control him.

To build and run this sample on your own machine, you will require Visual Studio 2017 with the Python workload, the Python Native Development option, and Python 3.6 32-bit. If you already have Visual Studio 2017, these can be added by launching “Visual Studio Installer” and modifying the existing install.

Note: When you install Python 3.6 32-bit through Visual Studio, it automatically includes debugging symbols. If you install it yourself, you will need to select “Customize installation” and include debugging symbols. If you have already installed it, you can use Programs and Features to modify your install and add debugging symbols.

Clone our repository using git clone --recurse-submodules https://github.com/zooba/ogre3d-python-embed.git or using Team Explorer in Visual Studio. There is a PowerShell script in the root of the repository called get_externals.ps1 that will download and extract the version of the Ogre3d and SDL2 runtimes needed, and will prompt if you are missing either Python 3.6 32-bit or the DirectX redistributables (you can download the latter here, but be aware that this is a very old installer that will offer to install a browser toolbar – feel free to deselect that).

Once everything is installed, open srcPythonCharacter.sln in Visual Studio 2017 and press Ctrl+F5 to build and run the sample. While running, the sample will capture your mouse cursor, but you can use Alt+Tab to switch to another window. We will do that next to look at some of the code.

In Visual Studio, open Solution Explorer and then open the following files. We will be looking at each in the next few sections.

• ogre.pyi
• ogre_module.h
• SinbadCharacterController.h
• ai.py

Modifying without recompiling

The Python module ai.py is where we define the behavior of Sinbad, our dancing ogre. Collapse the command definitions region by clicking the “-” symbol to the left of #region Command definitions and look at the SCRIPT variable. Each item in this list is the movement we want Sinbad to do, including the amount of time he should do it for. There are already some extra movements listed but commented out, so try uncommenting them or adding your own. You can do this while the demo is running in the background.

Once you’ve made changes, save this file, use Alt+Tab to go back to the running sample, and press F5. The F5 key will reload the script, and you will see your changes (or an error) immediately. Compare this to normal C++ development, where you would have had to stop running, modify the code, recompile (and wait!), start running again, and return to the same state you were previously.

This is possible because of CPython’s support for embedding, and made simple by the powerful pybind11 library. Embedding allows you to host the Python runtime in any native application, on any platform and using any compiler supported by CPython. So rather than launching “python.exe” with a script, you can load python36.dll into your own application and use it directly.

It is very easy to make Python representations of your C++ classes with pybind11. Switch to the ogre_module.h file to see what we did for this sample.

This file defines the mapping between a Python module named “ogre” and the classes and functions we want to make available. Using the metaprogramming features added in C++11, the pybind11 library automatically generates code to do the type conversions necessary to make Python code transparently interact with your C++ code. If you switch back to ai.py, you will see that the ogre.CharacterController class is imported and used in Python code to call back into C.

But how can we be sure that it is really doing all this? It seems pretty magical, and barely enough work on our part to make a game engine suddenly support Python scripting. In the next section, we will look at the proof that it is doing what we claim.

Debugging Python and C++

If you’ve got the demo running, now is the time to exit it by clicking on the Stop Debugging button. Visual Studio is famous for its debugging features, and this one is pretty cool. When you installed the Python Native Development option, we included the ability to do mixed Python/C++ debugging, whether you’re in a Python project or a C++ project. You can find information about doing this from Python project in our documentation, but in this case we are going to launch a C++ project with Python debugging enabled.

Find the Start debugging button on the toolbar. Depending on your settings and the file you have open, it be labelled “Start”, “Local Windows Debugger”, “Python/Native Debugging” or “Attach…”. Clicking the small down arrow next to the button will display your options.

Select “Python/Native Debugging” here to make it your new default. All the settings required to make the sample correctly load are already configured for this project (see this page for the details), so you can now press F5 to launch the sample again, but this time with the debugger attached.

Open ai.py again and set a breakpoint in the on_frame function near the end of the file. This function is called for each frame, but normally returns quickly until it’s time to run the next step in the script. So while the demo is running, sooner or later this function will be hit. When it is, you’ll see a mixed call stack showing both Python and C++ frames. (In this screenshot, I’ve hidden external code, so you may see some extra frames from pybind11.)

As you press F11 to step through the code, you will jump between Python and C++ just as naturally as either language on its own, and anywhere you view a Python object you’ll be able to see the regular Python view. Breakpoints set in native code or Python code will be hit, and switching up and down the call stack will let you view both kinds of frames.

Type hints for Python code

Finally, let’s take another look at the Python code and how Visual Studio helps you be a productive developer. In general, when you have defined a Python class in C++ you are not going to get IntelliSense when editing the code using it. This is unfortunate, as IntelliSense is most useful when using code that does not have easily read sources, but we now have the ability to provide IntelliSense separately.

Open the ogre.pyi file. While it looks very much like Python code, it’s actually a type stub file that exists solely for its type hints. We include class and function definitions, but no bodies. With function annotations, we can specify the expected types and the return type of each function, and we will also extract and present documentation strings.

As this file is named ogre.pyi, when the ai.py module imports the native ogre module, IntelliSense will import the type stub instead. Without the type stub, we would not be able to resolve the ogre module at all, as it is generated at runtime by pybind11 and there is no other way for us to find it.

Switch back to ai.py and find the on_frame function. As this is called from C++, we have no information about the arguments that are passed to it, so to get IntelliSense we use type annotations to specify the parameter types. If you start typing character. within this function then you will see all the members that were specified in the type stub.

While this is very convenient and necessary in some situations, most of the time we are able to show you good IntelliSense without needing type hints. If you hover over command in this function you will see all the possible command types; using Go To Definition (F12) on command.execute will take you to either of the implementations of that function; and even though there are no type hints on these functions, we are still able to provide all the completions on character through our type analysis engine. Type stub files are useful for when you want to provide users with IntelliSense for code that cannot be automatically analyzed, and annotations can fill in the gaps that occur when using complex or unusual Python code.

Summary

If you develop native applications that have frequently-changing logic or business rules, it is easy to move that logic from C or C++ into Python, where it can be easily modified and updated without needing to recompile or restart your application. The combination of Visual Studio 2017 with the official CPython releases and pybind11 is the most productive way to develop and debug all aspects of your hybrid C++ and Python project. Download the free Visual Studio 2017 Community to get started today.

Steve Dower Software Engineer, Python

Follow