Microsoft Graph Engine is a distributed in-memory data processing engine. It’s an open source project but the official instructions claim that for Linux, it only supports Ubuntu 16.04. But what if you’re using other distributions of Linux, like Red Hat? Well, it’s possible but it does take quite some effort to build the source. Let’s get started.
Prerequisites:
- You need 64 G of RAM. Otherwise, you’ll run into “virtual memory exhausted” errors during the build process.
- As of 12/11/2018, if you build the latest commit of the source, the output binary won’t work with Linux. I filed this issue and until it’s resolved, you should check out earlier commits.
Install required software packages:
You need git, dotnet, openssl, and c++
sudo yum upgrade -y sudo yum install git rh-dotnet22 openssl-devel -y sudo yum install gcc-c++ libgcc.i686 glibc-devel.i686
Build GCC
No, devtoolset won’t work due to ABI compatibility issue. You HAVE TO build GCC 5.4 or higher. This also takes the longest time (a couple of hours).
Please follow the official instruction. I chose GCC 7.4 and I only built C & C++. So the commands I used were
wget ftp://ftp.gnu.org/gnu/gcc/gcc-7.4.0/gcc-7.4.0.tar.gz $PWD/../gcc-7.4.0/configure --prefix=$HOME/GCC-7.4.0 --enable-languages=c,c++
Set the right environment variables
These are necessary for building CMake and later the Graphic Engine.
export PATH=~/GCC-7.4.0/bin:$PATH export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/GCC-7.4.0/lib:~/GCC-7.4.0/lib64 export CC=~/GCC-7.4.0/bin/gcc export CXX=~/GCC-7.4.0/bin/g++
Build CMake
Since CMake 3 is needed, you have to build it from its source. It’s very quick though. One thing to note is, when doing the “make install”, you have to be root. You need to re-export LD_LIBRARY_PATH
sudo -s export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<absolutepathtoGCClib>:<absolutepathtoGCClib64> make install
Build Graph Engine
Finally, we’re ready to build the source of Microsoft Graph Engine. Exciting?! As mentioned earlier, I checked out an earlier commit:
git clone https://github.com/Microsoft/GraphEngine.git git checkout 4175d13a084453bd2a2bf43adf1cc4e76f9fe417 cd GraphEngine/ scl enable rh-dotnet22 bash tools/build.sh
Voila, you have the Nuget package built under the build/ folder!