Running PETSC on windows


 

PETSc, stands for portable, extensible toolkit for scientific computation, it is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations.  It employs the MPI standard for all message-passing communication.  There are hundreds of applications that uses petsc in the area of nano-simulation, medical, fusion, Geo sciences, surface flow, CFD, optimization, and much more.  

As one of the most important scientific libraries that run on top of MPI, I decided to try it out for our windows HPC cluster.  It turned out that it was quite easy to get running using visual c++ 2008, Microsoft windows MPI stack, and cygwin as a build environment. 

I took some simple notes to help anyone that needs to get it running on windows:

1. Install the following:   petsc, windows HPCS 2008 SDK, Microsoft Visual C++ 2008, and Cygwin with python.

2. run visual studio command prompt for 32 bit build, and visual studio command prompt win64 for 64bit builds…

This ensures that you have CL (the ms c++ compiler in your path). Then, run c:\cygwin\cygwin.bat to get into the cygwin environment.

3. unpack petsc:  gzip -cd petsc*.tar.gz |tar xf –

 

4. cd into the petsc directory, then Configure petsc, but first run:  export PETSC_DIR=`pwd` # THIS SETS UP bash environment for you.

5. Run the command below (one line), this is a 32 bit build.   You will have to change the path if you are building 64bit to x64 instead of i386.

config/configure.py –with-cc=”win32fe cl” –with-fc=0 –download-c-blas-lapack=1
–with-mpi-include=”/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include”
–with-mpi-lib=”/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Lib/i386/msmpi.lib”
–useThreads=0 –with-shared=0

6.(only 32bit version).  

edit python/BuildSystem/config/packages/MPI.py and comment out the line
> >     self.functions        = [‘MPI_Init’, ‘MPI_Comm_create’

You’ll also need to patch up the code, because 32 bit MSMPI stack uses STDCALL convention, thus it needs a little patch from petsc folks. to add MSMPI.  64 bit always uses fastcall convention, thus this step is not needed.

The patch file basically puts a MPIAPI macro in front of all their calls.  This patch is either available upon request or by emailing petsc support and refer to [PETSC #17869].

7.  Once this is all configured, type make all.

This will take a few hours……  it’s a BIG library to compile.

8.  Let’s build example 23….in dir:  ~/petsc-2.3.3-p13/src/ksp/ksp/examples/tutorials

type make ex23

$ make ex23

output:
/cygdrive/c/Users/wenmingy/petsc-2.3.3-p13/bin/win32fe/win32fe cl -o ex23.o -c –
wd4996 -MT -Z7 -I/cygdrive/c/Users/wenmingy/petsc-2.3.3-p13/src/dm/mesh/sieve -I
/cygdrive/c/Users/wenmingy/petsc-2.3.3-p13 -I/cygdrive/c/Users/wenmingy/petsc-2.
3.3-p13/bmake/cygwin-c-debug -I/cygdrive/c/Users/wenmingy/petsc-2.3.3-p13/includ
e -I/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include -D__SDIR__=”sr
c/ksp/ksp/examples/tutorials/” ex23.c

Petsc shows you all the link options…

9.  Now, let’s run it: 

$ mpiexec -n 4  ex23     // running 4 processors locally
KSP Object:
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-030
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-007, absolute=1e-050, divergence=10000
  left preconditioning
PC Object:
  type: jacobi
  linear system matrix = precond matrix:
  Matrix Object:
    type=mpiaij, rows=100, cols=100
    total: nonzeros=298, allocated nonzeros=700
      not using I-node (on process 0) routines
Norm of error 0.000101234, Iterations 502
  

10.  That’s it, you just solved it.  and you can submit this job to the compute cluster job scheduler, but I am on my laptop, that would be another blog.

As you can see, with some minor tweak, PETSc plays nicely with MSMPI stack.

I would also like to thank Serguei O. for his help!


Comments (7)

  1. leberecht says:

    Have you guys really tried this on Windows Server 2008 x64? The Cygwin python is broken as described in http://cygwin.com/ml/cygwin/2007-12/msg00441.html . We can’t cross-compile, since the configure will try to run the x64 bins on an x86 machine and we are kind of stuck. What am I missing?

  2. hpctrekker says:

    My build env was vista 64 bit & win 7beta 64bit.

    Some additional notes,  the patch mentioned above is no longer needed for petsc2.3.3p15.

    My configure line for 64 bit:

    Start visual studio x64 command prompt.  start cygwin by typing: c:cygwincygwin.bat within that command prompt.  

    cd into petsc directory.

    export PETSC_DIR=`pwd`

    make sure win32fe is your path.

    Then here’s the configure line:

    ./config/configure.py –with-mpi-include="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include"  –with-mpi-lib="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Lib/amd64/msmpi.lib"  –with-fortran=0 –with-x=0 –with-debugging=no –useThreads=0 –with-shared=0 –download-c-blas-lapack=1 –with-cc="win32fe cl –nodetect" -AR="win32fe lib"

    You can swap cl –nodetect with icl –nodetect if you are using intel compilers.

  3. hpctrekker says:

    Just tried to build  with intel compilers.  Only issue I had was that you must run it with –with-batch, then restart the configure process…  

    ./config/configure.py –with-mpi-include="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include"  –with-mpi-lib="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Lib/amd64/msmpi.lib"  –with-fortran=0 –with-x=0 –with-debugging=no –useThreads=0 –with-shared=0 –download-c-blas-lapack=1 –with-cc="win32fe icl –nodetect" -AR="win32fe lib" –with-batch

    then follow instructions….

  4. The 10th LCI ( Linux Cluster Institute ) international Conference on High Performance Clustered Computing

  5. Alp says:

    I did exactly the below

    Wenming Ye:

    "

    My build env was vista 64 bit & win 7beta 64bit.

    Some additional notes,  the patch mentioned above is no longer needed for petsc2.3.3p15.

    My configure line for 64 bit:

    Start visual studio x64 command prompt.  start cygwin by typing: c:cygwincygwin.bat within that command prompt.  

    cd into petsc directory.

    export PETSC_DIR=`pwd`

    make sure win32fe is your path.

    Then here's the configure line:

    ./config/configure.py –with-mpi-include="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Include"  –with-mpi-lib="/cygdrive/c/Program Files/Microsoft HPC Pack 2008 SDK/Lib/amd64/msmpi.lib"  –with-fortran=0 –with-x=0 –with-debugging=no –useThreads=0 –with-shared=0 –download-c-blas-lapack=1 –with-cc="win32fe cl –nodetect" -AR="win32fe lib"

    "

    but I get an error message, "Cannot find a C preprocessor"

    What should I do? Any suggestions?

    Thank you…

  6. Henrik says:

    We have  built a nice version of PETSc with a Windows installer. You can find it here:

    http://www.mic-tc.ch/…/PETScforWindows.aspx