Compiling JOREK

  • Check the code out from the git repository as explained here.
  • Compile JOREK as explained at the end of this page.

Specialized guides:

SCOTCH

Scotch is a software package for graph and mesh/hypergraph partitioning, graph clustering, and sparse matrix ordering.

tar -xvzf scotch_5.1.12b.tar.gz

make clean
make -j 8 scotch
make -j 8 ptscotch

PaStiX (before version 6.x)

PaStiX is a high performance parallel solver for very large sparse linear systems based on direct methods. It provides also an adaptive blockwise iLU(k) factorization that can be used as a parallel preconditioner.

tar -xvjf pastix_5.2.3.tar.bz2

make clean
make -j 8
make -j 8 install
  • It is possible to validate the PaStiX installation after it has been compiled:
# simplest example
make examples
mpirun -np <Nprocs> ./example/bin/simple -rsa matrix/small.rsa -t <nthreads> -v 4

# Calling PaStiX with separated steps (as in Jorek)
mpirun -np <Nprocs> ./example/bin/step-by-step -rsa matrix/small.rsa -t <nthreads> -v 4

# Calling PaStiX with several communicators (as in Jorek)
mpirun -np <Nprocs> ./example/bin/multi-comm -rsa matrix/small.rsa -t <nthreads> -v 4

PaStiX 6

The new version of PaStiX (with new features such as block-low-rank compression).

tar -xvzf pastix-6.0.2.tar.gz
  • Go to the PaStiX 6 base directory, create a build subdirectory and build with CMake. Several options can be set, most of which will be set automatically but you might need to specify your ordering library (Scotch or Metis) and the compiler used to compile it, as well as the installation prefix (where PaStiX 6 should be installed). In this case, the command line argument is as follows (replacing ... with the appropriate paths):
mkdir build
cd build
cmake .. -DSCOTCH_DIR=... -DCMAKE_C_COMPILER=... -DCMAKE_INSTALL_PREFIX=...
  • Just like previous versions, PaStiX 6 can now be installed from the PaStiX home directory with
make clean
make -j 8
make -j 8 install

FFTW

JOREK also uses FFTW for fast Fourier transformations. This library is usually available on machines via module load fftw. See also here to include FFTW.

Compile JOREK

We use Make to compile JOREK, with dependency generation via util/makedepend. See Makefile for more information.

  • Create the JOREK configuration file trunk/Makefile.inc for your system, examples are available in trunk/Make.inc/.
  • Add the following lines to link to the libraries:
# SCOTCH Library: Automatically included via the pastix-conf script calls below

# PaStiX Library:
USE_PASTIX        = 1
USE_PASTIX_MURGE  = 1  # Different way for matrix assembly, the choice is up to you
PASTIX_HOME       = <path-to-pastix>/install/
LIB_PASTIX        = `$(PASTIX_HOME)/pastix-conf --libs`
LIB_PASTIX_MURGE  = `$(PASTIX_HOME)/pastix-conf --libs_murge`
LIB_PASTIX_BLAS   = `$(PASTIX_HOME)/pastix-conf --blas`
INC_PASTIX        = `$(PASTIX_HOME)/pastix-conf --incs`

# FFTW Library: FFTW_HOME might be set by "module load fftw", otherwise do so manually
LIBFFTW           = $(FFTW_HOME)/lib/libfftw3.a
  • For PaStiX after revision 3945, additionally the following is required (see also preprocessor flags):
FFLAGS := $(FFLAGS) -DFUNNELED -DWORLDWAR2

  • In case of PaStiX 6.x, instead of the parameters shown above, USE_PASTIX6=1, PASTIX6_HOME, LIB_PASTIX6, LIB_PASTIX6_BLAS, and INC_PASTIX6 need to be set correctly. In that case, USE_PASTIX must be 0.

  • Set the hard-coded parameters correctly. For example:

./util/config.sh model=303 n_tor=3 n_period=8 n_plane=8   # set hard-coded parameters
./util/config.sh                                          # check hard-coded parameters
  • Compile the main program and/or diagnostics tools:
make                       # compile main program jorek_modelxxx
make jorek2vtk
make jorek2_postproc
...
  • Parallel compilation is possible, e.g. with 8 threads:
make -j 8

Special Compile Options for Single Source Files

  • It is now possible to specify special compile options (added to the normal FFLAGS) for individual source files. An example for the file models/model333/element_matrix_fft.f90 (to put in Makefile.inc):
.obj/element_matrix_fft.o: FFLAGS+=-O3

Remarks

  • When switching the physics model or hard-coded parameters, you should always call a make clean before recompiling. When dependencies between files differ for different models (preprocessor directives), this is not sufficient and a make cleanall is required. ./util/config.sh has been adapted to do this automatically.
  • Compiler Optimization: (Preliminary -- will be updated)
    • Tests in Garching on our local Linux Cluster and on our supercomputer Hydra showed that the best optimization options seem to be simply -fast -DUSE_BLOCK without anything else like -O2 -xSSE4.2 etc. Both systems are equipped with Intel Ivy Bridge processors very similar to Helios.