3rd Party Software Installations

XGC uses the following external libraries:

Dependency	Version	Required for XGC	Required for kernels
PETSc	>=3.12.0	YES	NO
ADIOS2	>=2.3.0	YES	NO
FFTW3	Any	YES	NO
Kokkos	>=3.0	YES	YES
LAPACK	Any	YES	YES
Cabana	>=0.3.0	YES	YES
GTest	Any	NO	Testing only
FusionIO	Any	NO	NO
GINKGO	Any(**)	NO	NO

**N.B.: Please use the “batch_develop” branch of Ginkgo.

It also depends on PSPLINE and CAMTIMERS, which are inside the repository in lib/, and built automatically if needed.

Maintaining shared installations

The shared installations are located here:

System	Installation Directory
Aurora	/gecko/XGC_aesp_CNDA/Software
Cori Haswell	/project/projectdirs/m499/Software
Cori KNL	/project/projectdirs/m499/Software
Greene	/p/xgc/Software
Stellar	/projects/XGC/STELLAR/Software
Theta	/projects/TokamakITER/Software
Traverse	/projects/XGC/TRAVERSE/Software

If you want to update shared installations, please use the following convention when creating subdirectories:

[Software directory]/[Package name]/[version]/[target system (if applicable)]/[compiler]/...

For example:

/project/projectdirs/m499/Software/petsc/3.7.6/cori_knl/intel/…

Each available package has softlinks DEFAULT for [version] and [compiler]. Module files may be provided in the future to make it easier to access 3rd party software that is not installed systemwide.

Installation instructions

You may want to run XGC on an unsupported system or to customize your dependency installations. Here are some examples of installing the dependencies manually. Feedback is appreciated to make sure these stay useful.

Once you have installed the necessary dependencies, you will need to provide CMake with the location of XGC’s dependencies if they are already installed on your system. You can either do this on the CMake configuration line, e.g. adding -DPETSc_ROOT=<path to PETSc>, or by creating an XGC_PLATFORM by analogy to the existing ones, found in the XGC-Devel/CMake directory, e.g.:

cp CMake/find_dependencies_generic.cmake CMake/find_dependencies_new_platform.cmake
export XGC_PLATFORM=new_platform

PETSc

#!/usr/bin/python
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--COPTFLAGS=-fast -no-ipo',
    '--CXXOPTFLAGS=-fast -no-ipo',
    '--FOPTFLAGS=-fast -no-ipo',
    '--download-parmetis',
    '--download-metis',
#    '--download-superlu', does not work with 64 !!!
#    '--download-superlu_dist',
#    '--download-hypre',
#    '--download-mumps',
    '--download-scalapack',
    '--with-ssl=0',
    '--with-cc=cc',
    '--with-clib-autodetect=0',
    '--with-cxx=CC',
    '--with-cxxlib-autodetect=0',
    '--with-debugging=0',
    '--with-fc=ftn',
    '--with-fortranlib-autodetect=0',
#    '--with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/',
    '--with-shared-libraries=0',
    '--with-static-libraries=1',
    '--with-x=0',
    '--with-mpiexec=srun',
    'LIBS=-lstdc++',
    '--with-64-bit-indices',
    '--with-batch=1',
    '--prefix=/project/projectdirs/m499/Software/petsc/bdea225/cori_knl/intel-static/',
    'PETSC_ARCH=arch-cori-knl-opt64-intel',
  ]
  configure.petsc_configure(configure_options)

ADIOS2

mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=$ADIOS2_INSTALL_DIR \
    -DADIOS2_BUILD_EXAMPLES=OFF \
    -DADIOS2_BUILD_TESTING=OFF \
    ..

FFTW

cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$FFTW_INSTALL_DIR \
..

Kokkos

cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
    -DCMAKE_CXX_STANDARD=17 \
    -DBUILD_TESTING=OFF \
    -DKokkos_ENABLE_OPENMP=ON \
    -DKokkos_ENABLE_SERIAL=ON \
    -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \
    ..

An additional option is used for Cori:

-DKokkos_ENABLE_LIBDL=OFF \

Additional options for CUDA:

-DKokkos_ENABLE_CUDA=ON \
-DKokkos_ENABLE_CUDA_LAMBDA=ON \
-DKokkos_ARCH_VOLTA70=ON \
-DKokkos_ARCH_POWER9=ON \

Cabana

cmake \
    -DCMAKE_BUILD_TYPE="Release" \
    -DCMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR \
    -DCMAKE_INSTALL_PREFIX=$CABANA_INSTALL_DIR \
    -DCabana_ENABLE_CAJITA=OFF \
    -DCabana_ENABLE_TESTING=Off \
    -DCabana_ENABLE_EXAMPLES=Off \
    ..

GTest

cmake \
    -DCMAKE_INSTALL_PREFIX=/Users/aaronscheinberg/Projects/googletest/install \
    -DCMAKE_CXX_STANDARD=14 \
    ..

Ginkgo

For CoriGPU:

cmake \
        -DCMAKE_CXX_COMPILER=nvc++ \
        -DGINKGO_COMPILER_FLAGS=-pedantic \
        -DCMAKE_INSTALL_PREFIX=/global/homes/d/dhruva/XGC-Devel/ginkgo/build/install_ompoff \
        -DGINKGO_CUDA_ARCHITECTURES=Volta \
        -DGINKGO_BUILD_OMP=OFF \
        -DCMAKE_CXX_FLAGS=-Mint128 \
        -DCMAKE_BUILD_TYPE:STRING=RELEASE \
        -DGINKGO_BUILD_TESTS=off \
        -DGINKGO_BUILD_BENCHMARKS=off \
        -DGINKGO_BUILD_EXAMPLES=off \
        -DGINKGO_BUILD_HWLOC=off \
        ..