3rd Party Software Installations

XGC uses the following external libraries:

Dependency

Version

Required for XGC

Required for kernels

PETSC

>=3.12.0

YES

NO

ADIOS2

>=2.3.0

YES

NO

FFTW3

Any

YES

NO

Kokkos

>=3.0

YES

YES

LAPACK

Any

YES

YES

Cabana

>=0.3.0

YES

YES

GTest

Any

NO

Testing only

FusionIO

Any

NO

NO

It also depends on PSPLINE and CAMTIMERS, which are inside the repository in lib/, and built automatically if needed.

Maintaining shared installations

The shared installations are located here:

System

Installation Directory

Cori Haswell

/project/projectdirs/m499/Software

Cori KNL

/project/projectdirs/m499/Software

Greene

/p/xgc/Software

Summit

/gpfs/alpine/world-shared/phy122

Theta

/projects/TokamakITER/Software

Traverse

/home/rhager/Software

If you want to update shared installations, please use the following convention when creating subdirectories:

[Software directory]/[Package name]/[version]/[target system (if applicable)]/[compiler]/...

For example:

/project/projectdirs/m499/Software/petsc/3.7.6/cori_knl/intel/…

Each available package has softlinks DEFAULT for [version] and [compiler]. Module files may be provided in the future to make it easier to access 3rd party software that is not installed systemwide.

Installation instructions

You may want to run XGC on an unsupported system or to customize your dependency installations. Here are some examples of installing the dependencies manually. Feedback is appreciated to make sure these stay useful.

Once you have installed the necessary dependencies, you will need to provide CMake with the location of XGC’s dependencies if they are already installed on your system. You can either do this on the CMake configuration line, e.g. adding -DPETSc_ROOT=<path to PETSc>, or by creating an XGC_PLATFORM by analogy to the existing ones, found in the XGC-Devel/CMake directory, e.g.:

cp CMake/find_dependencies_generic.cmake CMake/find_dependencies_new_platform.cmake
export XGC_PLATFORM=new_platform

PETSc

#!/usr/bin/python
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--COPTFLAGS=-fast -no-ipo',
    '--CXXOPTFLAGS=-fast -no-ipo',
    '--FOPTFLAGS=-fast -no-ipo',
    '--download-parmetis',
    '--download-metis',
#    '--download-superlu', does not work with 64 !!!
#    '--download-superlu_dist',
#    '--download-hypre',
#    '--download-mumps',
    '--download-scalapack',
    '--with-ssl=0',
    '--with-cc=cc',
    '--with-clib-autodetect=0',
    '--with-cxx=CC',
    '--with-cxxlib-autodetect=0',
    '--with-debugging=0',
    '--with-fc=ftn',
    '--with-fortranlib-autodetect=0',
#    '--with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/',
    '--with-shared-libraries=0',
    '--with-static-libraries=1',
    '--with-x=0',
    '--with-mpiexec=srun',
    'LIBS=-lstdc++',
    '--with-64-bit-indices',
    '--with-batch=1',
    '--prefix=/project/projectdirs/m499/Software/petsc/bdea225/cori_knl/intel-static/',
    'PETSC_ARCH=arch-cori-knl-opt64-intel',
  ]
  configure.petsc_configure(configure_options)

ADIOS2

mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=$ADIOS2_INSTALL_DIR \
    -DADIOS2_BUILD_EXAMPLES=OFF \
    -DADIOS2_BUILD_TESTING=OFF \
    ..

FFTW

cmake -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=$FFTW_INSTALL_DIR \
..

Kokkos

cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
    -DCMAKE_CXX_STANDARD=14 \
    -DBUILD_TESTING=OFF \
    -DKokkos_ENABLE_OPENMP=ON \
    -DKokkos_ENABLE_SERIAL=ON \
    -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \
    ..

Cabana

cmake \
    -DCMAKE_BUILD_TYPE="Release" \
    -DCMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR \
    -DCMAKE_INSTALL_PREFIX=$CABANA_INSTALL_DIR \
    -DCabana_ENABLE_CAJITA=OFF \
    -DCabana_ENABLE_TESTING=Off \
    -DCabana_ENABLE_EXAMPLES=Off \
    ..

GTest

cmake \
    -DCMAKE_INSTALL_PREFIX=/Users/aaronscheinberg/Projects/googletest/install \
    -DCMAKE_CXX_STANDARD=14 \
    ..