Build Instructions

  • XGC has several external dependencies. If you are building XGC at one of the following HPC facilities, indicate which one by setting the environment variable XGC_PLATFORM before configuring, e.g.:

    export XGC_PLATFORM=summit
    

    System

    XGC_PLATFORM

    Cori Haswell

    cori_haswell

    Cori KNL

    cori_knl

    Crusher

    crusher

    Greene

    greene

    JLSE

    jlse

    Perlmutter

    perlmutter_nvhpc

    Perlmutter CPU Nvidia

    perlmutter_cpu_nvidia

    Polaris

    polaris

    Spock

    spock

    Stellar

    stellar

    Summit

    summit

    Theta

    theta

    Traverse

    traverse

    After that, please set the environment variable and load the modules specified in the section at the end of this page.

    If you are not using one of the above facilities or want custom-built dependencies, then you have two options:

    1. Install the libraries manually (see 3rd Party Software Installations)

    2. Use the Superbuild, which will download, build, and install its dependencies for you. To enable this feature, pass

-DBUILD_DEPENDENCIES=ON to cmake when configuring the XGC build below.

  • Load modules and set environment variables

    Compiling and running on the supported systems may require modules and environment variables. See Building at HPC Facilities at the bottom of this page for the commonly used ones on your system.

  • Create and enter a directory.

    mkdir build; cd build
    
  • Run CMake to configure a build of XGC.

    cmake ..
    

    Additional settings can be passed as -D flags, e.g.:

    cmake -DBUILD_DEPENDENCIES=ON -DCONVERT_GRID2=ON -DSOLVERLU=OFF ..
    

    To interactively edit configuration settings, use ccmake . for a CLI or cmake-gui for a GUI.

    For a full list of XGC configure options, see XGC Preprocessor Macros.

  • Build all available targets:

    make -j
    

    Or just the one you want, e.g.:

    make -j xgc-es-cpp
    

    The executables will be in build/bin. Current available targets are: xgc-es-cpp, xgc-es-cpp-gpu, xgca-cpp, xgca-cpp-gpu, xgc-eem-cpp, xgc-eem-cpp-gpu, as well as kernels and tests (see Kernels and Tests).

Environment at HPC facilities

Compiling and running on the supported systems may require modules and environment variables. Below are the commonly used sets.

Cori

The following script sets the environment for the Haswell partition (same as login nodes):

source /project/projectdirs/m499/Software/bin/nersc_config

Cori (KNL partition)

For switching between the Haswell and KNL partitions use:

source /project/projectdirs/m499/Software/bin/haswell2knl
source /project/projectdirs/m499/Software/bin/knl2haswell

Crusher

 module reset
 module unload perftools-base
 module load cmake
 module load PrgEnv-amd
 module swap amd amd/5.1.0
 module load rocm/5.1.0
 export CRAYPE_LINK_TYPE=dynamic
 export PATH=${CRAY_MPICH_PREFIX}/bin:${PATH}
 export PATH=${ROCM_COMPILER_PATH}/bin:${PATH}
 export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0
 export MPICH_GPU_SUPPORT_ENABLED=1
 export XGC_PLATFORM=crusher
 export OMP_NUM_THREADS=4
 export OMP_PROC_BIND=true
 export MPICH_CXX=`which hipcc`

Extra CMake variables needed:
-DCMAKE_CXX_COMPILER=${CRAY_MPICH_PREFIX}/bin/mpicxx
-DCMAKE_C_COMPILER=${CRAY_MPICH_PREFIX}/bin/mpicc
-DCMAKE_Fortran_COMPILER=${CRAY_MPICH_PREFIX}/bin/mpifort

Greene

source /p/xgc/Software/greene_config

Perlmutter

module unload gpu
module load cmake/3.22.0
module load nvidia/22.7
module load cudatoolkit/11.7
module load cray-libsci/22.06.1.3
module load cray-fftw/3.3.10.1
module unload darshan
export XGC_PLATFORM=perlmutter_nvhpc
export CRAYPE_LINK_TYPE=dynamic
export NVCC_WRAPPER_DEFAULT_COMPILER=CC

Perlmutter CPU Nvidia

module unload gpu
module load cmake/3.22.0
module load nvidia/22.7
module load cray-libsci/22.06.1.3
module load cray-fftw/3.3.10.1
module unload darshan
export XGC_PLATFORM=perlmutter_cpu_nvidia
export CRAYPE_LINK_TYPE=dynamic

cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..

Polaris

module load cmake kokkos cabana cray-fftw
export XGC_PLATFORM=polaris

cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..

Stellar

source /home/amollen/Software/bin/set_up_xgc.stellar

Summit

module load nvhpc/21.7
module load spectrum-mpi
module load python
module load netlib-lapack
module load hypre
module load fftw
module load hdf5
module load cmake/3.20.2
module load libfabric/1.12.1-sysrdma
export XGC_PLATFORM=summit
export OMP_NUM_THREADS=14
export FC=mpifort
export CC=mpicc
export NVCC_WRAPPER_DEFAULT_COMPILER=mpiCC
export CXX=/gpfs/alpine/world-shared/phy122/lib/install/summit/kokkos/nvhpc21.7/bin/nvcc_wrapper

Theta

module load cray-libsci
module load cray-hdf5-parallel
module load cray-fftw
module use -a /projects/TokamakITER/Software/modulefiles
module load adios2/DEFAULT
module load cmake/3.20.4
export CRAYPE_LINK_TYPE=dynamic
export XGC_PLATFORM=theta

Traverse

  source /home/rhager/Software/bin/set_up_xgc.traverse

CMake version 3.22.2 is available and can be used via ``xgc_cmake`` and ``xgc_ccmake``.