Build Instructions¶
XGC has several external dependencies. If you are building XGC at one of the following HPC facilities, indicate which one by setting the environment variable
XGC_PLATFORM
before configuring, e.g.:export XGC_PLATFORM=summit
System
XGC_PLATFORM
Cori Haswell
cori_haswell
Cori KNL
cori_knl
Crusher
crusher
Greene
greene
JLSE
jlse
Perlmutter GCC
perlmutter_gcc
Perlmutter Nvidia
perlmutter_nvhpc
Perlmutter CPU Nvidia
perlmutter_cpu_nvidia
Polaris
polaris
Spock
spock
Stellar
stellar
Summit
summit
Theta
theta
Traverse
traverse
After that, please set the environment variable and load the modules specified in the section at the end of this page.
If you are not using one of the above facilities or want custom-built dependencies, then you have two options:
Install the libraries manually (see 3rd Party Software Installations)
Use the Superbuild, which will download, build, and install its dependencies for you. To enable this feature, pass
-DBUILD_DEPENDENCIES=ON
to cmake when configuring the XGC build below.
Load modules and set environment variables
Compiling and running on the supported systems may require modules and environment variables. See Building at HPC Facilities at the bottom of this page for the commonly used ones on your system.
Create and enter a directory.
mkdir build; cd build
Run CMake to configure a build of XGC.
cmake ..
Additional settings can be passed as
-D
flags, e.g.:cmake -DBUILD_DEPENDENCIES=ON -DCONVERT_GRID2=ON -DSOLVERLU=OFF ..
To interactively edit configuration settings, use
ccmake .
for a CLI orcmake-gui
for a GUI.For a full list of XGC configure options, see XGC Preprocessor Macros.
Build all available targets:
make -j
Or just the one you want, e.g.:
make -j xgc-es-cpp
The executables will be in
build/bin
. Current available targets are:xgc-es-cpp
,xgc-es-cpp-gpu
,xgca-cpp
,xgca-cpp-gpu
,xgc-eem-cpp
,xgc-eem-cpp-gpu
, as well as kernels and tests (see Kernels and Tests).
Environment at HPC facilities¶
Compiling and running on the supported systems may require modules and environment variables. Below are the commonly used sets.
Cori¶
The following script sets the environment for the Haswell partition (same as login nodes):
source /project/projectdirs/m499/Software/bin/nersc_config
Cori (KNL partition)¶
For switching between the Haswell and KNL partitions use:
source /project/projectdirs/m499/Software/bin/haswell2knl
source /project/projectdirs/m499/Software/bin/knl2haswell
Crusher¶
module reset
module unload perftools-base
module load cmake
module load PrgEnv-amd
module swap amd amd/5.2.0
module swap cray-mpich cray-mpich/8.1.17
module load craype-accel-amd-gfx90a
export ROCM_PATH=/opt/rocm-5.2.0
export OLCF_ROCM_ROOT=${ROCM_PATH}
export CRAYPE_LINK_TYPE=dynamic
export PATH=${CRAY_MPICH_PREFIX}/bin:${PATH}
export PATH=${ROCM_COMPILER_PATH}/bin:${PATH}
export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0
export MPICH_GPU_SUPPORT_ENABLED=1
export XGC_PLATFORM=crusher
export OMP_NUM_THREADS=8
export OMP_PROC_BIND=true
export MPICH_CXX=`which hipcc`
Extra CMake variables needed:
-DCMAKE_CXX_COMPILER=${CRAY_MPICH_PREFIX}/bin/mpicxx
-DCMAKE_C_COMPILER=${CRAY_MPICH_PREFIX}/bin/mpicc
-DCMAKE_Fortran_COMPILER=${CRAY_MPICH_PREFIX}/bin/mpifort
Greene¶
source /p/xgc/Software/greene_config
Perlmutter GCC¶
March 14, 2023 update: GNU/GCC XGC build with GPU-awre MPI is crashing compute nodes due to an issue with Perlmutter software. Until the issue is resolved, please build XGC with the Nvidia compiler (without GPU-aware MPI).
module load cmake
module load cray-fftw
module unload darshan
module unload cray-libsci
export XGC_PLATFORM=perlmutter_gcc
export CRAYPE_LINK_TYPE=dynamic
export NVCC_WRAPPER_DEFAULT_COMPILER=CC
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn -DUSE_GPU_AWARE_MPI=ON ..
Perlmutter Nvidia¶
March 14, 2023 update: the Nvidia compiler is the recommended compiler for building XGC on Perlmutter.
Note: as of March 10, 2023, still cannot build an XGC binary with GPU-aware MPI with Nvidia compiler. Working with vendor to resolve the issue.
module unload gpu
module load cmake
module load PrgEnv-nvidia
module swap nvidia nvidia/22.9
module load cray-fftw
module unload darshan
module unload cray-libsci
export XGC_PLATFORM=perlmutter_nvhpc
export CRAYPE_LINK_TYPE=dynamic
export NVCC_WRAPPER_DEFAULT_COMPILER=CC
export CPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.9/math_libs/11.7/targets/x86_64-linux/include:$CPATH
export LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/22.9/math_libs/11.7/targets/x86_64-linux/lib:$LIBRARY_PATH
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..
Perlmutter CPU Nvidia¶
module unload gpu
module load cmake
module load nvidia/22.7
module load cray-libsci/22.11.1.2
module load cray-fftw/3.3.10.2
module unload darshan
export XGC_PLATFORM=perlmutter_cpu_nvidia
export CRAYPE_LINK_TYPE=dynamic
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..
Polaris¶
module load cmake kokkos cabana cray-fftw
export XGC_PLATFORM=polaris
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..
Stellar¶
source /home/amollen/Software/bin/set_up_xgc.stellar
Summit¶
module load nvhpc/21.7
module load spectrum-mpi
module load python
module load netlib-lapack
module load hypre
module load fftw
module load hdf5
module load cmake/3.20.2
module load libfabric/1.12.1-sysrdma
export XGC_PLATFORM=summit
export OMP_NUM_THREADS=14
export FC=mpifort
export CC=mpicc
export NVCC_WRAPPER_DEFAULT_COMPILER=mpiCC
export CXX=/gpfs/alpine/world-shared/phy122/lib/install/summit/kokkos/nvhpc21.7/bin/nvcc_wrapper
Theta¶
module load cray-libsci
module load cray-hdf5-parallel
module load cray-fftw
module use -a /projects/TokamakITER/Software/modulefiles
module load adios2/DEFAULT
module load cmake/3.20.4
export CRAYPE_LINK_TYPE=dynamic
export XGC_PLATFORM=theta
Traverse¶
source /home/rhager/Software/bin/set_up_xgc.traverse
CMake version 3.22.2 is available and can be used via ``xgc_cmake`` and ``xgc_ccmake``.