Build Instructions¶
XGC has several external dependencies. If you are building XGC at one of the following HPC facilities, indicate which one by setting the environment variable
XGC_PLATFORM
before configuring, e.g.:export XGC_PLATFORM=summit
System
XGC_PLATFORM
Frontier
frontier
Greene
greene
JLSE
jlse
Perlmutter GCC
perlmutter_gcc
Perlmutter CPU GCC
perlmutter_cpu_gcc
Perlmutter Nvidia
perlmutter_nvhpc
Perlmutter CPU Nvidia
perlmutter_cpu_nvidia
Polaris
polaris
Stellar
stellar
Summit
summit
Sunspot
sunspot
Theta
theta
Traverse
traverse
After that, please set the environment variable and load the modules specified in the section at the end of this page.
If you are not using one of the above facilities or want custom-built dependencies, then you have two options:
Install the libraries manually (see 3rd Party Software Installations)
Use the Superbuild, which will download, build, and install its dependencies for you. To enable this feature, pass
-DBUILD_DEPENDENCIES=ON
to cmake when configuring the XGC build below.
Load modules and set environment variables
Compiling and running on the supported systems may require modules and environment variables. See Building at HPC Facilities at the bottom of this page for the commonly used ones on your system.
Create and enter a directory.
mkdir build; cd build
Run CMake to configure a build of XGC.
cmake ..
Additional settings can be passed as
-D
flags, e.g.:cmake -DBUILD_DEPENDENCIES=ON -DCONVERT_GRID2=ON -DSOLVERLU=OFF ..
To interactively edit configuration settings, use
ccmake .
for a CLI orcmake-gui
for a GUI.For a full list of XGC configure options, see XGC Preprocessor Macros.
Build all available targets:
make -j
Or just the one you want, e.g.:
make -j xgc-es-cpp
The executables will be in
build/bin
. Current available targets are:xgc-es-cpp
,xgc-es-cpp-gpu
,xgca-cpp
,xgca-cpp-gpu
,xgc-eem-cpp
,xgc-eem-cpp-gpu
, as well as kernels and tests (see Kernels and Tests).
Environment at HPC facilities¶
Compiling and running on the supported systems may require modules and environment variables. Below are the commonly used sets.
Frontier¶
module reset
module unload perftools-base
module load cmake
module load PrgEnv-amd
module swap amd amd/5.2.0
module swap cray-mpich cray-mpich/8.1.25
module load craype-accel-amd-gfx90a
export CRAYPE_LINK_TYPE=dynamic
export PATH=${CRAY_MPICH_PREFIX}/bin:${PATH}
export PATH=${ROCM_COMPILER_PATH}/bin:${PATH}
export MPICH_SMP_SINGLE_COPY_MODE=NONE
export MPICH_GPU_SUPPORT_ENABLED=1
export ROCM_PATH=/opt/rocm-5.2.0
export OLCF_ROCM_ROOT=/opt/rocm-5.2.0
export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH
export MPICH_CXX=${OLCF_ROCM_ROOT}/bin/hipcc
export LLVM_PATH=${ROCM_PATH}/llvm
export HIP_CLANG_PATH=${ROCM_PATH}/llvm/bin
export HSA_PATH=${ROCM_PATH}
export ROCMINFO_PATH=${ROCM_PATH}
export DEVICE_LIB_PATH=${ROCM_PATH}/amdgcn/bitcode
export HIP_DEVICE_LIB_PATH=${ROCM_PATH}/amdgcn/bitcode
export HIP_PLATFORM=amd
export HIP_COMPILER=clang
export HIPCC_COMPILE_FLAGS_APPEND="$HIPCC_COMPILE_FLAGS_APPEND --rocm-path=${ROCM_PATH}"
export XGC_PLATFORM=frontier
Extra CMake variables needed:
-DCMAKE_BUILD_TYPE=RelWithDebInfo #Release
-DCMAKE_CXX_COMPILER=`which mpicxx`
-DCMAKE_C_COMPILER=`which mpicc`
-DCMAKE_Fortran_COMPILER=`which mpifort`
-DCMAKE_CXX_FLAGS="-I${OLCF_ROCM_ROOT}/include -munsafe-fp-atomics"
-DCMAKE_EXE_LINKER_FLAGS="-L${OLCF_ROCM_ROOT}/lib -lamdhip64"
-DUSE_GPU_AWARE_MPI=On
Greene¶
source /p/xgc/Software/greene_config_gcc11_20230501
Perlmutter GCC¶
April 23, 2023 update: the GNU/GCC compilers are the recommended compilers for running on Perlmutter GPU compute nodes.
The assumption is that one is starting with the default list of modules before the following module load/unload commands, i.e. no module load/unload commands in the user’s .bashrc/.bash_profile/etc.
module load cmake/3.24.3
module load cray-fftw
module unload darshan
module unload cray-libsci
export XGC_PLATFORM=perlmutter_gcc
export CRAYPE_LINK_TYPE=dynamic
export NVCC_WRAPPER_DEFAULT_COMPILER=CC
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn -DCMAKE_BUILD_TYPE=Release ...
Perlmutter CPU GCC¶
For Perlmutter CPU-only compute nodes.
April 23, 2023 update: the GNU/GCC compilers are the recommended compilers for running on Perlmutter CPU-only compute nodes.
module unload gpu
module load cmake/3.24.3
module load cray-fftw
module unload darshan
module unload cray-libsci
export XGC_PLATFORM=perlmutter_cpu_gcc
export CRAYPE_LINK_TYPE=dynamic
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn -DCMAKE_BUILD_TYPE=Release ...
Perlmutter Nvidia¶
April 23, 2023 update: the recommendation is to use the GCC compilers for building XGC on Perlmutter. Still sorting out some issues with nvidia 23.1 + cuda 12.0.
module unload gpu
module load cmake/3.24.3
module load PrgEnv-nvidia
module swap nvidia nvidia/23.1
module load cudatoolkit/12.0
module load cray-fftw
module unload darshan
module unload cray-libsci
export XGC_PLATFORM=perlmutter_nvhpc
export CRAYPE_LINK_TYPE=dynamic
export NVCC_WRAPPER_DEFAULT_COMPILER=CC
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..
Perlmutter CPU Nvidia¶
For Perlmutter CPU-only compute nodes.
April 23, 2023 update: the recommendation is to use the GCC compilers for building XGC on Perlmutter.
module unload gpu
module load cmake/3.24.3
module load PrgEnv-nvidia
module swap nvidia nvidia/23.1
module load cray-fftw
module unload cray-libsci
module unload darshan
export XGC_PLATFORM=perlmutter_cpu_nvidia
export CRAYPE_LINK_TYPE=dynamic
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..
Polaris¶
module load cmake kokkos cabana cray-fftw
export XGC_PLATFORM=polaris
cmake -DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc -DCMAKE_Fortran_COMPILER=ftn ..
Stellar¶
source /scratch/gpfs/xgc/STELLAR/Software/bin/set_up_xgc.stellar
Summit¶
module load nvhpc/22.5
module load spectrum-mpi
module load python
module load netlib-lapack
module load hypre
module load fftw
module load hdf5
module load cmake/3.20.2
module load libfabric/1.12.1-sysrdma
module swap cuda/nvhpc cuda/11.7.1
export XGC_PLATFORM=summit
export OMP_NUM_THREADS=14
export FC=mpifort
export CC=mpicc
export KOKKOS_SRC_DIR=/gpfs/alpine/world-shared/phy122/lib/install/summit/kokkos/Aug31_23_nvhpc22.5
export CXX=$KOKKOS_SRC_DIR/bin/nvcc_wrapper
export NVCC_WRAPPER_DEFAULT_COMPILER=mpiCC
Sunspot¶
module load spack cmake
module load oneapi/eng-compiler/2023.05.15.006
module load kokkos/2023.05.15.006/eng-compiler/sycl_intel_aot
module load cabana/2023.05.15.006/eng-compiler/sycl_intel
export XGC_PLATFORM=sunspot
export OMP_NUM_THREADS=16
export ONEAPI_MPICH_GPU=NO_GPU
CXX="mpic++ -cxx=icpx" CC="mpicc -cc=icx" FC="mpifort -fc=ifort" I_MPI_CXX=icpx I_MPI_CC=icx I_MPI_F90=ifort\
cmake\
-DCMAKE_Fortran_FLAGS="-g -init=arrays,zero -fpp -O2 -fPIC -fopenmp -DCPP_PARTICLE_INIT=On -diag-disable=5462 -diag-disable=8291"\
-DCMAKE_C_FLAGS="-g -O2 -fPIC -fopenmp -DCPP_PARTICLE_INIT=On"\
-DCMAKE_CXX_FLAGS="-g -O2 -fPIC -fopenmp -Wno-tautological-constant-compare -DCPP_PARTICLE_INIT=On"\
-DCMAKE_EXE_LINKER_FLAGS=" -fsycl-max-parallel-link-jobs=8 -limf -lsvml -lintlc -lifcore" ..
Theta¶
module load cray-libsci
module load cray-hdf5-parallel
module load cray-fftw
module use -a /projects/TokamakITER/Software/modulefiles
module load adios2/DEFAULT
module load cmake/3.20.4
export CRAYPE_LINK_TYPE=dynamic
export XGC_PLATFORM=theta
Traverse¶
source /home/rhager/Software/bin/set_up_xgc.traverse
CMake version 3.22.2 is available and can be used via ``xgc_cmake`` and ``xgc_ccmake``.