How to Build XGC on macOS (Apple Silicon)

This document captures the exact workflow used to build xgca successfully on macOS.

1) Prerequisites

macOS with Xcode Command Line Tools
Homebrew
Git

2) Install required libraries with Homebrew

Run:

brew install llvm cmake make open-mpi fftw adios2 hdf5-mpi openblas libomp googletest pkgconf

Notes:

LLVM is installed with Homebrew, too, because it plays nicer with OpenMP.
cabana and ginkgo were intentionally disabled in CMake.
Homebrew kokkos was removed to avoid header conflicts with local Kokkos build:

brew uninstall kokkos

3) Build Kokkos locally (version 4.4.01)

XGC source in this setup required Kokkos 4.4.x compatibility.

cd /path/to/source/directory
git clone https://github.com/kokkos/kokkos.git
cd kokkos
git checkout -b v4.4.01 tags/4.4.01
cmake -S . -B kokkos-4.4.01-build \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_STANDARD=17 \
  -DKokkos_ENABLE_SERIAL=ON \
  -DKokkos_ENABLE_OPENMP=ON \
  -DKokkos_ENABLE_TESTS=OFF \
  -DKokkos_ARCH_NATIVE=ON \
  -DCMAKE_INSTALL_PREFIX="/path/to/kokkos-install"
cmake --build kokkos-4.4.01-build -j4
cmake --install kokkos-4.4.01-build

4) Build PETSc 3.24.4 locally with 64-bit indices

The XGC build here was done against PETSc 3.24.4 with 64-bit indices.

4.1 Get PETSc source tree for v3.24.4

cd /path/to/source/directory
git clone https://gitlab.com/petsc/petsc.git
cd petsc
git checkout -b v3.24.4 tags/v3.24.4

4.2 Configure PETSc

You can use this Python script to configure your build:

#!/usr/bin/env python3

if __name__ == '__main__':
    import os
    import sys
    import shutil

    sys.path.insert(0, os.path.abspath('config'))
    import configure

    brew = '/opt/homebrew'
    llvm_bin = f'{brew}/opt/llvm/bin'
    libomp = f'{brew}/opt/libomp'
    openmpi = f'{brew}/opt/open-mpi'
    openblas = f'{brew}/opt/openblas'
    make_dir = f'{brew}/opt/make'
    # When not installing as root, you may have to
    # create this directory first and adjust permissions
    prefix = '/opt/petsc/3.24.4-macos-clang-openmp'

    os.environ['PATH'] = f'{llvm_bin}:{os.environ.get("PATH", "")}'

    needed = [
        f'{llvm_bin}/clang',
        f'{openmpi}/bin/mpicc',
        f'{libomp}/include/omp.h',
        f'{openblas}/lib',
    ]
    missing = [p for p in needed if not os.path.exists(p)]
    if missing:
        raise RuntimeError('Missing required paths:\n  ' + '\n  '.join(missing))

    print('Using clang:', shutil.which('clang'))
    print('Using mpicc:', f'{openmpi}/bin/mpicc')

    configure_options = [
        'PETSC_ARCH=arch-macos-clang-openmp',

        '--with-batch=0',
        '--with-debugging=0',
        '--with-cuda=0',
        '--with-matlab=0',
        '--with-mpi=1',
        '--with-fortran-bindings=1',
        '--with-openmp=1',
        '--with-64-bit-indices=1',

        f'--prefix={prefix}',
        f'--with-make-dir={make_dir}',
        f'--with-mpi-dir={openmpi}',
        f'--with-openmp-dir={libomp}',
        f'--with-blaslapack-dir={openblas}',
        f'--download-sowing=1',

        '--download-metis',
        '--download-parmetis',

        '--COPTFLAGS=-O3',
        '--CXXOPTFLAGS=-O3',
        '--FOPTFLAGS=-O3',
    ]

    configure.petsc_configure(configure_options)

4.3 Build and install PETSc

The PETSc configure script will print a command to use for building, e.g.,

xxx=======================================================================================xxx
 Configure stage complete. Now build PETSc libraries with:
 make PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-macos-clang-openmp all
xxx=======================================================================================xxx

After building, PETSc will print a similar command for installing.

5) Configure and build XGC (Cabana OFF, Ginkgo OFF)

For configuring and building XGC use the provided MacOS template (XGC-Devel/CMake/find_dependencies_mac-os.cmake) via XGC_PLATFORM=mac-os. Depending on where you install Kokkos and PETSc, you might need to tweak the template a bit.

cd /path/to/XGC-Devel
mkdir _build
cd _build
export XGC_PLATFORM=mac-os

 cmake \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_Fortran_COMPILER=/opt/homebrew/bin/gfortran \
   -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang \
   -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++ \
   -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
   -DUSE_CABANA=OFF \
   -DCMAKE_DISABLE_FIND_PACKAGE_Ginkgo=TRUE \
   ../

6) Build XGC

make -j4 xgca

Successful binary path:

_build/bin/xgca