How to Build XGC on macOS (Apple Silicon)
=========================================

This document captures the exact workflow used to build ``xgca-cpp``
successfully on macOS.

.. _1-prerequisites:

1) Prerequisites
----------------

- macOS with Xcode Command Line Tools
- Homebrew
- Git

.. _2-install-required-libraries-with-homebrew:

2) Install required libraries with Homebrew
-------------------------------------------

Run:

.. code:: bash

   brew install llvm cmake make open-mpi fftw adios2 hdf5-mpi openblas libomp googletest pkgconf

Notes:

- LLVM is installed with Homebrew, too, because it plays nicer with OpenMP.
- ``cabana`` and ``ginkgo`` were intentionally disabled in CMake.
- Homebrew ``kokkos`` was removed to avoid header conflicts with local
  Kokkos build:

.. code:: bash

   brew uninstall kokkos

.. _3-build-kokkos-locally-version-4401:

3) Build Kokkos locally (version 4.4.01)
----------------------------------------

XGC source in this setup required Kokkos 4.4.x compatibility.

.. code:: bash
    cd /path/to/source/directory
    git clone https://github.com/kokkos/kokkos.git
    cd kokkos
    git checkout -b v4.4.01 tags/4.4.01
    cmake -S . -B kokkos-4.4.01-build \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CXX_STANDARD=17 \
      -DKokkos_ENABLE_SERIAL=ON \
      -DKokkos_ENABLE_OPENMP=ON \
      -DKokkos_ENABLE_TESTS=OFF \
      -DKokkos_ARCH_NATIVE=ON \
      -DCMAKE_INSTALL_PREFIX="/path/to/kokkos-install"
    cmake --build kokkos-4.4.01-build -j4
    cmake --install kokkos-4.4.01-build

.. _4-build-petsc-3244-locally-with-64-bit-indices:

4) Build PETSc 3.24.4 locally with 64-bit indices
-------------------------------------------------

The XGC build here was done against PETSc 3.24.4 with 64-bit indices.

.. _41-get-petsc-source-tree-for-v3244:

4.1 Get PETSc source tree for v3.24.4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: bash

   cd /path/to/source/directory
   git clone https://gitlab.com/petsc/petsc.git
   cd petsc
   git checkout -b v3.24.4 tags/v3.24.4

.. _42-configure-petsc

4.2 Configure PETSc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can use this Python script to configure your build:

.. code:: python

   #!/usr/bin/env python3

   if __name__ == '__main__':
       import os
       import sys
       import shutil

       sys.path.insert(0, os.path.abspath('config'))
       import configure

       brew = '/opt/homebrew'
       llvm_bin = f'{brew}/opt/llvm/bin'
       libomp = f'{brew}/opt/libomp'
       openmpi = f'{brew}/opt/open-mpi'
       openblas = f'{brew}/opt/openblas'
       make_dir = f'{brew}/opt/make'
       # When not installing as root, you may have to
       # create this directory first and adjust permissions
       prefix = '/opt/petsc/3.24.4-macos-clang-openmp'

       os.environ['PATH'] = f'{llvm_bin}:{os.environ.get("PATH", "")}'

       needed = [
           f'{llvm_bin}/clang',
           f'{openmpi}/bin/mpicc',
           f'{libomp}/include/omp.h',
           f'{openblas}/lib',
       ]
       missing = [p for p in needed if not os.path.exists(p)]
       if missing:
           raise RuntimeError('Missing required paths:\n  ' + '\n  '.join(missing))

       print('Using clang:', shutil.which('clang'))
       print('Using mpicc:', f'{openmpi}/bin/mpicc')

       configure_options = [
           'PETSC_ARCH=arch-macos-clang-openmp',

           '--with-batch=0',
           '--with-debugging=0',
           '--with-cuda=0',
           '--with-matlab=0',
           '--with-mpi=1',
           '--with-fortran-bindings=1',
           '--with-openmp=1',
           '--with-64-bit-indices=1',

           f'--prefix={prefix}',
           f'--with-make-dir={make_dir}',
           f'--with-mpi-dir={openmpi}',
           f'--with-openmp-dir={libomp}',
           f'--with-blaslapack-dir={openblas}',
           f'--download-sowing=1',

           '--download-metis',
           '--download-parmetis',

           '--COPTFLAGS=-O3',
           '--CXXOPTFLAGS=-O3',
           '--FOPTFLAGS=-O3',
       ]

       configure.petsc_configure(configure_options)

.. _43-build-and-install-petsc:

4.3 Build and install PETSc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The PETSc configure script will print a command to use for building, e.g.,

.. code:: bash

   xxx=======================================================================================xxx
    Configure stage complete. Now build PETSc libraries with:
    make PETSC_DIR=/path/to/petsc PETSC_ARCH=arch-macos-clang-openmp all
   xxx=======================================================================================xxx

After building, PETSc will print a similar command for installing.

5) Configure and build XGC (Cabana OFF, Ginkgo OFF)
---------------------------------------------------

For configuring and building XGC use the provided MacOS template (`XGC-Devel/CMake/find_dependencies_mac-os.cmake`) via `XGC_PLATFORM=mac-os`.
Depending on where you install Kokkos and PETSc, you might need to tweak the template a bit.

.. code:: bash

   cd /path/to/XGC-Devel
   mkdir _build
   cd _build
   export XGC_PLATFORM=mac-os

    cmake \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_Fortran_COMPILER=/opt/homebrew/bin/gfortran \
      -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang \
      -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++ \
      -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
      -DUSE_CABANA=OFF \
      -DCMAKE_DISABLE_FIND_PACKAGE_Ginkgo=TRUE \
      ../


6) Build XGC
---------------

.. code:: bash

   make -j4 xgca-cpp

Successful binary path:

.. code:: bash

   _build/bin/xgca-cpp