Build Instructions
========================

Note: If you are new to XGC or a new platform, our `Quickstart <quickstart.html>`__ may be helpful by providing a functioning example to work from.


* XGC has several external dependencies. If you are building XGC at one of the following HPC facilities, indicate which one by setting the environment variable ``XGC_PLATFORM`` before configuring, e.g.:

  .. code-block:: bash

     export XGC_PLATFORM=frontier

  =====================  ============
  System                 XGC_PLATFORM
  =====================  ============
  Aurora                 aurora
  Frontier               frontier
  Greene                 greene
  Perlmutter GPU         perlmutter
  Perlmutter CPU         perlmutter_cpu
  Polaris                polaris
  Stellar                stellar
  Traverse               traverse
  =====================  ============

  After that, please set the environment variable and load the modules specified in the section at
  the end of this page.

  If you are not using one of the above facilities or want custom-built dependencies, then you have two options:

  #. Install the libraries manually (see :doc:`third_party_software`)
  
  #. Use spack to install the libraries. See ``docker/rocm/spack.yaml`` for an example Spack environment file.

* Load modules and set environment variables

  Compiling and running on the supported systems may require modules and environment variables. See Building at HPC Facilities at the bottom of this page for the commonly used ones on your system.

* Create and enter a directory.

  .. code-block:: bash

     mkdir build; cd build

* Run CMake to configure a build of XGC.

  .. code-block:: bash

     cmake ..

  Additional settings can be passed as ``-D`` flags, e.g.:

  .. code-block:: bash

     cmake -DBUILD_DEPENDENCIES=ON -DCONVERT_GRID2=ON -DSOLVERLU=OFF ..

  To interactively edit configuration settings, use ``ccmake .`` for a CLI or
  ``cmake-gui`` for a GUI.

  For a full list of XGC configure options, see :doc:`/flags`.

* Build all available targets:

  .. code-block:: bash

     make -j

  Or just the one you want, e.g.:

  .. code-block:: bash

     make -j xgc-es-cpp

  The executables will be in ``build/bin``. Current available targets are: ``xgc-es-cpp``, ``xgc-es-cpp-gpu``, ``xgca-cpp``,
  ``xgca-cpp-gpu``, ``xgc-eem-cpp``, ``xgc-eem-cpp-gpu``, as well as kernels and tests (see :doc:`kernels_and_tests`).

Environment at HPC facilities
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Compiling and running on the supported systems may require modules and environment variables. The ones we typically use for our target platforms are found inside the repository in the directory ``quickstart/modules_and_env``.

Additionally, the platform may require additional ``CMake`` configuration specifications. These can be found in the directory ``quickstart/cmake_configs``.

For convenience, these two files are reproduced below for our supported platforms.

Aurora
~~~~~~~~~~

.. literalinclude:: ../quickstart/modules_and_env/aurora.sh
   :language: bash

.. literalinclude:: ../quickstart/cmake_configs/aurora.sh
   :language: bash

Frontier
~~~~~~~~~

.. literalinclude:: ../quickstart/modules_and_env/frontier.sh
   :language: bash

.. literalinclude:: ../quickstart/cmake_configs/frontier.sh
   :language: bash

Greene
~~~~~~

.. code-block:: bash

   source /p/xgc/Software/greene_config_gcc11_20230501

Perlmutter (GPU)
~~~~~~~~~~~~~~~~


.. literalinclude:: ../quickstart/modules_and_env/perlmutter.sh
   :language: bash

.. literalinclude:: ../quickstart/cmake_configs/perlmutter.sh
   :language: bash
   
Perlmutter (CPU)
~~~~~~~~~~~~~~~~

.. literalinclude:: ../quickstart/modules_and_env/perlmutter_cpu.sh
   :language: bash

.. literalinclude:: ../quickstart/cmake_configs/perlmutter_cpu.sh
   :language: bash

Polaris
~~~~~~~~

.. literalinclude:: ../quickstart/modules_and_env/polaris.sh
   :language: bash

.. literalinclude:: ../quickstart/cmake_configs/polaris.sh
   :language: bash

Stellar
~~~~~~~~~

.. code-block:: bash

    source /projects/XGC/STELLAR/Software/bin/set_up_xgc.stellar

Traverse
~~~~~~~~~

.. code-block:: bash

    source /projects/XGC/TRAVERSE/Software/bin/set_up_xgc.traverse

Build instructions for XGC-S
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

XGC-S can be compiled with GNU make.

Stellar
~~~~~~~~~

.. code-block:: bash

    source /projects/XGC/STELLAR/Software/bin/set_up_xgc.stellar
    cd XGC-Devel/XGC-S
    make -f Makefile.Stellar


Perlmutter
~~~~~~~~~~~

Works with the following modules loaded

.. code-block:: bash

    module list
    
    Currently Loaded Modules:
      1) craype-x86-milan     4) xpmem/2.6.2-2.5_2.33__gd067c3f.shasta   7) cray-mpich/8.1.28  10) perftools-base/23.12.0               13) cray-fftw/3.3.10.6 (math)
      2) libfabric/1.15.2.0   5) PrgEnv-gnu/8.5.0                        8) craype/2.7.30      11) cpe/23.12
      3) craype-network-ofi   6) cray-dsmml/0.2.2                        9) gcc-native/12.3    12) cmake/3.24.3           (buildtools)

.. code-block:: bash

    cd XGC-Devel/XGC-S
    make -f Makefile.Perlmutter

.. toctree::
   :maxdepth: 4