3rd Party Software Installations
================================

XGC uses the following external libraries:

============================================================   ============ ================ ====================
Dependency                                                     Version      Required for XGC Required for kernels
============================================================   ============ ================ ====================
`PETSc <https://www.mcs.anl.gov/petsc/download/index.html>`_   >=3.12.0     YES              NO
`ADIOS2 <https://github.com/ornladios/ADIOS2>`_                >=2.3.0      YES              NO
`FFTW3 <http://www.fftw.org/download.html>`_                   Any          YES              NO
`Kokkos <https://github.com/kokkos/kokkos>`_                   >=3.0        YES              YES
`LAPACK <http://www.netlib.org/lapack/>`_                      Any          YES              YES
`Cabana <https://github.com/ECP-copa/Cabana>`_                 >=0.3.0      YES              YES
`GTest <https://github.com/google/googletest>`_                Any          NO               Testing only
`FusionIO <https://github.com/nferraro/fusion-io>`_            Any          NO               NO
`GINKGO <https://github.com/ginkgo-project/ginkgo>`_           Any(**)      NO               NO
============================================================   ============ ================ ====================

\*\*N.B.: Please use the "batch_develop" branch of Ginkgo.

It also depends on ``PSPLINE`` and ``CAMTIMERS``, which are inside the repository in ``lib/``, and built automatically if needed.

Maintaining shared installations
================================

The shared installations are located here:

============  ======================
System        Installation Directory
============  ======================
Aurora        /gecko/XGC_aesp_CNDA/Software
Cori Haswell  /project/projectdirs/m499/Software
Cori KNL      /project/projectdirs/m499/Software
Greene        /p/xgc/Software
Stellar       /projects/XGC/STELLAR/Software
Theta         /projects/TokamakITER/Software
Traverse      /projects/XGC/TRAVERSE/Software
============  ======================

If you want to update shared installations, please use the following convention when creating subdirectories:

.. code-block:: bash

   [Software directory]/[Package name]/[version]/[target system (if applicable)]/[compiler]/...

For example:

.. code-block:: bash

   /project/projectdirs/m499/Software/petsc/3.7.6/cori_knl/intel/…

Each available package has softlinks ``DEFAULT`` for ``[version]`` and ``[compiler]``. Module files may be provided in the future to make it easier to access 3rd party software that is not installed systemwide.


Installation instructions
===========================================

You may want to run XGC on an unsupported system or to customize your dependency installations. Here are some examples of installing the dependencies manually. Feedback is appreciated to make sure these stay useful.

Once you have installed the necessary dependencies, you will need to provide CMake with the location of XGC's dependencies if they are already installed on your system. You can either do this on the CMake configuration line, e.g. adding ``-DPETSc_ROOT=<path to PETSc>``, or by creating an ``XGC_PLATFORM`` by analogy to the existing ones, found in the XGC-Devel/CMake directory, e.g.:

.. code-block:: bash

   cp CMake/find_dependencies_generic.cmake CMake/find_dependencies_new_platform.cmake
   export XGC_PLATFORM=new_platform

.. toctree::
   :maxdepth: 1

PETSc
################

.. code-block:: python

   #!/usr/bin/python
   if __name__ == '__main__':
     import sys
     import os
     sys.path.insert(0, os.path.abspath('config'))
     import configure
     configure_options = [
       '--COPTFLAGS=-fast -no-ipo',
       '--CXXOPTFLAGS=-fast -no-ipo',
       '--FOPTFLAGS=-fast -no-ipo',
       '--download-parmetis',
       '--download-metis',
   #    '--download-superlu', does not work with 64 !!!
   #    '--download-superlu_dist',
   #    '--download-hypre',
   #    '--download-mumps',
       '--download-scalapack',
       '--with-ssl=0',
       '--with-cc=cc',
       '--with-clib-autodetect=0',
       '--with-cxx=CC',
       '--with-cxxlib-autodetect=0',
       '--with-debugging=0',
       '--with-fc=ftn',
       '--with-fortranlib-autodetect=0',
   #    '--with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/',
       '--with-shared-libraries=0',
       '--with-static-libraries=1',
       '--with-x=0',
       '--with-mpiexec=srun',
       'LIBS=-lstdc++',
       '--with-64-bit-indices',
       '--with-batch=1',
       '--prefix=/project/projectdirs/m499/Software/petsc/bdea225/cori_knl/intel-static/',
       'PETSC_ARCH=arch-cori-knl-opt64-intel',
     ]
     configure.petsc_configure(configure_options)

ADIOS2
################

.. code-block:: bash

    mkdir build; cd build
    cmake -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=$ADIOS2_INSTALL_DIR \
        -DADIOS2_BUILD_EXAMPLES=OFF \
        -DADIOS2_BUILD_TESTING=OFF \
        ..

FFTW
#############

.. code-block:: bash

     cmake -DCMAKE_BUILD_TYPE=Release \
     -DCMAKE_INSTALL_PREFIX=$FFTW_INSTALL_DIR \
     ..

Kokkos
################

.. code-block:: bash

    cmake -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
        -DCMAKE_CXX_STANDARD=17 \
        -DBUILD_TESTING=OFF \
        -DKokkos_ENABLE_OPENMP=ON \
        -DKokkos_ENABLE_SERIAL=ON \
        -DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON \
        ..

An additional option is used for Cori:

.. code-block:: bash

   -DKokkos_ENABLE_LIBDL=OFF \

Additional options for CUDA:

.. code-block:: bash

   -DKokkos_ENABLE_CUDA=ON \
   -DKokkos_ENABLE_CUDA_LAMBDA=ON \
   -DKokkos_ARCH_VOLTA70=ON \
   -DKokkos_ARCH_POWER9=ON \

Cabana
################

.. code-block:: bash

    cmake \
        -DCMAKE_BUILD_TYPE="Release" \
        -DCMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR \
        -DCMAKE_INSTALL_PREFIX=$CABANA_INSTALL_DIR \
        -DCabana_ENABLE_CAJITA=OFF \
        -DCabana_ENABLE_TESTING=Off \
        -DCabana_ENABLE_EXAMPLES=Off \
        ..

GTest
##########

.. code-block:: bash

    cmake \
        -DCMAKE_INSTALL_PREFIX=/Users/aaronscheinberg/Projects/googletest/install \
        -DCMAKE_CXX_STANDARD=14 \
        ..

Ginkgo
##########

For CoriGPU:

.. code-block:: bash

	cmake \
		-DCMAKE_CXX_COMPILER=nvc++ \
		-DGINKGO_COMPILER_FLAGS=-pedantic \
		-DCMAKE_INSTALL_PREFIX=/global/homes/d/dhruva/XGC-Devel/ginkgo/build/install_ompoff \
		-DGINKGO_CUDA_ARCHITECTURES=Volta \
		-DGINKGO_BUILD_OMP=OFF \
		-DCMAKE_CXX_FLAGS=-Mint128 \
		-DCMAKE_BUILD_TYPE:STRING=RELEASE \
		-DGINKGO_BUILD_TESTS=off \
		-DGINKGO_BUILD_BENCHMARKS=off \
		-DGINKGO_BUILD_EXAMPLES=off \
		-DGINKGO_BUILD_HWLOC=off \
		..

.. toctree::
   :maxdepth: 4