Continuous Integration
######################

XGC uses `GitLab CI <https://docs.gitlab.com/ee/ci/>`_ and `GitHub Actions <https://docs.github.com/en/actions>`_ to test pull requests and changes to the master branch.
This allows us to verify that XGC continues to build and run successfully at selected HPC sites as we make
changes to the software.

.. image:: ./img/CI_tests.png
   :target: ./_images/CI_tests.png
   :alt: alt_text

*Before merging a pull request into the master branch all CI tests should finish successfully (show a green check mark).*

GitLab CI
*********

To configure GitLab CI for a new HPC Site:

1. On a new branch, write a new GitLab CI YAML file in the `gitlab/ subdirectory of XGC-Devel <https://github.com/PrincetonUniversity/XGC-Devel/tree/master/gitlab>`_. This directory contains working examples for you to follow. Other helpful resources include:

  * `XGC build instructions <https://xgc.pppl.gov/html/cmake_build_instructions.html>`_ -- be sure to follow any site-specific build instructions.
  * Any documentation from the HPC site on how to set up new GitLab jobs (ie. specific tags or enviorment variables to use).
  * `GitLab CI YAML syntax reference <https://docs.gitlab.com/ee/ci/yaml/>`_.

2. Create a new ``XGC-Devel`` project on this GitLab server.
3. Configure GitLab CI to use the YAML file you created in step #1 rather than the default ``.gitlab-ci.yml``. This allows us to define jobs specific to each GitLab server and their associated runners (eg. PPPL vs. NERSC).
4. Ensure that your new GitLab project is configured with CI enabled.
5. Manually push this topic branch of XGC-Devel (containing your new YAML file) to this GitLab instance and verify that XGC builds and tests correctly.


GitHub/GitLab integration
=========================

Although GitLab does support `integration with GitHub <https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html>`_, this built-in functionality does not support testing pull requests from forks of the mainline repository.
For this reason, instead of using GitLab's built-in integration with GitHub, we use a polling script called *SpackCIBridge*. This script achieves two goals for us:

* It copies branches to be tested from GitHub to GitLab.
* It posts status back to GitHub from completed GitLab pipelines.

The source code for this script lives here:
https://github.com/spack/spack-infrastructure/tree/main/images/gh-gl-sync

And it is available as a Docker image here:
https://ghcr.io/spack/ci-bridge

This script is intended to run every few minutes as a cron job or GitLab CI scheduled pipeline.
Each GitLab CI instance testing XGC will need to run its own separate copy of this script.

SpackCIBridge expects the following environment variables:

=========================  ============================================================  =========  ==============================
Name                       Description                                                   Required?  Notes
=========================  ============================================================  =========  ==============================
``GITHUB_TOKEN``           GitHub personal access token for ``xgcbot``                   Yes        Ask Robert for access
``GITLAB_SSH_KEY_BASE64``  base64 encoded private key that grants push access to GitLab  Yes        N/A
``GITLAB_TOKEN``           GitHub personal access token for API access                   No         Only needed for private GitLab projects
``GITHUB_STATUS_CONTEXT``  The name of the status posted to GitHub                       No         eg. ``gitlab-ci/pppl``
=========================  ============================================================  =========  ==============================

SpackCIBridge also requires four positional arguments. These are as follows:

.. code-block:: bash

  python SpackCIBridge.py \
    PrincetonUniversity/XGC-Devel            \  # GitHub project (this never changes for XGC)
    git@my.gitlab.url:mygroup/XGC-Devel.git  \  # Full SSH clone URL for GitLab
    https://my.gitlab.url                    \  # GitLab web host
    mygroup/XGC-Devel                           # GitLab project (org/repo or user/repo)

Here is an example GitLab CI YAML file that runs ``SpackCIBridge.py`` using a `Docker executor <https://docs.gitlab.com/runner/executors/docker.html>`_:

.. code-block:: yaml

  sync-prs-job:
    only:
      - schedules
    image:
      name: ghcr.io/spack/ci-bridge
      entrypoint: [""]
    variables:
      GIT_STRATEGY: none
      GITHUB_STATUS_CONTEXT: "gitlab-ci/pppl"
    script:
      - python /scripts/SpackCIBridge.py PrincetonUniversity/XGC-Devel git@git.pppl.gov:xgc/XGC-Devel.git https://git.pppl.gov xgc/XGC-Devel

Note that the aforementioned environment variables are saved in GitLab as `protected CI variables <https://docs.gitlab.com/ee/ci/variables/>`_
and thus are not hardcoded in this YAML file.

GitHub Actions
***************

XGC uses a self-hosted runner on the Princeton University CPU cluster Stellar for the GitHub Actions CI tests.
The test suite includes short build and run tests as well as longer physics tests that run during the weekend on the master branch.
It is also possible to trigger the physics tests for an open pull request by adding the label ``run-all-physics-tests`` to it.

**Attention:** It happens occasionally that the runner on Stellar switches off, e.g. after larger upgrades on the cluster. If you notice that it is not running (the tests never seem to start) please contact the XGC team.


.. image:: ./img/GitHub_Actions_tab.png
   :target: ./_images/GitHub_Actions_tab.png
   :alt: alt_text

*The GitHub Actions tab shows the latest workflow runs for pull requests or tests on the master branch. A green check mark indicates that a test finished successfully and a red cross that it failed.*

New tests can be added in the `.github/workflows subdirectory of XGC-Devel <https://github.com/PrincetonUniversity/XGC-Devel/tree/master/.github/workflows>`_.
Instructions on writing YAML files for GitHub Actions can be found at the `GitHub Actions <https://docs.github.com/en/actions>`_ website.

To configure GitHub Actions for a new HPC Site, follow the instructions to `add a self-hosted runner <https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository>`_ and to `configure a self-hosted runner application as a service <https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/configuring-the-self-hosted-runner-application-as-a-service>`_.
It is possible to run into issues if sudo privileges are missing. On Stellar we have the following workaround:

.. code-block:: bash

  #File actions.runner.service 

    [Unit]
    Description=self-hosted GitHub Actions Runner on stellar
    After=network.target

    [Service]
    ExecStart=/home/$USER/actions-runner/bin/runsvc.sh
    WorkingDirectory=/home/$USER/actions-runner
    KillMode=process
    KillSignal=SIGTERM
    TimeoutStopSec=5min

    [Install]
    WantedBy=default.target


1. ``mkdir -p ~/.config/systemd/user/``
2. ``cp actions.runner.service ~/.config/systemd/user/``
3. ``systemctl --user enable actions.runner``
4. ``systemctl --user start actions.runner``
5. Make sure the service is running correctly by checking the output of ``systemctl --user status actions.runner``. You should see active (running) in the output.
6. ``loginctl enable-linger $USER``