Continuous Integration¶

XGC uses GitLab CI and GitHub Actions to test pull requests and changes to the master branch. This allows us to verify that XGC continues to build and run successfully at selected HPC sites as we make changes to the software.

Before merging a pull request into the master branch all CI tests should finish successfully (show a green check mark).

GitLab CI¶

To configure GitLab CI for a new HPC Site:

On a new branch, write a new GitLab CI YAML file in the gitlab/ subdirectory of XGC-Devel. This directory contains working examples for you to follow. Other helpful resources include:

XGC build instructions – be sure to follow any site-specific build instructions.

Any documentation from the HPC site on how to set up new GitLab jobs (ie. specific tags or enviorment variables to use).

GitLab CI YAML syntax reference.

Create a new XGC-Devel project on this GitLab server.
Configure GitLab CI to use the YAML file you created in step #1 rather than the default .gitlab-ci.yml. This allows us to define jobs specific to each GitLab server and their associated runners (eg. PPPL vs. NERSC).
Ensure that your new GitLab project is configured with CI enabled.
Manually push this topic branch of XGC-Devel (containing your new YAML file) to this GitLab instance and verify that XGC builds and tests correctly.

GitHub/GitLab integration¶

Although GitLab does support integration with GitHub, this built-in functionality does not support testing pull requests from forks of the mainline repository. For this reason, instead of using GitLab’s built-in integration with GitHub, we use a polling script called SpackCIBridge. This script achieves two goals for us:

It copies branches to be tested from GitHub to GitLab.
It posts status back to GitHub from completed GitLab pipelines.

The source code for this script lives here: https://github.com/spack/spack-infrastructure/tree/main/images/gh-gl-sync

And it is available as a Docker image here: https://ghcr.io/spack/ci-bridge

This script is intended to run every few minutes as a cron job or GitLab CI scheduled pipeline. Each GitLab CI instance testing XGC will need to run its own separate copy of this script.

SpackCIBridge expects the following environment variables:

Name	Description	Required?	Notes
`GITHUB_TOKEN`	GitHub personal access token for `xgcbot`	Yes	Ask Robert for access
`GITLAB_SSH_KEY_BASE64`	base64 encoded private key that grants push access to GitLab	Yes	N/A
`GITLAB_TOKEN`	GitHub personal access token for API access	No	Only needed for private GitLab projects
`GITHUB_STATUS_CONTEXT`	The name of the status posted to GitHub	No	eg. `gitlab-ci/pppl`

SpackCIBridge also requires four positional arguments. These are as follows:

python SpackCIBridge.py \
  PrincetonUniversity/XGC-Devel            \  # GitHub project (this never changes for XGC)
  git@my.gitlab.url:mygroup/XGC-Devel.git  \  # Full SSH clone URL for GitLab
  https://my.gitlab.url                    \  # GitLab web host
  mygroup/XGC-Devel                           # GitLab project (org/repo or user/repo)

Here is an example GitLab CI YAML file that runs SpackCIBridge.py using a Docker executor:

sync-prs-job:
  only:
    - schedules
  image:
    name: ghcr.io/spack/ci-bridge
    entrypoint: [""]
  variables:
    GIT_STRATEGY: none
    GITHUB_STATUS_CONTEXT: "gitlab-ci/pppl"
  script:
    - python /scripts/SpackCIBridge.py PrincetonUniversity/XGC-Devel git@git.pppl.gov:xgc/XGC-Devel.git https://git.pppl.gov xgc/XGC-Devel

Note that the aforementioned environment variables are saved in GitLab as protected CI variables and thus are not hardcoded in this YAML file.

GitHub Actions¶

XGC uses a self-hosted runner on the Princeton University CPU cluster Stellar for the GitHub Actions CI tests. The test suite includes short build and run tests as well as longer physics tests that run during the weekend on the master branch. It is also possible to trigger the physics tests for an open pull request by adding the label run-all-physics-tests to it.

Attention: It happens occasionally that the runner on Stellar switches off, e.g. after larger upgrades on the cluster. If you notice that it is not running (the tests never seem to start) please contact the XGC team.

The GitHub Actions tab shows the latest workflow runs for pull requests or tests on the master branch. A green check mark indicates that a test finished successfully and a red cross that it failed.

New tests can be added in the .github/workflows subdirectory of XGC-Devel. Instructions on writing YAML files for GitHub Actions can be found at the GitHub Actions website.

To configure GitHub Actions for a new HPC Site, follow the instructions to add a self-hosted runner and to configure a self-hosted runner application as a service. It is possible to run into issues if sudo privileges are missing. On Stellar we have the following workaround:

#File actions.runner.service

  [Unit]
  Description=self-hosted GitHub Actions Runner on stellar
  After=network.target

  [Service]
  ExecStart=/home/$USER/actions-runner/bin/runsvc.sh
  WorkingDirectory=/home/$USER/actions-runner
  KillMode=process
  KillSignal=SIGTERM
  TimeoutStopSec=5min

  [Install]
  WantedBy=default.target

mkdir -p ~/.config/systemd/user/
cp actions.runner.service ~/.config/systemd/user/
systemctl --user enable actions.runner
systemctl --user start actions.runner
Make sure the service is running correctly by checking the output of systemctl --user status actions.runner. You should see active (running) in the output.
loginctl enable-linger $USER