XGC job composer

The XGC-GUI also exposes this composer through its experimental Job Composer page. See utils/input_gui/README.md for the form and package editor.

utils/python/job_composer/compose_job.py creates a job package for an existing XGC executable. It does not build XGC or submit a job. Run the composer with Python 3.9 or newer, including when composing on an HPC login node; use a suitable Python virtual environment if the system python3 is older. The previous quickstart/compose_job.py command remains a compatibility entry point. The package contains jobscript.sh, env_settings.sh, setup_rundir.sh, print_jobinfo.sh, job_artifacts.py, cleanup_failed_job.py, input_dir/, logs/, an empty rundir/, and a job_manifest.json recording the resolved configuration and input hashes. Submit from the package directory. At job start, the script copies the prepared input_dir/ into rundir/, stages its three required files there, creates rundir/restart_dir/, and launches XGC from rundir/. The Aurora DAOS option instead stages and runs XGC in its mounted DAOS work directory; its package has no local rundir/.

Package layout and saved settings

The version 2 manifest also records value_sources: chosen for supplied settings and default for settings resolved from the machine profile or composer defaults. The CLI can reload it with --config; provide a new --output-dir for the next package. The GUI can reload job specs and manifests for its available controls, using a path on the GUI server. It rejects custom companion recipes, which the GUI does not edit.

The manifest hashes files at composition time. Edit package input_dir/ before the first submission. On a restart, the existing run-local input files remain in place; edits to package input_dir/ are not staged again. Edits after composition are not reflected in the original manifest; fort.input.used captures XGC’s used namelist values, but it does not replace a full snapshot of every runtime input.

python3 utils/python/job_composer/compose_job.py \
  --platform perlmutter_gpu \
  --executable /path/visible/on/compute/nodes/xgca \
  --nodes 2 --account myproject --walltime 01:00:00 \
  --output-dir /path/to/new-run
cd /path/to/new-run
sbatch jobscript.sh

Use qsub jobscript.sh on Polaris and Aurora. --output-dir must name a new directory; the composer will not overwrite an existing run. The executable need not exist on the machine where the composer runs, but it must exist at the given absolute path on compute nodes. The default input directory is quickstart/inputs; --input-dir stages a prepared directory containing at least input, adios2cfg.xml, and petsc.rc. Review and edit these inputs before submission. The bundled default input is an example, not a scientific configuration for a particular campaign.

Platforms and resource settings

The nine canonical --platform values are perlmutter_cpu, perlmutter_gpu, flux, stellar, stellarai_cpu, stellarai_gpu, polaris, aurora, and frontier. Aliases perlmutter and stellarai select their GPU profiles. utils/python/job_composer/machines.json holds scheduler, environment, launcher, and resource defaults. GPU profiles use one MPI rank per GPU or GPU tile. CPU defaults are Perlmutter 16 ranks × 16 threads, Flux 8 × 16, and Stellar/Stellar-AI CPU 12 × 8. Aurora uses 12 ranks × 16 threads, keeping both hardware threads of each selected physical core in one rank. --tasks-per-node and --cores-per-task allow compatible overrides; Polaris currently requires its fixed four-rank NUMA mapping, and Aurora requires an even 2–16 threads per rank. The generated script overrides any OMP_NUM_THREADS value in the sourced environment file with the resolved value.

Profiles for Flux, Stellar, and both Perlmutter variants also record a suggested Python virtual environment. The GUI displays the path and offers a button to select it; other profiles omit the hint. A suggestion does not activate Python or change a direct job by itself. In the CLI, pass the desired path explicitly with --python-venv. Verify that the selected environment is visible on compute nodes and contains the packages needed by the chosen add-ons.

Other options are --queue, --job-name, --environment (an alternate source file staged as env_settings.sh), --python-venv /absolute/compute-visible/venv (sources its bin/activate after the site environment), and --debug (passes XGC’s --debug argument). PBS jobs also accept --filesystems with a colon-separated list, such as home:eagle; Polaris defaults to home:eagle. Aurora DAOS selects its required request automatically. The activated environment supplies python3 for artifact helpers and the supervisor; the built-in campaign packager runs with that venv’s bin/python. An explicit --optimizer-python can select a separate interpreter for the optimizer. --daos-pool and --daos-cont together enable the Aurora DAOS path: the script loads the DAOS module, calls launch-dfuse.sh, copies input_dir into the mounted work directory, and launches with DAOS MPI settings. Batch logs remain in the submission directory. DAOS pool and container must already exist and be accessible to the job. Consult site policy for the appropriate queue, account, filesystem, executable path, and walltime.

The Aurora profile currently supports the GPU-aware XGC build (USE_GPU_AWARE_MPI=On) and its matching runtime environment (MPIR_CVAR_ENABLE_GPU=1). DAOS is an independent filesystem choice: after loading its module, the script uses the module’s DAOS_PRELOAD value for both mpiexec and its ranks and adds --no-vni. A Flare run uses neither DAOS launch option. The composer does not yet offer a standard-MPI build mode; that mode would need its own verified runtime settings.

Python API

Import the Python API from utils.python.job_composer.compose_job. Its entry point is compose_job(output_dir, platform=..., executable=..., nodes=..., account=..., walltime=..., ...). resolve_job(...) computes machine and workflow defaults without writing files; plan_job(...) also validates the input source and records chosen/default provenance. save_job_spec(path, ...) writes a validated job_spec.json, and load_job_settings(path) restores actively chosen settings from either a saved spec or a package manifest. It deliberately omits the previous output directory.

Testing

With BUILD_TESTING=ON, CMake registers the standard-library composer suite as job_composer_python when it finds Python 3.9 or newer. If the same interpreter also has NumPy, CTest registers the input-package selection tests; if it also has Flask and Matplotlib, CTest registers the Job Composer GUI tests. Set Python3_EXECUTABLE at CMake configure time to select a suitable virtual environment. Run the registered tests from the build directory with ctest -L job_composer --output-on-failure. They do not need XGC-Analysis or an XGC executable.

To configure just these Python tests without XGC’s compiler and library dependencies, use the standalone CMake project:

cmake -S utils/python/job_composer/tests -B build-job-composer \
  -DPython3_EXECUTABLE=/path/to/python
ctest --test-dir build-job-composer -L job_composer --output-on-failure

The live ACA smoke test is kept separately in utils/python/job_composer/tests/integration_campaign.py and is not part of the default CTest suite. It requires a sibling XGC-Analysis checkout and an environment with ADIOS2, HPC-Campaign, and NumPy. Run it explicitly from the XGC-Devel source directory with python3 -m unittest discover -s utils/python/job_composer/tests -p integration_campaign.py -v.

Walltime and deadlock detection

Use --max-walltime to pass XGC -max_walltime as the scheduler limit minus a checkpoint margin, rounded down to whole minutes. --max-walltime-margin-minutes sets a positive margin (default 30) that must cover environment and input staging, the last time step, and writing the final restart; at least one whole minute must remain for XGC. The final restart also requires sml_restart_write_after_last_timestep, which defaults to true. Leave this feature off for short debug jobs. A selected deadlock detector, online campaign packager, or Diffusion Optimizer resolves the launch mode to supervised; otherwise it is direct. Deadlock detection has a 10-minute startup grace and a 20-minute default inactivity timeout, adjustable with --deadlock-timeout-minutes only when detection is enabled. The first inactivity interval begins after the startup grace, so no initial complete line causes a default stall after 30 minutes. Later complete lines reset the timer.

Generate a deadlock-monitored package without any companion commands:

python3 utils/python/job_composer/compose_job.py \
  --platform polaris --executable /path/to/xgc \
  --nodes 2 --account myproject --walltime 02:00:00 \
  --deadlock-detection --deadlock-timeout-minutes 25 \
  --output-dir /path/to/new-run

The supervisor observes its explicitly configured XGC stdout file every 10 seconds for normal minute-scale timeouts, counts complete newline-terminated lines, and returns status 124 when the log stalls. It records the path, cumulative line count, last observed progress time, and idle duration as a JSON event in batch stderr before terminating XGC and any companions. A file that has not appeared yet is treated as quiet; truncation or replacement restarts reading from the beginning. Unflushed stdout and partial lines cannot be observed as progress, so check buffering on a short site run before using the detector for production. utils.python.job_composer.deadlock_detector can also be imported by other workflow adapters or run as an observer-only CLI: python3 utils/python/job_composer/deadlock_detector.py /path/to/xgc.out --timeout-minutes 20 --startup-grace-minutes 10. Its CLI reports a stall with exit status 2 but does not cancel processes.

Boolean composer options also accept --no-... forms, so a CLI override can explicitly turn an option off. The GUI exposes both built-in companion recipes.

Online campaign packaging

The thin supervisor uses only Python’s standard library and can run under the site’s Python 3.6. The built-in --online-campaign-packaging recipe requires --python-venv /absolute/path/to/venv; its bin/python runs the packager. XGC-Analysis requires Python 3.10 or newer, and the venv must also provide ADIOS2 and HPC-Campaign. The batch script checks the interpreter version and imports before launching XGC. The earlier stage’s --campaign-python setting is not used in this stage; when loading an older manifest, provide --python-venv for a new package. An explicit custom companion command controls its own Python environment. Optional --campaign-interval-seconds (default 300), --campaign-settle-seconds (default 15), and --campaign-finalize-seconds (default 600) tune polling, BP settling, and the post-XGC scan limit. The packager writes xgc_run.aca and xgc_run_manifest.json in the XGC working directory. It references attempt files under the package root, separate from the run directory on all current layouts.

Generated live and final packager commands include --diagnostics. Their stderr logs show flushed timestamps and elapsed times for each catalog phase and each HPC-Campaign manager operation. XGC may create BP5 directories before writing their first step. The live packager therefore uses a filename-only catalog scan and defers BP products whose index is still empty, then checks the remaining sources for the configured quiet period before ACA registration. The final forced scan does not wait for a quiet period, but still omits BP products with empty indexes. The packager applies one ACA update through a single worker process using HPC-Campaign’s Manager API, so it opens the staged ACA once instead of starting one manager process per entry. The campaign Python environment must provide Manager.open, close, data, text, and add_time_series. HPC-Campaign opens registered BP products with ADIOS2; its reads remain in the worker, where a native crash fails that update without killing the long-lived packager. A nonempty index and quiet period do not guarantee that a writer will not resume.

After XGC exits, the supervisor gives the packager time to finish its scan before releasing other tasks. The batch script then snapshots per-job artifacts and runs one final ACA scan, so the archive includes the completed attempt’s logs, timing, and artifact references. A failed final scan returns a nonzero batch status when XGC itself succeeded. A short debug job without --max-walltime may still run out of scheduler time before this final scan; choose enough walltime for packaging. Concurrent BP reader/writer behavior and remote replicas require site validation.

Custom companions and supervision

Custom packager and optimizer commands remain available through a JSON array passed to --companion-tasks-file. A custom packager command replaces the built-in command. For example:

[{"name": "online_campaign_packaging", "argv": ["python3", "/path/to/packager.py"]}]

Each command runs as a local process on the batch host, oversubscribing CPUs on the first allocated node. The generated supervisor.json records the XGC launch command and scheduler, XGC stdout and stderr paths, companion commands, output files, placement, and shutdown grace. launch_xgc.sh holds the platform’s MPI launcher and XGC flags and uses exec so its shell is replaced by srun or mpiexec; the supervisor therefore tracks the local launcher PID directly. The batch script sources the staged site environment, prepares the working directory, and executes python3 -u supervise_job.py. Ensure that environment makes a standard-library Python 3 interpreter available on compute nodes. The supervisor owns task process groups. A companion that completes before XGC is treated as an error unless its entry sets "required_until_xgc_exit": false. Set "on_failure": "continue" to report such a failure while XGC continues, or "stop_xgc" to stop the workflow. Campaign packaging defaults to continue; Diffusion Optimizer defaults to stop_xgc. XGC completion stops remaining companions and returns XGC’s exit status. Cancellation or a fatal companion failure stops all tasks. On abnormal shutdown under Slurm, the supervisor also sends TERM and then KILL to active job steps through scancel --signal using SLURM_JOB_ID, while leaving the batch script to collect artifacts. The provided commands must themselves be valid on the target machine; site smoke jobs remain necessary to verify placement, concurrent launcher behavior, and remote-rank termination.

Diffusion Optimizer workflow

Select --diffusion-optimizer --optimizer-script /absolute/path/to/XGC-Analysis/Diffusion_Optimizer/diffusion_optimizer.py for the built-in optimizer recipe. The script path must be visible on compute nodes; provide either --python-venv or an absolute --optimizer-python path to a Python 3.10+ environment with the optimizer’s dependencies. The supervisor starts the optimizer in the XGC working directory 120 seconds after launching XGC by default, giving xgc.heatdiag2.bp time to become readable. Set --optimizer-start-delay-seconds to a nonnegative integer to change this delay; zero starts it immediately. If XGC exits before the delay, the optimizer is not started. The delay does not itself verify BP readiness. The optimizer waits for its required static files and communicates through XGC’s configured ADIOS streams. It uses the first node’s CPU resources without a separate scheduler step. The batch script checks that the entry point and Python version are available after environment and DAOS setup. supervisor.json records its exact argv, delay, per-attempt stdout/stderr paths, and failure policy. An optimizer exit with nonzero status stops XGC immediately. A normal optimizer exit gives XGC up to 600 seconds to finish; if XGC is still running afterward, the supervisor stops it. This accommodates stream closure during XGC teardown while keeping the optimizer’s 90-minute no-data timeout fatal to the workflow after that grace. For an unusually long teardown, edit the optimizer task’s normal_exit_grace_seconds in the generated supervisor.json before submission. A custom optimizer companion can still be supplied in --companion-tasks-file, but the GUI cannot reload or edit custom companion recipes.

Every batch-safe diffusion_optimizer.py argument is available as a composer flag with an --optimizer- prefix: for example, --optimizer-update-damping-factor 0.3, --optimizer-solver-method lsq, --no-optimizer-use-time-avg-system, --optimizer-momentum-rhs-taper 0.1 0.2 0.8 0.1, and --optimizer-online-response-window 0.2 0.8. Run python3 utils/python/job_composer/compose_job.py --help for the full list. The shared optimizer_options.json defines the default values, input types, and help used by the CLI and GUI. The resolved values and per-option chosen/default provenance are in job_manifest.json. Only explicitly chosen values are passed to the optimizer; its own checkout supplies the other defaults. Check explicitly chosen flags against the installed XGC-Analysis optimizer CLI whenever using an older checkout; the composer does not import its scientific code. Interactive --debug-solver is excluded because it opens X11 plots and breakpoints, and --use-wall-loss from older examples is absent from the current optimizer CLI.

The optimizer’s scientific inputs and XGC’s ADIOS stream configuration are not changed by the composer. The current recipe requires site smoke tests for the optimizer Python environment, live stream connection, concurrent launcher behavior, and clean shutdown on each target platform. The supervisor exposes XGC_STDOUT_PATH to companion processes.

Logs, artifacts, and restarts

The generated script records UTC XGC start and end times and propagates XGC’s exit status. Each scheduler submission writes its batch, XGC, and companion stdout/stderr under logs/<restart-count>/<job-id>/; XGC writes its timing files directly under timing/<restart-count>/<job-id>/ through its --timing-dir option. Timing output compatibility change: when launching the C++ XGC independently without --timing-dir, XGC now creates ./timing/ and writes its timing files there. Earlier builds wrote them in the run directory when ./timing/ did not already exist. Update scripts that read timing_all.txt or adios_timer.txt from the run directory. XGC-S continues to use the run directory for its timing files. The post-run hook copies fort.input.used, runinfo.dat, TASKMAP_*.txt, units.m, units.txt, Efield_excluded_*, xgc.used_inputs.bp, git_diff_XGC_core.txt, wallsource.dat, and m3dc1_eqd_out.eqd into artifacts/<restart-count>/<job-id>/ when present. Each artifact directory also contains job_attempt.json with timestamps, exit status, and the copied product names. These files remain at the working-directory root where XGC may need them. Legacy timing_*.dat files found there are copied to the per-job timing directory.

The count is zero padded to at least four digits. A fresh run uses 0001; for a restart the script uses the second integer from timestep.dat in the XGC working directory plus one, matching XGC’s checkpoint counter. The job ID keeps a retry distinct when it starts again from the same checkpoint. Composer jobs call setup_rundir.sh --in-place: on a new run it stages only missing input files, preserving edits in the working directory; on a checkpointed run it requires the existing input files and restart_dir/ without replacing them. The one-argument form remains for older quickstart scripts that create a separate rundir/; it also preserves existing runtime inputs and checkpoints and copies only missing inputs. On a checkpointed composer run, the artifact hook requires sml_param:sml_restart=.true. in the working-directory input before it archives the previous attempt or launches XGC. A missing or false setting stops the job to protect the checkpoint and BP outputs. Before a subsequent attempt, the hook snapshots the previous attempt again, covering interruptions before the post-run hook completed. It rejects an existing checkpoint without a composer attempt marker so that an unmanaged run is not silently relabeled. A later submission without a checkpoint is rejected because it cannot safely resume and may overwrite BP outputs.

For an uncheckpointed failed attempt, run ./cleanup_failed_job.py --dry-run from the idle package to see its archive destination, then ./cleanup_failed_job.py to move rundir/ to failed_runs/<restart-count>/<job-id>/ and recreate an empty rundir/. Existing per-job logs/, timing/, and artifacts/ stay in place. Correct the source files in package input_dir/ and submit again. The helper refuses a run with timestep.dat because it may be restartable, and refuses to act while the package lock is held. The hidden .xgc_run.lock file need not be removed: an unlocked file does not block another job. The helper also accepts older flat packages, archiving run-root entries outside their generated scaffold. DAOS cleanup needs its mounted work directory and is not supported by this local helper.

Package locking and scheduler output

The generated batch script acquires a nonblocking exclusive lock on .xgc_run.lock in the package before staging or launching XGC. It holds file descriptor 9 for the lifetime of the job, including child tasks. A second job using the same package exits rather than sharing its working directory. After an interrupted job has released the lock, the artifact hook can recover a marker still labeled running; direct callers of the hook must hold the same exclusive lock before requesting recovery. This uses Python’s standard-library fcntl module and requires working advisory file locks on the package filesystem. The batch host needs standard-library Python 3 even for direct jobs; job_artifacts.py uses Python 3.6 compatible APIs. A lock failure reports the OS error and errno so a filesystem that does not support flock can be distinguished from an active job. Include package locking on the intended filesystem in each site’s smoke test; the job stops if the lock is unavailable.

The scheduler captures startup failures before the script redirects its own streams to the per-job log directory. Slurm places those bootstrap files at logs/batch-<job-id>.out/.err; PBS uses logs/batch.out/.err, which can be overwritten by a later PBS submission. The per-job batch.out/.err capture the script after setup. PBS directives contain the package’s absolute path, so recompose after moving a PBS bundle. The post-run hook cannot execute after an abrupt scheduler kill; the next submission performs its recovery snapshot if the same working directory is available.