XGC job composer ================ The :doc:`XGC-GUI ` also exposes this composer through its experimental **Job Composer** page. See ``utils/input_gui/README.md`` for the form and package editor. ``utils/python/job_composer/compose_job.py`` creates a job package for an existing XGC executable. It does not build XGC or submit a job. Run the composer with Python 3.9 or newer, including when composing on an HPC login node; use a suitable Python virtual environment if the system ``python3`` is older. The previous ``quickstart/compose_job.py`` command remains a compatibility entry point. The package contains ``jobscript.sh``, ``env_settings.sh``, ``setup_rundir.sh``, ``print_jobinfo.sh``, ``job_artifacts.py``, ``cleanup_failed_job.py``, ``input_dir/``, ``logs/``, an empty ``rundir/``, and a ``job_manifest.json`` recording the resolved configuration and input hashes. Submit from the package directory. At job start, the script copies the prepared ``input_dir/`` into ``rundir/``, stages its three required files there, creates ``rundir/restart_dir/``, and launches XGC from ``rundir/``. The Aurora DAOS option instead stages and runs XGC in its mounted DAOS work directory; its package has no local ``rundir/``. Package layout and saved settings --------------------------------- The version 2 manifest also records ``value_sources``: ``chosen`` for supplied settings and ``default`` for settings resolved from the machine profile or composer defaults. The CLI can reload it with ``--config``; provide a new ``--output-dir`` for the next package. The GUI can reload job specs and manifests for its available controls, using a path on the GUI server. It rejects custom companion recipes, which the GUI does not edit. The manifest hashes files at composition time. Edit package ``input_dir/`` before the first submission. On a restart, the existing run-local input files remain in place; edits to package ``input_dir/`` are not staged again. Edits after composition are not reflected in the original manifest; ``fort.input.used`` captures XGC's used namelist values, but it does not replace a full snapshot of every runtime input. .. code-block:: bash python3 utils/python/job_composer/compose_job.py \ --platform perlmutter_gpu \ --executable /path/visible/on/compute/nodes/xgca \ --nodes 2 --account myproject --walltime 01:00:00 \ --output-dir /path/to/new-run cd /path/to/new-run sbatch jobscript.sh Use ``qsub jobscript.sh`` on Polaris and Aurora. ``--output-dir`` must name a new directory; the composer will not overwrite an existing run. The executable need not exist on the machine where the composer runs, but it must exist at the given absolute path on compute nodes. The default input directory is ``quickstart/inputs``; ``--input-dir`` stages a prepared directory containing at least ``input``, ``adios2cfg.xml``, and ``petsc.rc``. Review and edit these inputs before submission. The bundled default ``input`` is an example, not a scientific configuration for a particular campaign. Platforms and resource settings ------------------------------- The nine canonical ``--platform`` values are ``perlmutter_cpu``, ``perlmutter_gpu``, ``flux``, ``stellar``, ``stellarai_cpu``, ``stellarai_gpu``, ``polaris``, ``aurora``, and ``frontier``. Aliases ``perlmutter`` and ``stellarai`` select their GPU profiles. ``utils/python/job_composer/machines.json`` holds scheduler, environment, launcher, and resource defaults. GPU profiles use one MPI rank per GPU or GPU tile. CPU defaults are Perlmutter 16 ranks × 16 threads, Flux 8 × 16, and Stellar/Stellar-AI CPU 12 × 8. Aurora uses 12 ranks × 16 threads, keeping both hardware threads of each selected physical core in one rank. ``--tasks-per-node`` and ``--cores-per-task`` allow compatible overrides; Polaris currently requires its fixed four-rank NUMA mapping, and Aurora requires an even 2–16 threads per rank. The generated script overrides any ``OMP_NUM_THREADS`` value in the sourced environment file with the resolved value. Profiles for Flux, Stellar, and both Perlmutter variants also record a suggested Python virtual environment. The GUI displays the path and offers a button to select it; other profiles omit the hint. A suggestion does not activate Python or change a direct job by itself. In the CLI, pass the desired path explicitly with ``--python-venv``. Verify that the selected environment is visible on compute nodes and contains the packages needed by the chosen add-ons. Other options are ``--queue``, ``--job-name``, ``--environment`` (an alternate source file staged as ``env_settings.sh``), ``--python-venv /absolute/compute-visible/venv`` (sources its ``bin/activate`` after the site environment), and ``--debug`` (passes XGC's ``--debug`` argument). PBS jobs also accept ``--filesystems`` with a colon-separated list, such as ``home:eagle``; Polaris defaults to ``home:eagle``. Aurora DAOS selects its required request automatically. The activated environment supplies ``python3`` for artifact helpers and the supervisor; the built-in campaign packager runs with that venv's ``bin/python``. An explicit ``--optimizer-python`` can select a separate interpreter for the optimizer. ``--daos-pool`` and ``--daos-cont`` together enable the Aurora DAOS path: the script loads the DAOS module, calls ``launch-dfuse.sh``, copies ``input_dir`` into the mounted work directory, and launches with DAOS MPI settings. Batch logs remain in the submission directory. DAOS pool and container must already exist and be accessible to the job. Consult site policy for the appropriate queue, account, filesystem, executable path, and walltime. The Aurora profile currently supports the GPU-aware XGC build (``USE_GPU_AWARE_MPI=On``) and its matching runtime environment (``MPIR_CVAR_ENABLE_GPU=1``). DAOS is an independent filesystem choice: after loading its module, the script uses the module's ``DAOS_PRELOAD`` value for both ``mpiexec`` and its ranks and adds ``--no-vni``. A Flare run uses neither DAOS launch option. The composer does not yet offer a standard-MPI build mode; that mode would need its own verified runtime settings. Python API ---------- Import the Python API from ``utils.python.job_composer.compose_job``. Its entry point is ``compose_job(output_dir, platform=..., executable=..., nodes=..., account=..., walltime=..., ...)``. ``resolve_job(...)`` computes machine and workflow defaults without writing files; ``plan_job(...)`` also validates the input source and records chosen/default provenance. ``save_job_spec(path, ...)`` writes a validated ``job_spec.json``, and ``load_job_settings(path)`` restores actively chosen settings from either a saved spec or a package manifest. It deliberately omits the previous output directory. Testing ------- With ``BUILD_TESTING=ON``, CMake registers the standard-library composer suite as ``job_composer_python`` when it finds Python 3.9 or newer. If the same interpreter also has NumPy, CTest registers the input-package selection tests; if it also has Flask and Matplotlib, CTest registers the Job Composer GUI tests. Set ``Python3_EXECUTABLE`` at CMake configure time to select a suitable virtual environment. Run the registered tests from the build directory with ``ctest -L job_composer --output-on-failure``. They do not need XGC-Analysis or an XGC executable. To configure just these Python tests without XGC's compiler and library dependencies, use the standalone CMake project: .. code-block:: bash cmake -S utils/python/job_composer/tests -B build-job-composer \ -DPython3_EXECUTABLE=/path/to/python ctest --test-dir build-job-composer -L job_composer --output-on-failure The live ACA smoke test is kept separately in ``utils/python/job_composer/tests/integration_campaign.py`` and is not part of the default CTest suite. It requires a sibling XGC-Analysis checkout and an environment with ADIOS2, HPC-Campaign, and NumPy. Run it explicitly from the XGC-Devel source directory with ``python3 -m unittest discover -s utils/python/job_composer/tests -p integration_campaign.py -v``. Walltime and deadlock detection ------------------------------- Use ``--max-walltime`` to pass XGC ``-max_walltime`` as the scheduler limit minus a checkpoint margin, rounded down to whole minutes. ``--max-walltime-margin-minutes`` sets a positive margin (default 30) that must cover environment and input staging, the last time step, and writing the final restart; at least one whole minute must remain for XGC. The final restart also requires ``sml_restart_write_after_last_timestep``, which defaults to true. Leave this feature off for short debug jobs. A selected deadlock detector, online campaign packager, or Diffusion Optimizer resolves the launch mode to ``supervised``; otherwise it is ``direct``. Deadlock detection has a 10-minute startup grace and a 20-minute default inactivity timeout, adjustable with ``--deadlock-timeout-minutes`` only when detection is enabled. The first inactivity interval begins after the startup grace, so no initial complete line causes a default stall after 30 minutes. Later complete lines reset the timer. Generate a deadlock-monitored package without any companion commands: .. code-block:: bash python3 utils/python/job_composer/compose_job.py \ --platform polaris --executable /path/to/xgc \ --nodes 2 --account myproject --walltime 02:00:00 \ --deadlock-detection --deadlock-timeout-minutes 25 \ --output-dir /path/to/new-run The supervisor observes its explicitly configured XGC stdout file every 10 seconds for normal minute-scale timeouts, counts complete newline-terminated lines, and returns status 124 when the log stalls. It records the path, cumulative line count, last observed progress time, and idle duration as a JSON event in batch stderr before terminating XGC and any companions. A file that has not appeared yet is treated as quiet; truncation or replacement restarts reading from the beginning. Unflushed stdout and partial lines cannot be observed as progress, so check buffering on a short site run before using the detector for production. ``utils.python.job_composer.deadlock_detector`` can also be imported by other workflow adapters or run as an observer-only CLI: ``python3 utils/python/job_composer/deadlock_detector.py /path/to/xgc.out --timeout-minutes 20 --startup-grace-minutes 10``. Its CLI reports a stall with exit status 2 but does not cancel processes. Boolean composer options also accept ``--no-...`` forms, so a CLI override can explicitly turn an option off. The GUI exposes both built-in companion recipes. Online campaign packaging ------------------------- The thin supervisor uses only Python's standard library and can run under the site's Python 3.6. The built-in ``--online-campaign-packaging`` recipe requires ``--python-venv /absolute/path/to/venv``; its ``bin/python`` runs the packager. XGC-Analysis requires Python 3.10 or newer, and the venv must also provide ADIOS2 and HPC-Campaign. The batch script checks the interpreter version and imports before launching XGC. The earlier stage's ``--campaign-python`` setting is not used in this stage; when loading an older manifest, provide ``--python-venv`` for a new package. An explicit custom companion command controls its own Python environment. Optional ``--campaign-interval-seconds`` (default 300), ``--campaign-settle-seconds`` (default 15), and ``--campaign-finalize-seconds`` (default 600) tune polling, BP settling, and the post-XGC scan limit. The packager writes ``xgc_run.aca`` and ``xgc_run_manifest.json`` in the XGC working directory. It references attempt files under the package root, separate from the run directory on all current layouts. Generated live and final packager commands include ``--diagnostics``. Their stderr logs show flushed timestamps and elapsed times for each catalog phase and each HPC-Campaign manager operation. XGC may create BP5 directories before writing their first step. The live packager therefore uses a filename-only catalog scan and defers BP products whose index is still empty, then checks the remaining sources for the configured quiet period before ACA registration. The final forced scan does not wait for a quiet period, but still omits BP products with empty indexes. The packager applies one ACA update through a single worker process using HPC-Campaign's ``Manager`` API, so it opens the staged ACA once instead of starting one manager process per entry. The campaign Python environment must provide ``Manager.open``, ``close``, ``data``, ``text``, and ``add_time_series``. HPC-Campaign opens registered BP products with ADIOS2; its reads remain in the worker, where a native crash fails that update without killing the long-lived packager. A nonempty index and quiet period do not guarantee that a writer will not resume. After XGC exits, the supervisor gives the packager time to finish its scan before releasing other tasks. The batch script then snapshots per-job artifacts and runs one final ACA scan, so the archive includes the completed attempt's logs, timing, and artifact references. A failed final scan returns a nonzero batch status when XGC itself succeeded. A short debug job without ``--max-walltime`` may still run out of scheduler time before this final scan; choose enough walltime for packaging. Concurrent BP reader/writer behavior and remote replicas require site validation. Custom companions and supervision --------------------------------- Custom packager and optimizer commands remain available through a JSON array passed to ``--companion-tasks-file``. A custom packager command replaces the built-in command. For example: .. code-block:: json [{"name": "online_campaign_packaging", "argv": ["python3", "/path/to/packager.py"]}] Each command runs as a local process on the batch host, oversubscribing CPUs on the first allocated node. The generated ``supervisor.json`` records the XGC launch command and scheduler, XGC stdout and stderr paths, companion commands, output files, placement, and shutdown grace. ``launch_xgc.sh`` holds the platform's MPI launcher and XGC flags and uses ``exec`` so its shell is replaced by ``srun`` or ``mpiexec``; the supervisor therefore tracks the local launcher PID directly. The batch script sources the staged site environment, prepares the working directory, and executes ``python3 -u supervise_job.py``. Ensure that environment makes a standard-library Python 3 interpreter available on compute nodes. The supervisor owns task process groups. A companion that completes before XGC is treated as an error unless its entry sets ``"required_until_xgc_exit": false``. Set ``"on_failure": "continue"`` to report such a failure while XGC continues, or ``"stop_xgc"`` to stop the workflow. Campaign packaging defaults to ``continue``; Diffusion Optimizer defaults to ``stop_xgc``. XGC completion stops remaining companions and returns XGC's exit status. Cancellation or a fatal companion failure stops all tasks. On abnormal shutdown under Slurm, the supervisor also sends TERM and then KILL to active job steps through ``scancel --signal`` using ``SLURM_JOB_ID``, while leaving the batch script to collect artifacts. The provided commands must themselves be valid on the target machine; site smoke jobs remain necessary to verify placement, concurrent launcher behavior, and remote-rank termination. Diffusion Optimizer workflow ---------------------------- Select ``--diffusion-optimizer --optimizer-script /absolute/path/to/XGC-Analysis/Diffusion_Optimizer/diffusion_optimizer.py`` for the built-in optimizer recipe. The script path must be visible on compute nodes; provide either ``--python-venv`` or an absolute ``--optimizer-python`` path to a Python 3.10+ environment with the optimizer's dependencies. The supervisor starts the optimizer in the XGC working directory 120 seconds after launching XGC by default, giving ``xgc.heatdiag2.bp`` time to become readable. Set ``--optimizer-start-delay-seconds`` to a nonnegative integer to change this delay; zero starts it immediately. If XGC exits before the delay, the optimizer is not started. The delay does not itself verify BP readiness. The optimizer waits for its required static files and communicates through XGC's configured ADIOS streams. It uses the first node's CPU resources without a separate scheduler step. The batch script checks that the entry point and Python version are available after environment and DAOS setup. ``supervisor.json`` records its exact argv, delay, per-attempt stdout/stderr paths, and failure policy. An optimizer exit with nonzero status stops XGC immediately. A normal optimizer exit gives XGC up to 600 seconds to finish; if XGC is still running afterward, the supervisor stops it. This accommodates stream closure during XGC teardown while keeping the optimizer's 90-minute no-data timeout fatal to the workflow after that grace. For an unusually long teardown, edit the optimizer task's ``normal_exit_grace_seconds`` in the generated ``supervisor.json`` before submission. A custom optimizer companion can still be supplied in ``--companion-tasks-file``, but the GUI cannot reload or edit custom companion recipes. Every batch-safe ``diffusion_optimizer.py`` argument is available as a composer flag with an ``--optimizer-`` prefix: for example, ``--optimizer-update-damping-factor 0.3``, ``--optimizer-solver-method lsq``, ``--no-optimizer-use-time-avg-system``, ``--optimizer-momentum-rhs-taper 0.1 0.2 0.8 0.1``, and ``--optimizer-online-response-window 0.2 0.8``. Run ``python3 utils/python/job_composer/compose_job.py --help`` for the full list. The shared ``optimizer_options.json`` defines the default values, input types, and help used by the CLI and GUI. The resolved values and per-option chosen/default provenance are in ``job_manifest.json``. Only explicitly chosen values are passed to the optimizer; its own checkout supplies the other defaults. Check explicitly chosen flags against the installed XGC-Analysis optimizer CLI whenever using an older checkout; the composer does not import its scientific code. Interactive ``--debug-solver`` is excluded because it opens X11 plots and breakpoints, and ``--use-wall-loss`` from older examples is absent from the current optimizer CLI. The optimizer's scientific inputs and XGC's ADIOS stream configuration are not changed by the composer. The current recipe requires site smoke tests for the optimizer Python environment, live stream connection, concurrent launcher behavior, and clean shutdown on each target platform. The supervisor exposes ``XGC_STDOUT_PATH`` to companion processes. Logs, artifacts, and restarts ----------------------------- The generated script records UTC XGC start and end times and propagates XGC's exit status. Each scheduler submission writes its batch, XGC, and companion stdout/stderr under ``logs///``; XGC writes its timing files directly under ``timing///`` through its ``--timing-dir`` option. **Timing output compatibility change:** when launching the C++ XGC independently without ``--timing-dir``, XGC now creates ``./timing/`` and writes its timing files there. Earlier builds wrote them in the run directory when ``./timing/`` did not already exist. Update scripts that read ``timing_all.txt`` or ``adios_timer.txt`` from the run directory. XGC-S continues to use the run directory for its timing files. The post-run hook copies ``fort.input.used``, ``runinfo.dat``, ``TASKMAP_*.txt``, ``units.m``, ``units.txt``, ``Efield_excluded_*``, ``xgc.used_inputs.bp``, ``git_diff_XGC_core.txt``, ``wallsource.dat``, and ``m3dc1_eqd_out.eqd`` into ``artifacts///`` when present. Each artifact directory also contains ``job_attempt.json`` with timestamps, exit status, and the copied product names. These files remain at the working-directory root where XGC may need them. Legacy ``timing_*.dat`` files found there are copied to the per-job timing directory. The count is zero padded to at least four digits. A fresh run uses ``0001``; for a restart the script uses the second integer from ``timestep.dat`` in the XGC working directory plus one, matching XGC's checkpoint counter. The job ID keeps a retry distinct when it starts again from the same checkpoint. Composer jobs call ``setup_rundir.sh --in-place``: on a new run it stages only missing input files, preserving edits in the working directory; on a checkpointed run it requires the existing input files and ``restart_dir/`` without replacing them. The one-argument form remains for older quickstart scripts that create a separate ``rundir/``; it also preserves existing runtime inputs and checkpoints and copies only missing inputs. On a checkpointed composer run, the artifact hook requires ``sml_param:sml_restart=.true.`` in the working-directory ``input`` before it archives the previous attempt or launches XGC. A missing or false setting stops the job to protect the checkpoint and BP outputs. Before a subsequent attempt, the hook snapshots the previous attempt again, covering interruptions before the post-run hook completed. It rejects an existing checkpoint without a composer attempt marker so that an unmanaged run is not silently relabeled. A later submission without a checkpoint is rejected because it cannot safely resume and may overwrite BP outputs. For an uncheckpointed failed attempt, run ``./cleanup_failed_job.py --dry-run`` from the idle package to see its archive destination, then ``./cleanup_failed_job.py`` to move ``rundir/`` to ``failed_runs///`` and recreate an empty ``rundir/``. Existing per-job ``logs/``, ``timing/``, and ``artifacts/`` stay in place. Correct the source files in package ``input_dir/`` and submit again. The helper refuses a run with ``timestep.dat`` because it may be restartable, and refuses to act while the package lock is held. The hidden ``.xgc_run.lock`` file need not be removed: an unlocked file does not block another job. The helper also accepts older flat packages, archiving run-root entries outside their generated scaffold. DAOS cleanup needs its mounted work directory and is not supported by this local helper. Package locking and scheduler output ------------------------------------ The generated batch script acquires a nonblocking exclusive lock on ``.xgc_run.lock`` in the package before staging or launching XGC. It holds file descriptor 9 for the lifetime of the job, including child tasks. A second job using the same package exits rather than sharing its working directory. After an interrupted job has released the lock, the artifact hook can recover a marker still labeled ``running``; direct callers of the hook must hold the same exclusive lock before requesting recovery. This uses Python's standard-library ``fcntl`` module and requires working advisory file locks on the package filesystem. The batch host needs standard-library Python 3 even for direct jobs; ``job_artifacts.py`` uses Python 3.6 compatible APIs. A lock failure reports the OS error and errno so a filesystem that does not support ``flock`` can be distinguished from an active job. Include package locking on the intended filesystem in each site's smoke test; the job stops if the lock is unavailable. The scheduler captures startup failures before the script redirects its own streams to the per-job log directory. Slurm places those bootstrap files at ``logs/batch-.out/.err``; PBS uses ``logs/batch.out/.err``, which can be overwritten by a later PBS submission. The per-job ``batch.out/.err`` capture the script after setup. PBS directives contain the package's absolute path, so recompose after moving a PBS bundle. The post-run hook cannot execute after an abrupt scheduler kill; the next submission performs its recovery snapshot if the same working directory is available.