Data Access Architecture

This page describes the current XGC-Analysis data-access layout and the catalog/read-plan path that is being introduced to make readers independent of physical filenames. It is intended as an orientation page for developers. The API reference pages generated from docstrings remain the authoritative source for individual classes and functions.

Current implementation status

XGC-Analysis currently has a catalog/read-plan framework with two separable backend responsibilities:

• Catalog discovery backends answer which products, variables, and logical steps exist. The directory backend lives in xgc_analysis.catalog.directory_catalog; the campaign backend lives in xgc_analysis.catalog.campaign_catalog.

• Source-reader backends read selected variables and ADIOS steps from one physical source. Catalog reads use the FileReader-backed xgc_analysis.adios_file_reader.AdiosFileSourceReader interface. Directory catalogs open and close local BP sources per batched read. Campaign catalogs keep an ACA FileReader open for metadata discovery and refresh, while data reads use a fresh ACA FileReader per batched read by default because remote campaign handles can become unusable after earlier reads.

Everything above these two backend layers uses common catalog objects: xgc_analysis.catalog.DataProduct, xgc_analysis.catalog.SourceInfo, xgc_analysis.catalog.StepInfo, and xgc_analysis.catalog.ReadPlan. Product readers should only depend on those common objects and on raw values returned by a source reader; they should not care whether the source is a local BP directory or an HPC-Campaign file.

Runtime readers for field, moment, source, neutral, heat, one-dimensional, sheath, distribution-function, and read-only diffusion products now use this catalog/read-plan path. Static products used by Mesh, Plane, MagneticField, and VelocityGrid use the same catalog source-reader interface through static-product helpers. The standalone read_bp_file and csr_matrix.sparse_matrix_reader helpers remain available for low-level scripts that intentionally operate on a raw BP filename instead of a Simulation/catalog environment.

Simulation is a client of the reading framework. It owns shared interpretation objects such as xgc_analysis.mesh.Mesh, xgc_analysis.magnetic_field.MagneticField, and xgc_analysis.velocity_grid.VelocityGrid, but it should not define the backend mechanics.

Run manifests and text artifacts

Directory-backed catalogs now attach a run manifest generated by xgc_analysis.catalog.build_run_manifest(). The manifest is the shared filesystem scan that will also feed the campaign packager. It records:

• top-level BP products grouped into static data products or file-sequence time series;

• standard top-level text inputs such as input, fort.input.used, adios2cfg.xml, and petsc.rc;

• support files referenced by string values in input when a recursive filename match below the run root finds a text, BP, or HDF5 file. The manifest stores only matched artifacts, not every string candidate extracted from input;

• stdout-like text logs detected by the marker pair XGC_FLAGS: and GIT INFO. The run-info entry stores structured build metadata, preprocessor flags, Git remotes, the origin URL, the Git HEAD commit hash, and whether uncommitted changes were reported.

The directory catalog keeps manifest paths relative to the run root. Text artifacts are exposed through:

catalog.list_texts()
catalog.has_text("input")
input_text = catalog.read_text("input")

This gives Simulation and GUI code a catalog path for input/profile/config text instead of constructing local paths directly. Initialized Simulation objects require the catalog to provide the input text artifact, and file-based species profiles are read through catalog.read_text(...) using the path names from the input namelist. Campaign catalogs expose embedded text datasets through the same methods when the archive contains text artifacts added by the XGC campaign packager.

Catalog inspection APIs

The dataclass-returning APIs remain the structured Python interface used by readers and scripts:

catalog.list_products()
catalog.list_variables("xgc.2d.bp")
catalog.available_steps("xgc.2d.bp", "dpot")

For loosely coupled tools, GUI code, command-line wrappers, or JSON export, use the versioned machine summary:

summary = catalog.to_summary(
    include_variables=True,
    include_sources=False,
    include_texts=True,
    include_provenance=False,
    max_steps=20,
)

to_summary() returns only JSON-like builtins and starts with:

{
    "schema": "xgc-catalog-summary-v1",
    "backend": "directory",
    "root": "...",
    "product_count": 12,
    "products": {
        "xgc.2d.bp": {
            "type": "field_2d",
            "layout": "file_sequence",
            "variable_count": 18,
            "coordinates": {
                "gstep": {"count": 9, "first": 10, "last": 90, "sample": [10, 20]},
                "tindex": {"count": 9, "first": 0, "last": 8, "sample": [0, 1]},
                "time": {"count": 9, "first": 1.0e-6, "last": 9.0e-6, "sample": [...]},
            },
            "variables": {
                "dpot": {
                    "dtype": "double",
                    "shape": [1, 20698],
                    "units": "V",
                    "description": "...",
                }
            },
        }
    },
}

The gstep coordinate is the catalog’s logical XGC step coordinate. It is read from gstep when present and otherwise from the accepted legacy aliases step or timestep. tindex is the contiguous output-slice coordinate when an output writes that variable. time is the physical time in seconds when available. Missing coordinates are omitted rather than synthesized.

Set include_provenance=True when a caller needs backend/source context such as campaign archive path, source paths, modification times, file indices, and ADIOS step counts. The default omits full paths to keep summaries compact and safe to display.

For interactive browsing, use the name-only helpers and human-readable string helpers:

catalog.product_names()
catalog.variable_names("xgc.2d.bp")
catalog.text_names()
print(catalog)                         # calls catalog.overview()
catalog.overview(max_vars=3)
catalog.describe("xgc.2d.bp")
catalog.describe("xgc.2d.bp:dpot")
catalog.describe("dpot")

These methods are intentionally additive. They do not replace the dataclass APIs, and product readers should continue to use read plans instead of parsing human-readable text.

The command-line packager can be run as:

python -m xgc_analysis.catalog.package_xgc_run /path/to/run \
    --manifest /path/to/run/xgc_run_manifest.json

Adding --archive run.aca writes the manifest and then calls hpc_campaign manager to add BP data, file-sequence time-series definitions, and embedded text artifacts. --dry-run prints the campaign commands without executing them, which is the recommended way to inspect packaging for a new platform or HPC-Campaign version.

Catalog discovery

The catalog is constructed with:

from xgc_analysis.catalog import open_catalog, open_campaign_catalog

directory_catalog = open_catalog("/path/to/xgc/output")
campaign_catalog = open_campaign_catalog("/path/to/xgc/output/campaign.aca")

The directory backend scans the dataset root for entries ending in .bp. It groups physical files into logical XGC products. For example:

xgc.3d.00010.bp
xgc.3d.00012.bp

become one product keyed as:

xgc.3d.bp

During this scan the backend first builds an internal grouping equivalent to:

grouped = {
    "xgc.3d.bp": [
        (10, Path("xgc.3d.00010.bp")),
        (12, Path("xgc.3d.00012.bp")),
    ],
    "xgc.oneddiag.bp": [
        (None, Path("xgc.oneddiag.bp")),
    ],
}

For each physical source it creates a xgc_analysis.catalog.SourceInfo object:

SourceInfo(
    source_id="xgc.3d.00010.bp",
    path=Path(".../xgc.3d.00010.bp"),
    file_index=10,
    mtime=...,
    variables={
        "pot": VariableInfo(...),
        "time": VariableInfo(...),
    },
    step_values=[...],
    time_values=[...],
    adios_step_count=...,
)

step_values are read from the first available logical-step variable in this order:

• gstep

• step

• timestep

time_values are read from time when available. If metadata collection is disabled or a source cannot be opened, the catalog still records the source path and filename-derived information so the dataset can be inspected at a coarser level.

The source records are then grouped into a xgc_analysis.catalog.DataProduct:

DataProduct(
    key="xgc.3d.bp",
    label="xgc.3d.bp",
    product_type=ProductType.FIELD_3D,
    layout=ProductLayout.FILE_SEQUENCE,
    sources=[SourceInfo(...), SourceInfo(...)],
    variables={
        "pot": VariableInfo(...),
        "time": VariableInfo(...),
    },
    product_family="field",
)

The final catalog stores products in:

catalog.products: dict[str, DataProduct]

with product keys such as "xgc.3d.bp", "xgc.2d.bp", and "xgc.oneddiag.bp".

The campaign backend starts from campaign-qualified ADIOS variable names such as:

xgc.f2d.00010.bp/e_den
xgc.diffusion_coeff.bp/gstep

It strips the source prefix when building product metadata, so readers still see variables as "e_den" or "gstep". The prefix remains in SourceInfo.source_id so the campaign source reader can reconstruct the qualified ADIOS name at read time.

Logical steps

The catalog exposes the logical steps available for a product and optionally for a single variable:

steps = catalog.available_steps("xgc.3d.bp", "pot")

Internally this creates candidate source fragments:

candidates: dict[int, list[StepFragment]]

where the dictionary key is the logical XGC step. Each xgc_analysis.catalog.StepFragment maps one logical step to one physical source and ADIOS step:

StepFragment(
    source_id="xgc.3d.00010.bp",
    source_path=Path(".../xgc.3d.00010.bp"),
    logical_step=10,
    adios_step=0,
    file_index=10,
    time=...,
)

Logical step assignment follows this policy:

• Use explicit gstep, step, or timestep values when available.

• Otherwise, for file-sequence products, use file_index + adios_step.

• Otherwise, use the ADIOS step ordinal.

Several physical fragments can advertise the same logical step. The catalog selects one fragment using a newer-source-wins policy based primarily on source modification time. The returned data structure is a list of xgc_analysis.catalog.StepInfo objects:

StepInfo(
    logical_step=10,
    selected_fragment=StepFragment(...),
    duplicate_fragments=[StepFragment(...), ...],
    dedup_policy="newer_source_wins",
)

Read planning

A read plan resolves a user request in logical XGC steps into the physical source fragments that must be opened:

plan = catalog.plan_read(
    "xgc.3d.bp",
    "pot",
    [10, 11],
    missing="raise",
)

The catalog first builds:

step_map: dict[int, StepInfo]

Then it groups selected fragments by source:

grouped = {
    ("xgc.3d.bp", Path(".../xgc.3d.bp")): [
        (10, 0),
        (11, 1),
    ],
}

The returned xgc_analysis.catalog.ReadPlan contains one or more xgc_analysis.catalog.ReadFragment objects:

ReadPlan(
    product_key="xgc.3d.bp",
    variable="pot",
    requested_steps=[10, 11],
    fragments=[
        ReadFragment(
            source_id="xgc.3d.bp",
            source_path=Path(".../xgc.3d.bp"),
            variable="pot",
            logical_steps=[10, 11],
            adios_steps=[0, 1],
        ),
    ],
    missing_steps=[],
    missing_policy=MissingStepPolicy.RAISE,
)

Missing logical steps are handled during planning:

• missing="raise" raises immediately if a requested step is unavailable.

• missing="skip" omits unavailable steps from the fragments and records them in plan.missing_steps.

• missing="zero" also records unavailable steps in plan.missing_steps. The executor does not synthesize zero arrays because only the product reader knows the correct wrapped type, shape, and dtype.

Read-plan execution

The executor reads one or more plans through a source-reader backend:

from xgc_analysis.catalog import execute_read_plan, execute_read_plans

execution = execute_read_plan(plan)
batched_execution = execute_read_plans([plan_for_eden, plan_for_time])

For regular local BP files, the default backend is xgc_analysis.catalog.read_regular_bp_steps(), which now delegates to adios2.FileReader and reads exact ADIOS steps with step_selection. The batched executor groups all fragments by physical source, so each local BP source is opened once for the union of requested variables and ADIOS steps in that batch.

Campaign catalogs carry a default catalog.source_reader that exposes the same callable interface as the directory FileReader source reader. Campaign discovery keeps an ACA FileReader open for metadata refreshes, but data reads use a fresh ACA FileReader per batched read by default because remote campaign handles can become unusable after earlier reads. The source-reader result has the same shape for both backends:

{
    adios_step: {
        "variable_name": value,
    },
}

This is the main backend boundary.

Static product reads

Static support products are accessed with a thinner helper because they do not need a user-facing logical step range:

from xgc_analysis.catalog import read_static_variables

values = read_static_variables(
    catalog,
    "xgc.f0.mesh.bp",
    ["f0_nmu", "f0_nvp", "f0_dsmu", "f0_dvp"],
)

Internally this still builds ordinary one-step read plans at logical step 0 and executes them through xgc_analysis.catalog.execute_read_plans(), so directory and campaign reads use the same source-reader boundary. Static classes now require the requested products to be present in the catalog. Direct hard-coded local BP fallbacks are disabled so missing catalog metadata fails early and visibly.

Simulation is the preferred owner for shared static interpretation state. When a catalog is supplied, Simulation passes it into xgc_analysis.mesh.Mesh, xgc_analysis.magnetic_field.MagneticField, and xgc_analysis.velocity_grid.VelocityGrid. The velocity grid is then available as simulation.velocity_grid and should be reused by distribution-function workflows instead of being rediscovered by each reader.

Raw catalog reads

The low-level raw reader for finite source reads is xgc_analysis.adios_file_reader.read_adios_file_steps(). It returns plain values keyed by ADIOS step and variable name, and is the implementation used by the catalog source-reader classes.

The catalog convenience method SimulationCatalog.read_arrays reads advertised catalog variables directly into NumPy arrays without constructing Simulation or a product-specific reader. This supports lightweight GUI and scripting workflows that only need to inspect data values, dimensions, metadata, or quick plots. The helper sits below product-specific wrapping and uses the same read-plan/source-reader backend boundary:

arrays = catalog.read_arrays(
    product_key="xgc.2d.bp",
    variables=["eden", "time"],
    steps=[10, 20],
)

The returned xgc_analysis.catalog.CatalogArrayRead contains arrays[var][logical_step], raw read-plan records for source/step provenance, and missing_steps_by_variable. Specialized readers continue to return higher-level objects such as PlaneData, MeshData, and DistributionFunctionField.

The low-level source read result is step-first because it mirrors physical ADIOS iteration:

source_data = {
    0: {
        "eden": np.ndarray(...),
        "time": np.ndarray(...),
    },
    1: {
        "eden": np.ndarray(...),
        "time": np.ndarray(...),
    },
}

This structure is local to source reading. It is not the final reader storage layout.

The executor converts the source data into one xgc_analysis.catalog.ReadPlanRecord per variable and logical step:

ReadPlanRecord(
    logical_step=10,
    source_id="xgc.3d.bp",
    source_path=Path(".../xgc.3d.bp"),
    adios_step=0,
    variable="pot",
    value=np.ndarray(...),
)

The final xgc_analysis.catalog.ReadPlanExecution contains:

ReadPlanExecution(
    plan=plan,
    records=[ReadPlanRecord(...), ...],
    missing_steps=plan.missing_steps,
)

Convenience views are available:

execution.records_by_logical_step()
execution.values_by_logical_step()
execution.records_in_requested_order()

Existing reader storage

Existing reader classes store loaded values in a variable-first dictionary:

reader.data[var_name][step_index] = value

For field-like products, value may be a xgc_analysis.plane_data.PlaneData, xgc_analysis.mesh_data.MeshData, scalar value, or NumPy array. Distribution-function products additionally use xgc_analysis.distribution_function_data.DistributionFunctionField, which depends on both the mesh and simulation.velocity_grid for physical interpretation.

For example:

field.data["eden"][0] = PlaneData(...)
field.data["time"][0] = 1.234

Reader-local step metadata are stored separately:

field.step_index_info[0] = {
    "file_index": 10,
    "bp_step": 0,
}

The disabled legacy direct-file path used to perform these steps:

• Initialize BPReaderMixin state:

  self.requested_vars
  self.read_all_steps
  self.step_index_info = {}
  self._requested_var_templates = {}
  self.data = {}
  

• Construct filenames such as xgc.2d.00000.bp or xgc.3d.00000.bp.

• Open each BP source directly through a reader-local helper.

• Receive physical source data:

  file_data: dict[adios_step, dict[var_name, value]]
  

• Register each loaded ADIOS step:

  step_index = self._register_step(file_index, bp_step)
  

• Wrap arrays as product-specific objects.

• Store the wrapped values in:

  self.data[var_name][step_index] = wrapped_value
  

This path is retained only as historical context. Converted readers now raise clear errors from private direct-file helpers and read through catalog ReadPlan execution instead.

The catalog path preserves the same final reader storage layout:

ReadPlan
    -> execute_read_plan(plan)
    -> ReadPlanExecution.records
    -> reader.data[var_name][step_index]

The corresponding step metadata can then include both logical and physical coordinates:

reader.step_index_info[step_index] = {
    "logical_step": record.logical_step,
    "source_id": record.source_id,
    "source_path": record.source_path,
    "bp_step": record.adios_step,
}

Distribution-function reader ownership follows the same principle. The preferred construction style is:

f0 = DistributionFunctionData(
    simulation=simulation,
    steps=[2000],
    variables=["ion_f", "time"],
)

The reader inherits simulation.mesh, simulation.velocity_grid, and simulation.catalog unless explicit overrides are supplied. Legacy file_indices are still accepted as a selector and are interpreted as logical steps when steps is omitted. This keeps velocity-grid interpretation centralized in Simulation while still allowing tests and specialized workflows to pass separate objects.

Backend boundary

The executor accepts a source_reader callable with this shape:

source_reader(source_path, variables, adios_steps) -> {
    adios_step: {
        variable: value,
    },
}

The default implementation reads regular BP files with FileReader. The campaign-backed implementation uses the same callable interface with an open ACA FileReader and campaign-qualified variable names. Code above the executor can therefore consume ReadPlanExecution or BatchedReadPlanExecution records without knowing whether the data came from local BP files, a mixed local layout, or an HPC-Campaign source.