Data Access Architecture

This page describes the current XGC-Analysis data-access layout and the catalog/read-plan path that is being introduced to make readers independent of physical filenames. It is intended as an orientation page for developers. The API reference pages generated from docstrings remain the authoritative source for individual classes and functions.

Current implementation status

XGC-Analysis currently has a catalog/read-plan framework with two separable backend responsibilities:

Everything above these two backend layers uses common catalog objects: xgc_analysis.catalog.DataProduct, xgc_analysis.catalog.SourceInfo, xgc_analysis.catalog.StepInfo, and xgc_analysis.catalog.ReadPlan. Product readers should only depend on those common objects and on raw values returned by a source reader; they should not care whether the source is a local BP directory or an HPC-Campaign file.

Most runtime readers for field, moment, source, neutral, heat, one-dimensional, sheath, and distribution-function products now use this catalog/read-plan path. The standalone read_bp_file helper remains available for low-level scripts and for readers that have not yet been converted. The main remaining package reader in that category is diffusion_data.py, whose products have workflow-specific logical structure that still needs a separate catalog integration pass.

Simulation is a client of the reading framework. It owns shared interpretation objects such as xgc_analysis.mesh.Mesh, xgc_analysis.magnetic_field.MagneticField, and xgc_analysis.velocity_grid.VelocityGrid, but it should not define the backend mechanics.

Catalog discovery

The catalog is constructed with:

from xgc_analysis.catalog import open_catalog, open_campaign_catalog

directory_catalog = open_catalog("/path/to/xgc/output")
campaign_catalog = open_campaign_catalog("/path/to/xgc/output/campaign.aca")

The directory backend scans the dataset root for entries ending in .bp. It groups physical files into logical XGC products. For example:

xgc.3d.00010.bp
xgc.3d.00012.bp

become one product keyed as:

xgc.3d.bp

During this scan the backend first builds an internal grouping equivalent to:

grouped = {
    "xgc.3d.bp": [
        (10, Path("xgc.3d.00010.bp")),
        (12, Path("xgc.3d.00012.bp")),
    ],
    "xgc.oneddiag.bp": [
        (None, Path("xgc.oneddiag.bp")),
    ],
}

For each physical source it creates a xgc_analysis.catalog.SourceInfo object:

SourceInfo(
    source_id="xgc.3d.00010.bp",
    path=Path(".../xgc.3d.00010.bp"),
    file_index=10,
    mtime=...,
    variables={
        "pot": VariableInfo(...),
        "time": VariableInfo(...),
    },
    step_values=[...],
    time_values=[...],
    adios_step_count=...,
)

step_values are read from the first available logical-step variable in this order:

  • gstep

  • step

  • timestep

time_values are read from time when available. If metadata collection is disabled or a source cannot be opened, the catalog still records the source path and filename-derived information so the dataset can be inspected at a coarser level.

The source records are then grouped into a xgc_analysis.catalog.DataProduct:

DataProduct(
    key="xgc.3d.bp",
    label="xgc.3d.bp",
    product_type=ProductType.FIELD_3D,
    layout=ProductLayout.FILE_SEQUENCE,
    sources=[SourceInfo(...), SourceInfo(...)],
    variables={
        "pot": VariableInfo(...),
        "time": VariableInfo(...),
    },
    product_family="field",
)

The final catalog stores products in:

catalog.products: dict[str, DataProduct]

with product keys such as "xgc.3d.bp", "xgc.2d.bp", and "xgc.oneddiag.bp".

The campaign backend starts from campaign-qualified ADIOS variable names such as:

xgc.f2d.00010.bp/e_den
xgc.diffusion_coeff.bp/gstep

It strips the source prefix when building product metadata, so readers still see variables as "e_den" or "gstep". The prefix remains in SourceInfo.source_id so the campaign source reader can reconstruct the qualified ADIOS name at read time.

Logical steps

The catalog exposes the logical steps available for a product and optionally for a single variable:

steps = catalog.available_steps("xgc.3d.bp", "pot")

Internally this creates candidate source fragments:

candidates: dict[int, list[StepFragment]]

where the dictionary key is the logical XGC step. Each xgc_analysis.catalog.StepFragment maps one logical step to one physical source and ADIOS step:

StepFragment(
    source_id="xgc.3d.00010.bp",
    source_path=Path(".../xgc.3d.00010.bp"),
    logical_step=10,
    adios_step=0,
    file_index=10,
    time=...,
)

Logical step assignment follows this policy:

  • Use explicit gstep, step, or timestep values when available.

  • Otherwise, for file-sequence products, use file_index + adios_step.

  • Otherwise, use the ADIOS step ordinal.

Several physical fragments can advertise the same logical step. The catalog selects one fragment using a newer-source-wins policy based primarily on source modification time. The returned data structure is a list of xgc_analysis.catalog.StepInfo objects:

StepInfo(
    logical_step=10,
    selected_fragment=StepFragment(...),
    duplicate_fragments=[StepFragment(...), ...],
    dedup_policy="newer_source_wins",
)

Read planning

A read plan resolves a user request in logical XGC steps into the physical source fragments that must be opened:

plan = catalog.plan_read(
    "xgc.3d.bp",
    "pot",
    [10, 11],
    missing="raise",
)

The catalog first builds:

step_map: dict[int, StepInfo]

Then it groups selected fragments by source:

grouped = {
    ("xgc.3d.bp", Path(".../xgc.3d.bp")): [
        (10, 0),
        (11, 1),
    ],
}

The returned xgc_analysis.catalog.ReadPlan contains one or more xgc_analysis.catalog.ReadFragment objects:

ReadPlan(
    product_key="xgc.3d.bp",
    variable="pot",
    requested_steps=[10, 11],
    fragments=[
        ReadFragment(
            source_id="xgc.3d.bp",
            source_path=Path(".../xgc.3d.bp"),
            variable="pot",
            logical_steps=[10, 11],
            adios_steps=[0, 1],
        ),
    ],
    missing_steps=[],
    missing_policy=MissingStepPolicy.RAISE,
)

Missing logical steps are handled during planning:

  • missing="raise" raises immediately if a requested step is unavailable.

  • missing="skip" omits unavailable steps from the fragments and records them in plan.missing_steps.

  • missing="zero" also records unavailable steps in plan.missing_steps. The executor does not synthesize zero arrays because only the product reader knows the correct wrapped type, shape, and dtype.

Read-plan execution

The executor reads one or more plans through a source-reader backend:

from xgc_analysis.catalog import execute_read_plan, execute_read_plans

execution = execute_read_plan(plan)
batched_execution = execute_read_plans([plan_for_eden, plan_for_time])

For regular local BP files, the default backend is xgc_analysis.catalog.read_regular_bp_steps(), which now delegates to adios2.FileReader and reads exact ADIOS steps with step_selection. The batched executor groups all fragments by physical source, so each local BP source is opened once for the union of requested variables and ADIOS steps in that batch.

Campaign catalogs carry a default catalog.source_reader backed by the same FileReader source-reader class. Campaign discovery keeps an ACA FileReader open for metadata refreshes, but data reads use a fresh ACA FileReader per batched read by default because remote campaign handles can become unusable after earlier reads. The source-reader result has the same shape for both backends:

{
    adios_step: {
        "variable_name": value,
    },
}

This is the main backend boundary.

Static product reads

Static support products are accessed with a thinner helper because they do not need a user-facing logical step range:

from xgc_analysis.catalog import read_static_variables

values = read_static_variables(
    catalog,
    "xgc.f0.mesh.bp",
    ["f0_nmu", "f0_nvp", "f0_dsmu", "f0_dvp"],
)

Internally this still builds ordinary one-step read plans at logical step 0 and executes them through xgc_analysis.catalog.execute_read_plans(), so directory and campaign reads use the same source-reader boundary. Static classes now require the requested products to be present in the catalog. Direct hard-coded local BP fallbacks are disabled so missing catalog metadata fails early and visibly.

Simulation is the preferred owner for shared static interpretation state. When a catalog is supplied, Simulation passes it into xgc_analysis.mesh.Mesh, xgc_analysis.magnetic_field.MagneticField, and xgc_analysis.velocity_grid.VelocityGrid. The velocity grid is then available as simulation.velocity_grid and should be reused by distribution-function workflows instead of being rediscovered by each reader.

Raw catalog reads

The low-level raw reader for finite source reads is xgc_analysis.adios_file_reader.read_adios_file_steps(). It returns plain values keyed by ADIOS step and variable name, and is the implementation used by the catalog source-reader classes.

The catalog convenience method SimulationCatalog.read_arrays reads advertised catalog variables directly into NumPy arrays without constructing Simulation or a product-specific reader. This supports lightweight GUI and scripting workflows that only need to inspect data values, dimensions, metadata, or quick plots. The helper sits below product-specific wrapping and uses the same read-plan/source-reader backend boundary:

arrays = catalog.read_arrays(
    product_key="xgc.2d.bp",
    variables=["eden", "time"],
    steps=[10, 20],
)

The returned xgc_analysis.catalog.CatalogArrayRead contains arrays[var][logical_step], raw read-plan records for source/step provenance, and missing_steps_by_variable. Specialized readers continue to return higher-level objects such as PlaneData, MeshData, and DistributionFunctionField.

The low-level source read result is step-first because it mirrors physical ADIOS iteration:

source_data = {
    0: {
        "eden": np.ndarray(...),
        "time": np.ndarray(...),
    },
    1: {
        "eden": np.ndarray(...),
        "time": np.ndarray(...),
    },
}

This structure is local to source reading. It is not the final reader storage layout.

The executor converts the source data into one xgc_analysis.catalog.ReadPlanRecord per variable and logical step:

ReadPlanRecord(
    logical_step=10,
    source_id="xgc.3d.bp",
    source_path=Path(".../xgc.3d.bp"),
    adios_step=0,
    variable="pot",
    value=np.ndarray(...),
)

The final xgc_analysis.catalog.ReadPlanExecution contains:

ReadPlanExecution(
    plan=plan,
    records=[ReadPlanRecord(...), ...],
    missing_steps=plan.missing_steps,
)

Convenience views are available:

execution.records_by_logical_step()
execution.values_by_logical_step()
execution.records_in_requested_order()

Existing reader storage

Existing reader classes store loaded values in a variable-first dictionary:

reader.data[var_name][step_index] = value

For field-like products, value may be a xgc_analysis.plane_data.PlaneData, xgc_analysis.mesh_data.MeshData, scalar value, or NumPy array. Distribution-function products additionally use xgc_analysis.distribution_function_data.DistributionFunctionField, which depends on both the mesh and simulation.velocity_grid for physical interpretation.

For example:

field.data["eden"][0] = PlaneData(...)
field.data["time"][0] = 1.234

Reader-local step metadata are stored separately:

field.step_index_info[0] = {
    "file_index": 10,
    "bp_step": 0,
}

The disabled legacy direct-file path used to perform these steps:

  • Initialize BPReaderMixin state:

    self.requested_vars
self.read_all_steps
self.step_index_info = {}
self._requested_var_templates = {}
self.data = {}

  • Construct filenames such as xgc.2d.00000.bp or xgc.3d.00000.bp.

  • Open each BP source directly through a reader-local helper.

  • Receive physical source data:

    file_data: dict[adios_step, dict[var_name, value]]

  • Register each loaded ADIOS step:

    step_index = self._register_step(file_index, bp_step)

  • Wrap arrays as product-specific objects.

  • Store the wrapped values in:

    self.data[var_name][step_index] = wrapped_value

This path is retained only as historical context. Converted readers now raise clear errors from private direct-file helpers and read through catalog ReadPlan execution instead.

The catalog path preserves the same final reader storage layout:

ReadPlan
    -> execute_read_plan(plan)
    -> ReadPlanExecution.records
    -> reader.data[var_name][step_index]

The corresponding step metadata can then include both logical and physical coordinates:

reader.step_index_info[step_index] = {
    "logical_step": record.logical_step,
    "source_id": record.source_id,
    "source_path": record.source_path,
    "bp_step": record.adios_step,
}

Distribution-function reader ownership follows the same principle. The preferred construction style is:

f0 = DistributionFunctionData(
    simulation=simulation,
    steps=[2000],
    variables=["ion_f", "time"],
)

The reader inherits simulation.mesh, simulation.velocity_grid, and simulation.catalog unless explicit overrides are supplied. Legacy file_indices are still accepted as a selector and are interpreted as logical steps when steps is omitted. This keeps velocity-grid interpretation centralized in Simulation while still allowing tests and specialized workflows to pass separate objects.

Backend boundary

The executor accepts a source_reader callable with this shape:

source_reader(source_path, variables, adios_steps) -> {
    adios_step: {
        variable: value,
    },
}

The default implementation reads regular BP files with FileReader. The campaign-backed implementation uses the same callable interface with an open ACA FileReader and campaign-qualified variable names. Code above the executor can therefore consume ReadPlanExecution or BatchedReadPlanExecution records without knowing whether the data came from local BP files, a mixed local layout, or an HPC-Campaign source.