Data Access Architecture
========================

This page describes the current XGC-Analysis data-access layout and the
catalog/read-plan path that is being introduced to make readers independent of
physical filenames.  It is intended as an orientation page for developers.  The
API reference pages generated from docstrings remain the authoritative source
for individual classes and functions.

Current implementation status
-----------------------------

XGC-Analysis currently has a catalog/read-plan framework with two separable
backend responsibilities:

* Catalog discovery backends answer which products, variables, and logical
  steps exist.  The directory backend lives in
  :mod:`xgc_analysis.catalog.directory_catalog`; the campaign backend lives in
  :mod:`xgc_analysis.catalog.campaign_catalog`.
* Source-reader backends read selected variables and ADIOS steps from one
  physical source.  Catalog reads use the FileReader-backed
  :class:`xgc_analysis.adios_file_reader.AdiosFileSourceReader` interface.  Directory
  catalogs open and close local BP sources per batched read; campaign catalogs
  keep the ACA FileReader open and reuse it for read-plan execution.

Everything above these two backend layers uses common catalog objects:
:class:`xgc_analysis.catalog.DataProduct`,
:class:`xgc_analysis.catalog.SourceInfo`,
:class:`xgc_analysis.catalog.StepInfo`, and
:class:`xgc_analysis.catalog.ReadPlan`.  Product readers should only depend on
those common objects and on raw values returned by a source reader; they should
not care whether the source is a local BP directory or an HPC-Campaign file.

Most runtime readers for field, moment, source, neutral, heat, one-dimensional,
sheath, and distribution-function products now use this catalog/read-plan path.
The standalone ``read_bp_file`` helper remains available for low-level scripts
and for readers that have not yet been converted.  The main remaining package
reader in that category is ``diffusion_data.py``, whose products have
workflow-specific logical structure that still needs a separate catalog
integration pass.

``Simulation`` is a client of the reading framework.  It owns shared
interpretation objects such as :class:`xgc_analysis.mesh.Mesh`,
:class:`xgc_analysis.magnetic_field.MagneticField`, and
:class:`xgc_analysis.velocity_grid.VelocityGrid`, but it should not define the
backend mechanics.

Catalog discovery
-----------------

The catalog is constructed with:

.. code-block:: python

   from xgc_analysis.catalog import open_catalog, open_campaign_catalog

   directory_catalog = open_catalog("/path/to/xgc/output")
   campaign_catalog = open_campaign_catalog("/path/to/xgc/output/campaign.aca")

The directory backend scans the dataset root for entries ending in ``.bp``.  It
groups physical files into logical XGC products.  For example:

.. code-block:: text

   xgc.3d.00010.bp
   xgc.3d.00012.bp

become one product keyed as:

.. code-block:: text

   xgc.3d.bp

During this scan the backend first builds an internal grouping equivalent to:

.. code-block:: python

   grouped = {
       "xgc.3d.bp": [
           (10, Path("xgc.3d.00010.bp")),
           (12, Path("xgc.3d.00012.bp")),
       ],
       "xgc.oneddiag.bp": [
           (None, Path("xgc.oneddiag.bp")),
       ],
   }

For each physical source it creates a
:class:`xgc_analysis.catalog.SourceInfo` object:

.. code-block:: python

   SourceInfo(
       source_id="xgc.3d.00010.bp",
       path=Path(".../xgc.3d.00010.bp"),
       file_index=10,
       mtime=...,
       variables={
           "pot": VariableInfo(...),
           "time": VariableInfo(...),
       },
       step_values=[...],
       time_values=[...],
       adios_step_count=...,
   )

``step_values`` are read from the first available logical-step variable in this
order:

* ``gstep``
* ``step``
* ``timestep``

``time_values`` are read from ``time`` when available.  If metadata collection
is disabled or a source cannot be opened, the catalog still records the source
path and filename-derived information so the dataset can be inspected at a
coarser level.

The source records are then grouped into a
:class:`xgc_analysis.catalog.DataProduct`:

.. code-block:: python

   DataProduct(
       key="xgc.3d.bp",
       label="xgc.3d.bp",
       product_type=ProductType.FIELD_3D,
       layout=ProductLayout.FILE_SEQUENCE,
       sources=[SourceInfo(...), SourceInfo(...)],
       variables={
           "pot": VariableInfo(...),
           "time": VariableInfo(...),
       },
       product_family="field",
   )

The final catalog stores products in:

.. code-block:: python

   catalog.products: dict[str, DataProduct]

with product keys such as ``"xgc.3d.bp"``, ``"xgc.2d.bp"``, and
``"xgc.oneddiag.bp"``.

The campaign backend starts from campaign-qualified ADIOS variable names such
as:

.. code-block:: text

   xgc.f2d.00010.bp/e_den
   xgc.diffusion_coeff.bp/gstep

It strips the source prefix when building product metadata, so readers still
see variables as ``"e_den"`` or ``"gstep"``.  The prefix remains in
``SourceInfo.source_id`` so the campaign source reader can reconstruct the
qualified ADIOS name at read time.

Logical steps
-------------

The catalog exposes the logical steps available for a product and optionally
for a single variable:

.. code-block:: python

   steps = catalog.available_steps("xgc.3d.bp", "pot")

Internally this creates candidate source fragments:

.. code-block:: python

   candidates: dict[int, list[StepFragment]]

where the dictionary key is the logical XGC step.  Each
:class:`xgc_analysis.catalog.StepFragment` maps one logical step to one
physical source and ADIOS step:

.. code-block:: python

   StepFragment(
       source_id="xgc.3d.00010.bp",
       source_path=Path(".../xgc.3d.00010.bp"),
       logical_step=10,
       adios_step=0,
       file_index=10,
       time=...,
   )

Logical step assignment follows this policy:

* Use explicit ``gstep``, ``step``, or ``timestep`` values when available.
* Otherwise, for file-sequence products, use ``file_index + adios_step``.
* Otherwise, use the ADIOS step ordinal.

Several physical fragments can advertise the same logical step.  The catalog
selects one fragment using a newer-source-wins policy based primarily on source
modification time.  The returned data structure is a list of
:class:`xgc_analysis.catalog.StepInfo` objects:

.. code-block:: python

   StepInfo(
       logical_step=10,
       selected_fragment=StepFragment(...),
       duplicate_fragments=[StepFragment(...), ...],
       dedup_policy="newer_source_wins",
   )

Read planning
-------------

A read plan resolves a user request in logical XGC steps into the physical
source fragments that must be opened:

.. code-block:: python

   plan = catalog.plan_read(
       "xgc.3d.bp",
       "pot",
       [10, 11],
       missing="raise",
   )

The catalog first builds:

.. code-block:: python

   step_map: dict[int, StepInfo]

Then it groups selected fragments by source:

.. code-block:: python

   grouped = {
       ("xgc.3d.bp", Path(".../xgc.3d.bp")): [
           (10, 0),
           (11, 1),
       ],
   }

The returned :class:`xgc_analysis.catalog.ReadPlan` contains one or more
:class:`xgc_analysis.catalog.ReadFragment` objects:

.. code-block:: python

   ReadPlan(
       product_key="xgc.3d.bp",
       variable="pot",
       requested_steps=[10, 11],
       fragments=[
           ReadFragment(
               source_id="xgc.3d.bp",
               source_path=Path(".../xgc.3d.bp"),
               variable="pot",
               logical_steps=[10, 11],
               adios_steps=[0, 1],
           ),
       ],
       missing_steps=[],
       missing_policy=MissingStepPolicy.RAISE,
   )

Missing logical steps are handled during planning:

* ``missing="raise"`` raises immediately if a requested step is unavailable.
* ``missing="skip"`` omits unavailable steps from the fragments and records
  them in ``plan.missing_steps``.
* ``missing="zero"`` also records unavailable steps in ``plan.missing_steps``.
  The executor does not synthesize zero arrays because only the product reader
  knows the correct wrapped type, shape, and dtype.

Read-plan execution
-------------------

The executor reads one or more plans through a source-reader backend:

.. code-block:: python

   from xgc_analysis.catalog import execute_read_plan, execute_read_plans

   execution = execute_read_plan(plan)
   batched_execution = execute_read_plans([plan_for_eden, plan_for_time])

For regular local BP files, the default backend is
:func:`xgc_analysis.catalog.read_regular_bp_steps`, which now delegates to
``adios2.FileReader`` and reads exact ADIOS steps with ``step_selection``.  The
batched executor groups all fragments by physical source, so each local BP
source is opened once for the union of requested variables and ADIOS steps in
that batch.

Campaign catalogs carry a default ``catalog.source_reader`` backed by the same
FileReader source-reader class.  Campaign discovery keeps an ACA FileReader
open for metadata refreshes, but data reads use a fresh ACA FileReader per
batched read by default because remote campaign handles can become unusable
after earlier reads.  The source-reader result has the same shape for both
backends:

.. code-block:: python

   {
       adios_step: {
           "variable_name": value,
       },
   }

This is the main backend boundary.

Static product reads
--------------------

Static support products are accessed with a thinner helper because they do not
need a user-facing logical step range:

.. code-block:: python

   from xgc_analysis.catalog import read_static_variables

   values = read_static_variables(
       catalog,
       "xgc.f0.mesh.bp",
       ["f0_nmu", "f0_nvp", "f0_dsmu", "f0_dvp"],
   )

Internally this still builds ordinary one-step read plans at logical step ``0``
and executes them through :func:`xgc_analysis.catalog.execute_read_plans`, so
directory and campaign reads use the same source-reader boundary.  Static
classes now require the requested products to be present in the catalog.  Direct
hard-coded local BP fallbacks are disabled so missing catalog metadata fails
early and visibly.

``Simulation`` is the preferred owner for shared static interpretation state.
When a catalog is supplied, ``Simulation`` passes it into
:class:`xgc_analysis.mesh.Mesh`,
:class:`xgc_analysis.magnetic_field.MagneticField`, and
:class:`xgc_analysis.velocity_grid.VelocityGrid`.  The velocity grid is then
available as ``simulation.velocity_grid`` and should be reused by
distribution-function workflows instead of being rediscovered by each reader.

Raw catalog reads
-----------------

The low-level raw reader for finite source reads is
:func:`xgc_analysis.adios_file_reader.read_adios_file_steps`.  It returns plain values
keyed by ADIOS step and variable name, and is the implementation used by the
catalog source-reader classes.

The catalog convenience method :meth:`SimulationCatalog.read_arrays
<xgc_analysis.catalog.directory_catalog.SimulationCatalog.read_arrays>` reads
advertised catalog variables directly into NumPy arrays without constructing
``Simulation`` or a product-specific reader.  This supports lightweight GUI and
scripting workflows that only need to inspect data values, dimensions, metadata,
or quick plots.  The helper sits below product-specific wrapping and uses the
same read-plan/source-reader backend boundary:

.. code-block:: python

   arrays = catalog.read_arrays(
       product_key="xgc.2d.bp",
       variables=["eden", "time"],
       steps=[10, 20],
   )

The returned :class:`xgc_analysis.catalog.CatalogArrayRead` contains
``arrays[var][logical_step]``, raw read-plan ``records`` for source/step
provenance, and ``missing_steps_by_variable``.  Specialized readers continue to
return higher-level objects such as ``PlaneData``, ``MeshData``, and
``DistributionFunctionField``.

The low-level source read result is step-first because it mirrors physical
ADIOS iteration:

.. code-block:: python

   source_data = {
       0: {
           "eden": np.ndarray(...),
           "time": np.ndarray(...),
       },
       1: {
           "eden": np.ndarray(...),
           "time": np.ndarray(...),
       },
   }

This structure is local to source reading.  It is not the final reader storage
layout.

The executor converts the source data into one
:class:`xgc_analysis.catalog.ReadPlanRecord` per variable and logical step:

.. code-block:: python

   ReadPlanRecord(
       logical_step=10,
       source_id="xgc.3d.bp",
       source_path=Path(".../xgc.3d.bp"),
       adios_step=0,
       variable="pot",
       value=np.ndarray(...),
   )

The final :class:`xgc_analysis.catalog.ReadPlanExecution` contains:

.. code-block:: python

   ReadPlanExecution(
       plan=plan,
       records=[ReadPlanRecord(...), ...],
       missing_steps=plan.missing_steps,
   )

Convenience views are available:

.. code-block:: python

   execution.records_by_logical_step()
   execution.values_by_logical_step()
   execution.records_in_requested_order()

Existing reader storage
-----------------------

Existing reader classes store loaded values in a variable-first dictionary:

.. code-block:: python

   reader.data[var_name][step_index] = value

For field-like products, ``value`` may be a
:class:`xgc_analysis.plane_data.PlaneData`,
:class:`xgc_analysis.mesh_data.MeshData`, scalar value, or NumPy array.
Distribution-function products additionally use
:class:`xgc_analysis.distribution_function_data.DistributionFunctionField`,
which depends on both the mesh and ``simulation.velocity_grid`` for physical
interpretation.

For example:

.. code-block:: python

   field.data["eden"][0] = PlaneData(...)
   field.data["time"][0] = 1.234

Reader-local step metadata are stored separately:

.. code-block:: python

   field.step_index_info[0] = {
       "file_index": 10,
       "bp_step": 0,
   }

The disabled legacy direct-file path used to perform these steps:

* Initialize ``BPReaderMixin`` state:

  .. code-block:: python

     self.requested_vars
     self.read_all_steps
     self.step_index_info = {}
     self._requested_var_templates = {}
     self.data = {}

* Construct filenames such as ``xgc.2d.00000.bp`` or ``xgc.3d.00000.bp``.
* Open each BP source directly through a reader-local helper.
* Receive physical source data:

  .. code-block:: python

     file_data: dict[adios_step, dict[var_name, value]]

* Register each loaded ADIOS step:

  .. code-block:: python

     step_index = self._register_step(file_index, bp_step)

* Wrap arrays as product-specific objects.
* Store the wrapped values in:

  .. code-block:: python

     self.data[var_name][step_index] = wrapped_value

This path is retained only as historical context.  Converted readers now raise
clear errors from private direct-file helpers and read through catalog
``ReadPlan`` execution instead.

The catalog path preserves the same final reader storage layout:

.. code-block:: python

   ReadPlan
       -> execute_read_plan(plan)
       -> ReadPlanExecution.records
       -> reader.data[var_name][step_index]

The corresponding step metadata can then include both logical and physical
coordinates:

.. code-block:: python

   reader.step_index_info[step_index] = {
       "logical_step": record.logical_step,
       "source_id": record.source_id,
       "source_path": record.source_path,
       "bp_step": record.adios_step,
   }

Distribution-function reader ownership follows the same principle.  The
preferred construction style is:

.. code-block:: python

   f0 = DistributionFunctionData(
       simulation=simulation,
       steps=[2000],
       variables=["ion_f", "time"],
   )

The reader inherits ``simulation.mesh``, ``simulation.velocity_grid``, and
``simulation.catalog`` unless explicit overrides are supplied.  Legacy
``file_indices`` are still accepted as a selector and are interpreted as logical
steps when ``steps`` is omitted.  This keeps velocity-grid interpretation
centralized in ``Simulation`` while still allowing tests and specialized
workflows to pass separate objects.

Backend boundary
----------------

The executor accepts a ``source_reader`` callable with this shape:

.. code-block:: python

   source_reader(source_path, variables, adios_steps) -> {
       adios_step: {
           variable: value,
       },
   }

The default implementation reads regular BP files with FileReader.  The
campaign-backed implementation uses the same callable interface with an open ACA
FileReader and campaign-qualified variable names.  Code above the executor can
therefore consume ``ReadPlanExecution`` or ``BatchedReadPlanExecution`` records
without knowing whether the data came from local BP files, a mixed local layout,
or an HPC-Campaign source.