Skip to content

S7 — historical inbound read_dataframe/PGMDataset retirement #92

Description

@Polichinel

Epic: #85 · S7 · [CROSS-REPO-GATED] · blocked · gate: pipeline-core C-40 · DO NOT MOVE YET

Background

The historical inbound is forced pandas: read_dataframePGMDataset (unfao.py:58-64), plus the pandas ops in _append_metadata (.reset_index/.set_index/.join, :142-149) and _validate (.isnull().sum(), :165,177). These are inherited from pipeline-core's PostprocessorManager base, whose _read/_transform/_validate contract is defined in terms of PGMDataset (pandas with object-dtype cells).

Why this is tracked but NOT actioned now

  • It is forced, not own-choice. Moving it unilaterally would fork the inherited base-class contract and maximally disturb cross-repo interconnectivity — the opposite of the strategy.
  • It unblocks only when pipeline-core ships its own DataFrame retirement (register C-40; pipeline-core epic #186/#207, issues #161/#159/#211).
  • Samples live on the forecast frame, not the historical frame — so S7 never blocks the sample work. This story exists to record the gate and the eventual change, not to be done in this epic.

Work (eventual, post-gate)

  • Replace read_dataframe/PGMDataset with frame-native loaders; convert _append_metadata/_validate pandas ops to numpy/frame operations.

Acceptance criteria (eventual)

  • Historical path holds a frame, not a PGMDataset; _append_metadata/_validate are pandas-free.
  • Parity: identical delivered output to today.

Dependencies / gate

Blocked on pipeline-core C-40 (#186/#207/#161/#159/#211). Independent of S1–S6.

Files

views_postprocessing/unfao/managers/unfao.py (_read_historical_data, _append_metadata, _validate).

Metadata

Metadata

Assignees

No one assigned

    Labels

    blockedBlocked on a dependency or decisionimplementationCode implementation workstoryA single reviewable unit of an epic

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions