Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion docs/ADRs/001_ontology_of_views_postprocessing.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,21 @@

# ADR-001: Ontology of views-postprocessing

**Status:** Accepted
**Status:** Superseded by [ADR-012](012_revised_ontology.md) (2026-06-27)
**Date:** 2026-06-02
**Deciders:** Project maintainers (PRIO MD&D Team)

---

> ⚠️ **Superseded.** This ADR's ontology was built around a runtime spatial-mapping
> engine and bundled shapefiles as the authoritative, stable core. That architecture was
> removed (ADR-011 / C-39): enrichment is now a precomputed GAUL lookup-join and the repo
> is a post-forecast delivery + input-integrity layer. The **current** ontology is
> [ADR-012](012_revised_ontology.md). This document is kept intact as the historical record
> of the original design.

---

## Context

The views-postprocessing repository bridges conflict prediction outputs from the VIEWS platform to external partner consumption formats. It operates at the intersection of spatial data, pipeline orchestration, and cloud delivery.
Expand Down
64 changes: 63 additions & 1 deletion docs/ADRs/002_topology_and_dependency_rules.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,20 @@
# ADR-002: Topology and Dependency Rules

**Status:** Accepted
**Status:** Accepted — amended 2026-06-27 (see [Amendment](#amendment-2026-06-27))
**Date:** 2026-06-02
**Deciders:** Project maintainers (PRIO MD&D Team)

---

> **Note (2026-06-27):** the *decision* below — dependencies must be acyclic and flow
> downward — still holds. Its **illustrative internal layering** (Pipeline Managers →
> Spatial Mapping Engine → Geographic Data Assets) is stale: the Spatial Mapping Engine and
> Geographic Data Assets layers were removed (ADR-011 / C-39). The current internal structure
> and the cross-repo topology this ADR originally omitted are in the
> [Amendment](#amendment-2026-06-27) at the bottom. Original text preserved as the record.

---

## Context

In complex systems, architectural fragility often emerges not from incorrect
Expand Down Expand Up @@ -115,3 +124,56 @@ It does not define:

Topology governs structure.
Contracts govern interaction.

---

## Amendment (2026-06-27)

This amendment updates the *illustration* of the dependency rule to current reality and adds
the **cross-repo topology** the original ADR omitted. The rule itself (acyclic, downward) is
unchanged.

### Cross-repo topology

views-postprocessing is one stage in a one-way platform pipeline. Dependencies flow **down**:

```
views-datafactory (produces data)
views-pipeline-core (the framework: lifecycle, data loader, dataset, datastore tools)
views-postprocessing (THIS REPO — post-forecast delivery + input-integrity)
views-faoapi (serves the delivered data; collapses draws)
```

- This repo **depends on** views-pipeline-core (it subclasses its postprocessor base —
Template Method) and views-frames (the frame contract). It **does not** depend on faoapi.
- **views-pipeline-core does not depend on this repo** (verified: zero imports). The
dependency is strictly one-way; a cycle here would be an architectural defect per the rule
above.
- **views-models** is the runner/composition root — it constructs this repo's manager and
calls `.execute()`; it is not a dependency *of* this repo.

### Current internal layering (replaces the stale illustration)

```
Pipeline Manager (UNFAOPostProcessorManager — orchestration; a pipeline-core subclass)
↓ calls
Delivery Invariants (views_postprocessing/delivery/ — representation-free rules)
↑ fed primitives by
Representation Seam (unfao/extraction.py — the only pandas-aware module)
↓ alongside
Enrichment (GaulLookupEnricher + data/gaul_lookup.parquet) · Producer Data-Facts (source_metadata.py)
```

- The manager **calls** the delivery invariants; it never inherits them. Invariants depend
only on primitives (DIP), so a representation change lands in the seam alone (OCP / C-40).
- Geography is a precomputed asset consumed by the enricher — there is no runtime spatial
engine layer anymore.

### See also

- The narrative version of this topology and the internal seams:
[`docs/architecture/role_and_seams.md`](../architecture/role_and_seams.md).
- The revised ontology these categories come from: [ADR-012](012_revised_ontology.md).
83 changes: 83 additions & 0 deletions docs/ADRs/012_revised_ontology.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# ADR-012: Revised ontology of views-postprocessing (post-lookup migration)

**Status:** Accepted
**Date:** 2026-06-27
**Deciders:** Project maintainers (PRIO MD&D Team)
**Supersedes:** [ADR-001](001_ontology_of_views_postprocessing.md)

---

## Context

[ADR-001](001_ontology_of_views_postprocessing.md) defined the repository's ontology around
a **runtime spatial-mapping engine** and **bundled shapefiles** as the authoritative,
stable core. That architecture no longer exists:

- [ADR-011](011_replace_runtime_mapper_with_precomputed_lookup.md) replaced the runtime
geopandas mapper with a **precomputed GAUL lookup table** (`GaulLookupEnricher`).
- The mapper, shapefiles, and caching machinery were deleted (C-39 / PR #42).
- Input-integrity invariants and structured delivery were added (epic #51).

So ADR-001's core categories ("Geographic Data Assets", "Spatial Mapping Engine") describe
deleted code. This ADR restates the ontology to match what the repository **actually is**:
a **post-forecast delivery + input-integrity layer**, not a spatial-mapping library and not
a statistical post-processor. (For the wider narrative — role vs the sibling repos and the
internal seams — see [`docs/architecture/role_and_seams.md`](../architecture/role_and_seams.md).)

---

## Decision

The repository defines a **closed set of conceptual categories**. Anything that does not
clearly belong to one is out of scope and must be redesigned or rejected.

### Core ontological categories (current)

| Category | Purpose | Authority | Stability |
|----------|---------|-----------|-----------|
| **Delivery Invariants** | Representation-free rules over primitives that a delivery must satisfy (coverage, forecast identity, observed-range, provenance). Live in `views_postprocessing/delivery/`. | Authoritative — they define what a valid delivery is | Stable — changes are governance decisions |
| **Representation Seam** | The single pandas-aware module (`unfao/extraction.py`) that turns the delivery's external representation into the primitives the invariants consume. | Derived — isolates the representation so invariants stay representation-free | Evolving — the one place a representation change (e.g. pandas → frames, C-40) lands |
| **Enrichment Asset + Engine** | The precomputed GAUL lookup (`data/gaul_lookup.parquet`) and the merge that joins it (`GaulLookupEnricher`). | Authoritative for geographic metadata | Stable — the lookup is rebuilt only when the producer (datafactory) releases new GAUL data |
| **Pipeline Manager** | The thin `UNFAOPostProcessorManager` — a concrete pipeline-core postprocessor (Template-Method subclass) that *orchestrates* read/transform/validate/save and **calls** the invariants (never inherits them). | Derived — implements delivery using the categories above | Evolving — changes as partner requirements change |
| **Producer Data-Facts** | Data-related facts sourced straight from the producer (datafactory) — e.g. `last_valid_month_id` — isolated in `unfao/source_metadata.py`. | Authoritative (the producer is the source of truth, D-07) | Evolving |
| **External Service Configurations** | Appwrite connection configs / env vars / bucket references. | Operational | Evolving |
| **Derived Outputs** | Enriched parquet files produced per run and delivered to the partner store. | Ephemeral | Ephemeral |

### Explicitly *not* in this repository's ontology

- **Spatial intersection / shapefiles / a runtime mapping engine** — removed (ADR-011/C-39);
geography is precomputed upstream and consumed as a lookup.
- **Statistical post-processing** — draw-collapse (MAP/HDI) is downstream in views-faoapi
(`views_frames_summarize`); reconciliation is in `views_frames_reconcile`. This repo
*preserves* forecast values and delivers them; it does not transform them.

---

## Rationale

- **Screaming architecture:** the categories now match the package layout — `delivery/`
(invariants), `unfao/extraction.py` (seam), `unfao/enrichment.py` + `data/` (enrichment),
`unfao/managers/` (orchestration). A reader can infer responsibilities from the structure.
- **DIP / OCP:** primitives are the abstraction the invariants depend on; the representation
seam is the single point of change for a representation migration (C-40), so the invariants
are closed against it.
- **Honest scope:** naming the repo a delivery + integrity layer (not a postprocessor that
does statistics) prevents the recurring confusion about where collapse/reconciliation live.

---

## Consequences

**Positive:** the ontology is verifiable against the code; onboarding docs (README, the
role-&-seams doc) can derive from it; the C-40 representation migration has a named seam.

**Negative:** when the representation migrates (pandas → frames) this ADR's "Representation
Seam" wording will need a light touch (the seam stays; its internals change).

---

## Notes

- Stability is a design constraint, not a preference (inherited from ADR-001 §Stability Rules).
- Dependency direction and cross-repo topology are governed by [ADR-002](002_topology_and_dependency_rules.md).
- Class-level contracts are governed by ADR-006 (see `docs/CICs/`).
11 changes: 9 additions & 2 deletions docs/ADRs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@ These ADRs define system philosophy and governance:
Establishes the ADR practice itself.

- **ADR-001** — Ontology of views-postprocessing
Defines what concepts exist.
⚠️ **Superseded by ADR-012** (the original shapefile/spatial-engine ontology; kept as the
historical record).

- **ADR-002** — Topology and Dependency Rules
Defines structural dependency direction.
Defines structural dependency direction. **Amended 2026-06-27** with the cross-repo
topology + current internal layering.

- **ADR-003** — Authority of Declarations Over Inference
Defines where semantic authority lives.
Expand Down Expand Up @@ -59,6 +61,11 @@ These ADRs form the architectural constitution of the repository.
- **ADR-011** — Replace Runtime Mapper with Precomputed Lookup Table
Replaces the 3,122-line runtime spatial mapper with a ~65K-row precomputed Parquet lookup table. Area-majority assignment confirmed as FAO contractual requirement.

- **ADR-012** — Revised Ontology (post-lookup migration)
Supersedes ADR-001. Restates the ontology to the current reality: a post-forecast delivery
+ input-integrity layer (delivery invariants, the representation seam, the lookup
enrichment, the thin manager) — no runtime spatial engine.

ADRs numbered 010 and above define:

- Domain-specific decisions
Expand Down
Loading