matthewmoorcroft · Sep 16, 2026 · Sep 17, 2026 · Sep 18, 2026 · Sep 18, 2026 · Sep 18, 2026
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
@@ -11,6 +11,7 @@
   "skills": [
     "./skills/flowx-setup",
     "./skills/flowx-discover",
+    "./skills/flowx-enrich",
     "./skills/flowx-convert",
     "./skills/flowx-package",
     "./skills/flowx-migrate"

diff --git a/README.md b/README.md
@@ -97,6 +97,7 @@ Or run individual phases:
 
 ```
 /flowx:flowx-discover    # Parse the source (ADF JSON / Airflow DAGs), produce inventory + complexity report
+/flowx:flowx-enrich      # Validate and record agent-authored insights on the inventory
 /flowx:flowx-convert     # Deterministic + agentic translation
 /flowx:flowx-package     # Generate DABs project
 ```
@@ -258,6 +259,7 @@ flowx_output/
   metadata/
     inventory.json            # discover: activity inventory
     source_graphs.json        # discover (ADF): saved source graphs the inventory is built from
+    agentic_insights.json     # enrich: validated agent-authored insights, rebuilt into inventory.json
     profile_report.csv        # discover: per-pipeline complexity report
     <pipeline>.arm.json       # discover: verbatim original ADF/ARM source
     configuration.json        # modify: collected configuration answers

diff --git a/app/README.md b/app/README.md
@@ -22,6 +22,7 @@ operation; `parameters` is its keyword-argument dict.
 | `convert` | `adapter convert` | ADF activities → Databricks IR |
 | `merge_agentic` | `adapter convert --merge-agentic` | Merge ADF agent-produced results into the report |
 | `resolve_agentic` | `adapter resolve-agentic` | Prepare, stage, and apply reviewed Airflow leaf-gap resolutions |
+| `enrich` | `adapter enrich` | Return `inventory.json` (`prepare`), then validate and record agent-authored insights (`apply`) |
 | `inspect` | `adapter inspect` | Surface pending translation options |
 | `apply_answers` | `adapter modify` | Apply answers → stamped IR |
 | `materialize_lookup` | `adapter materialize-lookup` | CSV → lookup-values JSON |
@@ -110,7 +111,7 @@ at `https://<app-url>/mcp`, and (3) **stateless** — this server sets
 `stateless_http=True` and adds CORS, so it qualifies. If a browser CORS error appears,
 set the app env var `FLOWX_ALLOWED_ORIGINS` to your workspace URL and redeploy.
 MCP access is capped at **20 tools** across all servers; flowx exposes just **one** tool
-(`flowx`, with 12 commands), so it uses a single slot.
+(`flowx`, covering every command in the table above), so it uses a single slot.
 
 > In serverless Genie Code, deploy with the `app/deploy_app.py` notebook (SDK-based). `./app/deploy.sh`
 > must run from a Databricks CLI session (workspace web terminal or a local machine), since

diff --git a/docs/content/docs/architecture.mdx b/docs/content/docs/architecture.mdx
@@ -41,7 +41,7 @@ The reporting layer (`reporting/`) can write per-run coverage to a Unity Catalog
                                   │ subprocess (same contract)
                     ┌─────────────────────────────┐
    MCP tool  ──────▶│  flowx.mcp (FastMCP)     │
-   (Claude / Genie) │  1 tool, 12 commands →       │
+   (Claude / Genie) │  1 tool, all commands →      │
                     │  adapter bridge              │
                     └─────────────────────────────┘
 ```
@@ -59,7 +59,7 @@ The unified `flowx.adapter` CLI is the single contract. Both surfaces go through
 | `mcp/runner.py`   | Subprocess bridge to `flowx.adapter` with artifact summarizers (for running translation without the `mcp` dependency) |
 | `mcp/__main__.py` | `python -m flowx.mcp` entry point (stdio default, `--http` for hosting)                                               |
 
-The `flowx` tool's `command` selects the adapter operation: `inputs`, `discover`, `convert`, `merge_agentic` (ADF only), `resolve_agentic` (Airflow only), `inspect`, `apply_answers`, `materialize_lookup`, `workspace_paths`, `package`, `migrate`, `record_results`, and `install_dashboard` (with `parameters` carrying that command's arguments).
+The `flowx` tool's `command` selects the adapter operation, with `parameters` carrying that command's arguments; the command table in `app/README.md` lists every command and the adapter call it wraps.
 
 ## Deployment topology
 

diff --git a/docs/content/docs/installation.mdx b/docs/content/docs/installation.mdx
@@ -132,7 +132,7 @@ flowx is distributed through a Claude Code marketplace. It can be installed as a
 You can also copy the skill folders straight into your local skills directory:
 
 ```shell
-cp -R skills/{flowx-setup,flowx-discover,flowx-convert,flowx-package,flowx-migrate} ~/.claude/skills/
+cp -R skills/{flowx-setup,flowx-discover,flowx-enrich,flowx-convert,flowx-package,flowx-migrate} ~/.claude/skills/
 ```
 </Tab>
 </Tabs>

diff --git a/skills/flowx-discover/SKILL.md b/skills/flowx-discover/SKILL.md
@@ -65,7 +65,7 @@ All under the shared `<output_dir>/metadata/` folder:
 | File | Description |
 |---|---|
 | `metadata/inventory.json` | Classified activity inventory for the convert phase |
-| `metadata/source_graphs.json` | (ADF) Saved, content-hashed source graphs (activities, lineage, detected motifs); the inventory is built from it and records its hash as `source_graphs_sha256` |
+| `metadata/source_graphs.json` | (ADF) The saved source graphs (activities, lineage and detected motifs), versioned and content-hashed; the inventory is built from it and records its hash as `source_graphs_sha256` |
 | `metadata/profile_report.csv` | Per-pipeline complexity report (counts + T-shirt size) |
 | `metadata/<pipeline>.arm.json` | (ADF) Verbatim original source for each pipeline (provenance) |
 
@@ -75,7 +75,29 @@ The inventory classifies every task into one of three strategies:
 - **Agentic** — requires LLM-assisted translation from the source definition.
 - **Unsupported** — no known translation path; needs manual intervention.
 
+## Step 3 — Enrich the inventory (default next step)
+
+Once the deterministic inventory is written, the standard flow **chains into enrichment**: you author
+a layer of judgment the parser cannot derive — a factory-wide architecture recommendation,
+per-pipeline intent + recommended Databricks patterns, and cross-pipeline relationships. The library
+records it in `metadata/agentic_insights.json` and rebuilds `inventory.json` with it under a single
+additive `insights` key. flowx contains no LLM: you author the JSON, the library validates and
+records it. The routing step (`flowx-route`) reads this block to present the
+agentic conversion option per pipeline group.
+
+**Continue with the `flowx-enrich` skill by default** — it guides authoring the insights and running
+`enrich`. Enrichment is additive and leaves every deterministic inventory key byte-identical, so it
+never destabilizes discover's output.
+
+**Deterministic-only skip path.** A standalone, headless discover — no agent, no LLM — is fully
+valid: `metadata/inventory.json` is complete and self-standing without an `insights` block, and
+`flowx-route` still recommends and records a plan from the deterministic structure alone. Skip
+`flowx-enrich` only when the caller explicitly asked for a deterministic-only pass, or when no human
+reader needs a migration narrative.
+
 ## Reference
 
 - `sources/adf.md` — Azure Data Factory discovery (ARM JSON, UC-volume download, complexity report)
 - `sources/airflow.md` — Apache Airflow discovery (DAG `.py` parsing, operator classification)
+- `flowx-enrich` skill — authoring the agentic `insights` layer and running `enrich` (the default
+  next step)
diff --git a/skills/flowx-enrich/SKILL.md b/skills/flowx-enrich/SKILL.md
@@ -0,0 +1,191 @@
+---
+name: flowx-enrich
+description: >
+  Enrich the discover inventory with an agent-authored layer of judgment — a factory-wide
+  architecture recommendation, per-pipeline intent + recommended Databricks patterns, and
+  cross-pipeline relationships. The library validates it, records it in agentic_insights.json and
+  rebuilds inventory.json from discovery plus that file. The default next step after flowx-discover
+  and the input the routing step consumes.
+triggers:
+  - "enrich inventory"
+  - "enrich pipelines"
+  - "author insights"
+  - "agentic insights"
+  - "recommend databricks patterns"
+  - "annotate inventory"
+  - "enrich discover"
+---
+
+# Enrich the Inventory with Agentic Insights
+
+The deterministic discover pass records what each source workflow **is**; it cannot record what to
+**do** about it. That judgment — a factory-wide architectural recommendation, each pipeline's intent
+and recommended Databricks patterns, and how pipelines couple — is authored by **you, the agent**,
+and recorded by the library in its own file, `metadata/agentic_insights.json`, bound to the saved
+`metadata/source_graphs.json` it was checked against. The library then rebuilds
+`metadata/inventory.json` from `source_graphs.json` plus that file, so it carries the same block under
+a single additive `insights` key; you never pass it anything for that step.
+
+This is the standard step **between discover and route** in the flowx workflow. `flowx-discover`
+chains into this skill by default; the routing step (`flowx-route`) reads the `insights` block to
+present the agentic conversion option per connected component. The one-shot MCP `migrate` command
+runs discover, convert and package without it, so it records no insights.
+
+## No LLM inside flowx: you author, the library validates and records
+
+**There is no LLM inside flowx.** You author the insights JSON; the library (`enrich`) only
+*validates and records* it, using the same author, validate, record, fingerprint-bound contract the
+agentic gap-resolution and routing paths use. That keeps the deterministic inventory trustworthy and
+every insight accountable: foreign keys must point at real pipelines, and every cross-pipeline edge
+is either an annotation of a proven lineage edge or an explicitly-flagged inference with cited
+evidence.
+
+`enrich` is **additive**: it writes `agentic_insights.json` and adds only the matching `insights`
+block to the inventory, leaving every existing inventory key byte-identical. It changes no conversion, IR, or routing decision on its own — the `insights` are
+descriptive data that `flowx-route` later consumes.
+
+## When to skip enrich (deterministic-only)
+
+Enrichment is on by default, but it is skippable. A standalone, deterministic-only discover — no
+agent, no LLM — is fully valid: `metadata/inventory.json` from discover is complete and self-standing
+without an `insights` block, and `flowx-route` still recommends and records a plan from the
+deterministic structure alone (the agentic option simply shows no recommended patterns). Skip enrich
+when the caller asked for a headless/deterministic pass, or when no human reader needs a migration
+narrative.
+
+## How to author (three steps)
+
+1. **Read the deterministic inventory.** Load `<output_dir>/metadata/inventory.json` (on the hosted
+   MCP server, through `enrich` with `action="prepare"`; see below). Note every
+   pipeline `name` (these are the valid foreign keys), and each pipeline's `lineage` block — in
+   particular `lineage.control_edges`, each `{source_workflow, target_workflow, via_task_key}`. A
+   deterministic **control** relationship you annotate must match one of these exactly, and only an
+   edge marked `"resolved": true` can be annotated. Such an edge can be annotated even when its
+   `target_workflow` is not a listed pipeline (discover ran with `--pipeline`, or the called pipeline
+   has no activities); every other relationship endpoint, and every `pipeline_insights[].pipeline`,
+   must be a listed pipeline. When the
+   inventory records a top-level `source_graphs_sha256` (ADF discover does), copy it into
+   `authored_against` by default, so enrich can refuse your insights if discover runs again before
+   you submit them; when it records none, leave `authored_against` out.
+2. **Read the source artifacts** you need to form judgment — the per-pipeline `raw` payloads in the
+   inventory, the ADF `metadata/<pipeline>.arm.json` provenance, or the DAG source — enough to state
+   each pipeline's *intent* and the Databricks patterns that fit. Ground every recommended pattern in
+   a **real, publicly-documented** Databricks capability; never invent a product name.
+
+   **Verify GA/Preview status before recommending a connector or Lakeflow Connect pattern, then
+   record it as a structured field.** Do **not** hardcode GA/Preview status or dates — release state
+   changes. Before recommending a connector or a Lakeflow Connect ingestion pattern (or naming it in
+   `recommended_patterns` / `system_recommendation`), you **must** verify against the **current public
+   Databricks docs** both that (a) the feature's GA-vs-Preview release state is acceptable and (b) it
+   is available in the target workspace. Verification happens **before** the recommendation, never
+   after: never emit a "recommend now, verify later" conditional.
+
+   Record the resolved state on the pattern with the **structured `release_state` field** — *not* prose
+   buried in `fit` / `conversion_notes` — and cite the doc that grounds it in `release_state_source`.
+   The validator enforces this. The state is a factual **disclosure**, not a warning; `flowx-route`
+   surfaces it as a neutral label (no alarm), tiered as:
+
+   - `ga` — Generally Available. **Silent** (not surfaced).
+   - `unknown` — the release state could not be verified. **Silent**, treated exactly like `ga` (there
+     is no separate label — the two are indistinguishable in the surfaced output).
+   - `public_preview` — surfaced as the informational label **"Public Preview (production-ready)"**:
+     Public Preview is generally production-ready and supported per Databricks. Still confirm the
+     feature is available in the target workspace. A cited `release_state_source` is required.
+   - `private_preview` — surfaced as the plain factual label **"Private Preview"**. A cited
+     `release_state_source` is required.
+   - `beta` — surfaced as the plain factual label **"Beta"**. A cited `release_state_source` is
+     required.
+
+   `release_state` is **required** on any pattern you mark `simplification_pattern: true` (a distinctive
+   capability must declare its verified release state). There is **no recommendation-eligibility gate** on
+   top of that: even a verified **Private Preview** feature may be suggested. **Disclose the release state
+   and let the customer decide** — record the verified state on the structured `release_state` /
+   `release_state_source` fields, note that Private Preview typically requires workspace
+   enrollment/entitlement, and leave the decision to proceed to the customer. The tool does **not**
+   unilaterally refuse to suggest it (there is no `doNotSuggest` gate). This is disclosure, not a gate —
+   and it does not relax the rule above: you still **verify the release state against the current public
+   Databricks docs before recommending** (never "recommend now, verify later").
+3. **Author the insights JSON, then call `enrich`.** The library validates it against the inventory
+   and the saved source graphs and, only when clean, writes `agentic_insights.json` and rebuilds the
+   inventory. On any violation both files are left untouched and you get the full list of
+   problems to fix in one pass.
+
+See **`insights.md`** in this skill directory for the exact insights shape, every field, and the
+validation rules the library enforces.
+
+## Run enrich — MCP tool or venv CLI
+
+Run the **`setup`** skill first if you haven't. Both paths run the same validate-and-record contract.
+
+- **MCP tool (Databricks Genie Code, or a local stdio registration):** call the single **`flowx`**
+  tool with `command="enrich"`. On the hosted server you cannot read `output_dir`, so first read the
+  inventory with `action="prepare"`. It returns `metadata/inventory.json` inline as
+  `bundle.files["inventory.json"]`, the same way `package` returns a bundle; pass `output_volume_path`
+  or `output_workspace_path` to have a large inventory uploaded there as `inventory.json` instead. An
+  inventory too large to return inline makes `prepare` fail with `ok: false`, naming those two options. Then
+  record the insights (the default `action="apply"`) with either inline insights or a file:
+
+  ```
+  flowx(command="enrich", parameters={"output_dir": "<dir>", "action": "prepare"})   # read inventory.json
+  flowx(command="enrich", parameters={"output_dir": "<dir>", "insights": { ... }})   # inline object
+  flowx(command="enrich", parameters={"output_dir": "<dir>", "insights_path": "<file>"})
+  ```
+
+  Provide **exactly one** of `insights` (inline object) or `insights_path`. `ok` reflects validation;
+  `result.violations` lists any problems. Run **no** `python3`/`$PY` commands on this path.
+
+- **venv CLI (local, no MCP server):** ensure the venv exists (`setup` / `bootstrap.sh`), then:
+
+  ```bash
+  export PYTHONPATH="<plugin_dir>/src"
+  PY="$(cat <plugin_dir>/.migration-venv)"
+  "$PY" -m flowx.adapter enrich --output-dir <dir> --insights-path authored_insights.json
+  ```
+
+  Both `--output-dir` and `--insights-path` are required; `--out <file>` optionally writes the result
+  JSON to a file instead of stdout. **Exit code 0** means the insights were recorded; **exit code 1** prints
+  the violations JSON and leaves the inventory untouched.
+
+## Idempotency & safety
+
+`enrich` is idempotent: it records the validated insights in
+`metadata/agentic_insights.json` (stamping `schema_version`, `inventory_sha256`, `source_graphs_sha256`
+and `agentic_insights_sha256`), then rebuilds `inventory.json` itself from the saved `source_graphs.json`
+(or, for an inventory that records no graphs hash, from its existing deterministic part) plus that
+document rather than patching it in place. The `insights` block is replaced wholesale (never stacks),
+and every deterministic inventory key stays byte-identical. Re-running with the same insights rewrites the same bytes; re-running
+with different insights replaces the block. A validation failure writes nothing.
+
+Absent a concurrent `discover`, only one `enrich` writes an output directory at a time (see the races
+below): it holds `metadata/.enrich.lock` from reading
+the inventory until both files are replaced, and a second call made meanwhile fails with a violation and
+writes nothing. `discover` does not wait for the lock: it clears `metadata/`, the lock with it. So right
+before writing, `enrich` checks that its lock and the `inventory.json` it read are both still there; if
+`discover` ran in between, it writes nothing and reports "discover ran during this enrich; prepare and
+enrich again". The two files are replaced one right after the other. If replacing `inventory.json`
+fails, `agentic_insights.json` is put back to the insights the unchanged `inventory.json` holds (or
+removed when it holds none). A process killed or interrupted (Ctrl-C)
+between those two steps, or a failure while putting that file back, can leave
+`agentic_insights.json` one write ahead of `inventory.json`. That run's lock then stays behind, and the
+next `enrich` refuses until the lock is cleared:
+
+- **Locally**, delete `metadata/.enrich.lock` and run `enrich` again; it rewrites both files.
+- **On the hosted MCP server** you cannot delete files in `output_dir`, so run `discover` again (it
+  clears `metadata/`, the lock with it), then `enrich` with `action="prepare"` and apply the insights again.
+
+Sequential use is safe. The checks above do not close every race, though: a `discover` plus two
+`enrich` runs on one output folder within a few milliseconds can slip past them. That can end in an
+error, but it can also overwrite silently. Every `enrich` uses the same temp file names, so one run can
+install another's files, leaving a stale `inventory.json` while it reports `ok: True` with
+`inventory_sha256` and `agentic_insights_sha256` values that do not match the files on disk. Two runs
+can also write together, so the two files may come from different runs and need not match the current
+inventory. Do not run `enrich` while another `enrich` or a `discover` is working on the same output folder.
+
+## Next step
+
+After the inventory is enriched, continue with **`flowx-route`** to recommend and record a
+per-connected-component conversion route (deterministic vs. agentic), then convert and package.
+
+## Reference
+
+- `insights.md` — the insights shape, every field, and the validator's rules.