matthewmoorcroft · Sep 16, 2026 · Sep 17, 2026 · Sep 17, 2026 · Sep 17, 2026 · Sep 17, 2026
diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
@@ -9,7 +9,7 @@
       "name": "flowx",
       "source": "./",
       "description": "Translate data pipelines to Databricks Lakeflow Jobs via Declarative Automation Bundles. Deterministic translation for known activity types with agentic fallback.",
-      "version": "0.1.0",
+      "version": "0.2.0",
       "license": "MIT",
       "keywords": ["adf", "databricks", "migration", "dabs", "lakeflow", "orchestration", "data-pipelines"]
     }

diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
 {
   "name": "flowx",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Translate Azure Data Factory pipelines to Databricks Lakeflow Jobs via Declarative Automation Bundles. Deterministic translation for known activity types with agentic fallback.",
   "author": {
     "name": "Greg Hansen",
@@ -12,6 +12,7 @@
     "./skills/flowx-setup",
     "./skills/flowx-discover",
     "./skills/flowx-enrich",
+    "./skills/flowx-route",
     "./skills/flowx-convert",
     "./skills/flowx-package",
     "./skills/flowx-migrate"

diff --git a/AGENTS.md b/AGENTS.md
@@ -86,6 +86,10 @@ intermediates under `.work/` (pruned by `package`).
 | `reporting/coverage.py` | Builds per-pipeline coverage rows from `metadata/` |
 | `reporting/results.py` | Writes per-run coverage to a UC table (run_id/run_date/run_by) via the SDK |
 | `reporting/dashboard.py` | Installs + publishes an AI/BI coverage dashboard over the results table |
+| `reporting/routing_review.py` | Renders `metadata/routing_review.html`, the standard eight-section routing review page, from `routing_review_template.html` (stdlib only, no timestamps) |
+| `routing.py` | Groups pipelines into connected components over control lineage; recommends deterministic/agentic per component (both options) and suggests groupings from the insights; writes the recommendation into `metadata/conversion_plan.json` with decisions pending and records the user's edits (additive; convert untouched). Agentic decisions are ADF-only; also the reader-side check that `agentic_insights.json` agrees with the inventory |
+| `route_agentic.py` | Applies a routing decision after convert: rebuilds the report from the never-rewritten baseline (`.work/route_baseline/`), the plan, and the agent's output in `metadata/agentic_conversion.json` (per-unit pipelines + fills of convert's own gaps); `fill-agentic` is the only fill for routed-agentic units |
+| `models/conversion_plan.py` | Typed `ConversionPlan` (schema 3): the library's load/validate/record API for `conversion_plan.json`, one decision per component (pending until made), suggested groupings, the routing conversation, bound to the inventory, `source_graphs.json` and `agentic_insights.json` hashes; package fails closed on a mismatch, a pending decision, or an unfilled agentic unit |
 
 ## Activity Types
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,26 @@ All notable changes to flowx will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [0.2.0]
+
+### Added
+- Enrich step: the agent adds insights on top of the discover inventory, and asks a few open
+  questions about the estate that the code can't answer
+- Route step: decides deterministic or agentic conversion per connected component, records the
+  decision in the conversion plan, and renders a routing review page; before suggesting an ingestion
+  pattern it checks it against the current platform limits, read live from the docs, as enrich does for
+  every pattern it recommends
+- Fill-agentic step: applies the agent's conversion output to routed-agentic units
+- Agentic component: for an activity the typed engine cannot express, the agent can author Databricks
+  tasks, files, pipeline resources and job environments, which package writes into the bundle while
+  flowx keeps the task name, dependencies and run policy
+
+### Changed
+- Re-running discover over an output folder that already holds a run now moves that run's outputs
+  into `metadata/archive/<UTC time>/`, with an `archive.json` of what moved, instead of deleting them;
+  re-running package does the same for any resource file and bundle folder it no longer writes, keeping
+  the list of bundle folders it wrote in `metadata/packaged_bundles.json`
+
 ## [0.1.0]
 
 ### Added

diff --git a/README.md b/README.md
@@ -99,6 +99,7 @@ Or run individual phases:
 /flowx:flowx-discover    # Parse the source (ADF JSON / Airflow DAGs), produce inventory + complexity report
 /flowx:flowx-enrich      # Validate and record agent-authored insights on the inventory
 /flowx:flowx-convert     # Deterministic + agentic translation
+/flowx:flowx-route       # Decide deterministic vs. agentic per pipeline group and fill the agentic groups (ADF)
 /flowx:flowx-package     # Generate DABs project
 ```
 
@@ -262,6 +263,11 @@ flowx_output/
     agentic_insights.json     # enrich: validated agent-authored insights, rebuilt into inventory.json
     profile_report.csv        # discover: per-pipeline complexity report
     <pipeline>.arm.json       # discover: verbatim original ADF/ARM source
+    conversion_plan.json      # route: the recommendation, suggested groupings and the user's per-component decisions
+    routing_review.html       # route: the standard review page drawn from the plan
+    agentic_conversion.json   # fill-agentic (ADF): the agent's conversion output per routed-agentic unit, re-applied by every route
+    route_audit.json          # package: routing decisions, outcomes, history, conversation and hashes, written when a plan is recorded
+    packaged_bundles.json     # package: the bundle folders it wrote, so the next package archives the ones it no longer writes
     configuration.json        # modify: collected configuration answers
   .work/                      # transient intermediates (translation report, IR, gaps.json); pruned by package
 ```

diff --git a/app/README.md b/app/README.md
@@ -23,12 +23,14 @@ operation; `parameters` is its keyword-argument dict.
 | `merge_agentic` | `adapter convert --merge-agentic` | Merge ADF agent-produced results into the report |
 | `resolve_agentic` | `adapter resolve-agentic` | Prepare, stage, and apply reviewed Airflow leaf-gap resolutions |
 | `enrich` | `adapter enrich` | Return `inventory.json` (`prepare`), then validate and record agent-authored insights (`apply`) |
+| `route` | `adapter route` | Write the recommendation into `conversion_plan.json` and the review page; apply the plan once decided |
+| `fill_agentic` | `adapter fill-agentic` | Fill a routed-agentic unit with agent-authored pipelines (ADF) |
 | `inspect` | `adapter inspect` | Surface pending translation options |
 | `apply_answers` | `adapter modify` | Apply answers → stamped IR |
 | `materialize_lookup` | `adapter materialize-lookup` | CSV → lookup-values JSON |
 | `workspace_paths` | `adapter workspace-paths` | Detect workspace paths / hosts |
 | `package` | `adapter package` | Emit the deployable DAB bundle |
-| `migrate` | discover→convert→package | Full non-interactive migration |
+| `migrate` | discover→convert→package | Deterministic-only migration in one call (never runs `enrich` or `route`; for a migration that enriches (any source) or routes (ADF), run the phases one at a time, passing the same source to `discover` and `convert`) |
 | `record_results` | `adapter record-results` | Write coverage to a UC table |
 | `install_dashboard` | `adapter install-dashboard` | Publish the coverage dashboard |
 

diff --git a/docs/content/docs/architecture.mdx b/docs/content/docs/architecture.mdx
@@ -83,7 +83,7 @@ See [Installation](/flowx/docs/installation#running-flowx-as-an-mcp-server) for
 <Callout type="info" title="Inputs and outputs on a hosted app">
 A Databricks App can't read the user's workspace / UC Volume files (`/Volumes/...` is **not** auto-mounted). Two ways to get data in/out of the `flowx` tool:
 
-- **Small jobs — inline.** `command="discover"`/`"migrate"` accept `adf_definitions` (a mapping of relative path → ARM JSON, Git-export layout), supplied by the agent and materialized to a temp dir; `package`/`migrate` return the DAB inline as `bundle = {"files": {relpath: text, …}}`. Inline data flows through the agent's context, so it's capped (~5 MB in).
+- **Small jobs — inline.** `command="discover"`/`"convert"`/`"migrate"` accept `adf_definitions` (a mapping of relative path → ARM JSON, Git-export layout), supplied by the agent and materialized to a temp dir; pass the same source to `discover` and `convert`; `package`/`migrate` return the DAB inline as `bundle = {"files": {relpath: text, …}}`. Inline data flows through the agent's context, so it's capped (~5 MB in).
 - **Large factories — by reference (recommended).** Point the server at the source: `adf_volume_path` (a UC Volume read via the SDK Files API) or `adf_workspace_path` (a `/Workspace` directory — e.g. an ADF Git folder — read via the SDK Workspace API). Write the DAB to `output_volume_path` (UC Volume, SDK Files API) or `output_workspace_path` (`/Workspace` directory, SDK Workspace API with `ImportFormat.RAW`), returned as `bundle_uploaded`. The bytes bypass the agent entirely, so it scales to thousands of pipelines; grant the app's service principal read on the source and write on the output target.
 
 Locally hosted, ordinary `adf_source_path` / `output_dir` paths and mounted volumes work as-is.

diff --git a/docs/content/docs/guide.mdx b/docs/content/docs/guide.mdx
@@ -56,8 +56,10 @@ flowx migrate pipelines in `/Volumes/main/default/adf_export`. Save output in `.
 Your agent will use flowx's `migrate` skill to run an end-to-end migration consisting of several phases:
 
 1. `discover` parses JSON files, builds an inventory, assigns a translation strategy for each resource, and creates a complexity report.
-2. `convert` converts each activity to an intermediate representation. The agent will ask for confirmation before running any LLM-based translation.
-3. `package` creates a [Declarative Automation Bundle](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) with job configuration, code, and setup scripts (e.g. to create Databricks Secret Scopes or Unity Catalog connections).
+2. `enrich` (default) has the agent author insights over the inventory and records them.
+3. `convert` converts each activity to an intermediate representation. The agent will ask for confirmation before running any LLM-based translation.
+4. `route` (ADF) recommends deterministic or agentic conversion per group of connected pipelines, shows the plan on a review page, and records your decisions; the agent then fills the groups you route agentic.
+5. `package` creates a [Declarative Automation Bundle](https://docs.databricks.com/aws/en/dev-tools/bundles/resources) with job configuration, code, and setup scripts (e.g. to create Databricks Secret Scopes or Unity Catalog connections).
 
 <Callout type="info" title="Running migration step-by-step">
 To run a single step, invoke its skill directly (e.g. `@flowx-discover`) or ask your agent to run a single step.
@@ -80,6 +82,11 @@ flowx_output/
 │   ├── inventory.json      # discover: activity inventory
 │   ├── profile_report.csv  # profile: per-pipeline complexity report
 │   ├── <pipeline>.arm.json # discover: verbatim original ADF/ARM pipeline source
+│   ├── conversion_plan.json # route: the recommendation and the user's per-component decisions
+│   ├── routing_review.html # route: the standard review page drawn from the plan
+│   ├── agentic_conversion.json # fill-agentic: the agent's conversion output, re-applied by every route
+│   ├── route_audit.json    # package: what each routing unit was converted to, and the routing conversation
+│   ├── packaged_bundles.json # package: the bundle folders it wrote, so the next package archives stale ones
 │   └── configuration.json  # modify: the collected configuration answers
 └── .work/                  # transient intermediates (translation report, IR, gaps.json); pruned by prepare
 ```

diff --git a/docs/content/docs/installation.mdx b/docs/content/docs/installation.mdx
@@ -132,7 +132,7 @@ flowx is distributed through a Claude Code marketplace. It can be installed as a
 You can also copy the skill folders straight into your local skills directory:
 
 ```shell
-cp -R skills/{flowx-setup,flowx-discover,flowx-enrich,flowx-convert,flowx-package,flowx-migrate} ~/.claude/skills/
+cp -R skills/{flowx-setup,flowx-discover,flowx-enrich,flowx-route,flowx-convert,flowx-package,flowx-migrate} ~/.claude/skills/
 ```
 </Tab>
 </Tabs>

diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [project]
 name = "databricks-flowx"
-version = "0.1.0"
+version = "0.2.0"
 description = "ADF to Databricks Lakeflow Jobs translator via Declarative Automation Bundles"
 readme = "README.md"
 requires-python = ">=3.12"

diff --git a/skills/flowx-convert/SKILL.md b/skills/flowx-convert/SKILL.md
@@ -42,7 +42,9 @@ agentic-gap flow with just-in-time configuration. Airflow converts deterministic
 Run the **`setup`** skill first if you haven't.
 
 - **MCP tool (Genie Code, or local stdio):** call the single **`flowx`** tool with
-  `command="convert"` and `parameters` including `"source": "<source>"` and `"output_dir": "<dir>"`.
+  `command="convert"` and `parameters` including `"source": "<source>"`, `"output_dir": "<dir>"` and
+  the same source key discover read (`adf_definitions`, `adf_volume_path`, `adf_workspace_path` or
+  `airflow_source_path`): convert reads the source again.
 
 - **venv CLI (local):**
 
@@ -70,7 +72,7 @@ across ADF and Airflow.
   `show_when` condition). Walk it locally; ask an option only when its `show_when` is satisfied.
 - `modify <report> --output-dir <dir> --answer OPTION_ID=VALUE ...` — validate and apply collected
   answers, writing `.work/translation_report.stamped.json` + `metadata/configuration.json`.
-- `merge_agentic --report <report> --agentic-results <dir>` — **ADF only**. Fold agent-produced per-activity translations into an ADF report. Airflow's legacy name-based merge is disabled.
+- `merge_agentic --report <report> --agentic-results <dir>` — **ADF only**. Fold agent-produced per-activity translations into an ADF report, filling convert's own agentic gaps. It refuses, writing nothing, any result landing in a pipeline the routing record routes agentic (fill those with `fill-agentic`). After routing, a merge written in place into the live `.work/translation_report.json` is also stored as a gap fill in `metadata/agentic_conversion.json`, so the next rebuild re-applies it on top of route's unchanged deterministic baseline; a merge written to a separate `--output` copy stores nothing, so a re-route does not keep it. Airflow's legacy name-based merge is disabled.
 
 ## Output artifacts (shared, transient under `<output_dir>/.work/`)
 

diff --git a/skills/flowx-convert/sources/adf.md b/skills/flowx-convert/sources/adf.md
@@ -164,7 +164,13 @@ Where the fragment leaves a value out, flowx adds only the plumbing the task nee
 ```
 
 Placeholders are replaced in place, status → `translated`. Exits non-zero if any result can't be
-matched. Add `--output <path>` to write a copy instead of overwriting.
+matched. Add `--output <path>` to write a copy instead of overwriting. An in-place merge into
+`.work/translation_report.json` also takes the merged activities' gaps out of `.work/gaps.json`. After
+routing, only such an in-place merge is also stored as a gap fill in `metadata/agentic_conversion.json`
+(so a re-route keeps it, while route's saved deterministic baseline stays as convert wrote it); a merge
+written to an `--output` copy stores nothing and leaves `gaps.json` alone. A merge into the live report
+refuses, writing nothing, while a route, fill-agentic or another merge is writing to the same output
+folder, so run merges one at a time (see "One at a time per output folder" in the `flowx-route` skill).
 
 ## Step 6.1 — Just-in-time translation configuration
 

diff --git a/skills/flowx-discover/SKILL.md b/skills/flowx-discover/SKILL.md
@@ -68,6 +68,11 @@ All under the shared `<output_dir>/metadata/` folder:
 | `metadata/source_graphs.json` | (ADF) The saved source graphs (activities, lineage and detected motifs), versioned and content-hashed; the inventory is built from it and records its hash as `source_graphs_sha256` |
 | `metadata/profile_report.csv` | Per-pipeline complexity report (counts + T-shirt size) |
 | `metadata/<pipeline>.arm.json` | (ADF) Verbatim original source for each pipeline (provenance) |
+| `metadata/archive/<UTC time>/` | Only on a re-run: the earlier run's `metadata/`, `.work/` and bundle, moved here with an `archive.json` of what moved, from where and when |
+
+Discover starts each run afresh. Over an output folder that already holds a run, it moves that run's
+files into a new archive folder instead of deleting them, and says so; package keeps `metadata/`,
+so the archive stays.
 
 The inventory classifies every task into one of three strategies:
 

diff --git a/skills/flowx-discover/sources/adf.md b/skills/flowx-discover/sources/adf.md
@@ -16,9 +16,9 @@ The directory should contain subdirectories or files for:
 - `linkedService/` or `linked_services/` — linked service JSON files (optional)
 - `trigger/` or `triggers/` — trigger definition JSON files (optional)
 
-On the MCP path the hosted server cannot read your workspace/volume files, so pass the ADF JSON
-inline as `adf_definitions` (a mapping of relative path → JSON content), or for large factories
-reference the source via `adf_volume_path` / `adf_workspace_path`.
+On the MCP path the hosted server cannot read your local files, so reference the source via
+`adf_volume_path` / `adf_workspace_path` (the server reads them through the SDK), or for small
+factories pass the ADF JSON inline as `adf_definitions` (a mapping of relative path → JSON content).
 
 ## Step 2 — Download from UC volumes if needed
 
@@ -104,6 +104,19 @@ from the activity's ARM JSON (no built-in deterministic translator exists), e.g.
 For `unsupported` activities, warn clearly, e.g. `ExecuteSSISPackage` — recommend manual conversion
 to a PySpark notebook.
 
+## Where to look for the facts enrich asks about
+
+`flowx-enrich` reads the raw activity JSON in `inventory.json` (and `metadata/<pipeline>.arm.json`) to
+find the facts that narrow the target pattern. In ADF they sit in:
+
+- a Lookup feeding `ForEach.items`, and an `ExecutePipeline` called once per item (scale, repeat);
+- dataset parameters and sink paths templated by item (fan-in);
+- query text, such as a version or watermark read before the copy, or change functions (change,
+  shaping);
+- `ForEach.batchCount` and trigger schedules (bar);
+- a Copy or Lookup `typeProperties.source.type` of `SqlServerSource`, `AzureSqlSource` or
+  `SqlMISource`, which marks a SQL Server source.
+
 ## Step 8 — Confirm output location
 
 Tell the user where the metadata files were written (`<output_dir>/metadata/`), summarise the

diff --git a/skills/flowx-discover/sources/airflow.md b/skills/flowx-discover/sources/airflow.md
@@ -56,6 +56,16 @@ For `agentic` tasks, name the operator that has no deterministic mapping yet (e.
 provider operator) and note it will be emitted as a placeholder notebook for the convert phase to
 fill via LLM-assisted translation.
 
+## Where to look for the facts enrich asks about
+
+`flowx-enrich` reads the DAG source to find the facts that narrow the target pattern. In Airflow they
+sit in:
+
+- dynamic task mapping (`expand`) and task groups, and the list a mapped task is given (scale,
+  repeat);
+- operator SQL, such as a watermark read before the load (change, shaping);
+- the DAG's `schedule` and `max_active_tasks` (bar).
+
 ## Coverage notes
 
 Current deterministic coverage: `PythonOperator` (callable body → generated PySpark notebook) and