[matthewmoorcroft]

Intent

Route each connected component of an ADF factory to a deterministic (1:1 engine) or an agentic (agent-authored re-architecture) conversion. The user decides, and the record of the decision is one readable file. Convert stays deterministic and untouched.

• Route writes its recommendation straight into metadata/conversion_plan.json, with every decision pending, and draws it as a standard review page.
• The user decides by editing the plan (or passing it), and route applies it on top of convert's unchanged baseline.
• The agent fills each routed-agentic unit with fill-agentic.
• Package ships an agentic decision only as the agent's output, never as the deterministic stand-in, and records what was converted in metadata/route_audit.json.

The plan can be edited and re-applied as often as needed. The same plan gives the same bytes, and only what changed is re-derived. Agentic routing is ADF-only.

This builds in the conversion-approach decisions of 8 Oct:

• Q1: package refuses until every agentic unit is filled.
• Q2: enrich-suggested groupings that the plan records once accepted, shown on a standardized HTML review page.
• Q3: convert's baseline is never rewritten; fills are stored separately.
• Q4: the agent's conversion output is its own file, metadata/agentic_conversion.json, renamed from agentic_combines.json.

It also builds in:

• the routing conversation recorded in the audit;
• the reader-side insights check carried from feat: add enrich step that records agent-authored insights on the discover inventory #95;
• the earlier decisions D3 (agentic refused on a non-ADF inventory) and D5 (package refuses a stale plan, with route_audit.json hashes).

What changed

• Files feat: add per-component routing and in-engine agentic conversion #97 reads and writes:

  Stage	Files
  describe	source_graphs.json, agentic_insights.json (optional), inventory.json
  decide	conversion_plan.json, routing_review.html
  convert	.work/translation_report.json, .work/gaps.json, agentic_conversion.json, .work/route_baseline/ (internal)
  after	the bundle, route_audit.json

  There is no separate recommendation or plan file.

• models/conversion_plan.py, the typed ConversionPlan (schema 3):

  • a decision may be null (pending);
  • suggested_groupings, each with accepted;
  • an optional conversation of {question, answer};
  • binds to agentic_insights_sha256 (feat: add enrich step that records agent-authored insights on the discover inventory #95's rename);
  • loads strictly: a missing, malformed or wrongly nested field is refused with a ValueError;
  • the reserved per-node assignments stay, as the Phase 2 extension point.

• routing.py:

  • Components and groupings:
    • Components are connected over proven control calls and are never split.
    • Groupings are suggested from the insights, joining components linked by an inferred relationship or by a simplification pattern recommended in more than one component.
    • A grouping's id is derived from its members, so it is stable across a re-enrich.
    • An accepted grouping becomes one agentic unit.
    • When an accepted grouping is no longer suggested, its components go back to pending and route reports it under dropped_groupings.
  • Recording the plan:
    • With no plan passed, route keeps the decisions in the recorded file (validated as written, so typos are refused).
    • It recomputes library-owned fields.
    • It applies the plan only once nothing is pending.
  • ADF-only: an inventory that records no source is not agentic-routable.
  • Reader-side insights check: insights_file_violations refuses when agentic_insights.json and the inventory's insights block disagree, or the file fails its own hash. The message says how to recover.

• route_agentic.py:

  • Rebuild: the pure rebuild runs baseline, then stored gap fills, then the plan's units, then the agent's outputs. Applied units drop their routed gaps, including gaps nested in If, ForEach, Switch or Until.
  • Gap ownership (13 Oct): a filled unit also owns convert's gaps for activities a motif collapsed (matched_activity_names) and for activities the inventory lists for its pipelines but the report no longer has as tasks. A name a pipeline that stays deterministic also holds is kept.
  • fill-agentic:
    • Replaces fill-agentic combine and stores outputs in agentic_conversion.json.
    • Keeps a replaced history, where the first fill is not a replacement.
    • Refuses authored pipelines that still hold a placeholder, or whose names share a bundle folder once normalised.
  • Route refuses:
    • a pending decision;
    • a baseline converted from a different discover (empty ADF pipelines allowed);
    • a missing report ("run convert first"). Route no longer triggers convert, has no interactive prompt, and does not read the plan from stdin.

• ir_serde.merge_agentic_results: after routing, an in-place merge of convert's own gaps is stored as a gap fill, and the baseline is never rewritten. The routing record's gap_fills_sha256 lets package spot a merge made after modify.

• bundler/dab_writer.py:

  • Preflight refusals:
    • insights files that disagree (plan or no plan);
    • a routed report with no plan;
    • a plan that is not a complete, valid decision;
    • a pending decision;
    • an unfilled agentic unit;
    • an edited agent output or gap fill;
    • any report that differs from a fresh rebuild.
  • route_audit.json: carries one canonical plan hash, per-unit outcomes and replaced history, the gap fills and the routing conversation.

• reporting/routing_review.py + routing_review_template.html: one eight-section page in the Databricks style, written by route and fill-agentic:

  1. at a glance
  2. what the source does
  3. the recommended conversion
  4. suggested groupings
  5. what happens next
  6. what you told us
  7. notes
  8. how to change a decision

  It uses the stdlib only, escapes all text, carries no timestamps, and has no logo. It is the only HTML page flowx generates.

• adapter, mcp/server.py: route and fill_agentic follow the flow above. MCP route returns the plan and the review page because a hosted agent can't read output_dir.

• Hosted MCP flow (13 Oct):

  • The flowx-migrate skill told Genie Code to use one MCP migrate call, which runs only discover, convert and package, so hosted runs never enriched or routed.
  • Its MCP path now runs the same steps as the CLI path, one flowx call per phase. It passes the same source to discover and convert, because convert reads the source again.
  • migrate is described as the deterministic-only shortcut in the skill, the server instructions, the tool description, app/README.md and the guide. Its behaviour is unchanged.
  • Enrich runs for every source; route and fill stay ADF-only.

• Inputs and SQL Server ingestion (14 Oct):

  • A: migrate asks each input at the step that needs it. The start asks only for the source, the output dir and full or deterministic-only; a deterministic-only pass skips enrich, route and fill. Convert options come at convert, and catalog and schema, the bundle name and other package options at package. The hosted MCP path says the same.
  • B: catalog and schema are one free-text answer, catalog.schema (default main.default), never a "Custom" choice that can't take text. The inputs package descriptions carry that format.
  • C: SQL Server ingestion is now qualified by the routine below (N1, N2), which replaces route's three fixed SQL Server questions.
  • D: the MCP instructions and the migrate skill say that on non-ADF sources route records deterministic decisions only, and that agentic decisions and fill-agentic are ADF-only.
  • Package: it now clears resources/*.yml before writing, so a unit switched back to deterministic no longer leaves its agent-built pipeline behind to deploy. Hand-added files in resources/ are cleared too.

• Questions at scale (14 Oct):

  • N1: enrich summarises what the analysis found, then asks at most three open questions about the facts that would rule a target pattern in or out (scale, repeat, fan-in, change, history, bar, policy, shaping), and records the answers in conversion_notes, with the source system and connector a pipeline reads from. The ADF and Airflow source guides say where each fact shows.
  • N2: route qualifies four target patterns (deterministic 1:1, Lakeflow Connect, a metadata-driven worklist, agentic re-architecture) with at most three follow-up questions, checks C1 to C13 (counting each gateway in gateway mode) and a suggestion order that matches them, and shows the failing check and its source beside a pattern it rules out. Answers stay free text in the plan's conversation, with no schema change.
  • N3: flowx keeps no copy of the platform limits. Enrich checks every pattern it recommends against the current Databricks platform limits, read live from the docs, and records each limit it relied on with its page and date in conversion_notes. Route reads the limits its checks rest on the same way, names each in words, and marks three claims the public docs don't state as assumptions: one pipeline per destination table, integrated CDC's release state, and whether a running gateway holds a concurrent-update slot. This replaced a stored platform-limits.json on 28 Oct.
  • N9: route and fill-agentic after a package now say that package pruned .work/ and to re-run convert, then route. The route, package and migrate skills describe how to change the conversion after a package, and what survives each step.

• Answers on the open calls (14 Oct):

  • Re-running migrate: discover, and so the hosted one-shot migrate, now moves every file it would remove (plan, review page, insights, route baseline, fills, bundle) into metadata/archive/<UTC time>/ with an archive.json of what moved, from where, when and each file's hash, and says so in its result. The hosted delivery puts the current bundle before the archive.
  • Grouping accepted: a value that is not a JSON boolean ("true", 1, null) is refused when the plan loads, and package refuses such a plan.
  • Review page redesign: the approved Databricks-style page is committed without the header logo. Its deterministic counts read "recommended deterministic", since a non-ADF component can be recommended deterministic with gaps left to fill.
  • Same style for every page: the routing review page is the only HTML page flowx generates, so nothing else changed.
  • fill-agentic names: an authored pipeline whose name is not a non-empty string is refused with "give every pipeline a name" instead of crashing.
  • Old bundle folders: the post-package loop text says a re-package after the pipeline count or names change can leave the earlier bundle folders, and to remove them.
  • Fills package can build, and insights for this inventory: fill-agentic refuses an authored pipeline whose tasks is not a non-empty list of task objects, or that is marked migration_status: "excluded"; the insights check also refuses agentic_insights.json whose recorded inventory_sha256 no longer matches the inventory, in route and package.

• sources/adf/translate.py:

  • a refused merge exits 1 with its message;
  • when convert finds no gaps it removes a stale .work/gaps.json, so the baseline is always convert's own output. Nothing in translation changes.

• Docs: flowx-route and flowx-migrate skills, convert notes, README and guide file trees, and the AGENTS.md routing rows. Docs about refactor: add source-neutral discovery graph contract and shared lineage core #84/feat(adf): save source graphs with dataset lineage at ADF discover #94/feat: add enrich step that records agent-authored insights on the discover inventory #95 internals were left to those PRs, and the Airflow gap-resolver skill registration to Peter.

• Version 0.2.0, with a CHANGELOG entry.

• Fixes from checking this PR against the points raised in review (26 Oct), each with a test that fails without it and a Codex cross-review:

  • MCP route returns ok: false, uploading nothing, when the plan or the review page is missing, unreadable or left out of the size-bounded inline bundle;
  • with a plan recorded, package refuses a report with no routing record while agentic_conversion.json stores a gap fill, as after a re-convert;
  • route, fill-agentic and any merge written to the live report hold metadata/.route.lock, built like enrich's lock, and refuse while another holds it;
  • agentic_conversion.json must hold exactly components and gap_fills;
  • route reads the inventory through enrich's guard and refuses one not shaped the way discover writes it.

  Follow-ups, not in this PR: MCP package and migrate also return ok with a truncated inline bundle, whose cap counts characters (main's code); discover, convert and package take no route lock, and route, fill and package don't hold enrich's; a configured or --report copy is matched by its routing record only; duplicate JSON names keep the last value; the routing record's own key set; the plan's authored defaults; route recording the plan before an apply that fails; route_audit.json rewritten on each package; duplicate inventory pipeline names; a fill stored before this carry isn't re-checked on replay; a component fill refused by the bundle writer reports an error rather than violations.

• Carries feat: add AgenticComponentActivity for agent-authored bundle components #96's head cb7f0d2 (with feat: add enrich step that records agent-authored insights on the discover inventory #95's lock fix and feat(adf): save source graphs with dataset lineage at ADF discover #94's round-four fix) through a no-conflict merge, 5edec1c. uv.lock is identical to feat: add AgenticComponentActivity for agent-authored bundle components #96's.

How it was tested

• Unit tests: about 4,700 lines across 13 routing, plan, review-page, insights-check, MCP and package-preflight test files, with a regression test for each change. The full suite (1816 passed), ruff and mypy pass, and CI is green.

• Gap ownership (13 Oct): three regression tests in test_route_agentic_flow.py, each red before the fix:

  • a collapsed motif plus an inventory-only activity, which leaves no gaps once filled;
  • two filled units sharing an inventory-only name;
  • the deterministic-pipeline guard.

  The no-mistakes test step also drove the real CLI on a hand-written ADF factory, 5 of 5 scenarios. They include package refusing the unfilled unit and switching back to deterministic, which restores gaps.json byte for byte. Deterministic-only output is still byte-identical to feat: add AgenticComponentActivity for agent-authored bundle components #96's head 07eae42.

• Questions at scale (14 Oct): a test that route and fill-agentic after a package give the pruned-.work/ message, red before the change. A test holds the skills to reading the limits live: no copy of them in the repository, enrich's rule and record form, route's live reading, each check's limits named in words, and the three assumptions marked (red before the 28 Oct change). The no-mistakes review then added the enrich database-source flag, aligned the suggestion order with the checks, counted gateways in gateway mode, and reverted a fix round's rewrite of uv.lock to an internal package mirror.

• Answers on the open calls (14 Oct): discover over a routed, filled and packaged folder archives every file it replaces with its bytes intact, and a fresh folder archives nothing (red on the old code); hosted migrate reports the archive and delivers the current bundle first; non-boolean accepted is refused at load and by package; a list or object pipeline name is refused by fill-agentic (each red before its fix). A fill whose tasks is a string, an object, null or a list of strings, or that is marked excluded, is refused and stores nothing, and a hand-edited deterministic inventory field is refused by route and package (red before; a missing or empty tasks list was already refused as an empty job).

• Hosted MCP flow (13 Oct): the server's commands were driven on the ADF fixture factory as Genie would, with the definitions inline: discover, enrich (prepare and apply), convert with the same definitions, route pending, route decided, then package, which wrote route_audit.json. Convert without the source fails with --source-dir is required, which is what the skill's earlier advice missed.

• Deterministic output: ADF and Airflow fixtures run through discover, convert and package give output trees (bundle, metadata, .work) byte-identical to feat: add AgenticComponentActivity for agent-authored bundle components #96's head 07eae42 (366 ADF files, 130 Airflow files).

• End to end, on the full ADF fixture factory:

  1. Discover, enrich with a shared Auto Loader pattern, convert.
  2. Route with no plan gives 35 pending components and the review page.
  3. Decide, and accept the suggested grouping, then route.
  4. Package refuses the unfilled grouping.
  5. Fill it with fill-agentic.
  6. Merge one of convert's gaps; the baseline is unchanged.
  7. A re-route is a no-op.
  8. Package passes, and the audit shows the grouping agentic-applied, the gap fill and the conversation.

• Codex cross-review, read-only:

  • Round 1: its feat: add per-component routing and in-engine agentic conversion #97 items are all fixed in the build 9221edc, each with a test:
    • a routed report with no plan;
    • names colliding after normalisation;
    • a strict plan load with package re-validation;
    • a baseline from another discover;
    • the first fill logged as a replacement, and history lost on switch-back.
  • Round 2 (on 11785e9): two P2 findings, both fixed red-green:
    • nested convert gaps surviving routing, in 517af7c;
    • malformed nested plan fields crashing instead of failing closed, in 9b8b9ae.

• No-mistakes review rounds: then fixed, among others:

  • empty ADF pipelines blocking agentic routing;
  • a merge after modify going unnoticed;
  • grouping identity across re-enrich (member-derived ids);
  • dropped groupings reset to pending and reported;
  • hand-edited plans validated as written;
  • stale gaps.json after a re-convert;
  • Until children.

• Deferred follow-ups:

  • the migrate and package skills' copied metadata/ lists;
  • dropped_groupings showing only on the run that drops the grouping;
  • Airflow skill distribution, which is Peter's;
  • a docs pass over refactor: add source-neutral discovery graph contract and shared lineage core #84 to feat: add enrich step that records agent-authored insights on the discover inventory #95 once the stack merges.

Merge order

Merge #84 first, then #94 retargeted to main, then #95, then #96, then this PR (retargeted to main once #96 has merged).

This pull request and its description were written by Isaac.

[matthewmoorcroft]

Thanks Greg. All ten are fixed, each with a test that failed first:

1. Gaps on rebuild: route's rebuild now drops convert's gap for each applied gap fill and for anything nested in the filled task. Convert's gaps carry no pipeline, so a (pipeline, activity_name) filter would never match them; the new drop_filled_gaps goes by name and keeps a gap only when a pipeline outside the routed units still has that name open. The tests now start from a real gaps.json. b1db89f, 40eba35
2. Gaps on the live merge: every in-place merge into .work/translation_report.json, routed or not, takes the same gaps out of .work/gaps.json with the same function, so the merge and a later route write the same gaps. A merge to an --output copy leaves gaps.json alone, and an unreadable gaps.json no longer stops a merge half way. b1db89f, 9d8ef25, 40eba35
3. taken_names: only the outputs the rebuild applies reserve names now, read from one helper the rebuild and the fill share, so switching a unit back frees its names. 867deb3
4. Carry-forward ids: each carried row takes its component's current id, so a decision follows its pipelines when the ids shift. ff554d9
5. Missing edge source: both ends are checked; an edge whose source is empty, not a string or not an inventory pipeline joins nothing and is a finding. ab84fc3
6. Private Preview: the route skill now says what enrich and the code do: the state is disclosed, with no doNotSuggest gate. d256374
7. NodeAssignment: removed with ASSIGNMENT_ROUTES and the loading and checking code; an assignments key is now refused like any other unknown key. ea070d9
8. agentic_pipeline_names: built from routing_units, so there is one answer. 4b387fa
9. Decided but not applied: a unit decided since the last route shows "Decided, not applied yet" with "Run route again", checked against the routing record with the same per-unit rule package uses. An accepted grouping with a component still pending shows as pending, since route applies nothing until it is decided. 3bb0475, 40eba35, bf07b01
10. ForEach and Switch: added to every nested-gap test, including the new gap-fill one. f4ad628

The branch also merges #96's round-three head (3406913, no conflicts). At bf07b01 the unit suite passes (2454) and ruff and mypy are clean.