All framework data moves to the tf-data branch (fix #1582) by suleimansh · Pull Request #1595 · gemstack-land/the-framework

suleimansh · 2026-08-18T23:23:40Z

Implements #1582: main is 100% code, 0% framework data. Everything The Framework writes — tickets/** (plans and locks included), TODO_AGENTS.md, the session archives — moves to one dedicated branch, the-framework_data, checked out at .the-framework/branches/the-framework_data with a root tickets symlink into it.

Net −1,300 LoC. Three mechanisms stop needing to exist:

queue-promote.ts — one queue, one branch, one local writer (the daemon). A drained entry is checked off by the daemon at settle, once the run's ending says the work was published. The Queue promote can clobber a newer main: an agent's stale TODO_AGENTS.md copy lands over externally-merged entries #1577 clobber can't happen anymore.
agent-commit.ts — archives are committed and pushed the moment a session settles, so the debounced committer and its flush phase go. Daemon commits bookkeeping to local main but never pushes it #1397's stranded bookkeeping dies with it.
ticket-locks' own commit/push machinery — a lock batch is just another funneled data write.

How writes work (data-branch.ts): every local write goes through one serialized cycle — sync with origin, apply, commit, push. A lost push race re-syncs and re-applies; a failed push keeps the commit local and the next cycle carries it out. The daemon also pulls the branch eagerly (~1 min), so machines and cloud sessions converge. Reads work from anywhere: the checkout, git show the-framework_data:…, or origin/… in a cloud clone.

Prompts (@brillout — yours to review, see my comment): one new "The data branch" section on the system channel; all data-touching presets point at it. Two deliberate changes: drain agents no longer check their entry off (the daemon retires it), and a plan is now a data write, not a PR — plan review moves to the human queue.

Tested:

Suites: 1455 node + 771 dashboard, both tsconfigs clean, CI green.
Manual end-to-end smoke on a scratch repo with the daemon on this build: first tick creates branch + checkout + symlinks + seeded queue; a dashboard queue-add lands as a pushed data commit; two real drain runs followed the new write protocol unprompted; check-off and archives landed as data commits; a run whose hand-off failed correctly left its entry open.
The smoke caught one real bug, fixed here: retiring the last ticket removes tickets/ itself, and the lock write didn't create parents — every later drain claim failed silently and the sweep stood down with a wrong message. Fix + regression tests in 5e3785b4; the whole retire loop is now also covered by three integration tests (1fa12a6b).
Synced with main past The post-merge review's nine findings, fixed as directed #1594 (one test needed to learn to wait for the archive commit — the archive is a git cycle now, not a file copy).

Hand-run migration (zero migration code, per the no-compat rule) — after merge, on the main checkout:

Let the daemon birth the branch (or git checkout --orphan the-framework_data && git rm -rf . && git commit --allow-empty -m "[The Framework] the-framework_data"), then work in .the-framework/branches/the-framework_data.
Copy tickets/** and TODO_AGENTS.md into the data checkout; move .the-framework/<user>/agents/** to agents/<user>/**; commit + push.
One commit on main: delete tickets/, TODO_AGENTS.md, and the committed archives; add the tickets symlink and the simplified .the-framework/.gitignore.
Restart the daemon; the reconcile creates the checkout + symlink on every other machine.

Closes #1582. Structurally closes #1577 and #1397.

The core module only, consumers follow: DATA_BRANCH checked out as a worktree at .the-framework/branches/the-framework_data (adopted from origin, or born parentless off the empty tree), a root tickets symlink into it, and one serialized write cycle — sync, apply, commit, push — with the op re-run against origin's state when a push loses a race, and stranded local commits carried out by the next cycle. Eager pull = the same cycle with a no-op, so it also drains what a network failure left behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…r one local writer (#1582) Readers: findTodoBacklog/nextQueuedTicket/readDocs' backlog read off the branch via readDataFile (checkout, local ref, then origin — so a project checkout, an agent worktree, and a cloud clone all read the same queue); cli's (fix #N) ticket read does too. Writers: appendTodoEntry/appendFlatTodoEntry are funneled withDataBranch cycles (an agent worktree resolves its project root via the git common dir); ticket locks acquire/release are funneled ops — the old commit/push/default-branch machinery in ticket-locks.ts goes, and a re-run op recognizes its own claims. The backlog loop's check-off is the framework's own funneled write now, not an agent file edit (the loop stalls only when the write cannot land, retried inline). queue-promote.ts is deleted whole: a pinned drain's entry is checked off by the daemon at settle, once the run's epilogue reports the work published — the same reported ending the #1583 lock release keys off, so auto-pm now holds any entry-carrying agent through the end-before-handoff gap. The daemon clock gains an eager 'data sync' pull, so every machine converges on what others pushed (#1577/#1397 die structurally when the remaining writers move in the follow-up commits). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tle (#1582) The lasting copy moves from main's .the-framework/<user>/agents/ to agents/<user>/ at the data branch's root: teardown funnels archiveWorktreeAgent through withDataBranch, so every archive is a pushed commit the moment the session settles — no debounce, no flush pass, no human. That deletes agent-commit.ts whole (the committer service, its clock job, and the shutdown flushAgents phase existed to commit archives a human would otherwise have to notice). Readers (listAgents/findArchive and friends) look at the data checkout first, then the transient .the-framework/agents/ a worktree-less agent still writes; the pre-#1582 <user>/agents location on main is not read — hand-run migration moves the existing history. Delete removes a data-branch record through the funnel too, so the deletion is itself a pushed commit. .the-framework/.gitignore collapses to 'ignore it all' (nothing under it is committed on main any more), and ensureArchiveIgnored goes with the rules it repaired. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… through it (#1582) New 'The data branch' section rides the system channel beside the two format specs: where tickets/ and TODO_AGENTS.md live, how to read them from any checkout (git show), and the fetch → detached worktree → commit → push → rebase-on-reject write cycle that works the same from a local worktree and a cloud clone. The presets that touch framework data (drain, plan-tickets, both triages, both suggests, update-tickets, maintenance, market-research), the two in-code pins in auto-pm.ts, and the system/on-before-mergeable TODO_FILE macros all point at it. The pinned drain contract changes shape: the agent no longer checks its entry off (the daemon retires it at settle) and the ticket/plan/lock cleanup is a data-branch commit instead of riding the PR; a plan is now a data write, not a PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e says why (#1582) Live smoke found the pair: retiring the last ticket removes tickets/ itself (git keeps no empty dirs, and the branch is born without it), and acquireTicketLocks' default write was a bare writeFile — so the next ticket-linked drain claim died on ENOENT inside the funnel, acquire resolved [], and the sweep stood the batch down with 'every entry in this batch links a ticket another agent already claimed', which was wrong on every word. The default write now creates parents, the way data-branch.ts' own default already does, and a cycle that fails before any lock lands is logged with its real error instead of only when part of the batch had already been claimed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ch funnel (#1582) Two textual conflicts: worktrees.ts' import line (union — their readFile for the strict meta read, our sep for the data-root prefix), and daemon-runtime.SPEC.md (their shutdown-refusal bullet kept; the archive-committer sentence stays deleted — this branch removes that service). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

#1594) The merge crossed two changes: #1594's lifecycle test keeps the handoff-local checkout and reads the archived replay right after the worktree shows on the Remove list — which is true the moment the meta flips done, before teardown runs — while #1582 made the archive a data-branch git cycle instead of a file copy. The read now waits for the journal the way the rest of the story waits for its states; the product surface was already fine (the dashboard polls). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Three integration tests on the real-git fixture, straight from the live smoke: a run whose archived record reports the hand-off gets its entry checked off as a data-branch commit (and only that entry); a run whose hand-off failed leaves its entry open — unpublished work is not retired; and a drain still claims an entry whose ticket file is gone, recreating tickets/ on the way — the batch-level proof of the missing-parent fix, which previously stood the sweep down with the wrong message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

suleimansh · 2026-08-19T00:58:05Z

@brillout prompt changes in 4052c45c want your look — one new "The data branch" section (tickets/queue/archives on the-framework_data, read via git show, write via detached worktree + direct push), all data-touching presets point at it. Drain agents no longer check their entry off — the daemon retires it at settle. Big call: a plan is now a data write, not a PR (plan review moves to the human queue).

Manually tested end to end on a scratch repo: dashboard queue-add, two real drain runs following the new protocol unprompted, check-off + archives landing as data commits. Objections to plans-as-data-writes, or wording tweaks?

…tignore keeps !LAYOUT

brillout · 2026-08-19T06:31:46Z

The prompt changes are messy, not DRY enough, and not succinct enough. WDYT? Ideas?

brillout · 2026-08-19T06:32:46Z

 | 83 | Handoff panel: push / open PR / merge, as buttons |
 | 84 | A withheld merge is reported with its reason |
-| 85 | Agent history archived in the repo under per-user directories |
+| 85 | Agent history archived on the `the-framework_data` branch under per-user directories — pushed the moment a session settles |


I thought we now prefix with tf-?

Right, missed #1581's convention — renaming to tf-data. All occurrences are new in this PR, so it's a pure rename.

suleimansh · 2026-08-19T09:08:27Z

Agreed — fix is deletion. The system prompt already says it twice (the tickets/TODO context bullets + the "The data branch" section), so all ten per-preset "(on the data branch — see …)" parentheticals go. Only behavior survives: drain's check-off step and update_tickets' one-commit rule. Prompt diff shrinks to the protocol file + two context-bullet tweaks.

…e data lives Review follow-ups on #1595: - the-framework_data -> tf-data everywhere: #1581 made tf- the prefix for every framework-minted branch, and the data branch was the one newcomer that ignored it. The name only exists on this PR's branch, so it is a pure rename. - The per-preset "(on the data branch — see ...)" reminders are deleted. The system prompt already carries the fact twice — the tickets/TODO context bullets and the "The data branch" protocol section — so the presets keep only genuine behavior: drain's check-off-when-published timing and update_tickets' whole-update-as-one-commit rule.

brillout · 2026-08-19T09:38:41Z

+6. `$ git worktree remove --force .tf-data`
+
+Rules:
+- Never commit these files to your session branch, and never mix a data commit with code commits — a data change is pushed directly, it does not ride your PR


Is that true? I think there are use cases when a PR also modifies data?

Actually, because a PR always targets a non-data branch, it kinda must be true, I think?

Yes, must be true — main carries zero data files now, so a PR with a ticket edit would re-land data on main at merge. The rule is there because committing tickets next to code is the old habit this branch breaks.

old habit

There shouldn't be any old habit left, since we should remove all old code. Zero compat code, as always.

Bad wording from me, there is no compat code — "habit" = the model's default to commit tickets next to code. The rule guards against that.
Tested e2e on a scratch repo: drain agent with the trimmed prompt did clean data commits, nothing leaked on its code branch.

brillout · 2026-08-19T09:39:35Z

+
+Rules:
+- Never commit these files to your session branch, and never mix a data commit with code commits — a data change is pushed directly, it does not ride your PR
+- With no remote, push is skipped: the commit on the local `tf-data` branch is the write


That's a weird exception, isn't it?

Agreed, it only reads weird — the write is the commit; push is just propagation, and no remote means nothing to propagate to. Folding it into the write sentence.

Why not ignoring this egde case? (AFAICT it would almost never happen.)

If anything such edge case is an error state that should be surface to the UI, right?

Swallowing is actually the worst way to handle this erroneous state?

Agree, dropping it, protocol just says commit + push, no special case.
No remote becomes an error surfaced to the UI.
Removing the clause + small ticket for the UI part.

brillout · 2026-08-19T09:41:57Z

+
+Only when the instructions tell you to edit tickets or the queue:
+
+1. `$ git fetch origin tf-data` (skip without a remote)


It feels like too much like micro management? Is it needed? Agents are quite proficient with git, so not sure it's needed? (I'd be curious how Claude Code Web handles git operations, but I'm inclined to think it doesn't micro manage agents.) WDYT?

Agree — dropping the command recipe, keeping just the contract: commit on tf-data, push immediately, rebase on rejection, never via your checkout or PR. Claude Code's own prompts don't micromanage git either. The recipe did pass the E2E drain runs, so if trimmed agents fumble, the cheap fallback is one worktree hint.

brillout · 2026-08-19T09:49:14Z

Brainstorming:

We mention the full path .the-framework/branches/tf-data/TODO_AGENTS.md
TF automatically rebases & pushes data

I think we don't need 1. and explaining where TODO_AGENTS.md lives is enough.

Not sure about 2.. For logs it seems clearly that eagerly pushing data is the way to go. Pushing ticket locks as well, but then we need to handle the situation there is a conflict — in that case I think it's better the agent tries to push and fails?

Review follow-ups on #1595: the six-step command recipe goes — agents get the contract instead (commit on tf-data pushed immediately, rebase on a rejected push, never via the checkout or the PR). The no-remote line stops being a standalone exception: the write is the commit, push is just propagation, so it folds into the write sentence. 24 lines down to 7.

suleimansh · 2026-08-19T10:29:03Z

Brainstorming:

We mention the full path .the-framework/branches/tf-data/TODO_AGENTS.md

TF automatically rebases & pushes data

I think we don't need 1. and explaining where TODO_AGENTS.md lives is enough.

Not sure about 2.. For logs it seems clearly that eagerly pushing data is the way to go. Pushing ticket locks as well, but then we need to handle the situation there is a conflict — in that case I think it's better the agent tries to push and fails?

1: agree, the trimmed protocol has no paths anymore.
2: the daemon already does this for its own writes. For agents push-and-fail is right, on locks the conflict IS the claim mechanism, loser backs off. And cloud runs have no daemon, so agents must push themselves.

Will test this and report back.

Per the review round: the protocol says commit + push, no special case. A project without a remote is an erroneous state for the daemon to surface, not one the prompt accommodates.

Verified the arbitration mechanics on a scratch remote: two agents claiming the same ticket race on the push — the loser's rebase hits an add/add conflict on the .lock.md. The lock section now tells the agent what that conflict means: back off, never resolve it in your own favor. Claims of different tickets rebase cleanly and both land.

suleimansh · 2026-08-19T10:46:31Z

Tested the lock race on a scratch remote: two agents claim the same ticket → second push rejected, rebase conflicts on the lock file → loser backs off, winner's lock intact.
Different tickets → both land cleanly.
So push-and-fail works as the claim mechanism, like you said. Added one line to the prompt so the loser knows the conflict means "back off", not "resolve it".

An uncommitted symlink at the repo root rides any sweeping git add -A onto a code branch — the leak the drain E2E surfaced. The info/exclude mechanism moves out of worktree-deps into git-exclude.ts so both callers share it; the rule is root-anchored and only written when the link is ours, so a user's own tickets path stays visible. The sibling leak (auto-onboard skipping installProject) is #1600. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

suleimansh · 2026-08-19T14:30:11Z

🤖 agent comment

One more commit after the review round: the drain E2E showed the new root tickets symlink was untracked, so any sweeping git add -A would commit it onto a code branch. a95e097b hides it through the repo-level exclude the moment it is created (mechanism extracted to git-exclude.ts, shared with the node_modules-links case; a user's own tickets path stays visible and untouched). The sibling leak — auto-onboarded repos never getting .the-framework/.gitignore at all — predates this PR and is filed as #1600.

Amendment (9e4d0324): the first cut used a bare /tickets rule, and the repo-level exclude speaks for every worktree — including the tf-data checkout itself, so data commits silently dropped ticket files (CI's control.test caught it). The rule is now the pair /tickets + !/tickets/: a trailing slash never matches a symlink, so the root link stays hidden while the data checkout's real directory keeps committing. Regression-tested in data-branch.test; full suites green (1468 + 772).

suleimansh · 2026-08-19T14:52:49Z

Before the migration runs after this merges: can you hand-publish @gemstack/the-framework from current main? #1598's layout gate only exists in unpublished code — latest on npm is 1.4.2 (Jul 27) — so until a gate-carrying build is out, a stale installed build would happily operate on the migrated layout with nothing to refuse it. Order would be: publish → merge this → migrate.

The repo-level exclude speaks for every worktree at once — including the tf-data checkout, whose root holds the real tickets/ the branch exists to carry. A bare /tickets rule swallowed it: data commits silently dropped every ticket file (CI's control.test caught it). The rule becomes a pair, /tickets then !/tickets/ — a trailing slash never matches a symlink, so the root link stays hidden while the directory keeps committing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… emitter: the data-branch sync (fix #1599, towards #1500) A per-project error state the daemon's background jobs set and clear (`project-errors.ts`): one slot per project and kind, with the detail and since when. The project list carries it (`onProjects` attaches each project's errors), the sidebar turns the project's dot red naming the error on hover, and the project's page shows a banner above the start form. The first emitter is the data-sync tick: a push origin rejects, or a repository with no remote at all, sets the `data-sync` error; the first sync that converges clears it. `pullDataBranch` now reports its outcome, counting no-remote as a failure to converge, since a data branch nobody else can reach is an error (#1595), not a mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

suleimansh and others added 8 commits August 19, 2026 01:09

SPECs, feature list, and changeset catch up with the data branch (#1582)

fdbefff

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

suleimansh marked this pull request as ready for review August 19, 2026 00:41

suleimansh mentioned this pull request Aug 19, 2026

A skewed framework build is refused by the repo's layout marker (fix #1575) #1598

Merged

Merge main: the layout gate learns the data branch, the simplified gi…

e1c4b90

…tignore keeps !LAYOUT

brillout reviewed Aug 19, 2026

View reviewed changes

The no-remote case stops being a supported mode of the data protocol

60c76d9

Per the review round: the protocol says commit + push, no special case. A project without a remote is an erroneous state for the daemon to surface, not one the prompt accommodates.

suleimansh mentioned this pull request Aug 19, 2026

Data-branch sync failures should surface in the dashboard #1599

Open

suleimansh mentioned this pull request Aug 19, 2026

Auto-onboarded repos never get .the-framework/.gitignore, so agents sweep framework state onto work branches #1600

Closed

suleimansh changed the title ~~All framework data moves to the the-framework_data branch (fix #1582)~~ All framework data moves to the tf-data branch (fix #1582) Aug 19, 2026

suleimansh merged commit 2b5fe13 into main Aug 19, 2026
2 checks passed

suleimansh deleted the issue-1582-data-branch branch August 19, 2026 15:43

suleimansh mentioned this pull request Aug 20, 2026

Cloud runs' work is adopted onto the run's record (fix #1601) #1603

Open


		Only when the instructions tell you to edit tickets or the queue:

		1. `$ git fetch origin tf-data` (skip without a remote)

Uh oh!

Conversation

suleimansh commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suleimansh commented Aug 19, 2026

Uh oh!

brillout commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suleimansh commented Aug 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brillout Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brillout Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brillout commented Aug 19, 2026

Uh oh!

suleimansh commented Aug 19, 2026

Uh oh!

suleimansh commented Aug 19, 2026

Uh oh!

suleimansh commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suleimansh commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

suleimansh commented Aug 18, 2026 •

edited

Loading

brillout commented Aug 19, 2026 •

edited

Loading

brillout Aug 19, 2026 •

edited

Loading

brillout Aug 19, 2026 •

edited

Loading

suleimansh commented Aug 19, 2026 •

edited

Loading