feat: intermediates manager - #321
Open
joshistoast wants to merge 15 commits into
Open
joshistoast wants to merge 15 commits into
joshistoast wants to merge 15 commits into
Conversation
Add a media_references table and the intermediates schema (project provenance and file sizes on images, videos and queue items), backfilled from existing project and workflow documents. Project and workflow writers keep the index current, and deleting an account drops the references its projects held.
…olicy Outputs and uploads inherit the project they were made for, and media records carry their on-disk size so reclaimable space can be reported per project. Every intermediate is classified as active, recent, referenced or safe in one query; previews freeze their targets and a guard re-applies the policy on the deleting transaction. The legacy clear endpoint now deletes only safe images.
Add /api/v1/intermediates for summaries, previews, starting, polling and retrying cleanup operations, and forward operation progress to the caller's socket room and to admins. Regenerate the legacy web OpenAPI document and types.
Canvas image uploads carry the owning project id so the server can attribute the resulting intermediates.
Per-project rows show owner, reclaimable size and used/unused counts, with bulk selection across pages, a preview-backed delete, a force-delete disclosure and live operation progress. Entry points open it from Data & workspace, the project actions menu and the Users page. Mock backend routes, fixtures and an accessibility journey cover the section; performance baselines are re-recorded for the new startup-reachable modules.
Explain the safe, referenced, active and recent classes, the force-delete disclosure and the API.
joshistoast
requested review from
JPPhoto,
Pfannkuchensack,
blessedcoolant and
lstein
as code owners
September 23, 2026 02:49
Persist bounded cleanup receipts and video purge journals; protect browser-held and newly referenced media. Restore UI operations across reloads, constrain project and admin scopes, and update API artifacts and coverage.
Hold cached output references under the cleanup transaction lock and recompute outputs when deletion wins the race. Cover active-session lifetimes and cleanup races with regression tests.
Add an upgrade migration with partial indexes for unmeasured intermediate images and videos. Verify bounded batch query work as measured rows leave the indexes.
Gate canvas uploads on persistence-owned server readiness and recheck account, project, and cancellation lifetimes. Cover delayed creation, retry, disposal, and stale-account uploads.
Track all-matching exclusions by identity and resolve current counts with a bounded summary query. Keep safe and force previews scoped to selected rows, with coverage for disappearing exclusions and query limits.
Show lookup errors with retry and dismissal controls, and stop polling missing operations. Cover restored receipts, read-only retries, and focus restoration after dismissal.
Preserve project ownership through batched workflow children and real SQLite retry coverage. Skip media holds for cache outputs without references.
Include optional project and file size fields in the legacy ImageDTO shape so its compile-time schema assertion passes.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Intermediates could only be cleared instance-wide, with no way to see which project or account made them, how much disk they hold, or whether a saved project still needs them. Clearing risked breaking saved documents and in-flight work.
This adds an Intermediates section to the Settings dialog: one row per project (cover, name, owner, reclaimable size, used/unused counts), search, bulk selection across pages, and a preview-backed Delete. Admins see every account; everyone else sees their own projects. Force delete lives inside the confirmation as an advanced disclosure that requires an acknowledgement and typing
CLEAR.The policy is server-owned. Every intermediate is classified in one query as active (a queued or running job owns it), recent (created in the last 30 minutes), referenced (a saved project or workflow names it, tracked by a new
media_referencesindex kept current by the project and workflow writers) or safe. Previews freeze their targets, and a guard re-applies the policy on the deleting transaction, so a save racing a delete can never lose a referenced file. Outputs and uploads now inherit the project that made them; existing rows stay unassigned rather than being inferred from board membership.Entry points: the Data & workspace section, the project actions menu (preselects that project) and the Users page (filters to that account).
Related Issues / Discussions
None.
QA Instructions
Manual, in Chromium against the mock backend (
scripts/mock-backend.mjs, representative profile): open Settings → Intermediates; select rows on two pages and confirm the estimate; Delete → preview → confirm; watch the operation panel settle and the rows refresh; toggle the force disclosure and confirm the affected-documents list, acknowledgement and typed word gate the button; search with no matches; 640px width. Focus returns to the trigger or falls back to the search box after the dialog closes.Not verified by hand: multiuser admin surfaces (the mock backend has no admin profile); those are covered by the router, service and browser tests only.
Performance baselines were re-recorded deliberately for the new startup-reachable modules (settings contribution metadata and entry points): launchpad initial raw +1.8 KB, editor initial raw +7.2 KB.
Review
Material findings resolved during review:
updated_at-only cache stamp; the stamp now includes status, sequence and session length.Remaining limitations:
GET /images/intermediatescount still counts every intermediate while the legacyDELETEnow removes only safe ones.Compatibility / Rollout
2026_09_22_add_intermediates_managementaddsproject_idtosession_queue,imagesandvideos,file_size_bytestoimagesandvideos, partial indexes on intermediate rows, and themedia_referencestable backfilled from existing project and workflow documents. Existing media stays unassigned; sizes are measured by a background worker after startup./api/v1/intermediatesroutes and aintermediates_operation_changedsocket event. The legacy webopenapi.jsonandschema.tsare regenerated. Upload endpoints accept an optionalproject_idthat must name one of the caller's own projects.Checklist
What's Newcopy (if doing a release after this PR)