Skip to content

feat: intermediates manager - #321

Open
joshistoast wants to merge 15 commits into
mainfrom
feat/intermediate-management
Open

joshistoast wants to merge 15 commits into
mainfrom
feat/intermediate-management

Conversation

@joshistoast

@joshistoast joshistoast commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Intermediates could only be cleared instance-wide, with no way to see which project or account made them, how much disk they hold, or whether a saved project still needs them. Clearing risked breaking saved documents and in-flight work.

This adds an Intermediates section to the Settings dialog: one row per project (cover, name, owner, reclaimable size, used/unused counts), search, bulk selection across pages, and a preview-backed Delete. Admins see every account; everyone else sees their own projects. Force delete lives inside the confirmation as an advanced disclosure that requires an acknowledgement and typing CLEAR.

The policy is server-owned. Every intermediate is classified in one query as active (a queued or running job owns it), recent (created in the last 30 minutes), referenced (a saved project or workflow names it, tracked by a new media_references index kept current by the project and workflow writers) or safe. Previews freeze their targets, and a guard re-applies the policy on the deleting transaction, so a save racing a delete can never lose a referenced file. Outputs and uploads now inherit the project that made them; existing rows stay unassigned rather than being inferred from board membership.

Entry points: the Data & workspace section, the project actions menu (preselects that project) and the Users page (filters to that account).

Related Issues / Discussions

None.

QA Instructions

Manual, in Chromium against the mock backend (scripts/mock-backend.mjs, representative profile): open Settings → Intermediates; select rows on two pages and confirm the estimate; Delete → preview → confirm; watch the operation panel settle and the rows refresh; toggle the force disclosure and confirm the affected-documents list, acknowledgement and typed word gate the button; search with no matches; 640px width. Focus returns to the trigger or falls back to the search box after the dialog closes.

Not verified by hand: multiuser admin surfaces (the mock backend has no admin profile); those are covered by the router, service and browser tests only.

Performance baselines were re-recorded deliberately for the new startup-reachable modules (settings contribution metadata and entry points): launchpad initial raw +1.8 KB, editor initial raw +7.2 KB.

Review

Material findings resolved during review:

  • A Python-side "temp table current" flag desynced when the deleting transaction rolled back, so a later delete could miss active inputs. The active-media temp table is now rebuilt on every classification, with a regression test.
  • Retrying an operation did not re-authorize the caller. Retries now run as the retrying caller and are covered for foreign, demoted and replayed callers.
  • Same-millisecond queue updates defeated an updated_at-only cache stamp; the stamp now includes status, sequence and session length.
  • Start was answered after the maintenance check, so an idempotent replay could 409 during image moves; replays are now answered first.

Remaining limitations:

  • Operation records live in memory. A server restart forgets running operations, though every delete is journaled and safe.
  • The legacy GET /images/intermediates count still counts every intermediate while the legacy DELETE now removes only safe ones.

Compatibility / Rollout

  • Migration 2026_09_22_add_intermediates_management adds project_id to session_queue, images and videos, file_size_bytes to images and videos, partial indexes on intermediate rows, and the media_references table backfilled from existing project and workflow documents. Existing media stays unassigned; sizes are measured by a background worker after startup.
  • New /api/v1/intermediates routes and a intermediates_operation_changed socket event. The legacy web openapi.json and schema.ts are regenerated. Upload endpoints accept an optional project_id that must name one of the caller's own projects.
  • The server must be restarted onto this build before the section shows data; the frontend degrades to an error state with retry against an older server.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

Add a media_references table and the intermediates schema (project provenance and file sizes on images, videos and queue items), backfilled from existing project and workflow documents.
Project and workflow writers keep the index current, and deleting an account drops the references its projects held.
…olicy

Outputs and uploads inherit the project they were made for, and media records carry their on-disk size so reclaimable space can be reported per project.
Every intermediate is classified as active, recent, referenced or safe in one query; previews freeze their targets and a guard re-applies the policy on the deleting transaction. The legacy clear endpoint now deletes only safe images.
Add /api/v1/intermediates for summaries, previews, starting, polling and retrying cleanup operations, and forward operation progress to the caller's socket room and to admins.
Regenerate the legacy web OpenAPI document and types.
Canvas image uploads carry the owning project id so the server can attribute the resulting intermediates.
Per-project rows show owner, reclaimable size and used/unused counts, with bulk selection across pages, a preview-backed delete, a force-delete disclosure and live operation progress.
Entry points open it from Data & workspace, the project actions menu and the Users page. Mock backend routes, fixtures and an accessibility journey cover the section; performance baselines are re-recorded for the new startup-reachable modules.
Explain the safe, referenced, active and recent classes, the force-delete disclosure and the API.
@joshistoast joshistoast changed the title feat: per-project intermediates manager with a server-owned cleanup policy feat: intermediates manager Sep 23, 2026
Persist bounded cleanup receipts and video purge journals; protect browser-held and newly referenced media.

Restore UI operations across reloads, constrain project and admin scopes, and update API artifacts and coverage.
Hold cached output references under the cleanup transaction lock and recompute outputs when deletion wins the race.
Cover active-session lifetimes and cleanup races with regression tests.
Add an upgrade migration with partial indexes for unmeasured intermediate images and videos.
Verify bounded batch query work as measured rows leave the indexes.
Gate canvas uploads on persistence-owned server readiness and recheck account, project, and cancellation lifetimes.
Cover delayed creation, retry, disposal, and stale-account uploads.
Track all-matching exclusions by identity and resolve current counts with a bounded summary query.
Keep safe and force previews scoped to selected rows, with coverage for disappearing exclusions and query limits.
Show lookup errors with retry and dismissal controls, and stop polling missing operations.
Cover restored receipts, read-only retries, and focus restoration after dismissal.
Preserve project ownership through batched workflow children and real SQLite retry coverage.
Skip media holds for cache outputs without references.
Include optional project and file size fields in the legacy ImageDTO shape so its compile-time schema assertion passes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant