Skip to content

perf: batch remote pipeline autosaves - #2780

Open
Mbeaulne wants to merge 43 commits into
mb/remote-pipeline-storagefrom
mb/batch-remote-pipeline-autosave
Open

Mbeaulne wants to merge 43 commits into
mb/remote-pipeline-storagefrom
mb/batch-remote-pipeline-autosave

Conversation

@Mbeaulne

@Mbeaulne Mbeaulne commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Description

Batch remote pipeline autosaves without delaying local recovery. This is the next layer above #2739 and contains only the autosave follow-up.

  • Save position-only edits three seconds after the first unsynced move; save other edits after a one-second pause. The earlier deadline uploads the latest complete pipeline.
  • Keep one save in flight and only the newest pending snapshot. Explicit Save and editor navigation bypass the timers.
  • Preserve browser-local recovery before the first upload, including edits made during slow or failed migration. Local-only and connected-folder timing stays unchanged.
  • Finish local reference migration when a first upload was confirmed by the server before the browser closed. Retrying keeps the confirmed server ID and latest recovery content.
  • Add focused timing, snapshot classification, recovery, and permission regression tests.
  • Show pending and in-progress remote saves with a soft yellow cloud and a small spinner. After a successful save, transition through green back to white over 750 ms. Keep routine status text available to screen readers and tooltips, retain visible errors and Retry, and respect reduced motion.

Related Issue and Pull requests

Stacked on #2739 (mb/remote-pipeline-storage), the final layer of the existing remote-pipeline stack.

Type of Change

  • Improvement
  • Documentation update

Checklist

  • I have tested this does not break current pipelines / runs functionality
  • I have tested the changes on staging

Screenshots (if applicable)

N/A.

Test Instructions

Review validation after the interrupted-migration fix:

  • pnpm exec vitest run src/routes/v2/pages/Editor/store/autoSaveStore.test.ts src/routes/v2/pages/Editor/store/autoSaveSnapshot.test.ts src/services/pipelineStorage src/routes/v2/pages/Editor/hooks/useLoadSpec.test.tsx src/routes/v2/pages/Editor/hooks/useSpecLifecycle.test.tsx src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.test.tsx --reporter=dot: 156 passed, including the new interrupted-migration regression.
  • pnpm exec tsc --noEmit and git diff --check: passed.
  • pnpm exec eslint src/services/pipelineStorage/RemotePipelineStore.ts src/services/pipelineStorage/RemotePipelineStore.test.ts: passed with the existing React-version configuration warning.
  • pnpm exec prettier --check src/services/pipelineStorage/RemotePipelineStore.ts src/services/pipelineStorage/RemotePipelineStore.test.ts: passed.
  • CI=1 pnpm run test:e2e:remote --reporter=line: 16 passed during review, before the fix. The tests use a mocked backend.
  • CI=1 pnpm run test:e2e:remote --reporter=line --grep "migrates a local pipeline|the first local edit|renaming a local pipeline": all three affected browser flows passed again after the fix.

Save indicator validation:

  • pnpm exec vitest run src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.test.tsx --reporter=dot: 12 passed.
  • pnpm exec eslint src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.tsx src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.test.tsx: passed with the existing React-version configuration warning.
  • pnpm exec prettier --check src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.tsx src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.test.tsx src/styles/global.css: passed.
  • pnpm exec tsc --noEmit and git diff --check: passed.
  • Isolated browser preview using the actual indicator component and CSS with mocked save state: 750 ms animation, repeated saves, newer edits, failed saves, and reduced motion passed. This does not verify real-backend save latency.

Earlier autosave validation, before the indicator changes:

  • pnpm exec vitest run src/routes/v2/pages/Editor/store/autoSaveStore.test.ts src/routes/v2/pages/Editor/store/autoSaveSnapshot.test.ts src/services/pipelineStorage src/routes/v2/pages/Editor/hooks/useLoadSpec.test.tsx src/routes/v2/pages/Editor/hooks/useSpecLifecycle.test.tsx src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.test.tsx --reporter=dot: 133 passed.
  • pnpm exec vitest run --reporter=dot: initial full run had four failures; a subsequent full run using a JSON reporter had 2,693 passes, three network-dependent failures, and two existing todo tests. All three remaining failures were in ComponentEditorDialog.test.tsx; pnpm exec vitest run src/components/shared/ComponentEditor/ComponentEditorDialog.test.tsx --reporter=dot passed all 11 tests when network access was allowed for its schema download. The other initial failure did not reproduce.
  • CI=1 pnpm run test:e2e:remote --reporter=line: 15 passed and one startup failure. CI=1 pnpm run test:e2e:remote --reporter=line --grep "remote mode preserves" passed the remaining test. These tests use a mocked backend.
  • pnpm exec tsc --noEmit, pnpm run knip, and git diff --check: passed.
  • pnpm exec eslint src/routes/v2/pages/Editor/store/autoSaveStore.ts src/routes/v2/pages/Editor/store/autoSaveStore.test.ts src/routes/v2/pages/Editor/store/autoSaveSnapshot.ts src/routes/v2/pages/Editor/store/autoSaveSnapshot.test.ts src/services/pipelineStorage/PipelineFile.ts src/services/pipelineStorage/PipelineStorageService.ts src/services/pipelineStorage/PipelineStorageService.test.ts src/services/pipelineStorage/RemotePipelineStore.ts src/services/pipelineStorage/RemotePipelineStore.test.ts: passed with the existing React-version configuration warning.
  • pnpm exec prettier --check docs/remote-pipelines.md src/routes/v2/pages/Editor/store/autoSaveStore.ts src/routes/v2/pages/Editor/store/autoSaveStore.test.ts src/routes/v2/pages/Editor/store/autoSaveSnapshot.ts src/routes/v2/pages/Editor/store/autoSaveSnapshot.test.ts src/services/pipelineStorage/PipelineFile.ts src/services/pipelineStorage/PipelineStorageService.ts src/services/pipelineStorage/PipelineStorageService.test.ts src/services/pipelineStorage/RemotePipelineStore.ts src/services/pipelineStorage/RemotePipelineStore.test.ts: passed.

Manual smoke test, not yet run against a real backend:

  1. Enable VITE_REMOTE_PIPELINES_ENABLED=true and connect to a compatible backend as a user with write permission. Watch pipeline PUT requests in the browser Network panel.
  2. Move nodes repeatedly. Expect the latest positions to upload about three seconds after the first unsynced move, without postponement by later moves.
  3. Change an input. Expect a save after a one-second pause; additional movement must not delay it.
  4. Make a change and immediately use Save or navigate away. Expect the latest snapshot to save without waiting for the timer.
  5. Throttle or fail an upload, make more edits, and confirm only the newest pending snapshot follows it. Reopen an unsaved draft and confirm it is recoverable without automatically retrying merely on open.
  6. Edit a writable remote pipeline. Expect a yellow cloud and spinner while changes are pending or saving, followed by green fading to white over 750 ms after the server save succeeds. A failed save must keep the error and Retry visible without showing the success animation.

Additional Comments

No service worker, backend, list-fetching, conflict-resolution, or automatic retry-policy changes. Cross-session saves remain last-writer-wins. Real-backend load testing and staging validation have not been performed.

Change Size

Relative to mb/remote-pipeline-storage for this PR only. Changed lines = additions + deletions.

Category Files Added Removed Changed
Application code 7 +446 -107 553
Tests 5 +961 -33 994
Documentation / configuration 1 +1 -0 1
Total 13 +1408 -140 1548

Application code excludes test files. Tests include unit, component, and browser test source; test-runner configuration, CI, package scripts, compiler configuration, and documentation are counted separately.

@Mbeaulne
Mbeaulne requested a review from a team as a code owner September 22, 2026 13:01
@github-actions

github-actions Bot commented Sep 22, 2026

Copy link
Copy Markdown

🎩 Preview

A preview build has been created at: mb/batch-remote-pipeline-autosave/05e000f

@Mbeaulne
Mbeaulne added this pull request to stack #2752 September 22, 2026 13:07
Comment thread src/services/pipelineStorage/PipelineStorageService.ts
Comment thread src/services/pipelineStorage/PipelineStorageService.ts
…ne-autosave

# Conflicts:
#	docs/remote-pipelines.md
#	src/components/Home/PipelineSection/PipelineFiltersBar.tsx
#	src/components/Home/PipelineSection/PipelineSection.tsx
#	src/components/Home/PipelineSection/useRemotePipelineList.test.ts
#	src/components/Home/PipelineSection/useRemotePipelineList.ts
#	src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.test.tsx
#	src/routes/v2/pages/Editor/components/EditorMenuBar/components/AutoSaveIndicator.tsx
#	src/routes/v2/pages/Editor/store/autoSaveStore.test.ts
#	src/routes/v2/pages/Editor/store/autoSaveStore.ts
#	src/services/pipelineStorage/PipelineFile.ts
#	src/services/pipelineStorage/PipelineStorageService.test.ts
#	src/services/pipelineStorage/PipelineStorageService.ts
#	src/services/pipelineStorage/RemotePipelineStore.test.ts
#	src/services/pipelineStorage/RemotePipelineStore.ts

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant