From aadffde8089600830befc4fd98d7eb13148cdaef Mon Sep 17 00:00:00 2001 From: Eliot Lim Date: Sun, 4 Oct 2026 01:52:18 +0800 Subject: [PATCH 1/6] feat(ui): add meeting recording, transcription and notes (MEET-5) --- _brief-meet5.md | 45 ++++ _brief.md | 32 --- _report.md | 87 ------- packages/sdk/src/client.ts | 2 + packages/server/src/localClient.ts | 2 + packages/ui/src/blockeditor/BlockEditor.tsx | 5 +- .../blockeditor/MeetingBlockPlaceholder.tsx | 14 -- .../ui/src/blockeditor/MeetingBlockView.tsx | 96 +++++++ packages/ui/src/blockeditor/SlashMenu.tsx | 2 +- .../__tests__/meetingRecorder.test.tsx | 237 ++++++++++++++++++ packages/ui/src/blockeditor/exportBlocks.ts | 30 ++- packages/ui/src/blockeditor/kit/index.ts | 2 +- .../ui/src/blockeditor/meetingRecorder.ts | 222 ++++++++++++++++ packages/ui/src/blockeditor/registry.tsx | 2 + .../ui/src/components/AssetBridgeHost.tsx | 1 + .../ui/src/components/history/VersionDiff.tsx | 2 + packages/ui/src/i18n/messages/de.ts | 36 +++ packages/ui/src/i18n/messages/en.ts | 36 +++ packages/ui/src/i18n/messages/ja.ts | 36 +++ packages/ui/src/i18n/messages/zh.ts | 36 +++ packages/ui/src/index.css | 15 ++ packages/ui/src/lib/assetBridge.ts | 6 + 22 files changed, 808 insertions(+), 138 deletions(-) create mode 100644 _brief-meet5.md delete mode 100644 _brief.md delete mode 100644 _report.md delete mode 100644 packages/ui/src/blockeditor/MeetingBlockPlaceholder.tsx create mode 100644 packages/ui/src/blockeditor/MeetingBlockView.tsx create mode 100644 packages/ui/src/blockeditor/__tests__/meetingRecorder.test.tsx create mode 100644 packages/ui/src/blockeditor/meetingRecorder.ts diff --git a/_brief-meet5.md b/_brief-meet5.md new file mode 100644 index 00000000..2ba8a8fd --- /dev/null +++ b/_brief-meet5.md @@ -0,0 +1,45 @@ +# MEET-5 — Recorder UI: meeting block view, progressive transcript, manual notes + +You are Ellis, an OpenBook Worker agent. Work in THIS worktree (`/Users/eliot/Workspaces/OpenBook-wt-meet-4`). Do NOT push. Conventional commits (`feat(ui): … (MEET-5)`), committed incrementally. DISK IS NEARLY FULL (~1.5 GiB free): do not download models or create worktrees; clean any large temp artifacts you produce. + +## Branch setup (do this first, carefully) +1. `git checkout -b feat/meet-5-recorder-ui` (from current HEAD = feat/meet-4-meeting-catalogue). +2. `git merge feat/meet-1-audio-assets` then `git merge feat/meet-2-transcribe`. Expected conflicts are trivial: + - `packages/server/src/ableOidc.test.ts`: all three branches contain the identical code line `expiresAt: new Date(Date.now() + 60_000)`; conflicts are comment-only. Resolve keeping MEET-2's comment wording (PAT validation uses the database clock, not the injected OIDC clock). + - Possible add/add in `packages/sdk/src/index.ts` export lines — keep BOTH exports, watch for duplicated re-export lines (known trap). + - `_brief.md`/`_report.md` conflicts: resolve by DELETING these files (they're being stripped from all branches). +3. After the merges, run `pnpm build:libs` (semantic-conflict check) before starting feature work. + +## What you're building +The real `meeting` block UI, replacing MEET-4's placeholder shell (registered via registerArtifactKit — find it and replace its renderer; keep registration/drift-guard structure intact). + +### Contracts you MUST follow (already merged into your branch) +- **Block props (MEET-4 contract, in `_report.md` at git ref 1f97ba06 — read `git show 1f97ba06:_report.md`):** props `status ('idle'|'recording'|'processing'|'done')`, `audioChunks [{assetId, durationMs, startedAtMs?}]` (startedAtMs = offset from meeting start), `transcript [{startMs, endMs, text}]` (offsets relative to meeting start, ms), `summary` (plain string), `startedAt` (epoch ms), `title?`. Arrays replace wholesale — batch and serialize your writes (single writer: the recording client). Manual notes = ordinary CRDT child blocks; MEET-4's shell stored them — render them as an editable notes region (children editing inside a kit block: see how form/KitFrame handles nested content; follow useKitLock + editor.readOnly semantics, BlockEditor.tsx ~:2429-2449). +- **Transcription API (MEET-2):** `client.transcribeAsset(assetId, pageId)` → `{text, segments?[{start,end,text}] (SECONDS), durationMs}`. Convert segment seconds → ms and offset by the chunk's startedAtMs when appending to `transcript`. Errors: 400 (unconfigured → show actionable copy pointing at Settings → AI), 403 (guest/paid gate), 404, 502. LocalDataClient rejects AI — degrade gracefully (recording + notes still work; transcription affordance disabled with clear copy). +- **Audio upload (MEET-1):** audio mimes webm/ogg/mpeg/mp4/wav accepted by the asset store; 10 MiB/asset cap. Upload chunks via the assetBridge (ui/src/lib/assetBridge.ts; ImageBlockView.tsx is the bytes↔objectURL reference). + +### Recording behavior +- getUserMedia + MediaRecorder. **Restart MediaRecorder per ~45s chunk so every chunk is a standalone playable file** (do NOT use one recorder with timeslice — those blobs aren't independently decodable). Prefer `audio/webm;codecs=opus`, probe with MediaRecorder.isTypeSupported and fall back (Safari → audio/mp4). +- Controls: record / pause / resume / stop; elapsed timer; per-chunk progress. Status prop transitions idle→recording→processing→done. +- Per chunk pipeline: finalize blob → upload asset → append to audioChunks prop → transcribeAsset → append converted segments to transcript prop. Transcript grows during the meeting. Chunk upload MUST succeed/retry even if transcription fails (flag untranscribed chunks, offer per-chunk retry). Exponential backoff; never lose audio; ephemeral state (recorder handle, retry queue, errors) in React state, durable state in props within editor.doc.transact. +- Mic-permission denied → graceful copy, no crash. Playback: simple per-chunk