Skip to content

Bound local audio transcription requests - #3175

Queued
robgruen wants to merge 11 commits into
mainfrom
fix/whisper-audio-input-limits
Queued

Bound local audio transcription requests#3175
robgruen wants to merge 11 commits into
mainfrom
fix/whisper-audio-input-limits

Conversation

@robgruen

@robgruen robgruen commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Accept audio files up to 30 MiB and recordings up to five minutes.
  • Check declared and streamed request sizes before multipart parsing.
  • Decode 16 kHz mono audio with bounded output and a 30-second FFmpeg timeout.
  • Process one request at a time, permit one waiting request, and report busy responses beyond that.
  • Keep processing reserved until its worker finishes, including repeated client cancellation.
  • Preserve successful transcription output and display input/processing errors in both clients through a shared helper.
  • Update the sample's bundled entry point and document the service limits and offline tests.

Validation

  • 11 Python endpoint tests passed with real FFmpeg decoding and a small model stand-in, including exact-byte/sample boundaries, compressed audio, streamed requests, timeouts, and cancellation.
  • Python syntax checks passed.
  • 9 compiled shared-client Jest tests passed.
  • Built sample client verified in Chromium for HTTP 200, 413, and 503.
  • Shared utility, sample client, and shell dependency-closure builds passed. Shell build used one package at a time and a bounded Node heap.
  • Repository-pinned formatting passed.
  • Lint, complexity, circular-dependency, and test-debt PR gates passed against origin/main, including tests.
  • Whitespace checks passed.

No Whisper model was downloaded or GPU inference run during offline validation.

Developed and validated in an isolated git worktree.

@robgruen
Bound local audio transcription requests
29e8799 
Apply upload, recording-length, decoding, and request-queue limits. Preserve transcription output and report processing failures in both browser clients using a shared helper, with focused tests and documentation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GeorgeNgMsft
George Ng (GeorgeNgMsft) reviewed Oct 15, 2026
View reviewed changes
Comment thread python/stt/whisperService/faster-whisper.py
GeorgeNgMsft
George Ng (GeorgeNgMsft) reviewed Oct 15, 2026
View reviewed changes
Comment thread ts/packages/shell/src/renderer/src/localWhisperClient.ts
GeorgeNgMsft
George Ng (GeorgeNgMsft) approved these changes Oct 15, 2026
View reviewed changes

@GeorgeNgMsft George Ng (GeorgeNgMsft) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple comments but otherwise looks good

@robgruen
Fix stalled Whisper uploads and isolate recording responses
7359b1c 
Bound total upload time and release admission capacity on timeout. Ignore stale transcription callbacks and silence checks after a new recording starts in both clients. Add endpoint and renderer regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@robgruen
robgruen marked this pull request as ready for review October 15, 2026 22:06
robgruen and others added 2 commits October 15, 2026 15:06
@robgruen
Merge branch 'main' into fix/whisper-audio-input-limits
701111d
@robgruen
Fix shell test configuration after merging main
b898d48 
Keep main's shell test model include and the renderer regression project reference in a single valid configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@robgruen
robgruen enabled auto-merge October 15, 2026 22:51
@robgruen
Merge remote-tracking branch 'origin/main' into fix/whisper-audio-inp…
69822a7 
…ut-limits

# Conflicts:
#	ts/packages/shell/test/tsconfig.json

Co-authored-by: robgruen <25374553+robgruen@users.noreply.github.com>
@robgruen
Exclude Jest and node tests from Playwright
8bf1044 
Co-authored-by: robgruen <25374553+robgruen@users.noreply.github.com>
Hillary Mutisya (hillary-mutisya) added a commit that referenced this pull request Oct 24, 2026
@hillary-mutisya
Reduce macOS CI latency while preserving platform coverage (#3213)
01533d9 
## Why this change

macOS test jobs have been holding up PR validation for more than an
hour, often after the corresponding Ubuntu and Windows jobs have
finished. We reviewed five recent completed PRs to separate slow initial
execution from time spent retrying failures.

The table shows median **Test-step duration in minutes**, including
retries, across each PR's configured Node versions. It does not include
installation or build time.

| PR / workflow run | macOS | Ubuntu | Windows |
| --- | ---: | ---: | ---: |
|
[#3175](https://github.com/microsoft/TypeAgent/actions/runs/39969763926)
| 93.03 | 19.92 | 25.33 |
|
[#3184](https://github.com/microsoft/TypeAgent/actions/runs/39912328526)
| 81.53 | 23.09 | 20.75 |
|
[#3210](https://github.com/microsoft/TypeAgent/actions/runs/39911017633)
| 71.80 | 16.62 | 25.73 |
|
[#3209](https://github.com/microsoft/TypeAgent/actions/runs/39856783535)
| 90.22 | 22.43 | 33.19 |
|
[#3203](https://github.com/microsoft/TypeAgent/actions/runs/39726876699)
| 72.02 | 18.82 | 28.16 |
| Median across all tested jobs | **75.63** | **19.55** | **25.58** |
| Jobs requiring a package retry | **12/12 (100%)** | **3/12 (25%)** |
**1/12 (8.3%)** |

This sample contains 36 executed test jobs. Some PRs had two Node
versions rather than three. Two other recent PRs never reached testing
and were excluded from the retry denominators. These are historical
observations, not a controlled benchmark.

Both initial execution and retries are expensive:

- macOS initial passes took **38–92 minutes**, with a **61.85-minute
median**, before retries. Every sampled macOS job retried, so the sample
has no retry-free macOS control.
- Memory-service retries reran **61–67 suites for only 1–6 failed
files**, costing another **7–30 minutes**. In one job, dispatcher reran
146 suites after one file failed, taking 11.5 minutes.
- Roughly 73% of macOS failed-case occurrences were timeouts or bounded
waits. All recovered on the observed serial package retries. The logs
suggest timing sensitivity but do not establish the underlying resource
or runtime cause.
- The slowdown extends beyond memory: in the separately investigated
[run
39985375882](https://github.com/microsoft/TypeAgent/actions/runs/39985375882),
dispatcher took 50–56 minutes and default-agent-provider took 34–37
minutes on the slow macOS jobs.

Removing a few scaling tests would not address these delays. This PR
keeps a smaller macOS compatibility suite blocking changes, moves full
macOS coverage to nightly runs, and reduces unnecessary retry work.

## Changes

1. **Split blocking and nightly macOS coverage.**
- PR, merge-queue and push jobs run an explicit selection of 13 package
suites on macOS, across all three supported Node versions.
- The selection covers file links, native SQLite, locking, memory
publication/persistence/privacy, small Git history, session lifecycle,
and representative cache merge/non-merge round trips.
- All root script tests remain blocking, including aliased-directory
healing and subprocess checks.
- Full macOS correctness coverage runs nightly at **08:17 UTC** on the
default branch, across all supported Node versions. Manual dispatch runs
full coverage on all three operating systems.
- Full Ubuntu/Windows blocking coverage is unchanged. The default local
test profile also remains full.

2. **Retry failed files where the runner supports it.**
- Recognized single-invocation Jest scripts retry the accumulated failed
files serially, with caching disabled, up to twice.
- Chained commands, lifecycle hooks and unsupported runners explicitly
fall back to the complete package command so retries do not bypass setup
or additional test phases.
- Missing failure artifacts and invalid file selections fail closed
rather than clearing failures.

3. **Fix test-phase accounting.**
- Previously, a package failure short-circuited the root script-test
command. A successful package retry could then report green without ever
running those script tests.
- Workspace tests/retries and root script tests now have independent
outcomes. Root script tests run even after unrecovered package failures,
and both phases must pass.
- GitHub summaries report each phase separately; workflows retain test
logs for 14 days.

4. **Apply consistent Jest settings.**
- Browser, browser-extension and CLI configurations now honor the shared
CI-only one-worker limit and retry cache setting.
   - CLI uses the shared failure reporter when reporting is enabled.

5. **Fix macOS aliased-path handling in generated-file healing.**
- Resolve generated-file and scratch-manifest destinations from
canonical roots. This avoids false containment failures when macOS
temporary paths resolve from `/var` to `/private/var`.
- Add an aliased-directory regression while preserving containment and
regular-file checks.

6. **Separate memory correctness from scaling coverage.**
   - Keep a ten-commit history-equivalence case in normal CI.
- Move the 100/1,000-commit variants to a dedicated regression command
and a weekly/manual workflow on Ubuntu, Windows and macOS.
- Keep bounded offline correctness scenarios in CI; the
external-data/model-backed rehearsal was already excluded.

## Coverage tradeoff

The blocking macOS selection is intentionally not the full correctness
suite. Full Ubuntu/Windows coverage still blocks PRs, but macOS-only
regressions outside the selection may first appear nightly. Maintainers
should triage nightly failures and add missing platform-sensitive
regressions to the blocking selection. Contract tests check the explicit
selection and event-specific workflow configuration.

This PR does not claim a measured macOS speedup yet. It also does not
fix the separately identified MCP shutdown test that waits for the
client's 60-second timeout.

## Validation

- Full workspace build passed.
- Compatibility profile: **13 suites / 276 tests passed** locally.
- Combined profile, workflow, Node-version, Jest-configuration and retry
checks: **78 tests passed**.
- After the retry-orchestration refactor, all **56 focused retry tests
passed**. The committed complexity, lint and test-debt gates passed
without threshold changes.
- Generated-file-healing tests: **32 passed**, with five existing
Windows skips.
- Local validation used Windows with Node 24.13.0, below the supported
minimum. The broader root script suite had one installer-test failure
because it requires a supported Node runtime; this PR does not change or
suppress that test.
- Actual macOS behavior and runtime still require CI validation.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@robgruen
Merge remote-tracking branch 'origin/main' into fix/whisper-audio-inp…
bcddb4d 
…ut-limits

# Conflicts:
#	ts/packages/shell/playwright.config.ts
#	ts/packages/shell/src/renderer/src/localWhisperClient.ts
#	ts/packages/shell/src/renderer/src/speech.ts

Co-authored-by: robgruen <25374553+robgruen@users.noreply.github.com>
@robgruen
Fix optional Whisper abort signal typing
808def6 
Co-authored-by: robgruen <25374553+robgruen@users.noreply.github.com>
@robgruen
Fix Whisper abort controller optional type
3b5c9c8 
Co-authored-by: robgruen <25374553+robgruen@users.noreply.github.com>
@robgruen
Correct final Whisper failure test expectation
c92a248 
Co-authored-by: robgruen <25374553+robgruen@users.noreply.github.com>
@robgruen
robgruen added this pull request to the merge queue Oct 25, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GeorgeNgMsft George Ng (GeorgeNgMsft) GeorgeNgMsft approved these changes

Assignees

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@robgruen @GeorgeNgMsft