fix(credentials): bound keyring calls by time so a dead backend cannot hang (#1181) - #1184
fix(credentials): bound keyring calls by time so a dead backend cannot hang (#1181)#1184frankbria wants to merge 4 commits into
Conversation
…t hang (#1181) `credentials.py` treated "keyring unavailable" as "raises, or is the fail backend". A backend that is installed and *selected* but unresponsive — SecretService on a headless box, container or SSH session with no session D-Bus — raises nothing. It blocks, with no timeout anywhere on the path. So the encrypted-file fallback never engaged and every credential read hung forever: the test suite wedged at 92%, and `GET /api/v2/settings/keys` hung a server request thread through `_build_status`. Every keyring call now runs through `_keyring_call`, which joins a daemon worker with a timeout and raises `KeyringTimeoutError` — a `KeyringError` subclass, so the existing "keyring failed, use the file" handling at each call site already does the right thing. The blocked call cannot be cancelled (it is waiting inside libdbus), so the worker is abandoned; the verdict is sticky per process and short-circuits the wrapper itself, so at most one thread is abandoned no matter how many stores were built before the backend died. `delete()` was re-raising on a keyring failure, which would have turned a timeout into a failed delete; a timeout there now degrades to the file cleanup that actually matters. - `CODEFRAME_DISABLE_KEYRING=1` skips the keyring outright - `CODEFRAME_KEYRING_TIMEOUT` tunes the bound (default 2.0s, read at call time rather than import — the #963 lesson) `test_keyring_availability_check` asserted "importable implies available", which is the assumption behind the bug; it now asserts a verdict arrives in bounded time. The sticky verdict is reset per test in the root conftest, the same treatment `_MIGRATION_COMPLETE` gets — one real timeout would otherwise push every later keyring test onto the file fallback. Verified: the full CI gate suite now runs to completion in 7m25s with PYTHON_KEYRING_BACKEND unset on the WSL2 box where it used to wedge forever. Closes #1181 Claude-Session: https://claude.ai/code/session_011pR6Wqk15b3ESqXDiheNMZ
Precision bug-hunting review in progress
|
…s a burst Post-PR `codex review` finding: without serialization the sticky flag bounds nothing under concurrency. A burst of concurrent first callers — the realistic shape for `GET /api/v2/settings/keys` — each read `_KEYRING_TIMED_OUT` as False, each start their own unkillable worker, and each wait the full timeout. One leaked thread per in-flight request. A process-wide lock around the wrapper fixes it: the losers wait on the lock and then see the verdict the winner set. Cheap — a healthy keyring call is milliseconds, credential reads are rare, and the backend serializes on its own socket anyway. The new test fails with "leaked 8 blocked keyring threads" when the lock is removed, so it is pinning the property rather than describing it. Claude-Session: https://claude.ai/code/session_011pR6Wqk15b3ESqXDiheNMZ
|
Reviewed the diff ( OverallSolid fix for a real class of bug: detecting keyring unavailability by exception alone genuinely can't catch a backend that's selected but blocks forever on D-Bus. Time-boxing every call via a daemon worker + Test coverage is thorough: both failure shapes (backend selection itself hanging vs. a selected backend hanging per-call), the sticky-verdict short-circuit (including the "stale store built before the timeout" case, which is the subtlest part of this fix), the escape hatch, the endpoint, and a regression check that a healthy backend is still preferred. The Findings (non-blocking)
None of these block the fix — the core problem (unbounded hang) is solved, and the design correctly treats timeout as a first-class |
Third-party review (post-PR) —
|
Precision bug-hunting review in progress
|
|
Follow-up review after the concurrency fix ( Recap of prior reviewI reviewed this PR earlier in this thread. Status of those findings on the current diff:
New check on this pass
Nothing here blocks merge. Item 1 is the only one I'd actually want tracked as a fast follow-up (a small, mechanical |
claude-review, on the PR: `list_key_status` is `async def` and calls `_build_status` directly in the route body, so the bounded-but-blocking `worker.join(timeout)` ran on the event loop thread. A backend that selects fast but hangs on `get_password` would therefore stall *every* in-flight request for the timeout, not just the caller — the same failure this PR bounds, one layer up. Every credential call in an async handler now goes through `run_in_threadpool` (already the file's pattern for the blocking provider-verification calls): `_build_status` in both list and store, `set_credential`, `delete_credential`, and the `get_credential` in the verify route. `test_that_the_event_loop_stays_responsive_during_the_timeout` drives the real route through an ASGI client with a heartbeat task running: mutation-checked at "event loop was blocked (only 0 ticks)" with the wrap removed. Also from the same review: an unparseable CODEFRAME_KEYRING_TIMEOUT fell back to the default silently, unlike the loud config validation elsewhere in this module. It now logs what it ignored. Claude-Session: https://claude.ai/code/session_011pR6Wqk15b3ESqXDiheNMZ
Review triage — both remaining
|
Precision bug-hunting review in progress
|
Review of the current HEAD (
|
Review nit: the timeout path read the env var twice — once to wait on, once to log — so a mid-call change would have logged a bound that was never applied. One read, and the log now names the value actually waited on. Claude-Session: https://claude.ai/code/session_011pR6Wqk15b3ESqXDiheNMZ
Precision bug-hunting review in progress
|
|
Reviewed the final state (HEAD Specific things I checked independently:
Code quality: the module docstring, Test coverage is strong: both failure shapes (selection-time hang vs. per-call hang), the sticky-verdict short-circuit including the "stale store built before the timeout" case, the concurrent-burst leak (mutation-checked per the PR description), the escape hatch, and the event-loop-responsiveness test via a real ASGI client with a heartbeat task. The No blocking issues. This is a solid, well-tested fix for a real unbounded-hang bug, and the known limitations section in the PR body honestly documents the remaining trade-offs (per-process sticky verdict, lock changes contention shape for a healthy-but-slow backend) rather than hiding them. |

Closes #1181
The bug
credentials.pyimplemented "keyring unavailable" as raises, or is the fail backend:A backend that is installed and selected but unresponsive raises nothing — it blocks. On a headless box (WSL2, a container, an SSH session with no session D-Bus) the SecretService backend is importable and gets selected, and the D-Bus call waits with no timeout. Detection by exception therefore never fires, the encrypted-file fallback never engages, and the caller hangs forever.
That is not just a test annoyance:
GET /api/v2/settings/keysreaches this through_build_status, so an unresponsive keyring hangs a server request thread indefinitely, and the same blocking read sits under every CLI/agent credential lookup.The fix
Bound it by time, not only by exception. Every keyring call now runs through
_keyring_call, which joins a daemon worker with a timeout and raisesKeyringTimeoutError— deliberately aKeyringErrorsubclass, so the "keyring failed, use the encrypted file" handling already present at each call site does the right thing with no new branching.A blocked call cannot be cancelled (it is parked inside libdbus), so the worker is abandoned. That is bounded: the timeout verdict is sticky per process and short-circuits the wrapper itself, so a store built before the backend died stops waiting too — at most one abandoned thread per process, and the timeout is paid once rather than once per request.
delete()re-raised on any keyring failure, which would have turned a timeout into a failed delete. A timeout there now degrades to the encrypted-file cleanup, which is the part that matters.New knobs, both documented in
CLAUDE.md:CODEFRAME_DISABLE_KEYRING=1CODEFRAME_KEYRING_TIMEOUT2.0, read at call time rather than import (#963)PYTHON_KEYRING_BACKEND=keyring.backends.fail.Keyringremains the keyring-native equivalent and is now documented next to them.Acceptance criteria
CredentialManager; it degrades to the encrypted file store within a bounded timeGET /api/v2/settings/keysreturns rather than hanging when the keyring is unresponsiveEvidence
The gate the issue reported as wedging:
It completes, on the same WSL2 box where it previously sat at 92% for over an hour. All 31 failures reproduce identically on
main(verified in a clean worktree,31 failed in 7.06s) — they are ANSI/terminal-width assertions in CLI output tests, an ambient property of this machine, untouched by this diff.Affected suites on the final code:
263 passed.ruff checkclean.Tests
tests/core/test_keyring_timeout_1181.py— 12 tests across two deliberately different failure shapes: a backend that is selected instantly but blocks on every credential op, and one wherekeyring.get_keyring()itself never returns (backend selection probes each candidate's priority, and SecretService's probe talks to D-Bus, so selection is a blocking call in its own right). Covers construction, store/retrieve/delete round-trip through the file fallback,get_credential, the sticky verdict on both new and pre-existing store instances,deletedegrading instead of raising, the escape hatch, the endpoint, and that a healthy backend is still used in preference to the file.Two changes to existing tests, both deliberate:
test_keyring_availability_checkasserted "importable implies available" — precisely the assumption that caused this bug. It now asserts a verdict comes back, and quickly.conftest.pyresets the sticky verdict per test, the same treatment_MIGRATION_COMPLETEgets. Without it a single real timeout silently pushes every later keyring test onto the file fallback, which is an order-dependent failure.Review
codex review(cross-family, pre-PR) raised two P2s, both fixed in this branch before opening:_check_keyring, so a store built before the timeout — andstore()'s owndelete_passwordretry after a timed-outset_password— would start another stuck worker and wait again._keyring_callnow short-circuits.get_keyringreturned instantly, so the tests did not actually prove the selection-time hang was bounded. Split into two fixtures, each with its own test.Follow-up fixes from review (all in this branch)
_check_keyring, so a pre-existing store — andstore()'s own cleanupdelete_password— started another blocked workercodex review, pre-PR_keyring_callshort-circuitsget_keyringreturned instantly, so the selection-time hang was never actually proven boundedcodex review, pre-PRcodex review, post-PR_KEYRING_CALL_LOCK; mutation-checked at "leaked 8 blocked keyring threads"list_key_statusisasync defand called_build_statuson the event loop, so the bounded-but-blocking join stalled every in-flight requestclaude-reviewrun_in_threadpool; mutation-checked at "event loop was blocked (only 0 ticks)"CODEFRAME_KEYRING_TIMEOUTfell back silentlyclaude-reviewclaude-reviewKnown limitations
https://claude.ai/code/session_011pR6Wqk15b3ESqXDiheNMZ