Skip to content

fix(errors): stop masking query-level errors as 'database_not_connected' - #448

Merged
cdeust merged 2 commits into
mainfrom
claude/fix-error-classifier-masking
Aug 24, 2026
Merged

fix(errors): stop masking query-level errors as 'database_not_connected'#448
cdeust merged 2 commits into
mainfrom
claude/fix-error-classifier-masking

Conversation

@cdeust

@cdeust cdeust commented Aug 24, 2026

Copy link
Copy Markdown
Owner

What

_classify_error routed exceptions to the PostgreSQL setup guide (database_not_connected) when their text matched any of "operationalerror", "role", "does not exist", or a bare "timeout". Those substrings match ordinary query-level failures, not connection failures.

The worst offender: sqlite3.OperationalError's class name is literally OperationalError, so type(exc).__name__.lower() == "operationalerror". Every SQLite query error — a FTS5 syntax error, no such table, database is locked — matched and was handed a brew install postgresql@17 guide. On the SQLite backend that guide is doubly wrong, and it buried the real error.

This is how it was found: while running the harness-comparison benchmark on a real SQLite store, a FTS5 syntax error in get_causal_chain (fixed in #447) came back to the client as "Cortex could not connect to PostgreSQL" — sending diagnosis in exactly the wrong direction.

The other keywords were similarly broad: "role" matched control/payroll/role-based; "does not exist" matched column "x" does not exist (a query/schema bug); bare "timeout" matched statement/lock timeouts.

Fix

Tighten the database_not_connected bucket to unambiguous connection/auth phrases only:

connection refused · could not connect · could not translate host name ·
no such host · connection reset · server closed the connection ·
the database system is starting up · password authentication failed ·
connection timed out · timeout expired

Genuine create-db/create-role first-run conditions stay fully actionable from their own honest OperationalError: ... FATAL: database "cortex" does not exist text, which the fall-through (type(exc).__name__, str(exc)) returns verbatim — precision over hand-holding, so a real bug is never masked.

Tests

Added TestQueryErrorsAreNotMaskedAsDbNotConnected: FTS5 syntax error, no such table, column "x" does not exist, a role substring in unrelated text, and a locked-db error all fall through to <Type>: <message>; a companion test asserts the seven genuine connection failures still get the setup guide. The pre-existing test_generic_connection_refused_still_classified_as_db_not_connected continues to pass unchanged.

Gates: ruff check ✓ · ruff format --check ✓ · craftsmanship gate ✓ · pytest tests_py/server/test_tool_error_handler_classification.py → 13 passed.

Pairs with #447 (the FTS5 crash this bug was hiding).

🤖 Generated with Claude Code


Generated by Claude Code

_classify_error matched the substrings "operationalerror", "role",
"does not exist", and a bare "timeout" when routing exceptions to the
PostgreSQL setup guide. Those match ordinary query failures, not connection
failures:

- sqlite3.OperationalError's class name is literally "OperationalError", so
  type(exc).__name__.lower() == "operationalerror" — EVERY SQLite query error
  (FTS5 syntax error, "no such table", "database is locked") was classified
  as database_not_connected and handed a `brew install postgresql@17` guide.
  On the SQLite backend that guide is doubly wrong, and it buried the real
  error (a FTS5 syntax error in get_causal_chain surfaced exactly this way
  during benchmarking).
- bare "role" matched "control", "payroll", "role-based"; bare
  "does not exist" matched `column "x" does not exist` (a query/schema bug);
  bare "timeout" matched statement/lock timeouts.

Tightened the bucket to unambiguous connection/auth phrases only
(connection refused, could not connect, could not translate host name,
server closed the connection, the database system is starting up, password
authentication failed, connection timed out, timeout expired). Genuine
create-db/create-role setup conditions remain fully actionable from their own
honest `OperationalError: ... FATAL: database "cortex" does not exist` text,
which the fall-through returns verbatim.

Regression tests pin that FTS5 syntax errors, no-such-table, column-does-not-
exist, a "role" substring in unrelated text, and a locked-db error all fall
through to their honest '<Type>: <message>', while the seven genuine
connection failures still get the setup guide.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fnomaXS71hgxMw2zr7HGR
@cdeust

cdeust commented Aug 24, 2026

Copy link
Copy Markdown
Owner Author

ZETETIC-REVIEW: REQUEST_CHANGES

Independent review (fresh-context reviewer, read-only via gh, head 7a0e1db).

Sound and kept: the masking diagnosis is correct (SQLite OperationalError class-name substring match + bare role/does not exist/timeout routing everything to the PostgreSQL guide), the fix resolves the exact reported FTS5/SQLite case, the truth-table test covers it, no keyword collisions across the four buckets, no bare except introduced, CI green on all required checks.

Blocking (one item):

  • mcp_server/tool_error_handler.py:84-142_classify_error grew from 42 to 59 lines: §4.2 hard cap (50) at High stakes (file imported by 7 production modules registering ~50 tools), no ADR. CI's craftsmanship gate missed it; a green automated gate is not a verdict. Fix (mechanical, Fowler Extract Variable/Constant): hoist the missing_extension and database_not_connected keyword lists into module-level named constants and attach the 16-line rationale comment to the constants — drops the function well under the cap without losing content.

Non-blocking:

  • The "truth-table" claim is not exhaustive: no such host, connection reset, timeout expired (3 of the 10 database_not_connected phrases) have no test — add them while touching the file.
  • Pre-existing, untouched: bare "extension" substring in the missing_extension bucket is broad — awareness only.

_classify_error grew to a 58-line AST span (cap: 40 local / 50 hard,
High stakes — tool_error_handler.py is imported by 7 production modules
registering ~50 tools), flagged blocking in the PR #448 review. The
violation was pre-baselined debt the diff-scoped craftsmanship gate
could not see growing.

Hoist the missing_extension and database_not_connected keyword lists
into module-level named constants (_MISSING_EXTENSION_PHRASES,
_CONNECTION_FAILURE_PHRASES), moving the connection-failure rationale
comment onto its constant. Pure Extract Variable — classification logic
and phrase lists are unchanged; span now 21 lines. Prune the now-stale
baseline entry for _classify_error (safe_handler's separate, untouched
entry stays).

Also close the non-blocking truth-table gap: add the three untested
database_not_connected phrases (no such host, connection reset,
timeout expired) to the genuine-connection-failures test loop.

Co-Authored-By: Claude <noreply@anthropic.com>
@cdeust

cdeust commented Aug 24, 2026

Copy link
Copy Markdown
Owner Author

ZETETIC-REVIEW: APPROVE

Delta re-review of commit e74ee3a (same fresh-context reviewer as the prior REQUEST_CHANGES):

  • §4.2 finding CLOSED: _classify_error re-derived independently at 22 lines (was 58). Clean Extract Variable — _MISSING_EXTENSION_PHRASES (3) and _CONNECTION_FAILURE_PHRASES (10) constants diffed byte-for-byte against the former inline lists: identical content, order, and comments. Behavior-preserving confirmed, not just claimed.
  • Truth-table gap CLOSED: the three missing phrases (no such host, connection reset, timeout expired) added — 10/10 coverage of the constant.
  • Baseline-prune AUDITED: the .craftsmanship-baseline.json diff removes exactly the one now-stale _classify_error method-size entry; the separate safe_handler entry and every other entry untouched; no threshold or config field changed. No rule configured around.
  • CI: all checks green on e74ee3a, including Craftsmanship Gate and CI Green.

Green AND verified — mergeable.

@cdeust
cdeust merged commit 17904c4 into main Aug 24, 2026
25 checks passed
@cdeust
cdeust deleted the claude/fix-error-classifier-masking branch August 24, 2026 18:22
cdeust added a commit that referenced this pull request Aug 24, 2026
… drift)

origin/main advanced past this branch's fork point (#447 FTS5 fix, #448
error-classification fix) and #448 fixed mcp_server/tool_error_handler.py
_classify_error's method-size violation (was in the branch-point
baseline; now 21 lines on main, verified by AST), pruning it from
main's .craftsmanship-baseline.json. This branch's copy of the file
still carried the stale entry, which the gate's ratchet rule flags as
"added without a base-ref match" once compared against a freshly
fetched origin/main (the ratchet may only shrink within a PR).

Not something this PR's own changes caused — reproduced by stashing
every change in this branch back to dddc001 and re-running the gate
against fresh origin/main; the same failure appears. Synced this
branch's baseline to match origin/main's exactly for this one entry
(diffed byte-for-byte confirmed no other divergence in either
direction) rather than grandfathering it or regenerating blind.

Co-Authored-By: Claude <noreply@anthropic.com>
cdeust added a commit that referenced this pull request Aug 24, 2026
…s instead of hanging (#449)

* fix(ap): bound interactive AP read calls so a wedged pipeline degrades

The AP MCP client runs with callTimeoutMs=0 — deliberately unbounded, because
a fresh index_codebase of a large tree legitimately exceeds any fixed wall
clock. The only backstop is the 600s wedge-silence window. That is correct for
ingestion but wrong for the interactive read path: search_codebase (behind
unified_search) and the get_symbol / get_context / get_impact / get_processes /
health_check lookups inherited the same 600s, so an AP that connects but then
wedges made unified_search / get_causal_chain hang for up to 10 minutes instead
of degrading to Cortex-only results — the "unreachable => status=partial"
promise in unified_search's docstring was not actually kept.

Fix: APBridge.call gains an optional timeout_s that wraps the client call in
asyncio.wait_for and degrades a timeout to None (recorded reason + stderr note),
exactly like every other AP failure. The interactive read wrappers pass
interactive_call_timeout_s() (30s default, env-overridable via
CORTEX_AP_INTERACTIVE_TIMEOUT_S, 20x below the 600s indexing window). The
indexing/write wrappers (index_codebase, analyze_codebase, resolve_graph,
cluster_graph, detect_changes) and the build-loop query_graph stay unbounded.

A None from a timed-out search_codebase flows through as_list(None) -> [] ->
unified_search returns status=partial with Cortex-only hits.

Tests: a hanging fake client returns None within a 50ms ceiling; the
interactive wrapper forwards a positive timeout while index_codebase forwards
none; the default ceiling is positive and strictly below the 600s wedge window
and honors/validates the env override.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017fnomaXS71hgxMw2zr7HGR

* fix(ap): extract APBridge._degrade to clear the 40-line method-size gate

APBridge.call spanned 42 lines (repo cap: 40, scripts/craftsmanship_rules.py
METHOD_LINE_LIMIT). Both except-branches duplicated the same "record
_unavailable_reason + stderr note + return None" degrade logic; pulled
into one _degrade(reason, note) helper, called from both branches. Fixes
the CI craftsmanship-gate FAILURE on 3debfda (PR #449 review finding 1).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ap): surface AP call failure in unified_search status, not just the config flag

unified_search.handler derived status solely from the static is_enabled()
config check. An AP that is enabled but times out or errors on the actual
search_codebase call returned [] indistinguishably from AP genuinely
finding nothing: status="ok", sources=["cortex","ap"], counts.ap=0 either
way. This silently hid the exact failure mode #3debfda's timeout fix was
meant to expose.

WorkflowGraphASTSource gains last_search_degraded_reason, reading the
outcome APBridge._degrade already records on its own call path (no
widened return type, no second round-trip) — None when the call
succeeded or was never attempted (AP disabled / no graph configured),
set when it genuinely failed. unified_search now sets status="partial"
and a degraded: {source, reason} field whenever AP was attempted but
failed, keeping status="ok", degraded=null for a genuine empty AP result.

PR #449 review finding 2.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(ap): remove false get_causal_chain AP-hang claim from source comments

get_causal_chain (mcp_server/handlers/get_causal_chain.py) imports only
MemoryStore and core.knowledge_graph — no AP dependency, so it cannot
hang on a wedged AP call. mcp_call_timeout.py's interactive-ceiling
comments and the interactive-timeout test's module docstring both
claimed otherwise; corrected to name unified_search only, with an
explicit note that get_causal_chain is unaffected. PR #449 review
finding 3 (PR body corrected separately via gh pr edit).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ap): extract search_codebase hit normalization to clear the 300-line/40-line gates

The degraded-tracking addition (bd74351) pushed
workflow_graph_source_ast.py to 316 lines (cap 300) and
search_codebase's body to 45 lines (cap 40) — a NEW craftsmanship
violation not in the base-ref baseline, caught by CI on 30b2d20.

Local `python3 scripts/check_craftsmanship.py` (no args) at the time
compared against a stale local origin/main; re-running after `git fetch
origin main` reproduces CI's FAILURE exactly.

Root fix: search_codebase's row-normalization loop (id/qualified_name/
file_path/score/snippet construction from the raw AP response) is
lifted into workflow_graph_ast_response.normalize_search_hits — the
same module that already owns "normalize an AP response shape" for the
symbol- and edge-loading concerns (issue #275's split), so this is the
same seam, not a new one. search_codebase now just resolves the graph
path, calls the bridge, and delegates. Docstrings on
search_codebase/last_search_degraded_reason trimmed to essentials.

workflow_graph_source_ast.py: 316 -> 288 lines. search_codebase: 45 ->
delegates (no longer a > 40-line method). Behavior-preserving; targeted
tests (ap_bridge/mcp_client/workflow_graph + the two new degraded-status
test files + response_budget_wiring) all still green.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ap): reset APBridge._unavailable_reason at the top of every call()

_unavailable_reason was only cleared in connect()'s slow (reconnect)
path. connect()'s fast path (client already connected) skipped the
clear, and call()'s success branch never reset it either. On a reused
instance — the codebase already does this, wiki_verify.py reuses one
WorkflowGraphASTSource across its candidate loop — a stale reason from
an earlier failed call leaked into a later successful one, so
last_search_degraded_reason could report a call as failed when it had
actually just succeeded. Contradicted its own docstring ("None if it
succeeded").

Fix: call() resets self._unavailable_reason = None right after the
tool-allowlist check, before touching connect()/the RPC — each call's
own outcome is now authoritative regardless of prior calls on the same
bridge instance. call() stays at 39 lines (cap 40).

New regression test: two calls on one APBridge instance, first via a
hanging client (fails), second via a client that succeeds, asserting
unavailable_reason is set after the first and None after the second.
Fails on pre-fix code (verified via git stash), passes after.

PR #449 review round 3 (Medium finding, house "no deferred coverage on
new code" rule).

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(craftsmanship): sync baseline ratchet to origin/main (unrelated drift)

origin/main advanced past this branch's fork point (#447 FTS5 fix, #448
error-classification fix) and #448 fixed mcp_server/tool_error_handler.py
_classify_error's method-size violation (was in the branch-point
baseline; now 21 lines on main, verified by AST), pruning it from
main's .craftsmanship-baseline.json. This branch's copy of the file
still carried the stale entry, which the gate's ratchet rule flags as
"added without a base-ref match" once compared against a freshly
fetched origin/main (the ratchet may only shrink within a PR).

Not something this PR's own changes caused — reproduced by stashing
every change in this branch back to dddc001 and re-running the gate
against fresh origin/main; the same failure appears. Synced this
branch's baseline to match origin/main's exactly for this one entry
(diffed byte-for-byte confirmed no other divergence in either
direction) rather than grandfathering it or regenerating blind.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants