[amrit110]

Part of #47.

Problem

install_compiled_blocks set torch._dynamo.config.recompile_limit (256 per GPU) and suppress_errors once, on the thread that installs the blocks. Dynamo's config overrides are per thread (each is a ContextVar). I verified this: a value set on one thread reads back as the default on another.

Serving runs fits and predicts on other threads: engine and worker threads, and asyncio.to_thread. Those threads saw dynamo's defaults:

• Recompile limit of 8. Prod's four A40 workers each logged [0/54] torch._dynamo hit config.recompile_limit (8), at the first graph after warmup's 54. After that, dynamo stops compiling the block for the life of the process, so every later TabICL fit ran eagerly, including shapes warmup had compiled.
• suppress_errors=False. A compile error would fail the tenant's fit instead of falling back to eager, as the docs promise.

I reproduced the exact prod warning on the A100 devbox with bench_stalls.py --call-thread, which makes every call on a separate thread, as serving does.

Fix

• New with_compile_settings: wraps the compiled forward in config.patch(...). One shared ConfigPatch serves every call; its saved state is itself a ContextVar stack, so it is safe across threads and nested calls.
• Cost: about 4 µs per block call.
• Scope: the settings no longer leak to other torch.compile users in the process.

Results

A100, TabICL, every call on a separate thread. Data in baselines/compile-call-thread-a100-{before,after}.jsonl.

fit	before	after
3,500 × 500	2.31 s	1.61 s
3,500 × 1,000	5.49 s	3.50 s
Graphs past warmup	0 (stopped at the limit)	6 (compiled once each)

Predicts are unchanged: 0.05–0.18 s across all shapes.

Still open in #47

Shapes warmup doesn't cover now compile once per worker, instead of running eagerly forever:

• Tables of about 30,000 rows hand the blocks differently strided q (recompile reason: stride mismatch, 128 vs 3,840,000). That costs 1–7 graphs, and the first such fit takes 5–25 s.
• 10,000 × 500 needs 1 more graph.

Covering these in warmup, or avoiding the stride guard, is the next step. This PR documents the cost in tabicl_compile.py.

Tests

• New tests/unit/test_tabicl_compile.py:
  • The settings hold on a new thread and under asyncio.to_thread.
  • They stay scoped to the call.
  • Nested calls, and calls from 8 concurrent threads, each restore their own state.
• Unit tests: 502 pass.
• bench_stalls.py gains --tasks and --call-thread.

Docs

• tabicl_compile.py docstring: the "Settings" section explains why the settings are applied per call, and the in-request compile costs use the measured numbers.
• ROADMAP compile item and CHANGELOG updated.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 15.38462% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.54%. Comparing base (9afb5ab) to head (c3c4c11).

Files with missing lines	Patch %	Lines
src/tabctx/backends/tabicl_compile.py	15.38%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
+ Coverage   79.45%   79.54%   +0.08%     
==========================================
  Files          42       42              
  Lines        3218     3226       +8     
==========================================
+ Hits         2557     2566       +9     
+ Misses        661      660       -1     

Flag	Coverage Δ
integration	30.00% <0.00%> (-0.08%)	⬇️
unit	76.03% <15.38%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/tabctx/backends/tabicl_compile.py	20.00% <15.38%> (+20.00%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amrit110]

Review fixes pushed. The independent review reproduced the fix end to end on CPU.

• accumulated_recompile_limit: dynamo also caps a code object at 256 graphs and checks that cap first, so the per-device limit above 256 did nothing. Both limits are raised now.
• Set at install as well as per call: on a torch whose config is process-wide, every call's shared patch then saves and restores the same values.
• Wording: past the limit, dynamo compiles no new graph; warmed shapes keep theirs. The CHANGELOG and docstrings are corrected. The 3,500-row gains in the sweep came from graphs that its earlier 30,000-row fits had built.
• Tests:
  • New end-to-end CPU regression (eager backend): 10 graphs on one thread, then a new shape on another thread must compile. It fails with install-time-only settings.
  • Restore after a raise, and both limits.
  • Unit tests: 508 pass.
• GPU check (A100, calls on a separate thread): no limit hits, and new graphs compile (59 total against 54 after warmup).

Next, a separate PR in #47: a first 3,500 × 500 fit still compiles 3 graphs (12.7 s), and 30,000-row tables compile 2. Warmup should cover those families.