test(aws_lambda): reduce suite from 39 to 14 tests without losing coverage - #6980
test(aws_lambda): reduce suite from 39 to 14 tests without losing coverage#6980ericapisani wants to merge 3 commits into
Conversation
Reduce the docker/SAM-heavy AWS Lambda integration suite from 39 to 14 test cases (-64%) with no loss of assertions or behavior coverage, and eliminate its flakiness. Reliability fixes: - SAM template sets Architectures matching the host CPU. x86_64 containers under QEMU emulation on arm64 Macs caused 10s function timeouts, lost envelopes, and 5-9 flaky failures per run. - lambda_client fixture waits for envelope delivery to settle after each invoke instead of reading the test server race-prone. Reductions (all assertions preserved; verified by per-merge branch analysis plus a final AST-level assertion audit against the base): - Delete 3 fully redundant tests (span_origin, timeout_error, trace_continuation) whose assertions exist verbatim elsewhere - Trim equivalence-class parametrize rows (non_dict_event 7->3, headers 5->1) - Merge config-arm tests into one-test-per-feature with sequential invokes: request_data (4 arms), url_query (3 arms), user_info (2 arms), error trace context (perf on/off x new/existing), span streaming (ok/error/trace-continuation) - Factor shared helpers (_request_data_payload, _assert_segment_span_attrs) - Remove the now-unused TimeoutError lambda function Runtime: ~210s -> ~95-115s per run; 12+ consecutive green runs. Coverage of sentry_sdk/ unchanged (guard checked at every step).
Codecov Results 📊✅ 94882 passed | ⏭️ 6343 skipped | Total: 101225 | Pass Rate: 93.73% | Execution Time: 345m 54s 📊 Comparison with Base Branch
➖ Removed Tests (1)View removed tests
All tests are passing successfully. ✅ Patch coverage is 100.00%. Project has 2511 uncovered lines. Coverage diff@@ Coverage Diff @@
## main #PR +/-##
==========================================
+ Coverage 89.88% 89.89% +0.01%
==========================================
Files 193 193 —
Lines 24832 24832 —
Branches 8912 8912 —
==========================================
+ Hits 22319 22321 +2
- Misses 2513 2511 -2
- Partials 1416 1415 -1Generated by Codecov Action |
There was a problem hiding this comment.
TBH I don't know how to approach reviewing this. Actually going through all existing test cases, understanding what they're testing, and then comparing with the end result would take a long time.
If you already did that work, I'm willing to stamp this because I trust your judgment.
But in general, I'm wary of these test compacting changes. I don't know if a slight speedup is worth it compared to making the tests less structured and harder to understand. Maybe it is! But the problem is, while the speedup is easily measurable, the quality of the test cases themselves isn't (coverage is just part of the story). So I can just go by what I know about our human vs AI written/assisted test cases, which is that I'd pick human-written test cases over AI anytime.
Follow-up to #6972, applying the same reduce-tests-without-losing-coverage experiment to the AWS Lambda integration suite — the most expensive suite in the repo to run locally, since every test round-trips through dockerized AWS SAM containers.
The suite goes from 39 to 14 test cases (-64%) with no loss of assertions or behavior coverage. Reductions come from three sources:
Result: ~210s → ~115s per run on local machine, and 18 consecutive green runs (previously 5–9 flaky failures per run). Verified on py3.13 and py3.11 (full docker runs), py3.8-compatible syntax, ruff check + format clean.
Additional notes
test_timeout_errorwas dropped because its assertions are a subset oftest_timeout_error_scope_modified, which is staying. The timeout-warning behaviour is still fully covered through the latter test.However, the two used different init paths - the removed one ran through the Lambda-layer auto-init path (
init_serverless_sdk), while the one we kept uses an explicitsentry_sdk.initcall.As a result, we lose timeout-specific coverage on the layer path. However, that path is still exercised by the other tests (BasicException, InitError, BasicOk), and the warning logic is identical.
I think it's worth the trade-off here, but if there are strong opinions about adding
test_timeout_errorback in to get that timeout-specific coverage, let me know.