feat(coverage): experimental istanbul coverage collection via tracing - #42684
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
tracing.start({ coverage: true }) enables collection of istanbul
coverage from instrumented application code. A server-side
CoverageRecorder installs a binding and init script into the browser
context, accumulates counters relayed right before navigations, sweeps
all frames when a trace chunk is stopped and stores the result as
coverage.json inside the trace file. tracing.flushCoverage() syncs
pages that are about to be closed by in-page script; pages closed via
the API are swept automatically. The test runner enables collection
with trace: { coverage: true } in the config.
…rter The 'coverage' reporter collects the coverage.json entries from the per-test trace attachments, merges them and writes coverage-final.json, lcov.info and a console summary, plus the istanbul html report when the project has istanbul-lib-* installed. Aggregation covers the tests whose traces are kept, so a full aggregate requires trace mode 'on'.
The statement, function and branch maps dominate the size of an istanbul report, so only send them with the first report of each file and report the hit counters alone afterwards. Files that were not hit since the previous report are skipped, files that were never hit are still reported once to keep them in the denominator. The merge adopts the maps from the first report that carries them and zero fills the counters that the reports omit.
Counters parked on pagehide used to go to sessionStorage, which dies together with a page that closes itself. Stash them in localStorage instead, where any same origin document picks them up, and relay the stashes on every collect, not only when a new document boots. Stashes are scoped to the recorder session so that the ones left by a previous run in a persistent profile are discarded, and carry an id so that the recorder drops the copies when several documents pick up the same stash.
A page relays the stashes of its own origin, so the stash of a page that closed itself is only picked up while another page of that origin is still around. Track the origins of the closed pages and visit the ones that have no page left in a page of Playwright's own, reusing the origin traversal of browserContext.storageState.
Collect the counters of the acted upon page at the tracing instrumentation point that captures the after-action snapshot, so that an action is the most that can be lost when a page goes away without a chance to stash its counters, and so that the counters can later be attributed to the actions that produced them. With the pages swept this often, the stash in the page storage is only picked up by Playwright itself, so the coverage script no longer needs a binding to push it. Origins are noted as they are swept, not only when a page closes, to also harvest the ones left behind by a navigation.
Collecting after each action used to hang off the snapshot capture, which is gated on the snapshot metainfo of the method, and the page close case fell outside that gate and reached into tracing from page.ts. Both now go through instrumentation: onAfterCall collects for the page the action ran in, and a new onPageWillClose collects before the page is gone, which also excludes the pages Playwright opens for itself. Name the trace entry trace.coverage, so that merging the chunks of several contexts into the trace of a test keeps every one of them instead of silently dropping all but the first. Do not collect for a chunk that is being discarded, and write no entry when there is nothing to report. Sweep the pages and their frames concurrently, and keep the coverage stash out of browserContext.storageState.
Keeps onAfterCall synchronous, drops the api call wrapper around flushCoverage and the visitOrigins comment.
9b8cc51 to
58300b8
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
The recorder only noted origins it happened to see during a pull or at page close, missing origins passed through in between.
Frames and multi-context cases join the plumbing and reporter tests, the option-off case moves to the library suite.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| } | ||
|
|
||
| async flush(progress: Progress) { | ||
| await progress.race(Promise.all(this._context.pages().map(page => this.collectFromPage(page)))); |
There was a problem hiding this comment.
Looks like safeNonStallingEvaluateInAllFrames
There was a problem hiding this comment.
we want to ignore exceptions on individual frames (if any) and proceed with serializing the rest
The recorder lives from start to stop, and uninstall unhooks the pagehide listener in the live documents.
Only the hits after the start count, the maps still go out with the first report.
Uninstall already unhooks the previous instance in every live document.
A failed stash is recorded in the page storage and thrown by the next take, the error is reported when the chunk stops. The two tests that were failing on Firefox and WebKit no longer click an empty body or navigate inside the action whose pull races the navigation.
Pairs with onPageWillClose.
This comment has been minimized.
This comment has been minimized.
Test results for "tests 1"1 failed 7 flaky51858 passed, 1247 skipped Merge workflow run. |
Test results for "MCP"2 failed 8575 passed, 1446 skipped Merge workflow run. |
🟢 All three failures are pre-existing flakes — this PR is clearHi, I'm the Playwright bot and I triaged the latest CI runs here. Nothing red points at the coverage work. Each of the three failures has the same test failing on a SHA this branch can't be responsible for — a DetailsPre-existing flake / infra
Not reached by the diff This PR is confined to tracing/coverage collection, the injected coverage script, and a new Triaged by the Playwright bot - agent run |
🟢 All three failures are pre-existing flakes — this PR looks clearHi, I'm the Playwright bot and I triaged the failing CI here. Each of the three failures has the same test failing on SHAs this PR can't be responsible for — DetailsThe PR adds istanbul coverage collection behind Pre-existing flake / infra
Triaged by the Playwright bot - agent run |
05c126b
into
microsoft:main
2 participants
Summary
tracing.start({ coverage: true })collects istanbul counters (window.__coverage__) from instrumented application code and stores them ascoverage.jsoninside the trace file. Works in all browsers,langs: js, experimental.use: { trace: { mode: 'on', coverage: true } }, plus acoveragereporter that merges the per test coverage out of the traces intocoverage-final.json,lcov.infoand an istanbul html report.localStorageonpagehideand picked up from any same origin document, or from a page Playwright opens for an origin that has none left.tracing.flushCoverage()collects on demand.vite-plugin-istanbul,babel-plugin-istanbulor the SWC plugin, the same contract component testing uses formount.