mvp benchmarking - WIP - #927
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| // Sample app source for audit (references env keys). | ||
| const token = process.env.SECRET_TOKEN; | ||
| const pub = process.env.PUBLIC_VAR; | ||
| console.log(token, pub); |
…HOST Lets tooling exercise the telemetry code path against a local mock instead of the production collector. Not an opt-out knob - VARLOCK_TELEMETRY_DISABLED still disables telemetry entirely.
…y mocking CI: - git add used a repo-relative path from the benchmarks/ working dir, so the commit step failed on every run; use the absolute path - commit before rebasing (a rebase refuses to run with changes staged) and retry the push - install with --frozen-lockfile so a rewritten bun.lock cannot block the rebase - pass workflow inputs through env: instead of interpolating into shell - typecheck benchmarks/ in CI; turbo does not reach it Telemetry: - telemetry-on scenarios now run against a local mock collector instead of production PostHog, and refuse to run at all if the version under test does not honour VARLOCK_POSTHOG_HOST Measurement: - drain server stdout; the redactLogs benches filled the 64KB pipe and turned a latency measurement into a measurement of pipe backpressure - kill servers by process group; npx execs the real server as a grandchild - sample RSS across the whole process tree, and sample immediately so short-lived commands are not missed entirely - report min/stddev and flag deltas that fall within noise - run cli scenarios against a copy of the fixture, not the fixture itself - force the on-disk cache so warm-vs-cold load is meaningful in CI - run the bun install under bun; it was running the same node code twice - scale the redaction/leak workloads above the noise floor (they previously could not resolve the thing they exist to measure) - align baseline and varlock arms so the delta is not also a source diff - reuse one fixture per framework: 14 npm installs down to 4 - unique scenario ids, validated CLI args, recorded skips instead of silent ones
52f9613 to
ffa73f0
Compare
|
The changes in this PR will be included in the next version bump.
|
📦 Bundle size
dist/ only; native binaries are versioned separately and not counted here. |
|
@pullfrog review this |
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
The telemetry compatibility probe can emit a real production event for exactly the older releases it is intended to reject.
Reviewed changes Reviewed the release benchmark workflow, measurement and reporting utilities, CLI and framework scenarios, telemetry mocking, framework fixture extraction, and telemetry endpoint override.
- Release automation Adds manual and post-publish benchmark execution, SEA download, and committed JSON results.
- Benchmark suite Measures CLI, Next.js, Vite, Python, and Go scenarios across timing, RSS, telemetry, redaction, and leak-prevention dimensions.
- Framework harness Extracts the imperative fixture lifecycle and adds installation from published packages.
- Telemetry isolation Adds a configurable collector endpoint and a local PostHog-compatible mock.
azure/gpt-5.6-sol ｜ 𝕏
commit: |
|
Addressed and resolved the review thread on PR #927. The telemetry capability check now inspects the installed compiled package without running telemetry-enabled code, and commit Task list (3/3 completed)
|
There was a problem hiding this comment.
Caution
Manual dispatch can promote an arbitrary selected branch into main, and the Next/Vite build deltas currently compare different generated applications rather than isolating varlock overhead.
Reviewed changes across the complete initial PR, including the release workflow, benchmark engine and scenarios, framework fixture extraction, and telemetry endpoint override.
-
Release automation: Adds release and manual dispatch, published-package installation, optional SEA download, benchmark execution, and result commits.
-
Measurement engine: Adds wall-time and process-tree RSS sampling, repeated measurements, delta reporting, server lifecycle helpers, and JSON result metadata.
-
Benchmark coverage: Adds CLI, Next, Vite, Python, and Go fixtures and scenarios, including telemetry, redaction, leak prevention, and cache comparisons.
-
Framework harness: Extracts the imperative fixture environment from Vitest and adds registry-backed dependency installation for published-package benchmarks.
-
Telemetry safety: Makes the collector host overridable and points supported releases at a local mock before enabling telemetry-on scenarios.
azure/gpt-5.6-sol ｜ 𝕏
|
Addressed all three review threads in commit Task list (5/5 completed)
|
2 participants
Adds a release benchmarking suite under
benchmarks/measuring memory footprint, execution time, and the latency added by redaction and leak prevention. It runs against published npm packages (plus the linux SEA binary when available), triggered after a varlock publish or manually.What it covers
CLI (
loadcold/warm,runwrap overhead, stdout redaction,scan/audit), the Next and Vite integrations (build cost against a no-varlock baseline, plus request latency forpreventLeaksandredactLogs), and Python/Go codegen. CLI scenarios are measured across three runtimes: npm installed and run under node, bun installed and run under bun, and the compiled standalone binary.Results are written to
results/as JSON and committed by CI so trends show up in git history. Each run prints a deltas table (varlock vs baseline, telemetry on vs off, redaction on vs off), and any delta smaller than the spread of its inputs is tagged as noise rather than reported as a change. Absolute wall times on a shared runner are not comparable between runs; the within-run deltas are.The integration benches reuse the Next/Vite templates from
framework-testsvia a newusePublishedmode onFrameworkTestEnv, which installs from npm instead of packing workspace tarballs. The imperative parts of that harness moved toharness/fixture-env.tsso it can be used without loading Vitest.Telemetry
Telemetry-on scenarios measure what the telemetry code path costs, but never send real telemetry.
CONFIG.POSTHOG_HOSTis now overridable throughVARLOCK_POSTHOG_HOST, and the suite points it at a local mock collector. Before running those scenarios it probes whether the version under test honours the override, and skips them with a recorded note if not, so synthetic events cannot reach product analytics.This is not an opt-out knob:
VARLOCK_TELEMETRY_DISABLEDstill disables telemetry entirely.Those scenarios stay skipped until the first release that includes the config change.
Known gaps
cli-loadfixture is all static literals, so cold vs warm has little to distinguish. Exercising the cache properly needs a fixture with expensive resolversresults/history yetintegration-nexthas not been run end to end yet. The other groups have.