feat(bench): Add comprehensive benchmarking, flamegraph profiling, and negentropy batched-build optimization - #263
Draft
Thunder-Blaze wants to merge 9 commits into
Draft
Conversation
…ptimize build without blocking writers
…filterFuzzTest - Replace JSON::XS with JSON::PP in dumbFilter.pl and filterFuzzTest.pl. JSON::PP is a Perl core module requiring no CPAN installation or C compiler, improving portability across build environments. - Fix 'push @topics' to 'push @$topics' in filterFuzzTest.pl. The variable $topics is declared as an array reference (line 10), so the original code was pushing to an implicit (undeclared) @topics array instead of the intended reference.
… documentation Benchmark improvements: - bench/run_stats.py: Gracefully skip Docker out-of-core test when daemon is unavailable instead of skipping the entire storage suite. Extract connection storm throughput (conn/sec) and latency (P50/P99) metrics from strfry-bench output. Enhanced report with per-connection- count breakdowns and churn throughput. - bench/ab_test.py: Fix --skip-heavy (was defaulting to True, making it always active). Wire up --full flag as the recommended mode before opening a PR — overrides --skip-heavy to run all suites including 1M event storage. Skip 'Status' rows in comparison table. Expand lower_better heuristic with 'memory' and 'sockets' keywords. Documentation: - Add docs/benchmarking.md with architecture overview, all 12 benchmark suites, CLI usage, pre-PR --full workflow, flamegraph profiling with perf + inferno, and identified performance hotspots. - Update README.md with Benchmarking & Performance section linking to the new documentation. Gitignore: - Add safety-net entries for transient benchmark directories (strfry-db-det, strfry-db-negentropy*) and ab_test temp files that could be left behind on process crash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a comprehensive benchmarking and performance profiling framework for strfry, along with an optimization to
strfry negentropy buildthat eliminates write-lock contention, and portability fixes for test scripts.Benchmarking Framework (
bench/)bench/run_stats.py: Orchestrator for 13 benchmark suites covering storage (in-core vs out-of-core), ingestion pipeline, concurrency, WebSocket connection storms, query engine & NIP-45, subscription fan-out, negentropy sync, write-policy plugin overhead, CLI import/export, OS-level resource metrics (WAF, CPU, disk I/O), adversarial stress tests, and backpressure performance. Outputs a structured Markdown report.bench/ab_test.py: Relative A/B comparison tool. Benchmarks the current branch against a base branch (default:master) under identical conditions, collecting hardware-agnostic deterministic metrics (retired CPU instructions, heap allocations, bytes allocated viaperf stat+alloc_tracker.c) alongside wall-clock metrics, and generates a delta report with regression indicators. Use--fullbefore opening a PR to run all suites including the 1M-event storage test.bench/alloc_tracker.c:LD_PRELOADshared library that interceptsmalloc/calloc/reallocto count heap allocations and total bytes allocated, providing deterministic, hardware-independent memory profiling.bench/bench_plugin.py: Minimal write-policy plugin for benchmarking plugin IPC overhead.Negentropy Build Optimization
src/apps/dbutils/cmd_negentropy.cpp: Refactorednegentropy buildto scan matching events in read-only batches of 10,000 (txn_ro), followed by short write-transaction windows (txn_rw) for BTree insertion. Previously, a single write transaction was held across the entire scan+insert phase, blocking all concurrent relay writes.Dockerfile Fixes
/app/strfry-dbdirectory at build time instead of attempting toCOPYit from the build stage (which may not produce the directory)..dockerignoreto exclude build artifacts and reduce context size.Test Script Portability Fixes
test/tests/dumbFilter.pl,test/tests/filterFuzzTest.pl: ReplacedJSON::XS(requires CPAN + C compiler) withJSON::PP(Perl core module, zero dependencies).Documentation
docs/benchmarking.mdwith detailed instructions on running benchmarks, A/B comparisons, pre-PR full comparison with--full, flamegraph profiling withperf+inferno, and documented identified performance hotspots.README.mdwith a Benchmarking & Performance section linking to the new documentation.