Add dashboard for Autobahn e2e - #4172
Conversation
Bind Grafana to the SSH CIDR by default, ship the merged peer list with genesis, and record main-loop phases on the existing sei_chain meter. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## giga-1 #4172 +/- ##
=========================================
Coverage ? 65.59%
=========================================
Files ? 2083
Lines ? 158064
Branches ? 0
=========================================
Hits ? 103689
Misses ? 54234
Partials ? 141
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
Solid, coherent port of the Autobahn E2E dashboard, distributed/colocated AWS topologies, and main-loop phase metrics; the metric name the dashboard queries lines up with the provider the node actually installs, and the phased deploy.sh preserves the existing local flow. No blockers — findings are a dead-but-hazardous SetupPrometheus, a README statement that contradicts the Grafana CIDR default, and a substring peer filter that this PR newly exposes to arbitrary AWS private IPs.
Findings: 0 blocking | 6 non-blocking | 3 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- [suggestion] No test asserts the metric name/labels the dashboard depends on.
giga/metricstests only check thatSetPhasedoesn't panic, so renamingtimerName(or thesei_chainnamespace inutils/metrics) silently blanks the two "Main loop — time spent" panels. A test that callsSetPhasetwice, gathersprometheus.DefaultGatherer, and asserts a family namedautobahn_main_loop_phase_duration_seconds_totalwith aphaselabel would pin the contract between the Go code anddocker/monitornode/dashboards/autobahn-e2e-dashboard.json. - [suggestion]
sei-load's own metrics are never scraped. Deploy's printed instructions and the README both run it with--metricsListenAddr 0.0.0.0:19698, butprometheusScrapeConfigemits only validator<private-ip>:26660targets, so nothing collects that endpoint. Either add a scrape job for the load host or drop the flag from the instructions. - [suggestion]
--grafana-cidr 0.0.0.0/0publishes Grafana with the hardcodedadmin/admincredentials fromdocker-compose.aws-load.yml(GF_SECURITY_ADMIN_PASSWORD=admin). The default (SSH CIDR) is right and the README does warn to tear the cluster down, but since this is the first port the tool opens beyond SSH, consider also settingGF_AUTH_ANONYMOUS_ENABLED=falseexplicitly and/or generating a random admin password into the printed URL line. - 3 suggestion(s)/nit(s) flagged inline on specific lines.
| ) | ||
|
|
||
| // SetupPrometheus installs a Prometheus MeterProvider on the default registerer. | ||
| func SetupPrometheus() error { |
There was a problem hiding this comment.
[suggestion] SetupPrometheus has no production caller — only autobahn_loop_test.go — and it conflicts with the provider the node actually uses. app/app.go:511 calls utils/metrics.SetupOtelMetricsProvider, which installs a Prometheus exporter with WithNamespace("sei_chain") and chain_id as a constant label; that namespace is exactly what makes the dashboard's sei_chain_autobahn_main_loop_phase_duration_seconds_total resolve. This function installs a provider with no namespace, and otel.SetMeterProvider is last-write-wins, so any future caller ordered after NewSeiApp would rename the series to autobahn_main_loop_phase_duration_seconds_total and strip chain_id from every OTel series in the process — the dashboard panels go blank with no error anywhere.
Suggest deleting SetupPrometheus (and the now-unneeded client_golang/prometheus, otlptranslator, otelprometheus, sdkmetric imports) so app.go remains the single choke point that owns the meter provider. MainLoop()'s lazy otel.Meter already delegates correctly through the global provider, so nothing else has to change.
| compile, and initialize in parallel. The load instance is brought up | ||
| afterward with Prometheus and Grafana; `sei-load` is left for you to | ||
| start. The security group admits SSH from the caller, Grafana (`:3000`) | ||
| from the internet, and all TCP between the five instances. |
There was a problem hiding this comment.
[suggestion] This contradicts the code and line 151 of this same file. resolveGrafanaCIDR defaults --grafana-cidr to the resolved --ssh-cidr, so Grafana is admitted from the caller's IP, not from the internet — the internet case only happens if you pass --grafana-cidr 0.0.0.0/0. Suggest "SSH and Grafana (:3000) from the caller (widen Grafana with --grafana-cidr), and all TCP between the five instances."
| # Override up persistent peers | ||
| NODE_IP=$(hostname -i | awk '{print $1}') | ||
| if [ -n "$ADVERTISE_IP" ]; then | ||
| NODE_IP="$ADVERTISE_IP" |
There was a problem hiding this comment.
[suggestion] Now that NODE_IP can be an arbitrary AWS private address, the grep -v "$NODE_IP" on line 46 is a substring match over the whole <nodeid>@<ip>:<port> line, so a validator can silently drop a legitimate peer: with NODE_IP=172.31.1.1, the entry for 172.31.1.10:26656 also matches and is filtered out. In the local 4-in-1 cluster the fixed 192.168.10.1{0,1,2,3} addresses made this unreachable; on AWS-assigned IPs it is a plausible collision. Anchoring the match fixes it, e.g. grep -v "@$NODE_IP:" (P2P_PORT is not defined in this script, so either use that form or define it to 26656 as step1_configure_init.sh does).
PR SummaryMedium Risk Overview Grafana & metrics. Local/AWS monitoring now provisions an Autobahn E2E dashboard (TPS, blocks/sec, finalize latency, main-loop phase split). New AWS Docker/Makefile. New Reviewed by Cursor Bugbot for commit 2e7c4aa. Bugbot is set up for automated code reviews on this repo. Configure here. |
| } | ||
|
|
||
| for n := next; ; n += 1 { | ||
| gigametrics.SetPhase(gigametrics.PhaseConsensus) |
There was a problem hiding this comment.
I'm fine adding this in for now, but this will mostly be wait time, not really CPU.
There was a problem hiding this comment.
Feel free to add more proper metrics for consensus
2 participants
Describe your changes and provide context
Port of #4167 onto
giga-1. Adds an Autobahn E2E Grafana dashboard, main-loop phase metrics, and an AWS--topologyflag.Dashboard. Local and AWS clusters provision an Autobahn E2E Grafana board that shows executed TPS, blocks/sec, produce-to-execute finalize time, and the execute-goroutine split across consensus wait, EVM execution, and storage. Those last panels read
sei_chain_autobahn_main_loop_phase_duration_seconds_total, recorded from the Autobahn execute path.AWS topology.
autobahn-e2e deploy --target awstakes--topology:distributed(default): five EC2 hosts — one validator per instance, plus a load/monitoring host. Validators clone, compile, and initialize in parallel. Prometheus and Grafana come up on the load box;sei-loadis left for you to start.colocated: one EC2 host running the same four Docker validators used locally, plus monitoring.The local Docker 4-in-1 path is unchanged. EVM JSON-RPC stays private and is reached through
forward.Defaults and access. AWS deploy uses Ubuntu 24.04 AMD64,
r7i.12xlarge, and a 40m readiness timeout. SSH and Grafana (:3000) default to the caller's public IP. Use--grafana-cidrto widen Grafana (for example0.0.0.0/0). Login remainsadmin/adminon these temporary test hosts.This branch also includes the #4167 review fixes: genesis ships the merged
persistent_peers.txt, siblingrun-instancescalls no longer cancel each other, and phase instruments attach to the existingsei_chainmeter instead of installing a second MeterProvider.Testing performed to validate your change
go test ./cmd/autobahn-e2e/ ./giga/metrics/./autobahn-e2e deploy --target aws --topology distributed— Grafana reachable from the SSH CIDR only; validators form a mesh./autobahn-e2e deploy --target aws --topology colocated— four Docker validators on one instance./autobahn-e2e deploy --target localstill starts the 4-in-1 Docker cluster