Record and expose perfetto timing traces for inventory collection - #11023
Open
smklein wants to merge 3 commits into
Open
Record and expose perfetto timing traces for inventory collection#11023smklein wants to merge 3 commits into
smklein wants to merge 3 commits into
Conversation
The Trace and TraceEvent types from support-bundle-collection are now in a standalone perfetto-trace crate, so that other crates can emit traces without depending on the full support bundle machinery. The old support_bundle_collection::perfetto path still works via a re-export. The new crate also gains a TraceSpan recording type, a timed() future bracketing helper, and an assemble() function that packs spans into the minimum number of tid lanes (greedy interval packing), so the lane count in the viewer reflects the maximum observed concurrency.
The Collector now records a TraceSpan for each collection phase, each request made within the concurrent phases (sled agents, timesync, keepers, DNS generations), each MGS client, and each SP queried through MGS. The per-SP work moves into a new collect_one_sp helper so it can be bracketed as a unit; its behavior is unchanged. collect_all returns the spans alongside the Collection. The inventory background task assembles them into a Chrome Trace Event format trace and reports it in its activation status via the new shared InventoryCollectionStatus type, so the last collection's timing can be retrieved from the running Nexus and loaded into ui.perfetto.dev. The trace is not persisted anywhere: a failed activation, or asking a Nexus that has not collected since it started, yields no trace.
The inventory_collection task printer now deserializes the shared
InventoryCollectionStatus type and prints per-phase durations when the
status includes a trace. A new subcommand,
omdb nexus background-tasks inventory-trace --output <path>
writes the last collection's full trace as Chrome Trace Event format
JSON for loading into https://ui.perfetto.dev/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #11022.
Now that inventory collection runs concurrently within each phase, this PR adds visibility into where collection time actually goes. The collector records a timing span for each phase, each request in the four fan-out phases (sled agents, timesync, keepers, DNS generations), each MGS client, and each SP queried through MGS. The spans are assembled into a Chrome Trace Event format JSON document that loads directly into https://ui.perfetto.dev/ or chrome://tracing.
What's in this PR
New
perfetto-tracecrate. TheTrace/TraceEventtypes move out ofsupport-bundle-collection. The new crate adds aTraceSpanrecording type, atimed()future-bracketing helper, andassemble(), which packs spans into the minimum number oftidlanes (using greedy interval packing).Span recording in the collector. Fan-out tasks time themselves and return their span through the existing result tuples; the per-SP MGS work moves into a
collect_one_sphelper (a pure extraction of the loop body) so it can be bracketed as a unit.collect_allreturns the spans alongside theCollection.Exposure via background task status and omdb. The task's success JSON becomes the shared
InventoryCollectionStatustype with an optional embedded trace.omdb nexus background-tasks showprints per-phase durations, and a newomdb nexus background-tasks inventory-trace --output <path>writes the full trace JSON for the perfetto UI.