mz-debug: symbolization script#37793
Draft
SangJunBak wants to merge 3 commits into
Draft
Conversation
Because multi-process replicas can have multiple pods per service, we account for that when dumping resources.
Engineers symbolizing mz-debug profiles previously followed a manual guide: install go, google/pprof, and graphviz, discover build IDs from pprof errors, curl the executable and debug info per build ID from the debuginfo service, and re-run pprof. This automates all of it. profiles/symbolize.sh now ships inside every dump (embedded via include_str!) and also lives in the repo behind bin/pprof-symbolize. It needs only Docker and network access: it builds a small content-hashed tool image (pinned google/pprof, graphviz, llvm), reads the build IDs out of each profile's mappings, fetches executables and debug info from https://debuginfo.dev.materialize.com into a local cache, and writes shareable <name>.symbolized.pprof.gz files. A --http mode serves the interactive pprof web UI. 404s from expired PR-build artifacts and junk files degrade gracefully with per-profile statuses. zip_debug_folder now preserves unix permissions so the script's executable bit survives unzipping. Verified end to end against profiles scraped from a kind cluster: all profiles symbolize with full Rust frames, re-runs hit the cache, the web UI serves, and the script embedded in a fresh dump zip runs as shipped.
SangJunBak
force-pushed
the
jun/mz-debug-symbolize-script
branch
from
July 23, 2026 13:18
9c265b9 to
e84d7dc
Compare
SangJunBak
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/cputurns on CPU profiling. Before, users could only turn it on via a webpage. We also ensure CPU and memory profiling cannot be turned on at the same time. This is because memory profiling and CPU profiling at the same time can cause deadlocks (see #3922 for more context). Thus before we start CPU profiling, we turn off memory profilng, then create a drop guard for the remainder of the CPU profiling execution that ensures memory profiling is turned back to its previous state after a drop./modereturns what profiling is enabled and allows users to turn on/off memory profiling.Remove these sections if your commit already has a good description!Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
How do you know this change is correct? Describe new or existing automated
tests, or manual steps you took.