fix: exclude the dyld shared cache from macOS value scans#50
Merged
Merged
Conversation
On macOS the dyld shared cache (~5.8 GB of read-only library text/data the kernel maps into every process) was scanned on every value/pattern scan because vm_region_basic_info's `shared` flag reports FALSE for it. Linux and Windows already exclude their equivalent file-backed library mappings, so the same scan walked ~6.6 GB on macOS versus ~1 GB elsewhere — making scans (and the test suite) run 4-6x slower. Recognize the cache via its vm_region_extended_info user_tag (VM_MEMORY_SHARED_PMAP) and mark those regions as shared so default_scan_filter drops them, matching the Linux/Win32 backends. Address-list reads and pointer scans are unaffected — they filter on readability/writability, not the shared flag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On macOS, every value/pattern scan walked the dyld shared cache — the ~5.8 GB of read-only library text/data the kernel maps into every process. Linux and Windows already exclude their equivalent file-backed/shared library mappings via
default_scan_filter, but on macOS the filter excluded nothing: it relies onMemoryRegion.is_shared, which on macOS reads thesharedflag ofvm_region_basic_info_64— and that flag reports FALSE for the shared cache.Measured on a real process (current Python interpreter under pytest):
is_shared(before)So the pure-Python scan loop chewed through ~6.6 GB on macOS versus ~1 GB on the other platforms — making scans (and the CI test suite, which runs without the NumPy fast path) 4-6x slower: ~15-25 min on macOS vs ~4-6 min on Linux/Windows.
The three offending regions (2176 / 2016 / 1632 MB here) all carry
vm_region_extended_info.user_tag == VM_MEMORY_SHARED_PMAP (32)— the kernel's identifier for the shared-cache submap. The private heap regions we do want to scan carry malloc tags instead.Fix
Recognize the shared cache by its
user_tag(read via aVM_REGION_EXTENDED_INFOquery, since the basic-info struct has no tag) and mark those regions asis_shared, sodefault_scan_filterdrops them — matching the Linux/Win32 exclusion of file-backed library mappings.macos/types.py— add thevm_region_extended_infostruct +VM_REGION_EXTENDED_INFO/VM_MEMORY_SHARED_PMAPconstants.macos/libsystem.py— generalize themach_vm_regionprototype (infoas an opaque pointer) so both basic- and extended-info structs can be passed.macos/functions.py—_region_user_tag()reads the tag;_region_is_shared()returns the basicsharedflag ORuser_tag == VM_MEMORY_SHARED_PMAP. Used byget_memory_regionsand_query_region.tests/test_macos_protect.py— regression test: the shared cache must be flaggedis_shared, anddefault_scan_filtermust exclude it.Pointer scans and address-list reads are unaffected — they filter on
is_readable/is_writable, notis_shared, so the shared cache is still a valid pointer target and a valid explicit-address read.Results
search_by_value: 11.94 s → 1.56 s (7.6x)346 passed, 13 skipped; flake8 and mypy clean.