gh-149584: `_remote_debugging`: Map remote pages with `mach_vm_remap` on macOS instead of copying them by maurycy · Pull Request #161011 · python/cpython

maurycy · 2026-10-31T15:43:39Z

The PR adds vm_remap support to profiling.sampling (Tachyon) on macOS, instead of always using mach_vm_read_overwrite().

In plain English :-) instead of copying remote proces memory, just map the same page and read from it directly. Direct memcpy() is much cheaper than a syscall. The trick works especially for typically stably mapped objects, hence explicit syscall read in #160888 for data stack.

The performance gain is relatively good:

Metric	Baseline (`main` e891a91 + #160888)	The PR	Change
Work rate	335314.2 calls/sec	1074395.9 calls/sec	3.20x faster
Sample rate	326132.5 samples/sec	986112.9 samples/sec	3.02x faster
Avg call time	2.98 µs	0.93 µs	-68.8%
Total attempts	978,399	2,958,340	3.02x more
Successful samples	978,399	2,958,340	-
Failed samples	0	0	-
Success rate	100.00%	100.00%	-
Work efficiency	97.3%	91.8%	-

On top of this, there's little loss in accuracy:

Oracle, this PR (12 cases, 3 runs, 0.5 s each)

3.16.0a1+dev (heads/tachyon-vmremap:d561805beb0, Nov 10 2026, 02:17:28) [Clang 21.0.0 (clang-2100.4.1.1)]
cases=async_running_task,asyncio,basic,code_object_reuse,deep_recursion,deep_static,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf,threads runs=3 duration=0.5 rate_khz=100.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=21
mode                 samples    stacks     empty           µs      errors% impossible  impossible%
blocking-nocache      147831    295662      2085    7.67±0.05    0.00±0.00          0    0.00±0.00
live-nocache          142859    285280      7011    5.68±0.02    0.09±0.01        987    0.35±0.01

asyncio (get_async_stack_trace) ref_keys=28
mode                 samples    stacks     empty           µs      errors%
blocking-nocache      142993    239764      6921    8.34±0.03    0.00±0.00
live-nocache          146337    179529      2798    6.00±0.23    0.17±0.02

basic (get_stack_trace) ref_keys=61
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      149983    6.54±0.08    0.00±0.00          ref          ref          -
blocking-cache        149990    2.92±0.16    0.00±0.00  0.541±0.024  0.529±0.024      0.012
live-cache            150000    1.29±0.04    0.00±0.00  0.673±0.030  0.662±0.030      0.012
live-nocache          150000    3.86±0.03    0.00±0.00  0.673±0.030  0.661±0.030      0.012

code_object_reuse (get_stack_trace) ref_keys=10
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.84±0.07    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    5.35±0.04    0.00±0.00          0    0.00±0.00  0.008±0.003  0.005±0.003      0.003
live-cache            150000    1.34±0.04    0.00±0.00          0    0.00±0.00  0.338±0.038  0.335±0.038      0.003
live-nocache          150000    3.82±0.06    0.00±0.00          0    0.00±0.00  0.011±0.001  0.007±0.001      0.003

deep_recursion (get_stack_trace) ref_keys=984
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      102063    7.85±0.25    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache         40683   14.63±0.29    0.00±0.00          0    0.00±0.00  0.675±0.033  0.606±0.035      0.070
live-cache             31419   32.85±1.22    5.63±0.08       5177   16.49±0.89  0.692±0.020  0.610±0.022      0.082
live-nocache           31382   31.81±1.86    5.41±0.33       5105   16.28±0.80  0.705±0.023  0.623±0.027      0.083

deep_static (get_stack_trace) ref_keys=1
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       11814   26.99±0.12    0.00±0.00          ref          ref          -
blocking-cache         14125    5.99±0.10    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-cache             14255    5.03±0.03    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-nocache           12107   25.53±0.28    0.00±0.00  0.000±0.000  0.000±0.000      0.000

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.84±0.05    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.78±0.12    0.00±0.00          0    0.00±0.00  0.039±0.019  0.035±0.019      0.004
live-cache            150000    4.47±0.03    0.00±0.00       1175    0.78±0.12  0.242±0.004  0.238±0.004      0.004
live-nocache          150000    3.84±0.03    0.00±0.00       1012    0.67±0.08  0.230±0.010  0.226±0.010      0.004

gen_alternating (get_stack_trace) ref_keys=19
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    7.47±0.08    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    7.16±0.02    0.00±0.00          2    0.00±0.00  0.077±0.004  0.068±0.004      0.008
live-cache            125571    4.88±0.04   16.29±0.23      30897   24.60±0.95  0.302±0.008  0.293±0.008      0.010
live-nocache          126010    4.24±0.01   15.99±0.22      33092   26.26±0.14  0.306±0.003  0.296±0.003      0.010

nested_alternating (get_stack_trace) ref_keys=10
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.80±0.04    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    5.89±0.12    0.00±0.00          0    0.00±0.00  0.017±0.005  0.012±0.005      0.005
live-cache            150000    1.52±0.00    0.00±0.00          0    0.00±0.00  0.137±0.002  0.132±0.002      0.005
live-nocache          150000    3.92±0.03    0.00±0.00          0    0.00±0.00  0.138±0.008  0.133±0.008      0.005

oversized_chunk (get_stack_trace) ref_keys=15
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      149999    6.79±0.10    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149999    5.88±0.08    0.00±0.00          0    0.00±0.00  0.023±0.012  0.018±0.012      0.005
live-cache            150000    2.84±0.01    0.00±0.00         31    0.02±0.01  0.069±0.002  0.064±0.002      0.005
live-nocache          150000    3.91±0.02    0.00±0.00          0    0.00±0.00  0.065±0.001  0.060±0.001      0.005

shared_leaf (get_stack_trace) ref_keys=10
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      150000    6.94±0.08    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    5.69±0.03    0.00±0.00          0    0.00±0.00  0.008±0.004  0.005±0.004      0.003
live-cache            150000    1.33±0.01    0.00±0.00       9776    6.52±1.21  0.114±0.005  0.111±0.005      0.003
live-nocache          150000    3.81±0.05    0.00±0.00          0    0.00±0.00  0.120±0.003  0.117±0.003      0.003

threads (get_stack_trace) ref_keys=7
mode                 samples    stacks           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       42983    270716   29.08±2.82    0.00±0.00          ref          ref          -
blocking-cache        101658    752153    9.04±3.48    0.00±0.00  0.056±0.038  0.054±0.038      0.002
live-cache            125166    872286    6.26±1.21    0.00±0.00  0.064±0.013  0.062±0.013      0.002
live-nocache           46214    297139   26.83±2.59    0.00±0.00  0.054±0.015  0.052±0.015      0.002

v.

Oracle, baseline (12 cases, 3 runs, 0.5 s each)

3.16.0a1+dev (heads/main-dirty:e891a911798, Nov 10 2026, 02:17:30) [Clang 21.0.0 (clang-2100.4.1.1)]
cases=async_running_task,asyncio,basic,code_object_reuse,deep_recursion,deep_static,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf,threads runs=3 duration=0.5 rate_khz=100.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=2
mode                 samples    stacks     empty           µs      errors% impossible  impossible%
blocking-nocache       46324     92648      4435   27.45±2.58    0.00±0.00          0    0.00±0.00
live-nocache           53286    105811     12068   21.17±0.29    0.41±0.18       4824    4.56±0.11

asyncio (get_async_stack_trace) ref_keys=8
mode                 samples    stacks     empty           µs      errors%
blocking-nocache       61051     84824      2975   22.55±4.23    0.00±0.00
live-nocache           70833     83103      2158   19.19±0.18    0.66±0.24

basic (get_stack_trace) ref_keys=19
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      136952    9.43±0.21    0.00±0.00          ref          ref          -
blocking-cache        149933    5.26±0.03    0.00±0.00  0.580±0.116  0.572±0.117      0.007
live-cache            150000    3.63±0.06    0.00±0.00  0.724±0.080  0.716±0.081      0.007
live-nocache          150000    7.20±0.03    0.00±0.00  0.723±0.080  0.716±0.081      0.007

code_object_reuse (get_stack_trace) ref_keys=1
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      141779    8.94±0.31    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.72±0.06    0.00±0.00          0    0.00±0.00  0.511±0.016  0.511±0.016      0.000
live-cache            150000    3.66±0.03    0.00±0.00          0    0.00±0.00  0.906±0.010  0.906±0.010      0.000
live-nocache          150000    7.11±0.01    0.00±0.00          0    0.00±0.00  0.504±0.000  0.504±0.000      0.000

deep_recursion (get_stack_trace) ref_keys=574
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      126099    9.42±0.46    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache         58342   12.37±1.10    0.00±0.00          0    0.00±0.00  0.716±0.056  0.687±0.059      0.028
live-cache             29822   21.94±0.49   45.48±0.39       8167   27.39±0.54  0.905±0.012  0.862±0.016      0.041
live-nocache           30243   22.68±1.24   43.74±4.06       9693   32.04±0.48  0.909±0.009  0.866±0.014      0.042

deep_static (get_stack_trace) ref_keys=1
mode                 samples           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       11524   29.85±0.66    0.00±0.00          ref          ref          -
blocking-cache         13986    8.21±0.18    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-cache             13905    7.46±0.11    0.00±0.00  0.000±0.000  0.000±0.000      0.000
live-nocache           11686   28.64±0.11    0.00±0.00  0.000±0.000  0.000±0.000      0.000

flat_alternating (get_stack_trace) ref_keys=5
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      140998    9.07±0.30    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149999    7.32±0.02    0.00±0.00          0    0.00±0.00  0.115±0.049  0.111±0.049      0.004
live-cache            150000    7.57±0.04    0.00±0.00        678    0.45±0.01  0.117±0.010  0.113±0.010      0.004
live-nocache          150000    7.02±0.06    0.00±0.00        844    0.56±0.06  0.127±0.010  0.123±0.010      0.004

gen_alternating (get_stack_trace) ref_keys=14
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      122707   10.86±0.48    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149998    7.97±0.07    0.00±0.00          0    0.00±0.00  0.312±0.061  0.304±0.061      0.008
live-cache            124422    8.97±0.07   17.05±0.08      22501   18.08±2.06  0.444±0.025  0.435±0.026      0.009
live-nocache          125425    8.37±0.02   16.38±0.11      20824   16.60±1.41  0.440±0.022  0.431±0.022      0.009

nested_alternating (get_stack_trace) ref_keys=3
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      123026    9.78±0.11    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.71±0.09    0.00±0.00          0    0.00±0.00  0.472±0.056  0.469±0.057      0.003
live-cache            150000    3.83±0.03    0.00±0.00         20    0.01±0.02  0.466±0.060  0.463±0.060      0.003
live-nocache          150000    6.97±0.08    0.00±0.00          0    0.00±0.00  0.466±0.060  0.463±0.060      0.003

oversized_chunk (get_stack_trace) ref_keys=7
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      132653    9.44±0.11    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        150000    6.75±0.03    0.00±0.00          0    0.00±0.00  0.108±0.023  0.103±0.022      0.005
live-cache            150000    5.65±0.01    0.00±0.00         65    0.04±0.03  0.132±0.028  0.127±0.028      0.005
live-nocache          150000    7.03±0.07    0.00±0.00          0    0.00±0.00  0.164±0.031  0.159±0.031      0.005

shared_leaf (get_stack_trace) ref_keys=2
mode                 samples           µs      errors% impossible  impossible%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache      137680    9.22±0.15    0.00±0.00          0    0.00±0.00          ref          ref          -
blocking-cache        149999    6.74±0.03    0.00±0.00          0    0.00±0.00  0.066±0.070  0.063±0.070      0.003
live-cache            150000    3.58±0.00    0.00±0.00        100    0.07±0.12  0.169±0.080  0.166±0.080      0.003
live-nocache          150000    6.94±0.10    0.00±0.00          0    0.00±0.00  0.167±0.081  0.164±0.080      0.003

threads (get_stack_trace) ref_keys=8
mode                 samples    stacks           µs      errors%      raw_tvd   tvd_excess  tvd_floor
blocking-nocache       35587    213522   36.43±1.80    0.00±0.00          ref          ref          -
blocking-cache         63645    410209   17.86±1.85    0.00±0.00  0.049±0.056  0.047±0.055      0.002
live-cache             65711    471700   17.11±2.86    0.00±0.00  0.089±0.053  0.086±0.053      0.002
live-nocache           32077    233688   41.52±9.23    0.00±0.00  0.090±0.064  0.087±0.063      0.002

(this is where #153804 shines in my opinion.)

To make things even better, the CPU usage is also relatively better:

case / frame cache	base user µs	base sys µs	base CPU %	alias (PR) user µs	alias sys µs	alias CPU %
async_running_task/cache	2.67	10.53	13.2	2.46	5.12	7.6
async_running_task/nocache	3.44	13.05	16.5	3.35	7.17	10.5
asyncio/cache	4.63	15.06	19.7	5.32	10.43	15.8
asyncio/nocache	7.75	22.67	30.4	5.48	10.61	16.1
basic/cache	1.59	5.73	7.3	1.61	2.92	4.5
basic/nocache	2.41	8.87	11.3	2.38	5.33	7.7
code_object_reuse/cache	2.36	8.88	11.2	2.30	3.72	6.0
code_object_reuse/nocache	3.80	14.22	18.0	2.91	5.99	8.9
deep_recursion/cache	12.46	37.53	50.0	10.78	25.61	36.4
deep_recursion/nocache	10.24	32.43	42.7	11.66	29.89	41.6
deep_static/cache	5.77	5.33	11.1	5.85	2.60	8.4
deep_static/nocache	14.49	18.95	33.4	14.82	16.02	30.8
flat_alternating/cache	2.69	9.70	12.4	3.50	7.65	11.2
flat_alternating/nocache	2.68	9.55	12.2	2.79	5.75	8.5
gen_alternating/cache	3.12	11.89	15.0	3.19	6.79	10.0
gen_alternating/nocache	3.01	11.36	14.4	3.15	6.29	9.4
nested_alternating/cache	2.38	8.18	10.6	2.62	4.96	7.6
nested_alternating/nocache	2.78	9.75	12.5	3.03	6.13	9.2
oversized_chunk/cache	2.89	10.24	13.1	3.08	6.73	9.8
oversized_chunk/nocache	2.79	9.80	12.6	2.91	5.95	8.9
shared_leaf/cache	1.96	6.69	8.6	2.02	3.61	5.6
shared_leaf/nocache	2.69	9.63	12.3	2.86	5.99	8.9
threads/cache	4.76	25.05	29.8	5.65	15.28	20.9
threads/nocache	14.06	60.64	74.6	11.55	38.28	49.8

(everything on PGO+LTO builds.)

Some considerations:

vm_remap is a Mach 3, so it's at least 1992? The XNU kernel is extremely helpful, but there's also https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/osfmk/man/vm_remap.html etc. I used Fable 5.1, and previous Claude models, a lot when asking questions about the kernel. I recommend doing the same.
I feel that _Py_RemoteDebug_ReadRemoteMemory and _Py_RemoteDebug_PagedReadRemoteMemory are increasingly leaky abstractions; the optimal way of reading depends on what (simply: datastacks have different volatility / are less stable than interpreters etc.), the operating system; this is visible more since gh-149584: Fix excessive overhead in the Tachyon profiler regarding the cache behavior #149649 We can reorganize this as a follow up

The Oracle redone with changes from #161674

Issue: _remote_debugging: reading whole pages over and over #149584

maurycy · 2026-10-31T15:55:15Z

+}
+
+static int
+alias_maybe_probe_entry(proc_handle_t *handle, alias_page_entry_t *entry)


mach_vm_region_recurse is expensive, hence adaptive probing here, as the exact churn depends on the target

maurycy requested a review from pablogsal as a code owner October 31, 2026 15:43

bedevere-app Bot added the awaiting review label Oct 31, 2026

bedevere-app Bot mentioned this pull request Oct 31, 2026

_remote_debugging: reading whole pages over and over #149584

Open

maurycy mentioned this pull request Oct 31, 2026

Use mach_vm_remap in Tachyon on macOS maurycy/cpython#3

Closed

maurycy commented Oct 31, 2026

View reviewed changes

maurycy force-pushed the tachyon-vmremap branch from 99d1803 to 69c085e Compare November 9, 2026 23:55

maurycy added 2 commits November 10, 2026 01:02

see #3

b4747ab

news

d561805

maurycy force-pushed the tachyon-vmremap branch from 69c085e to d561805 Compare November 10, 2026 00:02

Navigation Menu

Repository navigation

gh-149584: `_remote_debugging`: Map remote pages with `mach_vm_remap` on macOS instead of copying them - #161011

gh-149584: `_remote_debugging`: Map remote pages with `mach_vm_remap` on macOS instead of copying them#161011
maurycy wants to merge 2 commits into
python:mainfrom
maurycy:tachyon-vmremap

maurycy commented Oct 31, 2026 •

edited

Loading

Uh oh!

maurycy Oct 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

maurycy commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

maurycy commented Oct 31, 2026 •

edited

Loading

maurycy Oct 31, 2026 •

edited

Loading