[pytorchbot]

What is wrong today

A method can call the same lowered module several times, for example a denoising step that is lowered once and called 10 times in a loop. The emitter then writes one BackendDelegate per call. So at load time the runtime calls the backend's init() once per call and keeps every handle alive, even though all of them come from the same compiled blob. This wastes load time and memory.

What this change does

A backend can now return PreprocessResult(..., share_handle_across_calls=True). Then every call of that lowered module in a method uses one BackendDelegate, so init() runs once. Each call still passes its own arguments.

• It is off by default, so existing backends behave as before. Some backends keep argument pointers on the handle, so sharing is not safe for all of them.
• A delegate that declares scratch memory is never shared, because scratch is planned per call and the runtime keeps it on the delegate. Method::init now rejects a program where such a delegate has more than one call site. It also checks every delegate index at load, not only at run time.
• Only calls of one lowered module share a handle. Separate partitions keep separate delegates, even when their blobs are identical.
• The flag is saved and loaded like scratch_specs, including through exir.save and exir.load, so the serde schema moves to version 5.6. The .pte file format does not change.
• The demo test backend opts in.

Before #14564, the emitter shared one delegate between calls with the same blob for every backend. Now it shares by lowered module, and only for backends that ask for it.

What was tested

• New tests cover one delegate for repeated calls with the same output as separate calls, the partitioner path, separate modules keeping their own delegates, save and load, the default, and scratch, including a hand-made program that the runtime rejects. The runtime test for scratch fails without the new runtime check. The load-time index check, calls inside control flow, the multi-method lowering path and the number of init() calls have no dedicated test.
• On an NVIDIA Jetson, a TensorRT step that a method calls 10 times went from 10 delegates to 1 shared delegate: load time 435 ms to 74 ms, memory 1060 MiB to 284 MiB, with the same outputs and speed.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24207

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 2bc573a with merge base 4db778e ():

NEW FAILURE - The following job has failed:

• Test QNN Windows MSVC build / build-qnn-windows-x64 / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.