Stop shipping vendored third-party sources in the wheel - #22584
Open
shoumikhin wants to merge 2 commits into
Open
Stop shipping vendored third-party sources in the wheel#22584shoumikhin wants to merge 2 commits into
shoumikhin wants to merge 2 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22584
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ab12a82 with merge base 5cdcc02 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
shoumikhin
force-pushed
the
slim-wheel-tests-thirdparty
branch
from
September 6, 2026 01:24
360f24a to
4711a4e
Compare
shoumikhin
changed the title
The NXP backend keeps a table of short names for edge operators, for example `AddTensor = exir_ops.edge.aten.add.Tensor`. It lives in the backend's test package, but three modules under backend/ read it at import time: backend/edge_helper.py backend/node_format_inference.py edge_passes/move_auxiliary_operator_into_separate_qdq_cluster_pass.py Over twenty operator converters import those modules, so the backend cannot load at all without a test package. The dependency points the wrong way: the tests should depend on the backend, not the backend on the tests. The table has no test logic in it, so this moves it to backend/ops_aliases.py, next to the code that reads it. The same applies to calibration_dataset.py, a pair of small dataset classes that an example export pipeline imports the same way. Both follow the shape the Arm backend already uses for shared operator constants in backends/arm/constants.py. No behavior changes. The names, values, and every importer stay the same, including the roughly fifty tests that use the table. Test plan: imported the moved modules and every updated importer from an installed wheel, and confirmed the table exposes the same names from the same values. Checked that every NXP test target can still reach the table through the backend package. Ran the formatter and linter over the changed files.
A pip install of ExecuTorch carries 348 files copied out of the vendored
third-party checkouts: a full copy of the MLX Python package, the XNNPACK
codegen scripts and their operator yaml, the Vulkan header registry, and
similar. Those exist to build the C++ targets. Once the libraries are built
nothing in an installed wheel imports them.
The cause is one missing setting. setup.py passes a `packages` list only for
the minimal build, so the full build falls back to setuptools auto discovery
and picks up everything under src/executorch. This adds the list for the full
build and closes the two ways those files arrive:
- `packages` excludes third-party and third_party directories.
- The package_data manifest is filtered with the same list. It is built from
the source tree rather than from `packages`, so the *.yaml glob still named
60 files inside those directories.
One directory list feeds both, so they cannot drift apart.
Test packages deliberately stay. The suites here import each other through the
installed name, for example `from executorch.backends.arm.test import common`,
about 1800 times, so dropping them stops collection under a non-editable
install. Pruning them needs that dependency removed first and is left alone
here. The TODO in pyproject.toml is updated to say so.
Result per wheel: 348 files and 3.8 MB uncompressed, 0.7 MB of the download.
Test plan: built wheels for Linux x86_64, Linux x86_64 with CUDA, and macOS
arm64. Confirmed no vendored directory remains in any of them, that the test
packages are still present, and that the shipped headers, cmake files and
flatbuffer schemas are unchanged in number. Installed each into a clean
environment and exported, loaded and ran a model through the portable kernels
and the XNNPACK delegate, checking outputs against eager; on a GPU also
through the CUDA delegate. Added a unit test next to the existing wheel checks
and confirmed it fails when the exclusion is removed.
shoumikhin
force-pushed
the
slim-wheel-tests-thirdparty
branch
from
September 6, 2026 07:33
4711a4e to
ab12a82
Compare
shoumikhin
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pip install of ExecuTorch carries 348 files copied out of the vendored third-party checkouts: most of the MLX Python source tree, the XNNPACK codegen scripts and their operator yaml, the Vulkan header registry. Those exist to build the C++ targets. Nothing in an installed wheel imports them.
The cause is one missing setting.
setup.pypasses apackageslist only for the minimal build. The full build has no list, so setuptools discovers packages by itself and picks up everything undersrc/executorch.First commit: move the NXP op alias table out of the test package
The NXP backend keeps a table of short names for edge operators in its test package, and three shipped modules read it at import time. Two dozen production files import those modules, so the backend cannot load without a test package.
The table has no test logic in it, so it moves next to the code that reads it, the way
backends/arm/constants.pyalready does. The calibration dataset classes move too. No behavior changes.Second commit: stop shipping vendored third-party sources
This adds the
packageslist for the full build, excluding the vendored directories, and filters the data-file manifest with the same list. The manifest is built from the source tree, not from that list, so a*.yamlpattern still matched 60 files inside them. One list feeds both, so they cannot drift apart.Test packages deliberately stay. The suites here import each other through the installed name, for example
from executorch.backends.arm.test import common, about 1800 times. Dropping them stops a non-editable install from collecting most tests. An editable install still works, because it exposes the whole source tree. Pruning the test packages needs that cross-import removed first, so it is not done here. The note inpyproject.tomlnow says why.Result: 348 files per wheel. The macOS wheel goes from 18.8 MB to 17.9 MB, and unpacked from 65.9 MB to 59.4 MB.
Test plan
Built wheels for macOS arm64, Linux x86_64, and Linux x86_64 with CUDA. Confirmed no vendored directory remains, the test packages are still there, and the shipped headers, cmake files and schemas are unchanged.
Installed into a clean environment and, from outside the checkout, exported, loaded and ran a model through the portable kernels and the XNNPACK delegate, checking outputs against eager. Added a unit test beside the existing wheel checks, and confirmed it fails when the package list stops excluding the vendored directories.