feat(rules): describe runfiles groups for rules_runfiles_group by FrankPortman · Pull Request #4254 · bazel-contrib/rules_python

FrankPortman · 2026-10-12T22:42:45Z

Gives py_library, py_binary/py_test and py_runtime_pair runfiles group describers for rules_runfiles_group, so packagers like rules_img's image_from_binary can split a Python binary into layers instead of shipping one big one.

A packager walks the binary with its own aspect and asks each target what it adds, so nothing runs unless you package. No flags. The groups:

rules_python#runtime: interpreter and stdlib, merged in from the Python toolchain. Every binary on that toolchain shares it.
rules_python#pypi/<name>: one per wheel. With venvs_site_packages=yes, the venv's links to a wheel's shared libraries go in that wheel's group too, so they stay next to the files they point at.
One per py_library, and one per rule target in data.
The binary itself, as the executable group.

A binary whose pyc_collection differs from the configuration's, or that generates legacy __init__.py files, keeps its deps in its own group, since their groups wouldn't match its runfiles.

Notes:

Walking the toolchain needs Bazel 8.8+. On older versions packaging fails; normal builds are unaffected.
Adds PyInfo.direct_implicit_pyc_files and PyExecutableInfo.venv_runfiles, since describers only see providers.
Only python/private/BUILD.bazel loads rules_runfiles_group, so there's no autoload cycle on Bazel 8.

Before merging:

Needs a rules_runfiles_group release. The DO NOT MERGE commit points the nested workspaces at the dev registry and gets reverted then.

Tested with runfiles_group_analysis_test, which checks the groups add up to exactly default_runfiles with no overlap, across a bunch of binary shapes, and end to end with rules_img.

Supersedes #3962. Written with LLM help (Claude); I've been through all of it and can own it in review.

Fixes #3757

FrankPortman · 2026-10-21T14:44:28Z

Note to maintainers, I think it's probably better to pull the following out into an upstream PR, but just leaving it monothlic until initial review:

PyInfo.direct_implicit_pyc_files, plus wiring it through the PyInfo builder and create_py_info
PyExecutableInfo.venv_runfiles, plus the small py_executable.bzl change that builds it once and reuses it
The behavior-neutral refactors:
- py_info_runfiles_files in common.bzl
- get_runtime_files(ctx, runtime) pulled out of _get_runtime_details
- exporting get_package_and_version
news/3757-provider-fields.added.md

malt3

The implementation looks good to me and works.
I tested it here: https://github.com/malt3/runfiles_group_demo and on some non-public code.
I also created a little visualisation here (AI generated): https://malt3.github.io/runfiles_group_demo/

aignas

I think @gfrankliu has raised valid points in the ticket and I would like to ask if it would be possible to get some examples in the form of tests to show how user can do changes to the config?

Overall, the approach is nice, it is in one place and once commented, maybe I'll be able to read it.

One ask would be to move refactors to internals that are for consistency or exposing to separate PRs if possible. Reading everything I thought that most of the internal changes could be factored out, but I could be wrong.

EDIT: regarding the PR split, I see there was a comment above, so all is good from that side.

aignas · 2026-10-25T00:42:35Z

+(rules) `py_binary` and `py_test` describe their runfiles groups for
+[rules_runfiles_group](https://github.com/bazel-contrib/rules_runfiles_group),
+so packaging rules like rules_img's `image_from_binary` can put the runtime,
+each PyPI package and each library in a layer of its own. Needs Bazel 8.8 or


I am not sure if this is ideal, but it can be a good default.

yeah, it's meant as the default. there's now a test showing how to change it per binary through aspect_hints. central config is the open question, I put the options in a top level comment.

aignas · 2026-10-25T00:45:14Z

+        "_aspect_rule_hints": attr.label_list(default = [
+            "//python/private:py_executable_runfiles_group_describer",
+            # What //python:toolchain_type is an alias of.
+            "@bazel_tools//tools/python:toolchain_type",


IIRC there is an alias for this in our own repo, please use that.

rules_runfiles_group checks the hint by its resolved label, and //python:toolchain_type resolves to @bazel_tools//tools/python:toolchain_type, so naming the alias makes the toolchain check fail.

@malt3 is that something that could match on what the rule declares instead? I forget if we already talked about this on Slack and maybe you have changed it on the latest version already.

@malt3 is that something that could match on what the rule declares instead? I forget if we already talked about this on Slack and maybe you have changed it on the latest version already.

I don't think so. The reason is that in Bazel, the aspect propagation context is called with the resolved toolchain name (and no way to resolve this back to the alias). That would require changes in Bazel.

ack makes sense, @aignas let me know what you think the lesser of evils is here. perhaps we can define and export a string constant that is used in both places, with a comment explaining why.

aignas · 2026-10-25T01:03:09Z

+
+# Stamped on every group, so Python groups stay together when a packager has to
+# merge groups to fit a layer limit.
+_MERGE_AFFINITY = "rules_python"


This is probably what @gfrankliu was talking about. Configuring this per python package would be nice. Would it be possible to have some config here?

Thanks @aignas, yes, this is the point I raised in #3757, and it's still unaddressed. The new test in fa42c2d documents that a per-binary aspect_hints transform is "the only way to configure grouping today", but that's the escape hatch I already said doesn't scale: it has to be added to every py_binary/py_test, and a macro that packages an existing binary (like our py_oci_image(binary = ...)) can't add it at all.

The example also doesn't show bucketing: setting do_not_merge = True on the ML groups keeps each wheel in its own layer. What ML images need is the opposite: torch, tensorflow, jax and nvidia-* merged with each other into one or two stable layers, and small, often-bumped packages kept out of them. With every group hard-coded to merge_affinity = "rules_python", there's no way to express that.

Could this be configurable centrally, e.g. a package → affinity mapping (on the pip extension, or a label flag the describers read), or a transform that packagers like layer_from_binary accept as an attribute? Without one of those, our in-house wrapper still does the job better, and I'd consider this a blocker for ML users rather than a follow-up.

(Thanks for the namespace-package fix in 5c82a34; I'll re-test that separately.)

Re-tested 4b94656 on the same image: the namespace-package fix works. The binary's layer is back to 0.3MB (same as our wrapper), and a one-line code change only re-pushes that layer. The 76 duplicate paths (1.3MB, mostly protobuf plus wheels' bin/ scripts) are still there, and the affinity question above stands.

FrankPortman · 2026-10-25T21:41:50Z

@aignas chewing through the other comments but for now I pulled #4341 out. PTAL 🙏

FrankPortman · 2026-10-26T13:32:46Z

@aignas / @malt3 on configuring grouping, today it's a transform in the binary's aspect_hints (see bucket_ml in the tests). if we want something more ready-made later, like a default set by a flag and overridable per binary, I see two places it could live:

rules_python: a flag pointing at a config target that maps distributions to merge_affinity / do_not_merge, read by the describers when they create the groups. works with no upstream changes since describers have ctx, but it's python-only and adds a public flag.

rules_runfiles_group / rules_img: a default transform set by a flag, plus a transform that takes parameters, e.g. runfiles_group_overrides(groups = {"rules_python#pypi/torch": {"merge_affinity": "ml"}}). works for every language, but needs upstream changes, since a transform today has to be a module level def and only gets the resolved groups, so there's no way to pass it names. this does mean that rules_python has to "advertise" how it names packages and thoroughly document how it interacts with the runfiles_group_overrides mechanism. and vice versa. but that feels warranted for what is attempting to be an ecosystem wide change and UX.

IMO it's important for it to be overridable for any one build or target, play well with with_cfg.bzl and so on. The above two proposals have that.

I lean towards the packager side since bucketing is a packaging decision, but either works. I'd keep it out of this PR.

…runfiles (bazel-contrib#4341) `PyInfo` had a direct counterpart for every pyc field except the implicit pyc files, and `PyExecutableInfo` exposed the venv's interpreter runfiles but not the application's. Tools that only see providers, like a packager's aspect, need both to tell a target's own files apart from what it inherits. Also pulls a few internals out so they can be reused, with no change in behavior: `py_info_runfiles_files()` for the files a binary takes from a dependency's PyInfo, `get_runtime_files()` for the runtime's files, and `get_package_and_version()` taking the data files instead of the ctx. Split out of bazel-contrib#4254, which uses all of these.

Give py_library, py_binary and py_test runfiles group describers, so packaging rules that use rules_runfiles_group (e.g. rules_img's image_from_binary) can split a binary's runfiles into layers: the runtime, one group per PyPI package, one per library and per rule target in data, and the binary's own files. Groups are collected on demand by a packager's aspect, so nothing runs otherwise. The runtime group comes from a singleton target resolving the same toolchain as the binary, so a packager builds it once per configuration. A binary whose pyc_collection differs from the configuration's, or that generates legacy __init__.py files, keeps its dependencies in its own group. Describers only see providers, so this adds PyInfo.direct_implicit_pyc_files and PyExecutableInfo.venv_runfiles. Fixes bazel-contrib#3757

Port to rules_runfiles_group's current API: the describers are held by _aspect_rule_hints, and the runtime group is merged in from the Python toolchain, described by py_runtime_pair. This drops the singleton python_runtime_runfiles target. Also fix two issues found testing against a real image: * A Python target in srcs (whl_library's :pkg wraps its :srcs) describes its own files, so the wrapper no longer claims them too. * With venvs_site_packages=yes, the venv's links to a wheel's files (shared libraries) go into that wheel's group instead of the binary's, so packagers keep them next to the files they point to.

…kspaces Revert once rules_runfiles_group is released.

Picks up the analysis test comparing symlinks by path and target.

rules_runfiles_group's lib.bzl loads its packager code, which needs a recent bazel_features that WORKSPACE users don't have. Its new describe.bzl doesn't, so loading the Python rules works in WORKSPACE again without extra setup. The analysis tests still go through the packager, so the dev WORKSPACE gets bazel_features 1.51.0, and the tests are skipped on Bazel versions whose aspects can't walk toolchains.

…r toolchain walking

Give every function in runfiles_groups.bzl a docstring with typed args, list the groups a binary gets in the module docstring, and drop the copyright header. A wheel's group is named after its distribution: the name from its .dist-info directory, normalized by get_package_and_version (lowercase, runs of -, _ and . turned into _), e.g. rules_python#pypi/google_cloud_storage.

…group Where distributions ship the same package directory (pkgutil-style namespace packages, common under google/), the venv links each file in it to the file itself, instead of linking the directory. Those links weren't found in any distribution's group, so they stayed in the binary's group and a packager copied their files into it. Look links up by every file a venv entry covers, not just the one it links to.

A transform in a binary's aspect_hints can change the metadata of its groups, e.g. to keep a large distribution's group from being merged into others. This is the only way to configure grouping today.

Same tarball bzlmod resolves from the dev registry. The GitHub archive of d105dfa can go away now that the commit isn't on a branch.

whl_library lists each console script in the wheel's data twice: as the script's own target and inside the :data filegroup. The script target got a group of its own, and the filegroup's file was taken as a plain file of the wheel, so the script landed in two groups. Merge everything in a wheel's data into its distribution's group.

…oolchain Revert once rules_runfiles_group is released.

The example set do_not_merge alongside the affinity, which made the affinity a no-op. Give two distributions a shared affinity instead and check that the others keep the default one.

malt3 · 2026-10-27T15:44:40Z

@FrankPortman I agree that we should make grouping more customizable on the packager side (in rules_runfiles_group and rules_img).

I would be happy to extend the regrouping and overriding mechanism on the packager side with more expressive ways to define how runfiles are grouped. There are a few generic, language-agnostic groupings that make sense out of the box: per-target grouping (the current fine-grained default) and per-repo grouping. For repos, targets could be merged either as one group per repo and merge affinity, or by overriding the merge affinity entirely to produce a single group per repo.

In addition, we should make it straightforward to define custom groupings, such as clustering third-party dependencies by an explicit list or a repository name prefix.

None of this requires changes to the describer part of rules_runfiles_group. Merging fine-grained, per-target groups happens in the packaging rule implementation after the aspect finishes walking the dependency graph. rules_runfiles_group also already includes a weight-aware merging algorithm that merges groups up to a configured limit. While that does not cover every custom use case, it handles grouping well out of the box with the data currently available.

For that reason, I do not think we need to block this PR on landing the final customization mechanism on the packager side.

FrankPortman · 2026-10-27T16:14:50Z

thanks @malt3, agreed w/ both bucketing on the packager side, and not blocking this PR on the final customization mechanisms.

the tests are now hopefully illustrative enough on how to hint as to what groups to collapse. once the packager side has more expressive groupings, we can adapt to that.

This was referenced Oct 12, 2026

feat(rules): emit RunfilesGroupInfo from py_binary, py_test, and py_runtime #3962

Closed

Emit RunfilesGroupInfo in py_binary targets #3757

Open

FrankPortman marked this pull request as ready for review October 19, 2026 13:43

FrankPortman requested review from aignas, dougthor42 and rickeylev as code owners October 19, 2026 13:43

malt3 approved these changes Oct 22, 2026

View reviewed changes

aignas reviewed Oct 25, 2026

View reviewed changes

FrankPortman mentioned this pull request Oct 25, 2026

feat: add PyInfo.direct_implicit_pyc_files and PyExecutableInfo.venv_runfiles #4341

Merged

FrankPortman added 13 commits October 27, 2026 10:59

DO NOT MERGE: use the rules_runfiles_group dev registry in nested wor…

cbc1133

…kspaces Revert once rules_runfiles_group is released.

chore: bump rules_runfiles_group to 99.0.0-dev.20261019.1.1cee2dd

6024e17

Picks up the analysis test comparing symlinks by path and target.

style: sort the runfiles group attributes, name the Bazel versions fo…

5a7dc55

…r toolchain walking

test: show configuring a binary's runfiles groups through aspect_hints

35c4cc5

A transform in a binary's aspect_hints can change the metadata of its groups, e.g. to keep a large distribution's group from being merged into others. This is the only way to configure grouping today.

build: fetch rules_runfiles_group from its dev registry in WORKSPACE

b585663

Same tarball bzlmod resolves from the dev registry. The GitHub archive of d105dfa can go away now that the commit isn't on a branch.

DO NOT MERGE: use the rules_runfiles_group dev registry in prebuilt_t…

654e903

…oolchain Revert once rules_runfiles_group is released.

test: show bucketing distributions together through aspect_hints

1b40985

The example set do_not_merge alongside the affinity, which made the affinity a no-op. Give two distributions a shared affinity instead and check that the others keep the default one.

FrankPortman force-pushed the runfiles_group_describers branch from b233c33 to 1b40985 Compare October 27, 2026 14:59

FrankPortman mentioned this pull request Oct 27, 2026

Support RunfilesGroupInfo bazelbuild/rules_java#368

Open

Conversation

FrankPortman commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FrankPortman commented Oct 21, 2026

Uh oh!

malt3 left a comment

Choose a reason for hiding this comment

Uh oh!

aignas left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FrankPortman commented Oct 25, 2026

Uh oh!

FrankPortman commented Oct 26, 2026

Uh oh!

malt3 commented Oct 27, 2026

Uh oh!

FrankPortman commented Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

FrankPortman commented Oct 12, 2026 •

edited

Loading

aignas left a comment •

edited

Loading