docs(proposals): Deprecate PEP 508 Direct References (req.url) #1225
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Deprecate PEP 508 Direct References (`req.url`) | ||
|
|
||
| **Status:** Proposed | ||
| **Date:** 2026-07-02 | ||
| **Author:** Shanmukh Pawan | ||
|
|
||
| ## What | ||
|
|
||
| Deprecate and eventually remove support for PEP 508 direct reference URLs (the `package @ git+https://...` syntax) as top-level bootstrap inputs. Migrate users to the declarative `source` provider config introduced in the [new resolver and download configuration proposal](https://fromager.readthedocs.io/en/latest/proposals/new-resolver-config.html). | ||
|
|
||
| ## Why | ||
|
|
||
| PEP 508 direct references (parsed into `Requirement.url` by the `packaging` library) let users specify a git URL inline in a requirement string. Fromager added this in [5f3fa68](https://github.com/python-wheel-build/fromager/commit/5f3fa68) (2025-02-22) to support building packages whose PyPI sdists were broken or unavailable. | ||
|
|
||
| This feature sits outside fromager's normal pipeline. Because the `Requirement` object carries the URL instead of a version specifier, every pipeline phase (resolution, download, prepare, build sdist) needs a separate `if req.url:` branch to bypass its normal logic. The codebase currently has **19 such branches across 6 files**. Each subsequent commit ([4a080bb](https://github.com/python-wheel-build/fromager/commit/4a080bb), [45008ce](https://github.com/python-wheel-build/fromager/commit/45008ce), [e50247d](https://github.com/python-wheel-build/fromager/commit/e50247d), [5687cf7](https://github.com/python-wheel-build/fromager/commit/5687cf7)) added special-case handling to accommodate `req.url` in a pipeline that wasn't designed for it. | ||
|
Member
There was a problem hiding this comment. This is a good reason to consider changing or removing this feature. |
||
|
|
||
| The scope is inherently narrow: | ||
|
|
||
| - Only top-level requirements. Transitive dependencies cannot carry URLs (PyPI rejects them per PEP 440). | ||
| - Only git URLs. No other VCS or URL type is supported. | ||
| - Only command-line input. Not declarative, not version-controlled, not per-variant. | ||
| - Requires cloning just to discover the version. The normal pipeline resolves versions without fetching source. | ||
|
|
||
| The new `source` provider config covers every `req.url` use case declaratively, without special-casing: | ||
|
|
||
| | Use case | `req.url` approach | `source` config replacement | | ||
| | -- | -- | -- | | ||
| | Build from git tag | `pkg @ git+https://repo@v1.0` | `provider: pypi-git` with `clone_url` and `tag` | | ||
| | Build from GitHub tag | `pkg @ git+https://github.com/org/repo@v1.0` | `provider: github-tag-git` with `project_url` | | ||
| | Build from GitLab tag | `pkg @ git+https://gitlab.com/org/repo@v1.0` | `provider: gitlab-tag-git` with `project_url` | | ||
| | Build from specific commit | `pkg @ git+https://repo@abc123` | `provider: versionmap-git` with `versionmap` | | ||
| | Build from branch HEAD | `pkg @ git+https://repo@main` | Same providers with appropriate tag/ref config | | ||
|
Member
There was a problem hiding this comment. This is the primary case we really cared about. Could you expand on the "same providers with appropriate config" explanation? How would we sometimes build from the HEAD of a branch and other times build a tagged version?
Contributor
Author
There was a problem hiding this comment. When I wrote "same providers with appropriate config," I was thinking of the case where a user could resolve the current HEAD to a known tag and passes that into the settings. I didn't mean to imply the existing providers support dynamic HEAD resolution out of the box. I'll update the proposal to make that distinction clear. For supporting builds from branch HEAD directly, I think we'd want a new provider, something like git-branch that clones the repo first and discovers the version from package metadata. The source config proposal already mentions VCS support as future work in the git clone section, so a git-branch provider could fit naturally into that scope. That said, I don't think we'd be able to support both building from HEAD and from a tagged release for the same package in the same run. The source config settings allows only one provider per package (or per variant). The current req.url flow has the same limitation. If requirements.txt contains both mypkg==1.2.3 and mypkg @ git+https://...main, the bootstrap loop is keyed on (package_name, version). Whichever entry resolves first gets processed; the other is silently marked as "already seen" and skipped. The behavior is order-dependent. There's also a cache staleness concern with branch HEAD builds. If the package uses a static version in pyproject.toml (not setuptools-scm or similar), then two different commits can produce the same version string and since fromager's wheel cache is keyed by (name, version), a rebuild the next day would find the cached wheel from the previous run and reuse it, even though the source has changed. This only works correctly when the version is dynamic and includes the commit hash (e.g., setuptools-scm producing 1.2.3.dev7+g3a4b5c6).
Member
There was a problem hiding this comment. Yes, the whole URL based scheme was only designed to work for builds where the version was dynamically determined from the tag. If the project is not using a tag to set the version, then all bets are off. Given that, I think the cache situation you raise is only a problem if the exact version specified in the build inputs is the same as the exact version that would be produced by the tag. In that case, it's not really a big deal because we only want one build anyway. The feature was designed to be used in a completely separate pipeline build, though. Like having a new collection for nightly builds of projects. Those might not even necessarily go to the same place as a production build.
Contributor
Author
There was a problem hiding this comment. Thanks, Doug! This is incredibly useful information. We'll review it during the next Fromager office hours and update you.
Member
There was a problem hiding this comment. Sounds good, and to reiterate what I said elsewhere earlier today: I do think we should review whether this feature is useful enough to keep the code complexity. It's possible we could handle it a different way (a pre-processor to produce a config file or a different set of phase implementation functions are 2 ways), but it's also possible we could just delete the code and not worry about it. |
||
|
|
||
| ## Decision | ||
|
|
||
| Deprecate `req.url` support with a warning, then remove it after users have migrated to `source` provider configs. | ||
|
|
||
| The deprecation is reversible. If unforeseen use cases emerge, the warning can remain indefinitely without blocking users. | ||
|
|
||
| ## How | ||
|
|
||
| Phase 1: When a top-level requirement has `req.url` set, emit a deprecation warning with guidance pointing to the equivalent `source` provider config. No behavior change. | ||
|
|
||
| Phase 2: Add a migration guide documenting the before/after for each use case (git tag, git commit, branch HEAD). | ||
|
|
||
| Phase 3: After downstream migration is confirmed, remove the `req.url` code paths: the 19 `if req.url:` branches, `_resolve_version_from_git_url()`, and related helper methods in `bootstrapper.py`. | ||
|
|
||
| `gitutils.py` is retained. It is used by git-clone source providers via `default_download_source()`. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also added it to allow building from source for nightly builds, to differentiate from building only tagged versions of packages. We did that originally for one of our downstream apps that was regularly adding new dependencies when they did tag and we wanted to pull those dependencies into the nightly builds to catch issues before releases.