Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .claude/port-package-prompt.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
Port a new package to this repo (riscv64 wheel → pypi.riseproject.dev). Follow
the porting playbook, working process, and gotchas in CLAUDE.md — that's the
source of truth; everything below is just this package's coordinates.
the `python-project-porting` skill (skills/python-project-porting/SKILL.md and its
references/ — start by skimming references/gotchas-index.md) — that's the source of truth;
everything below is just this package's coordinates.

Create the branch and the worktree first then proceed with research. Always work out of the worktree!

Expand All @@ -15,4 +16,5 @@ Do the full loop end to end: branch + worktree, add build-<pkg>.yml, validate
locally, push, open the PR, then watch CI and iterate until every matrix job is
green and publish dry-runs cleanly. Wire up real testing the way upstream tests
its own wheels. When it's working, fold any new project-agnostic learnings back
into CLAUDE.md.
into the matching themed file under the skill's references/gotchas/ (and add a row to
references/gotchas-index.md).
1 change: 1 addition & 0 deletions .claude/skills
5,381 changes: 25 additions & 5,356 deletions CLAUDE.md

Large diffs are not rendered by default.

192 changes: 192 additions & 0 deletions skills/python-project-porting/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
---
name: python-project-porting
description: Playbook for adding a package's riscv64 wheel build to the RISE python-wheels repo (published to pypi.riseproject.dev). Use when porting a Python package to riscv64, authoring or debugging a .github/workflows/build-<pkg>.yml, wiring cibuildwheel/maturin/setuptools-rust builds for riscv64, triaging manylinux_riscv64 build or test failures, deciding whether a package is feasible to port, or handling the PR/publish/licensing steps for a python-wheels contribution.
license: See repository LICENSE
compatibility: Designed for Claude Code in the riseproject-dev/python-wheels repo. Uses git, gh (needs workflow and project token scopes), and docker/podman for local QEMU + aarch64 validation. Builds run on ubuntu-24.04-riscv self-hosted runners.
metadata:
source: Distilled from the python-wheels porting playbook (originally CLAUDE.md), generalized from the protobuf port onward.
---

# Python project porting (riscv64 wheels)

Guidance for adding a new package's riscv64 wheel build to this repo and publishing it to
`pypi.riseproject.dev`. The 186 hard-won "gotchas" are split by theme under
[`references/gotchas/`](references/gotchas/) and routed by
[references/gotchas-index.md](references/gotchas-index.md) — **skim the index before you start,
then read the themed file for the step you're on.** Many gotchas each cost a full CI cycle
(minutes for a simple package, hours for one that compiles a large C++ world).

This SKILL.md is the navigator: the core loop and the rules that are always in force. Depth
lives in the reference files linked at the bottom — load them as the task calls for them.

## What this repo does

Builds riscv64 wheels for packages that don't ship them on public PyPI, and
publishes them to `pypi.riseproject.dev`. Each package gets a
`.github/workflows/build-<pkg>.yml`. Wheels are consumed on `ubuntu-24.04-riscv`
self-hosted runners.

Four structural goals (from the [development guide](https://pypi.riseproject.dev/python-wheels/development.html)):
1. give users a simple index to install riscv64 wheels from;
2. build them with workflows that **closely mirror each upstream project's own CI**,
narrowed to riscv64;
3. carry tooling that tracks upstream releases, automates version bumps, and makes
deprecation easy once upstream ships riscv64 itself;
4. serve as evidence to upstream maintainers that riscv64 support is cheap to add.

Goal 2 is the one that constrains daily work: **a workflow that diverges from upstream's
for no reason is a defect**, because these files are meant to be handed to upstream as a
working precedent.

## Non-negotiable rules

These are low-freedom guardrails — each has been reinforced by a revert, a rejected commit,
or a repeated ask. Follow them exactly; the reference files explain the why.

- **Never write outside the repository.** Worktrees go in `.claude/worktrees/<pkg>`, scratch
files in `.git/pw-scratch/<pkg>`, local lock state in `.git/pw-locks/`. No files in `$HOME`,
`~/.local/bin`, `/tmp`, or sibling directories, and **no installing software** on the host
(brew/apt/dnf/npm/pip). If you think you need either, ask first.
- **A port adds files only under `.github/workflows/` and `patches/<pkg>/<version>/`.** Never
create a `ci/` directory or any helper script, Dockerfile, or test file elsewhere — not for
a build step, not for a smoke test, not "just this once." Anything a job needs that is not a
patch is **written by the workflow at run time** from a `run:` heredoc (gotcha 7). Treat a
new top-level path as a hard stop, not a judgement call. Full rules:
[references/environment-and-auth.md](references/environment-and-auth.md).
- **Commit identity is `Ludovic Henry <git@ludovic.dev>`** and is already configured. Never
pass `-c user.email`/`user.name` or set `GIT_AUTHOR_*`/`GIT_COMMITTER_*` (in particular not
the address from your own session context — it differs). A `pre-commit` hook rejects any
other identity and any workflow adding `BUILD_VERBOSITY`; if it fires, fix the command,
don't bypass the hook.
- **Both workflow triggers, always** (`workflow_dispatch` **and** `pull_request: paths`). The
`pull_request` trigger is the only thing that registers a new workflow with GitHub; without
it dispatch and `Trigger:` lines both 404 (gotchas 45/54). Shipping `workflow_dispatch`
alone is why #364 was reverted by #391.
- **Default to NO comments in workflows** — they are read as reference. One "why" line only for
a genuine non-obvious deviation; never narrate standard steps.
- **Never set `CIBW_BUILD_VERBOSITY`** — drop it if you inherit it from a template.
- **Start from upstream's own build/test workflow, then delete** everything that isn't Linux
glibc/musl, and only then apply the riscv64 changes. See
[references/workflow-anatomy.md](references/workflow-anatomy.md).
- **Pushing workflow files needs `workflow` scope** on the gh token; the post-merge steps need
`project` scope. Refresh with `gh auth refresh -h github.com -s workflow` (and `-s project`).

## Working process

Given a package to port, the loop is always the same (project-specific inputs —
name, repo, version, upstream build docs — come from the invoking prompt):

1. Branch `<pkg>` from `origin/main` and work in a dedicated git worktree, created at
**`.claude/worktrees/<pkg>`** inside this repo (locally ignored via `.git/info/exclude`).
Never put a worktree — or anything else — outside the repository.
2. Add `.github/workflows/build-<pkg>.yml` following the playbook below and
[references/workflow-anatomy.md](references/workflow-anatomy.md).
3. Validate locally (gotcha 9), then push to `origin` and open a PR. The
`pull_request: paths` trigger is what produces the **first** run of a new workflow, and
that run is what registers it with GitHub. **A `Trigger:` line alone cannot start a new
package's build** — dispatch resolves the workflow through the registry and answers
`HTTP 404` until a `pull_request` run exists (gotcha 54). Once the workflow is
registered (or already on `main`), a `Trigger: <pkg>:<tag>` line in the **PR
description** — one per version, `Trigger: numpy:v2.5.1` — lets `pr-trigger.yml` build
a different version without editing the workflow.
4. Watch CI, triage failures, iterate until every matrix job is green and the
`publish` job dry-runs cleanly.
5. When the wheels build and tests pass, reply to any review threads, then
record reusable, project-agnostic learnings back into this skill — add them to the
matching themed file under [`references/gotchas/`](references/gotchas/) as the next unused
number, and add a row to [references/gotchas-index.md](references/gotchas-index.md) (see the
index header for the numbering rules).

## Porting playbook (do these in order)

1. **Read the upstream project's own build + release docs first.** Find how *they*
build their wheels and sdist. Don't assume `python -m build` works (see gotcha 1).

2. **Fetch and inspect the real PyPI sdist** to learn its layout and whether it's
self-contained:
```
pip download <pkg>==<ver> --no-binary :all: --no-deps -d /tmp/x
tar tzf /tmp/x/<pkg>-*.tar.gz | head -50
```
Then try to build a wheel from it locally (works even on x86/aarch64 — proves
portability before you burn a riscv CI cycle):
```
pip wheel /tmp/x/<pkg>-*.tar.gz --no-deps --no-build-isolation -w /tmp/out
```
If that succeeds with no special toolchain, the riscv bdist job can be minimal.

3. **Decide where the sdist comes from** (only relevant for the sdist→bdist shape).
**Always build the sdist yourself from an upstream checkout** — never wire the
prebuilt PyPI sdist in as the CI build input (fetch it only for the local
*inspection* in step 2). Use `python -m build --sdist` from the checkout if the
project supports it; otherwise whatever the project uses (protobuf: Bazel
`//python/dist:source_wheel`). Heads-up for Rust/maturin projects: a
locally-built sdist may pin dependencies *differently* than the released PyPI
sdist — see gotcha 10.

4. **Map the git tag to the Python version** (see gotcha 3). Take the tag as the
workflow input; derive `package_version` from the built sdist filename.

5. **Identify native deps** the bdist needs at build time. Three cases: (a) none
bundled → add `CIBW_BEFORE_BUILD` to build them in-container (cffi builds libffi
that way); (b) the sdist bundles its C sources (protobuf bundles upb/utf8_range)
→ no before-build needed; (c) the dep is another Python wheel we already ship →
`pip install` it from our registry in `CIBW_BEFORE_BUILD` (see gotcha 17).

6. **Wire up real testing** — mirror how upstream tests its wheels (gotcha 6).

7. **Validate locally, then push** (gotcha 9). Open a PR; the `pull_request` path
trigger runs CI. Watch, triage, iterate.

## Reference files

Load these on demand — they are one level deep from here.

- **[references/workflow-anatomy.md](references/workflow-anatomy.md)** — the anatomy of a
`build-<pkg>.yml`: standard triggers, UV/env vars, the interpreter matrix, `setup-uv`
vs `setup-python`, the two build shapes (sdist→bdist, build-from-checkout), driving the
container yourself, and the shared `publish` job.
- **[references/gotchas-index.md](references/gotchas-index.md)** — the router for all 186
gotchas: a topic→file table and the full number→file lookup. Start here when you have a
symptom but not a number, or to resolve a "gotcha N" citation to its file. The gotchas
themselves live in [`references/gotchas/`](references/gotchas/), split by theme (listed
under "Finding the right gotcha" below).
- **[references/patching-and-licensing.md](references/patching-and-licensing.md)** — when a
patch is justified, the `patches/<pkg>/<version>/` mechanics, the five `Upstream-Status:`
types, and licence/GPL-sources compliance (the `gpl_sources` job).
- **[references/pr-and-publishing.md](references/pr-and-publishing.md)** — the post-merge
publish/issue/project steps, the PR description template (use it verbatim), and the PR/CI
conventions (draft status, no hard-wrapping, dry-run checks).
- **[references/environment-and-auth.md](references/environment-and-auth.md)** — where files
may and may not go, commit identity, token scopes, and remotes.

### Finding the right gotcha

The 186 gotchas are split into themed files under `references/gotchas/`. **Read the one file
that matches your current step** rather than loading them all — each file opens with an
`## In this file` list of its entries. Three ways in:

1. **By symptom (no number yet)** — jump to the file whose theme matches, or grep the whole
set: `grep -rn '<term>' references/gotchas/`. The map:
- `feasibility-and-triage.md` — is this worth porting? all-`py3-none-*` wheels, vendored blobs, conda/CUDA-blocked deps, source-only distros.
- `sdist-source-and-versioning.md` — where the sdist comes from; git-tag≠version; dirty-tree/`setuptools_scm`/`tag_build` version poisoning; no-tag upstreams.
- `cibuildwheel-matrix-and-abi3.md` — `{project}` vs `{package}`, the interpreter matrix, abi3 tag collapse, the `CIBW_ENVIRONMENT` cascade, YAML folding, heredocs.
- `rust-maturin-and-pyo3.md` — maturin/setuptools-rust/pyo3, rustup targets, `MATURIN_PEP517_ARGS`, cargo features, cross-compile pre-flight.
- `native-build-bazel-and-drivers.md` — Bazel bootstrap, driving the container yourself, rules_python, per-interpreter loops, vcpkg-image replacement.
- `manylinux-image-and-toolchain.md` — Rocky 10 packages, EPEL/CRB, GCC/binutils versions, RVV/SIMD gates, perl/gconv.
- `native-deps-and-linking.md` — build-once C++, the dep-wheel pattern, static-vs-shared, auditwheel `--exclude`, `patchelf` RPATH, missing symbols.
- `compiled-vs-pure-detection.md` — is the wheel actually compiled? the `.so` proof, mislabeled pure wheels, the require-extension knob, free-threading declaration.
- `dependencies-and-registry.md` — checking pypi.riseproject.dev, per-interpreter coverage, `PIP_ONLY_BINARY`, matrix trimming, test-venv rebuilds.
- `build-tool-drift-and-pins.md` — Cython/setuptools/numpy version drift, `--no-build-isolation`, `PIP_BUILD_CONSTRAINT`, a published wheel breaking another package.
- `testing-and-shadowing.md` — tests importing the checkout instead of the wheel: `CIBW_TEST_SOURCES`, rootdir shadowing, renaming the staged package, in-container build products.
- `pytest-config-servers-and-selection.md` — staging the pytest ini (`addopts`/`markers`/`log_level`), servers in `before-test`, `-W error`, choosing which tests run.
- `test-failures-and-flakes.md` — a job fails/segfaults/flakes: refcount bugs, xdist crashes, slow-runner races, libgomp/OpenMP, numeric divergence, native backtraces.
- `licensing-and-gpl.md` — vendored-dep LICENSE files, PEP 639 vs setuptools globs, REUSE `LICENSES/`, the `gpl_sources` job, SBOMs.
- `local-validation-and-rehearsal.md` — local `pip wheel`, QEMU, the aarch64 rehearsal and its traps, `pip download` resolution checks.
- `pr-ci-and-maintainer.md` — registering a new workflow, `Trigger:` lines, action-SHA pins, maintainer holds/cancellations, post-merge publish.
2. **By number** — a "gotcha N" citation (in these files or in workflow comments). Find its
file in the number→file table of [references/gotchas-index.md](references/gotchas-index.md),
then `grep -n '^N\. ' references/gotchas/<file>`.
3. **Numbers are stable IDs, not positions** — they are *not* contiguous, and four (33, 55,
56, 57) are **reused** with different content across two themes each. When a citation is
ambiguous, the topic decides which one; the index marks the reused rows.
33 changes: 33 additions & 0 deletions skills/python-project-porting/references/environment-and-auth.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## Environment / auth notes

- **Never write outside the repository.** Worktrees go in `.claude/worktrees/<pkg>`, scratch
files in `.git/pw-scratch/<pkg>`, local lock state in `.git/pw-locks/`. No files in `$HOME`,
`~/.local/bin`, `/tmp`, or sibling directories, and **no installing software** on the host
(brew/apt/dnf/npm/pip). If you think you need either, ask first.
- **A port adds files only under `.github/workflows/` and `patches/<pkg>/<version>/`.**
Never create a `ci/` directory, and never add a helper script, Dockerfile, or test file
anywhere else in the repo — not for a build step, not for a smoke test, not "just this
once" because the content is long. Anything a job needs that is not a patch is **written
by the workflow at run time** from a `run:` heredoc (gotcha 7): into `$RUNNER_TEMP` for a
docker build context, or into the upstream checkout for something cibuildwheel must carry
into the container (`{project}/<name>` then names it). This has been asked for three times
— `ci/memray`, `ci/pyogrio`, `ci/pyroscope-io` were each removed after the fact — so treat
a new top-level path as a hard stop, not a judgement call. If a file genuinely cannot be
inlined, ask before writing it.
- Reproduce the file byte for byte when inlining: quote the heredoc marker (`<<'EOF'`) so
nothing expands, and diff the extracted block against the original before pushing.
- `cat >` drops the executable bit, so a script invoked by path needs `chmod +x`; one
invoked as `bash <script>` does not. Getting this wrong costs a full image build.
- **Commit identity is `Ludovic Henry <git@ludovic.dev>`** and is already configured. Never
pass `-c user.email`/`-c user.name` or set `GIT_AUTHOR_*`/`GIT_COMMITTER_*` — in particular
do not use the user's address from your own session context, which is a *different*
address. A `pre-commit` hook rejects any other identity (and any workflow adding
`BUILD_VERBOSITY`); if it fires, fix the command, don't bypass the hook.

- **Pushing workflow files needs `workflow` scope** on the gh token, else the push is
rejected ("refusing to allow an OAuth App to create or update workflow … without
`workflow` scope"). Fix: `gh auth refresh -h github.com -s workflow` (interactive).
- `origin` (`riseproject-dev/python-wheels`) is canonical; there is no separate
`upstream` remote. Branch from `origin/main`.
- Use `gh` extensively for anything requiring access to GitHub.

Loading
Loading