ci: add riscv64 manylinux wheels with native RISE runners - #234
Conversation
|
Correction: the linked CI run was from the fork's own scheduled build, not RISE runners. However, the build succeeds on native riscv64. build time: ~2m39s (https://github.com/gounthar/cffi/actions/runs/23103252300). Also verified on BananaPi F3 hardware. The PR's native runner approach (ubuntu-24.04-riscv) is ready for when RISE runners are installed on the cffi org. |
|
Successful riscv64 build on native RISE runners (ubuntu-24.04-riscv): https://github.com/gounthar/cffi/actions/runs/23706627020 Build, wheel verification, and release all passed. Build time under 3 minutes on native hardware. |
2399473 to
5a09612
Compare
| - { spec: cp314t-manylinux_s390x, arch: s390x, omit: ${{ env.skip_slow_jobs }} } | ||
|
|
||
| # riscv64 manylinux | ||
| - { spec: cp39-manylinux_riscv64, arch: riscv64, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs }} } |
There was a problem hiding this comment.
I guess we can remove ${{ env.skip_slow_jobs }} now it runs on native runners
There was a problem hiding this comment.
Good catch. Removed skip_slow_jobs from all riscv64 entries. Kept skip_ci_redundant_jobs for the intermediate Python versions, matching the pattern of the other architectures.
There was a problem hiding this comment.
Also other non emulated targets don't specify test_args: '{package}/src/c' which seems to run only a subset of the test suite. Maybe we should remove it here also?
I started an action on my fork, and with the full test suite, riscv64 build takes 34 minutes (see https://github.com/justeph/cffi/actions/runs/23733528422/job/69141241075) compared to x86 which succeeded in 4 minutes..
For what it worth, this is still better than the 1 hour with emulation :) https://github.com/justeph/cffi/actions/runs/21716002341/job/62632136017
So makes me wondering if we should remove the skip_slow_jobs or running the full test suite?
But then, do we need native runners? Emulated riscv64 job with test_args: '{package}/src/c' runs in 5 minutes https://github.com/justeph/cffi/actions/runs/21716002341/job/62632136043 when it runs in 6 minutes on native runner https://github.com/justeph/cffi/actions/runs/23731763287/job/69126983344...
Maybe that will be a decision for @DimitriPapadopoulos or @mattip (or someone else 😅 ) ?
There was a problem hiding this comment.
Makes sense. Removed test_args so riscv64 runs the full test suite like the other native targets. 34 minutes is reasonable for a native build, and much better than the hour under emulation.
433c715 to
84162b7
Compare
|
@mattip @nitzmahone @DimitriPapadopoulos pinging you on this. Any chance to add riscv64 support?
I guess this PR fixes the qemu issue? any chance to merge it? |
84162b7 to
77cb165
Compare
|
Hi all, Is there a timeline where we might get this PR merged (after conflicts are resolved)? |
|
I think if qemu is good enough for NumPy it should be good enough for CFFI. |
77cb165 to
4051ba9
Compare
riscv64 is the last mainstream Linux architecture without cffi wheels on
PyPI, so anything downstream of it has to compile from sdist. That bites
hardest in the maturin/Rust-to-Python toolchain, where cffi and patchelf
are the two remaining source builds.
The matrix entries follow the x86_64/aarch64 shape rather than the
s390x/ppc64le one: no test_args trimming and gated on
skip_ci_redundant_jobs instead of skip_slow_jobs, because these run on
real hardware and are not slow. Note that anything gated on
skip_slow_jobs never runs on a pull_request or push at all, since
ci.yaml sets that variable true for both events and dynamatrix drops the
entry.
Two riscv64-specific wrinkles in the linux job:
- docker/setup-qemu-action is skipped, since the host is already
riscv64 and there is nothing to emulate.
- actions/setup-python publishes no riscv64 builds, so cibuildwheel is
driven by an interpreter that uv fetches instead. The alternative,
apt-installing a system python, ties the tool interpreter to
whatever the runner image ships.
manylinux_2_39 is the first manylinux spec with riscv64 images, hence
the differing default for CIBW_MANYLINUX_RISCV64_IMAGE.
The runner label ubuntu-24.04-riscv comes from the RISE runners GitHub
App (https://github.com/apps/rise-risc-v-runners), which the
organisation would need installed before these jobs can pick up a
machine. Builds on the riscv64 matrix entries proposed by @justeph in
python-cffi#227.
Signed-off-by: Bruno Verachten <gounthar@gmail.com>
4051ba9 to
8c3035c
Compare
|
@gounthar would you like to submit a QEMU version of your PR, or should I? |
|
As you please. 😉 |
4 participants
Summary
Add riscv64 manylinux wheel builds using native RISE RISC-V runners instead of QEMU emulation.
Based on the excellent work by @justeph in #227. This PR builds on that approach with native runner support for significantly faster builds.
Changes
ubuntu-24.04-riscvnative runner (no QEMU needed)docker/setup-qemu-actionfor riscv64CIBW_MANYLINUX_RISCV64_IMAGEdefaulting tomanylinux_2_39(first manylinux with riscv64)Why native runners over QEMU
QEMU emulation (as in #227) works but is slow. Native RISE runners provide real riscv64 hardware at no cost to the project, with build times comparable to other architectures.
Evidence
Credits
Supersedes #227