Add smithy-checksums package with Rust-backed CRC32 and CRC32C by jonathan343 · Pull Request #867 · smithy-lang/smithy-python

jonathan343 · 2026-10-25T03:28:13Z

Note

This is the first of three PRs. Follow-ups add CRC64NVME and then XXHash64, XXHash3, and XXHash128 to this package.

Summary

S3's flexible checksums support CRC32, CRC32C, CRC64NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants. The standard library only covers CRC32, SHA, and MD5. botocore gets the rest from awscrt, which is an optional dependency, so those algorithms are unavailable without it. smithy-python has no checksum implementations yet, so generated clients can't support flexible checksums.

This PR adds a smithy-checksums package, starting with CRC32 and CRC32C. Each algorithm has a Rust implementation (PyO3 and the crc-fast crate, which the AWS SDK for Rust also uses) and a pure-Python fallback that gives identical results. SHA-1, SHA-256, SHA-512, and MD5 come from hashlib through the same new() entry point. The fallback means the package installs everywhere, so clients can take a hard dependency on it. Platforms with native wheels get fast checksums.

Example

import base64

import smithy_checksums

checksum = smithy_checksums.new("CRC32C")
checksum.update(b"Hello ")
checksum.update(b"world")
print(checksum.name, checksum.digest().hex(), base64.b64encode(checksum.digest()))
print("native:", smithy_checksums.HAS_NATIVE_EXTENSION)

print(type(smithy_checksums.new("SHA256")))

try:
    smithy_checksums.new("crc16")
except ValueError as error:
    print(error)

Native wheel

crc32c 72b51f78 b'crUfeA=='
native: True
<class '_hashlib.HASH'>
Unsupported checksum algorithm: 'crc16'

Pure-Python wheel

crc32c 72b51f78 b'crUfeA=='
native: False
<class '_hashlib.HASH'>
Unsupported checksum algorithm: 'crc16'

Decisions

One package, two build backends. An in-tree build backend uses maturin when cargo is available and hatchling (pure Python) otherwise. A separate native package couldn't be a safe hard dependency, since there's no way to require it only where a wheel exists.
Fall back only when Rust is missing. Without Rust, source builds quietly produce the pure-Python wheel. If Rust is installed but too old or the build fails, the build stops and suggests SMITHY_PYTHON_NATIVE_BUILD=never.
Native wheels ship third-party licenses. They contain compiled MIT and Apache-2.0 crates, so a generated LICENSE-THIRD-PARTY is included and CI checks it's current.

Performance

Throughput hashing 1 MiB per update() call. Each figure is the median of 15 rounds,
measured on an EC2 m7i.xlarge (Intel Xeon Platinum 8488C, 4 vCPUs) with CPython 3.12,
pinned to one CPU, with input that isn't 64-byte aligned:

Algorithm	Native (Rust)	awscrt 0.32	Pure Python
CRC32	68 GB/s	2.8 GB/s	5.0 GB/s (zlib)
CRC32C	92 GB/s	66 GB/s	109 MB/s

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

kuhe · 2026-10-26T21:42:59Z

+trait State: Clone + Send {
+    fn new() -> Self;
+    fn update(&mut self, data: &[u8]);
+    fn digest(&self) -> Vec<u8>;


This almost implies digest can be variable length.

The digest length can be set as a compile time property of the trait. What do you think? Makes it too unreadable?

kuhe · 2026-10-27T20:18:00Z

+def build_sdist(sdist_directory: str, config_settings: _ConfigSettings = None) -> str:
+    return import_module("hatchling.build").build_sdist(
+        sdist_directory, config_settings
+    )


this file looks like it could be part of a generic rust-extension builder/validator

Agreed, most of this file isn't checksum-specific. Only a few details like the package name.

Pulling it out is a bit more than moving the file though. Python packaging requires a package's build backend to live inside that package's directory, so packages can't share one from a common folder in the repo. We'd either need to publish the builder as its own package or keep a synced copy in each package.

Since this is our only Rust package right now, I'd like to hold off until we add a second one. Then we can decide which way to go based on two real use cases.

kuhe · 2026-10-27T20:25:35Z

@@ -0,0 +1,135 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.


how about calling this folder src-ext or src-rust?

are we going to have tests-ext or tests-rust too? or colocated with the source?

I personally lean towards keeping rust/. Happy to discuss further if there are good reasons to rename or you feel strongly.

For tests, Rust ones would live next to the code, but I'm not planning any right now. The Python tests already check the Rust and pure-Python versions against each other and our rust code is just wrapping the crc-fast crate.

that's fine, no information is lost and it's not confusing.

It only looks asymmetrical to have "src" and "rust" when the rust code is also source, and src is not "python".

Adds the smithy-checksums package, which provides the checksum algorithms Smithy clients need for flexible checksums. CRC32 and CRC32C are backed by a native Rust extension (PyO3, crc-fast) with a pure-Python fallback, and SHA-1, SHA-256, SHA-512, and MD5 come from hashlib through new(). Wheels are built by an in-tree PEP 517 backend: maturin when a Rust toolchain is available, hatchling (pure Python) otherwise. Source distributions are always built with hatchling. SMITHY_PYTHON_NATIVE_BUILD (auto, always, never) overrides the choice. If Rust is present but too old, or the native build fails, the build stops with instructions instead of silently falling back. Native wheels target abi3 for 3.12+, cp314t, and abi3t for 3.15t+. Native and pure-Python checksum objects are safe to share between threads. Repository setup: - CI builds the native extension in the main matrix, checks Rust formatting, lints, dependencies (cargo-deny), and third-party license attribution (cargo-about), builds and tests native wheels on Linux, macOS, and Windows, and tests the pure-Python wheel and an sdist install without Rust. - Adds make targets check-rust, lint-rust, and third-party-licenses. - License header checks cover .rs and .pyi files. - Dependabot updates Cargo dependencies.

Pins the toolchain used for development and CI in rust-toolchain.toml, so a new stable release can't introduce clippy lints that fail CI without a change here. Builds from a source distribution are unaffected and use the installed toolchain. CI now caches Rust builds with Swatinem/rust-cache, saving caches only on develop, and adds a job that checks each native extension builds with the minimum Rust version its Cargo.toml declares. The new jobs also get names and version comments on their pinned actions.

The byte-at-a-time table loop runs about a dozen bytecodes per byte, which caps it at roughly 15 MB/s however it's arranged; slicing-by-8 and 16-bit tables only reach 20-35 MB/s. A CRC without its initial value and final XOR is linear over GF(2), so each bit of the CRC of a fixed-size chunk is the parity of the chunk's bits under a fixed mask. Input is now buffered into 16 KiB chunks, each read as one integer, and every CRC bit is computed with an AND and bit_count(), which do the per-byte work in C. The masks are built lazily by doubling, in a few milliseconds. Remainders under 1 KiB still use the table. CRC32C goes from about 16 MB/s to about 210 MB/s, whatever the update size.

maturin packages everything under python-source, so a native wheel built from a source tree that's been imported from picked up its __pycache__ directories, including bytecode for modules that no longer exist. The hatchling targets already exclude build artifacts; this gives maturin the same treatment.

The native extension read read-only buffers in place through an unsafe slice, on the grounds that nothing could change them. The read-only flag only stops this reader from writing, though: a read-only memoryview of a bytearray still changes when another thread writes to the bytearray, which can happen while the interpreter is detached. Only bytes is immutable, so it's the only input that's still hashed in place; every other buffer is copied. Copying runs at about 40 GB/s, against about 100 GB/s in place. Also: - State::digest returns a fixed-size array, given by an associated Digest type, instead of a Vec. - Import pyo3 items explicitly instead of through the prelude.

"Backend" is the PEP 517 term for the build backend, and it doesn't suggest that the native extension is optional. The module that picks the native or pure-Python implementations is now _private/selection.py, and the public classes' base is _BaseChecksum.

Handlebars escapes {{...}} for HTML, so LICENSE-THIRD-PARTY had " and ' in place of quotes. The template now uses {{{...}}} for every field, since the output is plain text.

Renaming _private/backend.py to _private/selection.py moved its imports out of order.

Strided byte views, such as memoryview(b"abcdefgh")[::2], were hashed while hashlib.sha256() rejects them, and views with other formats failed with a TypeError from the cast instead. new() returns hashlib objects for the SHA and MD5 algorithms, so the same update() call worked for some algorithms and raised for others. Every non-contiguous buffer now raises the same BufferError hashlib does.

The pure-Python update() copied every memoryview to bytes before hashing, which as_byte_view had just produced without copying. zlib.crc32 and the chunked CRC read any byte buffer, so the view is passed through, and the as_bytes helper is gone.

jonathan343 mentioned this pull request Oct 25, 2026

Add CRC64NVME to smithy-checksums #868

Draft

jonathan343 added this pull request to stack #869 October 25, 2026 04:26

jonathan343 mentioned this pull request Oct 25, 2026

Add XXHash64, XXHash3, and XXHash128 to smithy-checksums #870

Draft

jonathan343 force-pushed the feat/smithy-checksums branch 2 times, most recently from d6dc4fc to 7efa9ae Compare October 27, 2026 19:15

jonathan343 marked this pull request as ready for review October 27, 2026 20:06

jonathan343 requested a review from a team as a code owner October 27, 2026 20:06

kuhe reviewed Oct 27, 2026

View reviewed changes

stobrien89 requested changes Oct 28, 2026

View reviewed changes

Comment thread packages/smithy-checksums/rust/lib.rs Outdated

Comment thread packages/smithy-checksums/about.hbs Outdated

Comment thread packages/smithy-checksums/pyproject.toml

jonathan343 added 10 commits October 29, 2026 11:05

Describe what smithy-checksums adds in its changelog entry

0a78d46

Make the Checksum protocol runtime-checkable

4ff33b9

Stop HTML-escaping the third-party license file

2f880d8

Handlebars escapes {{...}} for HTML, so LICENSE-THIRD-PARTY had " and ' in place of quotes. The template now uses {{{...}}} for every field, since the output is plain text.

Sort the smithy_checksums imports

34c1c82

Renaming _private/backend.py to _private/selection.py moved its imports out of order.

jonathan343 force-pushed the feat/smithy-checksums branch from ad3b09e to 34c1c82 Compare October 29, 2026 15:09

jonathan343 added 2 commits October 29, 2026 11:42

jonathan343 requested review from kuhe and stobrien89 October 29, 2026 17:25

kuhe approved these changes Oct 29, 2026

View reviewed changes

Navigation Menu

Repository navigation

Add smithy-checksums package with Rust-backed CRC32 and CRC32C - #867

Add smithy-checksums package with Rust-backed CRC32 and CRC32C#867
jonathan343 wants to merge 12 commits into
developfrom
feat/smithy-checksums

jonathan343 commented Oct 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

kuhe Oct 26, 2026

Uh oh!

Uh oh!

Uh oh!

kuhe Oct 27, 2026

Uh oh!

jonathan343 Oct 29, 2026

Uh oh!

Uh oh!

kuhe Oct 27, 2026

Uh oh!

kuhe Oct 27, 2026

Uh oh!

jonathan343 Oct 29, 2026

Uh oh!

kuhe Oct 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,135 @@
		// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Conversation

jonathan343 commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example

Decisions

Performance

Uh oh!

Uh oh!

kuhe Oct 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kuhe Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan343 Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kuhe Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

kuhe Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

jonathan343 Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

kuhe Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jonathan343 commented Oct 25, 2026 •

edited

Loading