Skip to content

feat(python): pyrind, rind's API for Python - #459

Merged
kixelated merged 13 commits into
mainfrom
quest/m2/adoption/pyrind
Nov 7, 2026
Merged

kixelated merged 13 commits into
mainfrom
quest/m2/adoption/pyrind

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Python can't use rind. Quest: quest/m2/adoption/pyrind.md. Goal: pip install pyrind gives rind's own API (AEADs, hashes, HMAC, HKDF, PBKDF2, X25519, Ed25519, ECDSA and ECDH, RSA, ML-KEM) from wheels for every platform rind supports, and a published per-call comparison against cryptography, hashlib, and hmac.

Approach

  • Crate. New rind-python crate (publish = false), built by pyo3 0.30 and maturin as the pyrind package.

    • Each rind module is a Python submodule: pyrind.aead, .agreement, .cpu, .digest, .hkdf, .hmac, .kem, .pbkdf2, .rsa, .signature (with .sign and .verify).
    • Algorithms are typed constants. Legacy algorithms (SHA-1, MD5) live in each module's legacy submodule, as in rind.
  • Errors and import. Every rind error is a pyrind.Error subclass. KeyRejected carries rind's reason.

    • Importing raises ImportError naming any CPU feature rind requires that is missing.
  • Memory. Every rind value is boxed inside its Python object. pyo3 places a class's contents in memory CPython aligns to only 16 bytes, and ML-KEM's AVX2 loads need 32 (PyO3/pyo3#6526).

    • This was a real crash: about 1 in 40 test runs segfaulted in mlkem_basemul_k3. A regression test now holds 32 keys at once.
    • Wiping. The binding wipes nothing itself: every key or keyed state it holds is a rind value, wiped by rind's own Drop exactly when rind wipes. The published wheels build rind with its default features and forward no feature, as rind's other adapters do. On today's main that build wipes on drop; once moq-dev/rind#461 makes the wipe opt-in, the wheels skip it, as rind's default build does. A source build can turn it on with maturin build --features rind/wipe after feat: wiping keys on drop is opt-in, under a wipe feature #461.
  • Inputs, outputs, and the GIL.

    • Any bytes-like input is accepted. Non-bytes buffers are copied first, so no other thread can mutate them mid-call.
    • Outputs are written straight into a new bytes, with no zero-fill pass.
    • RSA keygen, sign, and decrypt, and PBKDF2, release the GIL.
  • Wheels (python.yml): abi3 for CPython 3.11 and later, on these targets:

    • Linux glibc 2.17: x86_64, aarch64, armv7.
    • Linux musl: x86_64, aarch64.
    • macOS: x86_64, arm64.
    • Windows: x64, arm64.

    There is also an sdist. Every runner that can load its wheel runs the unit tests, Wycheproof, the README example, and mypy stubtest. The workflow runs nightly, on dispatch, and from release.yml.

  • Release.

    • release-plz versions rind-python from git tags (git_only), so the next release PR carries 0.1.0 with a CHANGELOG.
    • After just release, sh/pyrind-unpublished.sh reports a tag that PyPI lacks.
    • release.yml then builds the wheels via python.yml and publishes them by trusted publishing, from the existing publish environment, with no token.
  • Checks. A py module in just check builds the wheel and runs the tests and stubtest whenever rind-python/ changes.

Impact

  • New crate and PyPI package. No change to rind's API or code.
  • The dev shell gains maturin, cryptography, and mypy.
  • admin/scan-license now fails if rind-python/LICENSE drifts from rind/LICENSE.

Per-call numbers

Measured on a held host and published in rind-python/README.md:

On an Intel Core i7-8700 (Coffee Lake, 2017), Linux 6.18, CPython 3.14.8,
against cryptography 50.0.2 on OpenSSL 3.5.9, one call at a time with its
inputs in hand, fastest of seven runs:

Operation pyrind cryptography hashlib, hmac cryptography ÷ pyrind
SHA-256, 21 B 262 ns 1.79 µs 526 ns 6.8x
HMAC-SHA256, 21 B 401 ns 2.54 µs 1.87 µs 6.3x
HKDF-SHA256, 32 B key 1.68 µs 5.07 µs - 3.0x
AES-128-GCM seal, 32 B 237 ns 667 ns - 2.8x
AES-128-GCM open, 32 B 218 ns 690 ns - 3.2x
AES-128-GCM seal, 1350 B 507 ns 924 ns - 1.8x
AES-128-GCM open, 1350 B 472 ns 926 ns - 2.0x
AES-128-GCM seal, 16384 B 3.04 µs 3.63 µs - 1.2x
AES-128-GCM open, 16384 B 3.02 µs 3.59 µs - 1.2x
ChaCha20-Poly1305 seal, 32 B 315 ns 858 ns - 2.7x
ChaCha20-Poly1305 open, 32 B 293 ns 894 ns - 3.0x
ChaCha20-Poly1305 seal, 1350 B 930 ns 1.54 µs - 1.7x
ChaCha20-Poly1305 open, 1350 B 901 ns 1.59 µs - 1.8x
ChaCha20-Poly1305 seal, 16384 B 7.13 µs 7.66 µs - 1.1x
ChaCha20-Poly1305 open, 16384 B 7.03 µs 7.72 µs - 1.1x
X25519 agree 19.93 µs 30.44 µs - 1.5x
ECDH P-256 agree 37.44 µs 62.57 µs - 1.7x
Ed25519 sign, 21 B 7.10 µs 28.96 µs - 4.1x
Ed25519 verify, 21 B 29.07 µs 93.03 µs - 3.2x
ECDSA P-256 sign, 21 B 14.41 µs 26.14 µs - 1.8x
ECDSA P-256 verify, 21 B 39.42 µs 60.53 µs - 1.5x
RSA-2048 PKCS#1 sign, 21 B 470.21 µs 483.67 µs - 1.0x
RSA-2048 PKCS#1 verify, 21 B 11.00 µs 20.14 µs - 1.8x
ML-KEM-768 encapsulate 3.22 µs 22.16 µs - 6.9x
ML-KEM-768 decapsulate 7.78 µs 33.81 µs - 4.3x

pyrind wins most where protocols spend their calls: small messages and per-message keys. On 16 KiB the two converge, and RSA signing is the same exponentiation in both.

Choices made without asking

  • API shape. Submodules mirror rind's modules. aad and label are keyword-only, so they can't be swapped with the plaintext. hkdf.Prk.expand returns an aead.Key or hmac.Key directly, so derived key bytes never reach Python.
  • Left out. cipher, key_wrap, constant_time, rand, quic, sign_digest, custom PSS salts, and Dit. These are protocol plumbing or covered by Python itself (hmac.compare_digest, os.urandom), and can be added on request.
  • Interpreters.
    • The abi3 floor is CPython 3.11, the first release with the buffer protocol in the limited API. 3.10 is past end of life.
    • PyPy and free-threaded CPython can't load abi3 wheels, so they build the sdist, which needs a Rust toolchain. There is no pure-Python fallback, per the quest.
    • No iOS or Android wheels.
  • Two safety-relevant implementation choices. The Box rule above, and one unsafe helper (bytes_with) that fills a new bytes without the zero-fill pass. Its SAFETY comment states the invariant.
  • Release. Reuses the publish environment rather than adding a pypi one. release-plz git_only, since the crate is never on crates.io.
  • Check scope. The py check covers rind-python/ changes only. A rind change that breaks the bindings shows up in the nightly python workflow.
  • Windows on Arm. pyca/cryptography's last win_arm64 wheel, 46.0.3, lacks ML-KEM. That runner therefore skips the comparisons against pyca and runs Wycheproof, the README example, and stubtest.

Validation

  • just check origin/main passes on fade49a, after merging main, whose new dart module sat beside py in sh/dispatch.sh (rs, msrv, cross, deny, license, py, dart, quest, gh, and the rest).
  • sh/python.sh test: 61 tests (pyca and stdlib comparisons, RFC vectors, and Wycheproof for every family in the API) and stubtest. 80 consecutive clean runs after the alignment fix.
  • Wheel matrix probed on a scratch branch (run). All nine wheels and the sdist build. All tested wheels pass, except Windows on Arm, which failed only on the pyca import and is fixed above. The scratch branch is deleted.
  • Three review rounds fixed the following; tests cover each:
    • A signed hmac.Context now wipes its boxed state.
    • Arguments take any buffer's raw bytes, through a single CPython copy that is safe on free-threaded builds.
    • HKDF and PBKDF2 refuse impossible lengths before allocating them.
    • A partly uploaded release retries until its sdist is on PyPI.
  • A dry release-plz update gives rind-python 0.1.0 with a CHANGELOG. actionlint, sh/pins.sh, and sh/alert.sh check pass.

Follow-ups

  • Planned in this PR: pyrind's musl and armv7 wheels are tested (S), beside pyrind in the m2 adoption epic. Those wheels are built but not loaded in CI.
  • abi3t wheels for free-threaded CPython 3.15, once pyo3 supports them, to replace the sdist build there. Recommendation: plan it once pyo3 ships abi3t; until then it waits on an outside party.

Models

  • Size: L
  • Implementer: Claude Code, claude-opus-5-5, high effort

Decisions

Settled by the maintainer on 2026-11-07:

  1. ✅ No ctgrind harness for the bindings (rejected: a CPython-under-valgrind harness, a follow-up quest for one). They only move secret bytes; every branch or comparison on a secret happens inside rind, which admin/ctgrind covers.
  2. ✅ Keep hmac.Context.sign's second wipe of the boxed heap copy (rejected: a new in-place sign(&mut self) in rind). No new rind API.
  3. ✅ Reuse the existing publish environment for PyPI trusted publishing (rejected: a new pypi environment). Adding the pending publisher on PyPI stays the maintainer's action. It gates the first release, not this merge, so it lives as a Maintainer: Waiting item in quest/m2/adoption/pyrind.md, not here.
  4. ✅ The binding's wipes follow rind's rule, through rind's own wiping (rejected: wiping unconditionally). It has no wipe of its own, so it needs no wipe feature of its own, and forwards none, matching feat: wiping keys on drop is opt-in, under a wipe feature #461's adapters. moq-dev/rind#461 is still open; whichever lands second needs no change to the binding's code. See Wiping under Approach for what the wheels build with.
  5. ✅ Plan "test-load the musl and armv7 wheels in CI" as a quest in this PR (rejected: leaving it as a suggested follow-up): quest/m2/adoption/pyrind-wheel-tests.md, sized S, beside pyrind in the m2 adoption epic.

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort
  • Session: 01a1b535-f5b4-7bd1-b4e3-ac3d7548ece1, round 5
  • Head: fade49a
  • Base: origin/main at 7af4efb
  • Score: 10/10, The merge preserves the reviewed Python implementation and both Python/Dart checks, all 61 tests pass against the existing wheel, and no new findings were identified.
  • Findings: none

c32c635 is a conflict-free merge of main onto the reviewed fade49a.

kixelated and others added 7 commits November 6, 2026 23:03
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README carries the benchmark against pyca/cryptography, hashlib, and
hmac, run on a held host. pyca/cryptography's last Windows on Arm wheel
lacks ML-KEM, so that runner tests pyrind against Wycheproof, the README,
and its stubs only. rustfmt applied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title pyrind on PyPI feat(python): pyrind, rind's API for Python Nov 7, 2026
kixelated and others added 2 commits November 7, 2026 01:05
- hmac.Context.sign signs a copy, so dropping the box wipes the heap state
  in place rather than freeing it unwiped after a move.
- Arguments take the raw bytes of any buffer, whatever its item format, as
  hashlib does.
- HKDF and PBKDF2 refuse a length no hash allows before allocating it.
- A partly uploaded release stays unpublished until its sdist, which twine
  uploads last, is on PyPI, and the retry skips what is already there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Arguments other than bytes are copied by one buffer export and
  PyBuffer_ToContiguous, so no Rust slice covers memory another thread may
  write on free-threaded CPython, and an exporter allowing one export works.
- PBKDF2 refuses a length past its own hash's limit before allocating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated and others added 4 commits November 7, 2026 10:36
The binding's wipes are rind's own drops, so they follow rind's wipe
rule; say so rather than claim an unconditional wipe. Plan the CI that
loads the musl and armv7 wheels, and say why the PyPI publisher stays
a maintainer step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 19:04
@pullfrog

pullfrog Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Your OpenCode Go account is out of credit. Top it up, then re-trigger Pullfrog.

OpenCode Go's response
provider error: Insufficient account funds

Top up OpenCode Go →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using glm-5.3

@coderabbitai

coderabbitai Bot commented Nov 7, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5550cfae-2eaa-486c-a581-b7d40775b2e9
Commits

Reviewing files that changed from the base of the PR and between 7af4efb and c32c635.

Files ignored due to path filters: 1

  • Cargo.lock is excluded by !**/*.lock
Files selected for processing: 55

  • .github/workflows/alert.yml
  • .github/workflows/python.yml
  • .github/workflows/release.yml
  • .gitignore
  • .release-plz.toml
  • .schema/latest.json
  • Cargo.toml
  • README.md
  • admin/scan-license
  • flake.nix
  • quest/m2/adoption/README.md
  • quest/m2/adoption/pyrind-wheel-tests.md
  • quest/m2/adoption/pyrind.md
  • rind-python/Cargo.toml
  • rind-python/LICENSE
  • rind-python/README.md
  • rind-python/bench/bench.py
  • rind-python/pyproject.toml
  • rind-python/python/pyrind/__init__.py
  • rind-python/python/pyrind/_rind.pyi
  • rind-python/python/pyrind/aead.pyi
  • rind-python/python/pyrind/agreement.pyi
  • rind-python/python/pyrind/cpu.pyi
  • rind-python/python/pyrind/digest/__init__.pyi
  • rind-python/python/pyrind/digest/legacy.pyi
  • rind-python/python/pyrind/hkdf/__init__.pyi
  • rind-python/python/pyrind/hkdf/legacy.pyi
  • rind-python/python/pyrind/hmac/__init__.pyi
  • rind-python/python/pyrind/hmac/legacy.pyi
  • rind-python/python/pyrind/kem.pyi
  • rind-python/python/pyrind/pbkdf2/__init__.pyi
  • rind-python/python/pyrind/pbkdf2/legacy.pyi
  • rind-python/python/pyrind/py.typed
  • rind-python/python/pyrind/rsa/__init__.pyi
  • rind-python/python/pyrind/rsa/legacy.pyi
  • rind-python/python/pyrind/signature/__init__.pyi
  • rind-python/python/pyrind/signature/sign.pyi
  • rind-python/python/pyrind/signature/verify/__init__.pyi
  • rind-python/python/pyrind/signature/verify/legacy.pyi
  • rind-python/src/aead.rs
  • rind-python/src/agreement.rs
  • rind-python/src/digest.rs
  • rind-python/src/hkdf.rs
  • rind-python/src/hmac.rs
  • rind-python/src/kem.rs
  • rind-python/src/lib.rs
  • rind-python/src/pbkdf2.rs
  • rind-python/src/rsa.rs
  • rind-python/src/signature.rs
  • rind-python/tests/test_api.py
  • rind-python/tests/test_readme.py
  • rind-python/tests/test_wycheproof.py
  • sh/dispatch.sh
  • sh/pyrind-unpublished.sh
  • sh/python.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated
kixelated added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 95928d2 Nov 7, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant