[jonathan343]

Summary

S3 also supports XXHash64, XXHash3 (64-bit), and XXHash128 checksums. This PR adds them to smithy-checksums as XXHash64, XXHash3, and XXHash128, also available through new(). The native side uses the twox-hash crate (MIT, added to LICENSE-THIRD-PARTY). The pure-Python side is a port of the xxHash reference implementation, limited to the seed of zero and default secret that checksums use. The second commit speeds it up: XXH3 and XXH128 key whole blocks with one big-integer XOR and accumulate in an unrolled loop, and XXH64 runs its four lanes at once packed into one integer. It now runs at about 47–68 MB/s, depending on the algorithm.

Example

import base64

import smithy_checksums

for name in ("XXHASH64", "XXHASH3", "XXHASH128"):
    checksum = smithy_checksums.new(name)
    checksum.update(b"Hello world")
    print(checksum.name, checksum.digest_size, base64.b64encode(checksum.digest()))

The native and pure-Python builds print the same thing. These are also the x-amz-checksum-xxhash* header values botocore's tests expect for the same input:

xxhash64 8 b'xQCwyRKzdtg='
xxhash3 8 b'tqy52Eo4/3Q='
xxhash128 16 b'c1H4mBL5c4K5HQWzHgTdfw=='

Testing

The known answers include 15 of xxHash's own sanity-test vectors per algorithm, from tests/sanity_test_vectors.h. They cover every XXH3 size class and block boundary. Both implementations are also compared against the CRT and against each other:

uv run pytest packages/smithy-checksums -k "xxhash and (known_answers or matches_crt or native_matches_pure)" --no-cov -q

18 passed, 196 deselected in 0.12s

I also ran all 12,483 seed-zero vectors in that file once, covering every input length from 0 to 4,160 bytes. Both implementations matched every one.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[kuhe]

please also review aws/aws-sdk-js-v3#8383 while this is fresh 😄

[stobrien89]

we should add xxHash’s BSD license here too since the Python implementation is described as a port of the reference code

[jonathan343]

Added in a0a9528