Skip to content

rust-serialization.md

Latest commit

 

History

History
175 lines (120 loc) · 8.86 KB

rust-serialization.md

File metadata and controls

175 lines (120 loc) · 8.86 KB
Raw

Home › Features › Rust Serialization

Rust Serialization (ByteStorage)

Available since v0.3.0

Note

This page describes the Rust-powered ByteStorage layer. For the pluggable serializer system (MessagePack, Arrow, Orjson, Pydantic), see Serializers.

What the Rust Layer Does

CacheKit includes a Rust extension (ByteStorage) that handles the low-level binary operations between the Python serializer and the storage backend:

Python object
    ↓
Serializer (StandardSerializer / AutoSerializer: MessagePack)
    ↓  [Rust ByteStorage takes over here]
LZ4 compression
    ↓
xxHash3-64 integrity hash  (detects corruption)
    ↓
[Optional] AES-256-GCM encryption  (if @cache.secure; AutoSerializer is refused)
    ↓
Storage backend (Redis / CachekitIO / File)

Retrieval is the exact reverse pipeline. ArrowSerializer and OrjsonSerializer skip ByteStorage; for them only @cache.secure's AES-256-GCM step runs in cachekit's Rust extension.

The Rust layer is transparent — you configure serializers and encryption at the Python level; ByteStorage handles the rest for the MessagePack serializers automatically.

Why Rust for This Layer?

ByteStorage runs LZ4 compression and xxHash3-64 checksums in Rust over the MessagePack bytes of StandardSerializer and AutoSerializer; ArrowSerializer and OrjsonSerializer do not use it. @cache.secure adds a separate Rust AES-256-GCM call for each serializer it wraps. cachekit-core's README publishes measured figures for those calls on named GitHub-hosted runners; their cost grows with payload size.

LZ4 Compression

LZ4 is chosen for its speed/ratio balance:

Data Type Compression Ratio
JSON strings 2-3x
Numeric arrays 3-5x
Repeated structures 5-10x
Already-compressed data ~1x (negligible overhead)
Random bytes ~1x

ByteStorage LZ4 compression of serialized envelopes (MessagePack: StandardSerializer and AutoSerializer) runs automatically whenever integrity checking is on (the default; @cache.secure forces it on for a serializer named by string, while a serializer instance you pass keeps its own setting); it has no separate switch. A payload under 64 bytes is not run through the compressor: ByteStorage writes it as one uncompressed LZ4 literal block, which any LZ4 decoder reads, so a small value is stored at its own size plus the envelope's overhead. Arrow (DataFrame) payloads use a separate Arrow IPC codec, selectable via the CACHEKIT_ARROW_COMPRESSION environment variable (zstd, lz4, or none) — that setting applies only to Arrow payloads and does not disable ByteStorage LZ4 for other serializers.

xxHash3-64 Integrity

With integrity checking on (the default), every value stored in a ByteStorage envelope includes an xxHash3-64 checksum (8 bytes, big-endian), computed over the original uncompressed payload bytes only. On retrieval:

  1. The payload is decompressed and its checksum is computed
  2. Stored checksum is compared
  3. Mismatch → SerializationError at the serializer layer (corrupted data, never returned to caller); a normal @cache-decorated read catches it, evicts the entry, and recomputes rather than propagating it to your code

Not every value travels in an envelope. With integrity checking off, StandardSerializer and AutoSerializer write no ByteStorage envelope, so MessagePack, columnar and NumPy values carry no checksum. Two AutoSerializer routes skip ByteStorage even with integrity on: NumPy arrays and Arrow DataFrames carry their own 8-byte xxHash3-64 prefix over the bytes they store, with no ByteStorage LZ4 step.

This protects the payload against Redis memory corruption, storage bugs, and bit rot. It does not cover the envelope's format field, so a rotted format passes it. AutoSerializer accepts only a format it writes, and only when a header claim that is present agrees; see Deserialization failed in error-codes.md for what still gets through. A corrupt frame header (truncated, bad version/length, unparseable JSON) takes the same path: SerializationError, evict, recompute.

Non-cryptographic — corruption detection only. xxHash3-64 is not a cryptographic hash and the comparison is a plain equality check: an attacker with backend write access can forge a valid checksum for arbitrary bytes in microseconds. On the plaintext @cache path the stored bytes are therefore attacker-forgeable. Tamper-resistance comes from AES-256-GCM (@cache.secure, or an explicit encryption= option), never from this checksum. See zero-knowledge-encryption.md for the corruption-vs-tamper telemetry split.

Standalone checksum API

Available since v0.12.0. On earlier releases, from cachekit._rust_serializer import checksum raises ImportError.

The same primitive is exposed directly — decoupled from LZ4 compression — for serializers where compression is ineffective (Arrow IPC, compact JSON):

from cachekit._rust_serializer import checksum, verify_checksum

digest = checksum(b"payload")  # 8 bytes, big-endian
assert len(digest) == 8
assert verify_checksum(b"payload", digest) is True
assert verify_checksum(b"tampered", digest) is False

verify_checksum raises ValueError unless the expected checksum is exactly 8 bytes. Both functions accept any buffer-protocol object (bytes, bytearray, memoryview), so a serializer holding its payload as a memoryview can hash it without a bytes copy. The output is byte-identical to the checksum embedded in every ByteStorage envelope and to xxhash.xxh3_64_digest from the xxhash package.

AES-256-GCM Encryption

When using @cache.secure, the Rust layer applies AES-256-GCM to the serializer's output. With StandardSerializer (the default), that output is a ByteStorage envelope, so encryption runs after LZ4 compression:

StandardSerializer:  LZ4(msgpack) → AES-256-GCM(compressed) → storage
OrjsonSerializer:    JSON bytes   → AES-256-GCM               → storage
ArrowSerializer:     Arrow IPC    → AES-256-GCM               → storage

OrjsonSerializer and ArrowSerializer skip ByteStorage, so ByteStorage's LZ4 step does not run before encryption. Arrow IPC bytes carry their own codec instead: zstd by default, set by the serializer's compression argument or CACHEKIT_ARROW_COMPRESSION. AutoSerializer cannot be encrypted: @cache.secure refuses it.

Encryption is authenticated — any tampering with ciphertext raises a decryption error before data reaches your application.

See Zero-Knowledge Encryption for full details.

Choosing a Serializer

Encryption works with any serializer whose class declares cross_sdk_compatible = True: Standard, Orjson, Arrow, or a custom serializer that opts in (see Under Encryption). AutoSerializer and unmarked custom serializers are refused. Mix and match:

Use Case Serializer Encryption
General caching Default (MessagePack) Optional
JSON APIs Orjson Optional
DataFrames / ML Arrow Optional
Typed models Pydantic Only if marked cross_sdk_compatible
Custom types Custom Only if marked cross_sdk_compatible

With integrity checking on, StandardSerializer, and AutoSerializer for MessagePack values, use ByteStorage (LZ4 + xxHash3-64); OrjsonSerializer and ArrowSerializer write their own xxHash3-64 prefix instead (Orjson only with integrity checking on). @cache.secure adds AES-256-GCM over the output of any of Standard, Orjson, Arrow or an opted-in custom serializer.

Troubleshooting

Q: Size exceeds limit A: Don't cache objects >100MB. Break into smaller pieces.

Q: Type not serializable A: Choose a serializer that supports your type. See Serializers.

Q: "Decryption failed: authentication tag verification failed" A: Key mismatch or data corruption. Check CACHEKIT_MASTER_KEY hasn't changed. See Zero-Knowledge Encryption.

Q: Compression ineffective A: Expected for already-compressed or random data. Overhead is negligible.

See Also