[Proposal] Java compatibility fixtures and PoC selection matrix for the Rust modernization groundwork

[dpol1]

Follow-up to the discussion in #3110. @imbajin confirmed the testing and validation groundwork can start before any Rust decision, so this issue tracks the Java-only part: a selection matrix for the first PoC candidate and golden compatibility fixtures for the codec classes.

Scope

Three deliverables, all plain Java against the current build:

1. A selection matrix for PoC candidates (first cut below, to be refined in this thread).
2. A golden fixture corpus for LongEncoding plus a replay test that runs it against the current implementation.
3. Short written notes on the behavioral contract the fixtures lock down (edge cases, error behavior, invariants).

Non-goals

No Rust, no JNI, no Cargo, no packaging changes, no new Maven modules, no runtime wiring. Those wait for the component decision, a mentor, and a design review with contributors who have deeper Rust experience, as discussed in #3110.

Selection matrix

Criteria: an existing test oracle, enough call sites to matter, work per call big enough to survive a JNI crossing, and, following the maintainer feedback in #3110, keeping the Java implementation alive has to stay cheap: graceful fallback is the maintainer-preferred direction, which this matrix adopts as a selection criterion.

Candidate	Oracle	Fallback cost	Notes
LongEncoding	LongEncodingTest (684 lines)	low, pure functions	output feeds storage keys, so the contract needs locking either way
NumericUtil	NumericUtilTest (475 lines)	low	coupled to LongEncoding.encodeNumber, candidate for the same corpus later
BytesBuffer	BytesBufferTest (1351 lines)	high	persisted format, 30+ callers, a fallback would have to stay byte-identical; the store's read path structurally parses server-encoded values via the struct copy, so the shared format surface spans ids, vints and property encoding (thread)
IntSet / IntMap	IntSetTest, IntMapTest	high	concurrent structures on Unsafe, a live double implementation risks divergence

With the fallback criterion the concurrent structures and the serializer drop out of round one. LongEncoding is the first fixture target. That is a choice about where to write fixtures first, not a decision that it becomes the Rust PoC.

Fixture shape

Plain ASCII, line-based (TSV with \uXXXX escapes for non-ASCII), checked into test resources so it needs no extra dependency to read. It covers: round trips across the full value range including Long.MIN_VALUE/MAX_VALUE (the encoding does intentional wrapping arithmetic), arbitrary UTF-16 input on the decode side (valid inputs are ASCII, invalid ones are not), the error cases (IllegalArgumentException vs NumberFormatException vs NPE on null), and the length limits.

The replay test is JUnit 4 and compiles at the Java 8 level, since that is the current hugegraph-commons baseline. Root is Java 11. Java 17 / commons Java 11 from #3117 are treated as declared targets, not assumptions.

Acceptance

• matrix reviewed in this thread
• corpus + replay test merged and running in the normal test suite
• contract notes written up here or in the PR

Future PoC evaluation (outside the current implementation scope)

The following diagram illustrates a later evaluation step. No Rust component is selected here; any future PoC should compare compatibility and end-to-end costs, including JNI and data conversion, against the Java baseline.

References

Annotated references and suggested practices: SQLite, Parquet, Arrow, Protobuf, Go, Insta, Wycheproof, encoding test vectors, and Java/Rust integration examples.

[dpol1]

Contract notes for the LongEncoding corpus (PR #3146), the invariants worth knowing before porting or refactoring:

Alphabet and ordering. B64_SYMBOLS = 0-9 A-Z _ a-z ~, 64 chars in ascending ASCII order. For canonical encodeSortable outputs, plain string comparison gives numeric order: negatives (prefix 0) sort before positives because positive outputs always start with a length char >= 1, the length character orders different magnitudes within one sign, and the digits decide between encodings of equal length.

Length symbols. LENGTH_SYMBOLS (0-9A-F) is exactly the first 16 characters of B64_SYMBOLS, and decodeSortable reads the declared length through B64_SYMBOLS.indexOf, so the prefix relationship is load-bearing for decoding: declared lengths from 0 to 63 are structurally accepted. Declared length 0 is reachable only in the negative form "00", since the initial str.length() >= 2 check rules out a bare positive length char.

Wrapping is intentional. encodeSortable maps negatives by num += Long.MIN_VALUE (overflow on purpose) and decodeSortable reverses it with value -= Long.MIN_VALUE; the digit loop accumulates with multiply/add that wrap silently. A port must use wrapping arithmetic (e.g. wrapping_add/wrapping_sub/wrapping_mul in Rust), not checked arithmetic.

Decode accepts more than encode produces. Structurally valid input goes up to 64 chars in the positive form and 65 with the negative prefix, and the accumulation wraps rather than failing. Concrete case: decodeSortable("00") returns Long.MIN_VALUE, while encodeSortable(Long.MIN_VALUE) produces "010". Round-trip equality holds encode->decode, not decode->encode.

Empty-string edge. decodeB64("") = 0, decodeSignedB64("") = 0, decodeSignedB64("-") = 0. No exception.

Specials. encodeSignedB64(Long.MIN_VALUE) returns the literal "-80000000000" (the b64 of 2^63). encodeB64 rejects any negative input with IllegalArgumentException.

Error taxonomy. IllegalArgumentException for argument/structural precondition failures (input shorter than 2 chars, declared length not matching the actual length, negative input to encodeB64); NumberFormatException for any character outside the symbol table during digit decoding, including non-ASCII BMP characters and surrogate code units; NullPointerException on null. Exception classes are contract; messages deliberately are not.

Domains. Encode outputs are always pure ASCII. Decode input is arbitrary UTF-16; the corpus exercises representative inputs on that side (non-ASCII BMP chars, surrogate pairs, lone surrogates), not an exhaustive sweep.

[dpol1]

Compatibility inventory, first pass. Analysis snapshot: master c10779dc6. Counting rule: production sources only — test code, generated sources, and the bundled dist copy are excluded. File:line evidence exists for every cell, happy to expand any row.

Candidate (module)	Wired production surface	Persisted or wire boundary	Existing oracle	Java fallback / PoC risk
LongEncoding (commons)	encodeNumber, encodeSignedB64, decodeSignedB64, validB64Char in 7 files; decodeSortable/decodeNumber have no production callers today	encoded strings embedded in vertex ids (primary-key strategy), edge sort keys, index ids and values, stored-string ids; the same contract reaches the store side via the struct copies of IdGenerator/ConditionQuery/PropertyKey	LongEncodingTest, golden corpus (PR #3146)	fallback cheap: final class, static pure functions, no state
NumericUtil (commons)	sortable conversions and byte helpers in 20 files	range-index row keys, LongId bytes, HBase counter cells; algorithm derived from Lucene NumericUtils, effectively frozen	NumericUtilTest; the corpus pins it only transitively via encodeNumber	fallback cheap: static, stateless; the byte-array surface has no golden corpus yet
BytesBuffer (server/core)	serializer stack, 26 files	the binary row format itself for every backend, raft log entries, client-visible page tokens; non-primitive property values embed Kryo bytes	BytesBufferTest, BinarySerializer round-trip tests	high: byte layout is the on-disk format, process-wide config state, Kryo coupling
BytesBuffer (hugegraph-struct, consumed by hstore modules)	store read/query paths, 16 files	gRPC ByteString id fields (wire) and RocksDB keys and values on the store side (disk)	none — no test imports this copy	high, and currently uncharacterized
IntSet/IntMap (server/core)	traversal records only, 4 files	none: per-request, in-heap	IntMapTest, IntSetTest, JMH benchmark	state cheap, but the concurrency contract is real (concurrent=true in K-neighbor/K-out) and the implementation sits on sun.misc.Unsafe, which is on the JEP 471 removal path

Findings that actually move the selection:

1. LongEncoding's wired surface is narrower than its API: production never calls decodeSortable/decodeNumber. The corpus still locks them (they are public contract), but a PoC can be judged on the four wired operations.
2. NumericUtil's byte-array surface (numberToSortableBytes and friends, used in range-index keys) is the natural next corpus once the shape of test(commons): add golden fixture corpus for LongEncoding #3146 settles.
3. Facts about the two BytesBuffer copies, recorded separately: the struct copy declares ID_LEN_MAX = 32768 but its writeId enforces a literal 16384 and the declared constant is referenced only in a comment; the core copy declares ID_LEN_MAX = 16384; BLOB_LEN is 5 in core and 4 in struct; git history shows the struct file was extracted from the core one in 2025-09 while the core copy kept evolving (latest 2026-06, feat(server) : make buffer max capacity configurable #3049); no test currently locks the struct copy. Whether the two copies must interoperate on the same byte format is one of the open points below. Until that contract is stated, the safe deliverable is a characterization fixture per copy, not a cross-copy equality test.
4. IntSet/IntMap: only 4 production call sites and transient state, so porting value looks low; the direct sun.misc.Unsafe usage matters for JDK upgrades independently of any Rust work. Side note: IntMapByDynamicHash has tests and a JMH benchmark but no production wiring.

Open points I could not settle from the tree — input welcome, and @dosu feel free to take a first pass:

• whether hugegraph-server's core BytesBuffer and the hstore struct copy ever need to read each other's bytes, or the two formats live behind separate boundaries;
• whether the store ingest path writes struct-BytesBuffer-encoded bytes, or only re-encodes on read and query;
• whether external tools (loader, hubble, tools) parse or generate the binary row format;
• whether stored Kryo-encoded property values are guaranteed readable across a Kryo upgrade, and whether page tokens are expected to survive a server upgrade.

I'll fold the answers into the matrix as they come.

Update (2026-08-09): the reply below answers the four points, and the mechanical claims check out against the tree (HstoreTable.insert forwards byte arrays unchanged; encoding is server-side; loader/hubble/tools stay on the client/REST layer; new Kryo() with no version header, pinned via TinkerPop 3.5.1; page tokens are ephemeral cursors). One design-level correction after reading the store code directly: the shared surface between the two BytesBuffer copies is wider than string ids. On the store's filtered-read path the struct copy structurally parses and re-encodes server-written column values — SelectIterator.select() does readId/readVInt/readProperty(dataType) and rewrites with writeId/writeVInt/writeProperty (hg-store-core), and BusinessHandlerImpl.scanIndexToElementId re-encodes element ids parsed from server-written index columns. So the struct copy is effectively a decoder of the core copy's output for ids, vints, the cardinality/type byte and per-type property values on that path. Whether the two copies are intended to track one format remains a maintainer question; until stated, per-copy characterization fixtures are still the deliverable, and where the read path overlaps they double as cross-copy compatibility evidence.

[dosubot]

Nice inventory. Here's a first pass on your four open points, traced through the current tree:

1. Core BytesBuffer vs struct BytesBuffer — do they need to read each other's bytes?

No, not at the gRPC boundary. The server's BinarySerializer encodes mutations using the core BytesBuffer into BackendColumn.name/BackendColumn.value byte arrays [1]. HstoreTable.insert() passes those byte arrays unchanged to the session/store [2]. The gRPC payload itself is an opaque bytes data field on StoreCommandRequest — no field-level protobuf structure for IDs or properties [3].

The struct BytesBuffer is used on the store side for its own key/query construction (prefix building, ID reading on the query path), not for re-encoding mutation payloads that crossed the wire [4] [5]. So the two copies live behind separate boundaries. The encoding logic for string IDs is compatible (same 0x80/0xc0 prefix-bit scheme, same effective 16384-byte limit enforced in writeId) [6] [7], but they don't cross-decode each other's output at the gRPC layer.

That said, the store does read bytes that the server encoded — so format compatibility matters for the string-ID encoding path even though the two BytesBuffer classes don't call each other. A characterization fixture per copy (as you proposed) is the right deliverable; a cross-copy equality test would only be meaningful for the shared string-ID encoding subset.

2. Store ingest path — does struct BytesBuffer encode on write, or only on read/query?

Encoding happens on the server side. The core BinarySerializer produces fully encoded BackendEntry columns , HstoreStore.mutate() forwards BackendAction objects without re-serializing [8], and the session passes byte arrays straight to graph.put() [9]. The struct BytesBuffer is used in the store for read-side and query-side key construction [10], not for encoding ingest payloads.

3. External tools — do loader/hubble/tools parse or generate the binary row format?

No. All three go through the client/REST API layer:

• The loader reads source records, converts to graph elements, and inserts via context.client().graph() [11] [12].
• Hubble communicates through HTTP/REST [13].
• Tools constructs a HugeClient from URL + graph name [14].

None import or use HugeGraph's BytesBuffer. The binary row format is internal to the server ↔ store boundary.

4. Kryo property values and page tokens across upgrades

Kryo: There's no explicit cross-version compatibility mechanism. The Kryo instance is a bare new Kryo() plus a UUID serializer, pinned transitively through TinkerPop 3.5.1 [15] [16]. Generic properties use writeClassAndObject()/readClassAndObject() with no application-level version header [17], and the read path has no fallback for incompatible data [18]. A Kryo or TinkerPop upgrade should not be assumed safe for stored generic property values without testing representative persisted data.

Page tokens: These are ephemeral, client-carried cursors — Base64-encoded PageState containing a byte-array position, offset, and total count [19] [20]. They survive normal restarts but there's no explicit cross-version compatibility guarantee [21]. Clients should be prepared to restart pagination from the first page after an upgrade.

Summary for the matrix: The two BytesBuffer copies live behind separate boundaries (server encodes, store reads), external tools don't touch the binary format, Kryo compatibility is unguaranteed, and page tokens are ephemeral. These findings reinforce that a per-copy characterization fixture is the right shape for BytesBuffer, and that LongEncoding → NumericUtil remains the natural fixture ordering since those are the codec paths with the cleanest boundaries.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[imbajin]

References: compatibility baselines and a later Java/Rust PoC

The Java-only groundwork here has useful precedents. LongEncoding remains the first fixture target; choosing a native component and demonstrating its value are later decisions.

Shared fixtures, golden outputs and differential testing

Reference	Existing practice	What is useful here
SQLite — sqllogictest	A reference engine fills in expected query results; validation replays the completed scripts against an engine under test. Independently developed engines provide additional oracles.	The closest example of recording reference results and replaying them elsewhere. Later Java/Rust differential tests can supplement the fixed corpus. This checks query results, not performance or concurrency.
SQLite — regression and fuzz testing	Historical fuzz cases are retained and replayed by fuzzcheck; boundary, malformed-input and fault tests complement ordinary regression tests.	Keep minimized failures as permanent regression inputs, alongside deliberately selected boundary cases. A finite corpus complements rather than replaces other test methods.
Apache Parquet — interoperability fixtures / parquet-testing	Shared data files verify that implementations in different languages can read and write each other's files; the repository also keeps bad-data reproducers.	Reuse one fixture collection across languages. If serialization becomes a candidate later, test old-writer/new-reader and new-writer/old-reader paths explicitly.
Apache Arrow — integration testing	Human-readable JSON datasets and pre-generated IPC files support cross-language integration tests.	Separate portable test data from language-specific runners. The fixture format need not resemble the implementation's internal objects.
Protocol Buffers — conformance suite	A common tester communicates with small adapters for individual language implementations.	Keep a common contract and thin runners. This is a conformance example, not a universal byte-equality rule: protobuf serialization is not canonical.
Go — go/doc golden files	Expected outputs are checked in as readable .golden files beside test inputs.	A lightweight precedent for reviewable expected-output files; this is an output-regression example, not a storage-format guarantee.
Rust Insta — snapshot files / review workflow	Committed snapshots are compared with new results, with an explicit review/accept workflow. Text snapshot line endings are normalized to LF.	Changes to expected results deserve review. The LF policy also illustrates why fixture transport details should not cause platform-dependent failures.
Project Wycheproof	Implementation-agnostic JSON vectors and schemas encode expected behavior for cryptographic edge cases and known bugs. Language-specific tests consume the same vectors.	Label the intent of unusual cases and keep the oracle reusable. The cryptographic requirements themselves are separate from HugeGraph's codec contract.
RFC 4648 — encoding test vectors	A small set of explicit input/output examples anchors encoding behavior.	Keep a few easy-to-explain examples beside the larger corpus. HugeGraph's ordered alphabet and sortable prefixes are its own format; RFC 4648 is a testing precedent, not the format to substitute.

The comparison rule must follow the contract: exact encoded strings for HugeGraph's canonical sortable keys; semantic equivalence where a format permits multiple representations; and explicit compatibility decisions for historical malformed-input behavior.

Java/Rust integration and operational boundaries

Reference	Existing practice	What is useful for a later PoC
Apache DataFusion Comet — architecture / compatibility	Spark/JVM integration delegates supported work to a Rust native engine, with documented compatibility and fallback behavior.	Evaluate a bounded component while preserving observable behavior. A native engine's success does not establish a speedup for individual LongEncoding calls.
Apache OpenDAL — Java binding	A Java API uses a Rust core and distributes platform-specific native artifacts.	A concrete reference for classifiers, native loading, platform support and release packaging. HugeGraph's preferred Java fallback still needs its own design.
jni-rs	Rust bindings expose Java native methods with explicit reference lifetimes and error/panic handling at the boundary.	Define ownership and Java-visible error mapping before integration; failures must not unwind across the FFI boundary.
JNI design/performance guidance	The official Android JNI guide recommends reducing marshalling volume and frequency.	Benchmark the complete call path, including crossing, conversion/copying and returned objects. Compare single-item and batched calls where appropriate; Android guidance is not a HugeGraph benchmark.
Rust wrapping arithmetic / Rust string invariants	Wrapping operations are explicit, while Rust str requires valid UTF-8.	Preserve the documented overflow behavior and decide how arbitrary Java UTF-16 inputs, including lone surrogates, reach the Rust boundary. Ordinary string conversion must not silently change the test input.

Applying these references without expanding the current scope

• Keep the committed corpus and the existing hand-written, ordering and round-trip tests. Record the baseline revision and fixture-format version; inspect behavior changes before regenerating expected results.
• A future native candidate should demonstrate compatibility and end-to-end value against the Java baseline on representative workloads. Java fallback should cover defined availability/compatibility cases; it is not recovery from a fatal native-process failure.
• One coverage clarification for the earlier inventory: test(commons): add golden fixture corpus for LongEncoding #3146 currently covers encodeSortable/decodeSortable, encodeB64/decodeB64, and encodeSignedB64/decodeSignedB64. It does not include encodeNumber or a shared NumericUtil corpus. Their existing Java unit tests and a future portable corpus are distinct forms of coverage.