fix: emit canonical Value key order in the fixture builder - #246
Open
nau wants to merge 2 commits into
Open
Conversation
buildValue combined lovelace and assets with foldl' (<>) over the
Data-backed PlutusLedgerApi.Data.V3.Value, whose Semigroup
(PlutusTx.Data.AssocMap.union) is insertion-ordered and appends the left
map's unmatched keys after the right map's entries. Any {lovelace,
assets} value spec therefore produced the custom currency symbol before
ADA's empty one - the reverse of the canonical (strictly ascending
byte-lexicographic) key order that real ledger-produced Values always
have, and that CIP-0153's unValueData builtin (plutus-core >= 1.65)
requires: it rejects non-canonical encodings outright instead of
normalising them. Any submission whose validator decodes a
fixture-provided multi-asset value with unValueData failed with a
machine error even though the validator was correct.
buildValue now re-sorts the built Value (outer currency-symbol map and
each inner token-name map) into strictly ascending byte-lexicographic
key order, making fixture values indistinguishable from real
ledger-constructed ones.
… order The canonical-ordering fix in the fixture builder re-orders the multi-asset Values in the linear_vesting script contexts (ADA's empty currency symbol now sorts first). Validators that walk the outer map now take a different number of steps to reach the vesting token's policy, so every committed linear_vesting submission's measured costs shift by roughly +2..8% cpu. All executions still succeed; no other scenario uses multi-asset values, and their metrics are byte-identical (spot-checked htlc and two_party_escrow). Only metrics.json files are touched; every .uplc and metadata.json is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Haskell test-fixture builder emits non-canonical
Valuekey order for any{lovelace, assets}value spec that mixes a native asset with lovelace: the custom policy'scurrency symbol appears before ADA's empty-bytestring policy in the outer map - the reverse of
canonical (strictly ascending byte-lexicographic) order. Real ledger-produced
Values are alwayscanonically sorted, and CIP-0153's
unValueDatabuiltin rejects non-canonical encodings outrightinstead of normalising them. So any submission - in any language - whose validator decodes a
fixture-provided multi-asset
ValuewithunValueDatafails with a machine error even though thevalidator logic is correct.
Observed against
scenarios/linear_vesting/cape-tests.json'ssuccessful_partial_unlockbaseline(the only committed scenario using multi-asset values): all 4
partial_unlock_*measurement testsof a CIP-0153-based submission abort identically with
The outer key order is
[0xdd..dd (28-byte custom policy), "" (ADA)]. The empty bytestring is thelexicographically smallest possible key, so a valid encoding must list ADA's entry first; this
value is in exactly reversed order. Both the spending input's value (quantity 1000) and the
continuing output's value (quantity 900) are affected - the same builder code path builds each.
Root cause
buildValue(lib/Cape/Tests.hs) combines the lovelace entry with the resolved asset entries asover
PlutusLedgerApi.Data.V3.Value- the Data-backed ledgerValue. That type'sSemigroupmakes no ordering guarantee:
instance Semigroup ValueisunionWith (+)(
plutus-ledger-api-1.45.0.0/1.65.0.0,src/PlutusLedgerApi/V1/Data/Value.hs:331-333/302-304), andunionWith'sunionValdelegates toMap.unionover the Data-backedPlutusTx.Data.AssocMap(Value.hs:431/:402).PlutusTx.Data.AssocMap.union(plutus-tx-1.45.0.0/1.65.0.0,src/PlutusTx/Data/AssocMap.hs:335-393) computesres = goLeft ls `safeAppend` goRight rs, andsafeAppendfolds the left map's entries intothe right map's list with
insert'(AssocMap.hs:146-164), which appends a missing key at theend of the list (its
nilCase).So for the disjoint-key case here,
adaValue <> assetValueproduces the right operand's keysfirst and the left operand's keys last:
[0xdd.., ""]. Empirical confirmation on the productionpin (plutus-core 1.45,
cabal repl lib:cape):The same code is present in plutus-tx 1.45 and 1.65, so both the production and preview measure
binaries are affected. This is not a plutus-tx bug: the Data-backed
AssocMapdocuments itself asan unordered association list, and on-chain nothing ever needs to reorder it. The bug is using an
insertion-ordered union to construct a transaction-context
Valuethat the real ledger wouldhave produced in canonical order.
Why canonical order is required (not a quirk of one compiler)
unValueData's implementation (plutus-core-1.65.0.0,src/PlutusCore/Value.hs:469-499)delegates to
buildValueWith(Value.hs:543+), whose documented contract is:Key comparison is newtype-derived from
ByteString's lexicographicOrd(Value.hs:81-82), underwhich the empty bytestring sorts first. The builtin deliberately does not normalise - it is a cheap
O(n) parse that assumes the canonical form every real ledger-constructed
Valuehas. EveryCIP-0153 consumer shares this behaviour, so every language/compiler that lowers value lookups to
unValueData(Plinth, Scalus, ...) hits this failure on the fixture-built values. The fixture JSONitself is fine; only the JSON-to-
Dataconversion is at fault.Fix
buildValuenow re-sorts the folded result into canonical order before returning it(
canonicalValueinlib/Cape/Tests.hs): theValue'sDataencoding is unpacked, the outercurrency-symbol map and (defensively) each inner token-name map are sorted into strictly ascending
byte-lexicographic key order, and the result is repacked. The fold is kept, so duplicate-key
merging semantics are unchanged; sorting a 1-entry map (every lovelace-only value) is a no-op.
There is no canonicalising constructor in the Data-backed ledger API to delegate to, hence the
explicit sort.
buildValueis the singleValue-construction point for script contexts, so thiscovers inputs, outputs, and any future value spec.
Blast radius: re-measured metrics (second commit)
Canonical order changes how many steps map-walking validators take to reach the vesting token's
policy (it now sorts after ADA instead of first), so measured costs of all committed
linear_vestingsubmissions shift by roughly +2..8% cpu. All executions still succeed - nosubmission's result flips. Regenerated
metrics.json(summed over measurement evaluations):Only
metrics.jsonfiles are touched; every.uplcandmetadata.jsonis byte-identical. Noother scenario's metrics change:
linear_vestingis the only committed scenario whose fixtures useassets(grep'"assets"'underscenarios/), and lovelace-only values takebuildValue'suntouched
[] -> pure adaValuebranch. Spot-checked by re-measuringhtlc/Plinth_1.65.0.0_Unisayand
two_party_escrow/Scalus_0.18.2_Unisaywith the fixed builder: byte-identicalmetrics.json.Verification
cabal test: 117 examples, 0 failures.cape submission verifygreen on all 7 current-tracklinear_vestingsubmissions(
Plinth_1.45/1.64/1.65 x {default, _plain},Scalus_0.18.2_Unisay), plusmetrics.schema.json/
metadata.schema.jsonvalidation for the 4 re-measured_previewsubmissions.treefmt(fourmolu): no reformatting needed.How to reproduce
Before the fix (on
main):With the fix,
buildValue's output for the same spec isMap [(B "", ...), (B "\221...\221", ...)]- canonical. End to end: measure anylinear_vestingsubmission whose validator usesunValueData(e.g. a Scalus 1.1.0 build with itsdefault CIP-0153
Valuelowering) withmeasure-preview; before the fix its 4partial_unlock_*tests abort in
unValueData, after the fix all 29 tests pass.