Conversation
First cycle of models-decisions, the System One tier: state in, typed probabilistic decisions out, with no decode loop anywhere. Written test-first. The 39 tests went red on 80 unresolved symbols before any of these types existed, and every figure they assert is one whose answer is known by hand rather than by running a model. What lands: - AnswerSpace, sealed over Noul (binary), Choice (2-255 unordered options) and Score (2-10 ordered levels). The space is declared before the decision, so the outcome can only be a label the caller named. That bound is structural, not a validation pass applied afterwards. - Verdict, bound to the space that produced it. The distribution is checked on construction, copied in and out, and resolves ties toward the earlier declaration so repeated runs over identical input agree. Confidence is one minus normalized entropy, and measures decisiveness rather than correctness. - Calibration: numerically stable temperature scaling that never moves the winner, plus expected calibration error and Brier score as the rules that decide whether a temperature helped. - DecisionHead and an affine LinearDecisionHead reading one hidden state in fixed accumulation order, so two reads of the same state agree bit for bit. A decision that moved between runs could not be calibrated, and the calibration is the product. Nothing here can generate text, and that is the point rather than a limitation to be lifted later. The module depends on models-api only. It pulls no backend: the evaluator that forks a shared physical KV prefix belongs to whichever backend implements the capability, and arrives in the next cycle. The one SpotBugs suppression is narrow and justified by a test: Choice and Score hold List.copyOf results that SpotBugs cannot see through, and AnswerSpaceTest pins their immutability behaviourally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second cycle. The economics of this tier rest on one claim: N decisions over a state cost one prefill and N head reads, because every question forks the same physical key/value prefix rather than recomputing it. SharedPrefixDecisionEvaluator is that claim in code. Written test-first; the ten new tests went red on three unresolved types. The load-bearing decision is that the evaluator asks the backend to prove the sharing, through sharesPrefixStorage, and carries the answer out in DecisionBatch beside the verdicts. Timing is not evidence of sharing, and a run that quietly stopped sharing would otherwise look like a run that merely got slower. Both directions are pinned: a backend whose forks do share reports true, one whose forks do not reports false, and a backend that cannot share a prefix or cannot expose a hidden state is refused up front rather than silently served. An arm cannot be inert everywhere and still pass as compliant. Also pinned: the state is frozen exactly once however many questions are asked, questions continue from the end of the state rather than restarting at zero, every branch session is closed even when a head rejects its hidden state, and an empty question list never touches the backend at all. The no-generation invariant is now enforced rather than documented. The stub backend fails the test if the evaluator ever reaches forward or forwardBatch, so the day someone reaches for a decode loop inside a decision, a test says so. 49 tests, full check green on the Intel workstation and reproduced on a Hetzner ccx33 under Temurin 25.0.4.1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A capability probe, run on the real artifact rather than reasoned from the code, establishing that the base this tier is built on genuinely supports the arrangement the plan assumes. It produces no number anyone should quote; it exists so the architecture was known to be viable before any harvest was paid for. The artifact is the qualified one: granite-4.1-3b-Q4_K_M pulled from the pinned HuggingFace revision, 2,099,501,664 bytes, SHA-256 662b0626... — byte count and digest both verified equal to the ModelJars catalog's recorded baseArtifactSha256. Its GGUF metadata reads general.architecture granite with no ssm.* keys, so it is a pure transformer rather than a Mamba hybrid, and it routes to LlamaDecoder. Established on the weights: - PureJavaBackend loads it and is a SharedPrefixInferenceBackend. - Hidden states and shared prefixes are both supported. - Two forks of one frozen prefix return true from sharesPrefixStorage. The sharing is proven by the backend, not inferred from timing. - The hidden state is 2560 wide and finite. That is the head's input width. - Identical suffixes over one shared prefix agree exactly. - SharedPrefixDecisionEvaluator drives the real backend end to end and returns a verdict inside its declared space, freezing the prefix once. What it does not establish: anything about quality, agreement with the decode verdict, calibration, or speed. The qualified answerability evidence is on rust-ffm and this probe used pure Java, so the two would not be comparable even had this produced figures. The probe skips cleanly when the artifact is absent, so it never fails a workstation build. backend-java is a test-only dependency; the module still pulls no backend of its own. One trap is recorded rather than quietly fixed. The first attempts reported SKIPPED and then UP-TO-DATE, because Gradle forwards neither -D to the forked test JVM nor an environment variable into task inputs. The probe skipped visibly rather than passing vacuously, but a cached skip could still be mistaken for a run, so integrationTest now declares DECISIONS_GRANITE_MODEL as an input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cycle 3 must run on rust-ffm for its numbers to be comparable with the qualified answerability evidence, so the native toolchain was brought up on the benchmark host ahead of it. Two findings are worth keeping. Ubuntu 24.04's packaged Cargo cannot build this repository's Rust 2024 crate, as the runbook already warned; the rustup bootstrap plus the repository's own pin selected rustc 1.96.0. More important: :backend-native:assemble does not build the native library. It finishes in ten seconds and produces only the Java jars. The cdylib comes from prepareNativePlatformResources, which compiled jmodels-kernels 0.3.42 and emitted libjmodels_kernels.so into the linux-x86_64 resource path. :backend-native:test then passes, including the bundled-library poll-budget case, so the binding loads the library built here. Had assemble been trusted, an arm labelled rust-ffm would have run with no native library present, and would either have failed outright or quietly executed pure Java under the wrong label. That is the ablation hazard the working agreement names, so it is recorded rather than silently worked around. No measurement was taken. This is toolchain readiness only.
…plit Written and committed before a single hidden state is harvested, so the floor cannot be chosen after seeing what the head managed. Setting the bar afterwards is the failure the memory campaign has already paid for once. Gates: agreement with the decode verdict at least 0.95 on a sealed split, expected calibration error at most 0.05 over 15 bins, and a median per-decision cost strictly below one decode turn at n=3. All of them must hold on both corpora. Two corpora in deliberately different genres, both published and neither authored here, both pinned by digest: SQuAD v2 dev (11,873 items, majority-class floor 0.5007, Wikipedia crowd-written) and MS MARCO v2.1 validation (101,093 items, floor 0.5503, real web search queries). One dataset measured carefully has produced a confident wrong answer in this tree before, so a pass on one and a failure on the other is a failure and gets reported as the disagreement it is. Splits are grouped by source passage so no passage spans two splits, which forecloses the leakage that would otherwise inflate agreement. The sealed 300 are read once, after the head and temperature are frozen. Two guards are declared in advance rather than discovered later. A sanity condition: if the decode verdict itself fails to beat its dataset's floor by 0.10 then the base cannot do the task, agreement is agreement with a coin flip, and the arm is UNINFORMATIVE rather than passed. And a note on sample size: with a fixed artifact, greedy decode and fixed seeds, agreement and calibration error are deterministic functions rather than draws, so n=1 is exact for them and error bars would be theatre; n=3 attaches to latency, which is the quantity that actually varies. Recorded too is what the design cannot test: transfer to any other base or task, arm C, anything at all about Jev, and whether calibration holds on a customer's data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cycle 3a, written test-first; the nine new tests went red on two unresolved types. Both fitters are checked against problems whose right answer is known by construction rather than by running a model. LogisticHeadTrainer is full-batch multinomial logistic regression over a fixed iteration count, in a fixed order, with no shuffling, no randomness and no early stopping. That is a deliberate choice, not a simplification. The pre-registration treats agreement and calibration error as exact functions of their inputs rather than as draws from a distribution, and that only holds if fitting the same evidence twice returns the same weights twice, which a test pins. Regularization reaches the weights but not the bias, so a class prior can still be learned at full strength -- which matters on a corpus whose base rate is skewed. TemperatureFitter minimizes negative log likelihood by fixed-iteration ternary search over log temperature, so it too is deterministic. Scaling cannot move the winner, so it cannot buy accuracy; all it can do is pull stated confidence toward observed frequency. Pinned: an overconfident set is cooled, an underconfident one is sharpened, a well-calibrated one is left near one, and the fitted temperature never scores worse than no scaling at all. The trainer serves every primitive, not just the binary one: a Choice over three options fits and decides inside its own declared space. 58 tests green, full check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s 14x Sizing for the harvest turned up a blocker worth more than the sizing was. On identical input, same process, same kernels: prefill returning logits runs at 26.6 tok/s because it takes the batched path, while prefillHiddenState runs at 1.9 tok/s because it does not. Fourteen times, and the hidden-state route is slower than decode despite skipping the vocabulary projection. The poll budget was the leading hypothesis, since vectors 0.1.22 warns that a caller running its own pool beside the persistent executor should set it to zero. Ablated in both directions: 1.86 tok/s at pollMillis=0 against 1.87 tok/s at the default. Identical within noise, hypothesis refuted, and recorded as refuted rather than quietly dropped. The cause is a missing branch, located exactly. prefill checks batchedPrefill at LlamaForwardPass:817 and delegates to prefillSessionBatched; prefillHiddenState at :851 has no such check and falls straight into a token-by-token loop. The batched machinery exists and is qualified. The hidden-state route never reaches it. The size of the gap is memory bandwidth: a single-token forward streams the whole two gigabytes of Q4_K_M weights to produce one row, where a batched block streams them once for the lot. This blocks the harvest rather than merely slowing it. A thousand items at roughly 160 tokens each is about 23 hours at the sequential rate against about 1.7 hours at the batched one, on a host billed hourly. Worse, a per-decision latency measured in this state would be measuring our own missing optimization and attributing it to the architecture. Two fixes are described, both in models, and deliberately neither is taken here: LlamaForwardPass is live territory for the GPU campaign, which edits the same file, and a decisions branch quietly rewriting its prefill would collide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…me host Records the payoff of models d23f440 against the measurement that motivated it. On the same box, artifact and kernels, prefillHiddenState over 128 tokens went from 66,538 ms to 4,283 ms, 1.9 tok/s to 29.9 tok/s. The hidden-state route now runs slightly faster than the logits route at 128 tokens and reaches parity by 512, which is the shape to expect: it skips the vocabulary projection on every row but the last, and that saving shrinks in relative terms as the prompt grows. Correctness is asserted rather than assumed, against the repository's existing SIMD reduction tolerance. The harvest this was blocking moves from roughly 23 hours to roughly 1.5 hours, so the pre-registered protocol is affordable as written, across all three corpora, without trimming N. Also recorded: the first version of the fix's tests passed vacuously, because an all-F32 fixture cannot reach the batched path at all and the equivalence assertions were comparing the sequential path against itself. The counter assertion was the only thing that failed, which is why it was written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…dware arms Written before any split was opened and before a single hidden state was harvested, so this remains pre-registration rather than a revision made after seeing a result. The original gates are untouched. CUAD joins as the third corpus because the case worth making is an enterprise one and the other two are academic and web genres. It is also the most awkward of the three, which is why it earns its place: a 70/30 base rate against the others' near-even split, which is the enterprise reality that most questions asked of a document have no answer in it. Accuracy is weak against a 0.7025 floor, so on that corpus the weight falls on agreement and calibration, stated now rather than discovered later. The gate must hold on all three; two out of three is a failure. Contracts are long, so a fixed four-thousand-token window is taken from the start of each, applied identically to answerable and unanswerable items with no answer-aware cropping, and the fraction of gold spans falling outside it is reported beside every CUAD figure. Those items cap achievable accuracy for reasons unrelated to the question being asked. An over-long input has silently destroyed two datasets here before. Energy becomes an axis. The VM was checked rather than assumed and exposes no powercap, no MSR, no hwmon and no rapl module, so joules cannot be measured on it; CPU-microseconds can, and a watt-hour figure derived from CPU-seconds and the part's TDP is recorded as computed and approximate, never as a measurement. The GPU arm carries real watts through NVML. One honesty condition is written in: the per-decision figure is defensible, a total energy reduction is not, and no write-up may convert one into the other. A GPU arm is added and is clean rather than contrived, because the Tornado kernel implements GgufBatchedMatrixKernel and so injects through PureJavaBackend.load while leaving the backend a SharedPrefixInferenceBackend -- the prefix sharing the tier depends on survives onto the device. Our own backend-cuda is excluded: its Q6_K kernel is not bit-exact past one super-block, and a quality gate measured on a kernel known to be numerically wrong would be a number about the defect. TPU is recorded as not reachable from this stack, which is a structural absence rather than work not yet done. Tornado targets OpenCL, PTX and SPIR-V; ONNX Runtime ships no TPU provider; libtpu is reached through XLA from C++ or Python; a Coral part cannot hold a 3B model; and backend-apple bridges Apple Foundation Models, a different model rather than an accelerator for our artifact. An external JAX control on Cloud TPU is described, and explicitly not adopted, because it would measure a different implementation of a different graph. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The split is where a result gets quietly invalidated, so it is built test-first and holds three properties a test can check. No passage reaches two splits. Questions are grouped by the passage they are asked about and each group is assigned whole, so a head cannot be scored on a context it was fitted on. The cost is that split sizes land near their targets rather than exactly on them, and the sizes actually drawn are what get reported. The draw preserves the corpus's natural base rate rather than evening it out. A skewed pool stays skewed, which matters because CUAD's is 70/30 and a quietly rebalanced sample would have made its majority-class floor look like 0.5 when the real one is 0.7025. SplitPlan computes that floor from the sample it actually drew rather than from a published figure. The draw is reproducible from its seed through a seeded Fisher-Yates rather than a library shuffle, so the same split comes back on any host. The pre-registration commits to seed 20260920, and that is only meaningful if a seed pins a draw. 66 tests green, full check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ne pass The tool the pre-registered comparison needs. For every item of a drawn split it records the verdict the base reaches through its ordinary decode loop and the hidden state a head would read instead, so the two can be held against each other on identical input. Both come from one prompt evaluation rather than two. The prompt is prefilled through the batched path, the final token is evaluated once for its hidden state, the session is rewound a position, and the same token is evaluated again for its logits. Hidden state and decode verdict therefore sit at the identical position, which is what the gate compares, and the prompt is not paid for twice -- which matters at roughly thirty tokens a second and three corpora. Hidden states are written as raw int bits rather than decimal text, so a harvest reloads bit-identically and the determinism the calibration rests on survives a round trip through disk. The four-thousand-token window from the amendment is applied here, uniformly and with no answer-aware cropping, and items truncated by it are flagged in the output so the out-of-window rate can be reported beside every CUAD figure rather than silently capping it. Per-item CPU microseconds are read from cgroup cpu.stat, which is the energy-proportional quantity this host can actually measure. It is recorded as CPU time and must not be reported as joules. backend-native enters the module's test scope only. The published module still pulls no backend: a decision head is wired to whichever backend is qualified, and the tool drives rust-ffm because that is where the answerability evidence lives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records a hardware verification of models PR #198 at 93b3feb, run on a Vultr A16-8Q at compute capability 8.6 -- a different device from the A40 the defect was diagnosed on, so the fix is not specific to the machine that found it. Seven device parity tests pass, including the two-super-block row that failed G1, at widths of 1, 2, 3, 32, 33 and 48 super-blocks. The whole module is 41 passed, 0 skipped, 0 failed. The zero skips are the load-bearing part and the note says so. These tests guard themselves with assumeTrue and skip silently on a host with no CUDA device, so a green build proves nothing by itself. Earlier in this same session a green backend-cuda build on this very host exercised no device code at all, because the branch under test did not contain these tests. Also recorded is a caveat that bites later rather than now. Installing the Ubuntu nvidia-cuda-toolkit pulled NVML 580.178 against a 550.90.07 kernel driver, so nvidia-smi fails with a version mismatch. The CUDA driver API is a separate library and was unaffected, which is why the parity tests ran after the install and still used the device. But NVML is exactly how the pre-registration amendment proposes to measure real watts, so an energy arm needs a matched toolkit or a CUDA-shipping image, and must prove nvidia-smi works before any power figure is trusted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deleted after the label was validated against the expected one rather than trusting the id alone. The exact id returns 404, the account holds zero instances and zero block-storage volumes. About 35 minutes at 0.236/hr, roughly USD 0.14 against a USD 2.00 ceiling. The host is not retained while the CPU harvest runs, since the parity question it was bought to answer is answered and the decisions GPU arm cannot start until a head exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written on registering for the preview and before any access existed, so the comparison protocol is fixed while there is still nothing to tune it against. No Jev output has been seen and no sealed split has been opened. The arm runs on the identical frozen splits, the identical state under the identical window policy, and one question phrasing fixed before either arm ran. That last constraint is the load-bearing one. An independent review reports that Jev's accuracy swings with how the question is asked, which is the easiest place in this whole comparison to manufacture a flattering result without technically lying: search phrasings for ours, take the default for theirs. Neither arm's phrasing may be tuned, and any sweep is run for both. Four numbers per arm per corpus, kept separate rather than blended, because the field's coverage of Jev notes its published accuracy is better understood as agreement with a model-derived reference than as verified correctness. Accuracy against published labels is the only one that is correctness; agreement with our own decode verdict is a different quantity; calibration is per corpus and never pooled; cost is wall, CPU and money. The majority-class floor sits beside all of them. Recorded in advance is what losing looks like. If Jev is more accurate on all three corpora that goes in the headline rather than a footnote. If our advantage is only latency and locality then the claim is latency and locality, and does not become a quality claim by adjacency. If the margin is inside the noise -- entirely possible at sealed splits of 300, 300 and 75 -- the finding is that the two are indistinguishable on this evidence, which is not a win. Use is evaluation only. Jev outputs do not become training data, do not fit a head and do not fit a temperature; our teacher stack is deliberately Apache-2.0 and MIT so no trained weight carries a licence question, and a benchmark arm must not reintroduce one. The preview terms are to be read and recorded before the first call, and if evaluation use is not permitted the arm does not run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured, not guessed: SQuAD ran at 6.09 s/item and MS MARCO at 7.20, both close to prediction, while CUAD's items are an order of magnitude larger because the 4,000-token window is doing what it was written to do. At about 121 s/item, N=1000 needs roughly 33.6 hours against a host deadline about 15.9 hours out. CUAD drops to N=250, split 125/50/75, same seed and same passage-grouped splitter. The window stays at 4,000 tokens. Shrinking the window would have been the faster fix and is explicitly rejected: the window is the realistic enterprise shape and the out-of-window gold-span rate is one of the numbers CUAD exists to report. SQuAD and MS MARCO both recorded zero out-of-window items, so CUAD is the only corpus where the policy has any effect, which is exactly why it must not move to buy runtime. The cost is written down rather than hidden. A sealed split of 75 carries a materially wider interval than the 300 of the other two, that n travels with every CUAD figure, and no CUAD margin narrower than 75 samples can support may be called a difference. If the enterprise arm turns on such a margin, the honest report is that this run could not resolve it. Recorded too is that seven CUAD items were harvested when the wrapper script advanced automatically at the end of MS MARCO. The run was stopped within four minutes and those rows are discarded, since they came from an N=1000 draw that this amendment supersedes. No CUAD hidden state has been read and no CUAD label looked at. Gates are unchanged, and the two-corpus requirement is unaffected: SQuAD and MS MARCO are complete at N=1000 each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…queue MS MARCO gives ten retrieved passages per query with an is_selected flag marking those holding the answer. The converter took the first three by position and ignored the flag. Sampled rows put the answer-bearing passages at indices [5], [4,5] and [] -- none in the window fed to the model. So items were given context that genuinely lacks the answer, labelled answerable because MS MARCO records a human answer somewhere, and the model was scored wrong for correctly refusing. It was penalised for being right. The pre-registered sanity condition caught it. Train-split decode accuracy was 0.4660 against a 0.5900 floor, a margin of -0.1240, with the verdict refusing 438 times in 500 while 295 were labelled answerable. Accuracy below the floor is the signature: a model finding a task hard lands near the floor, not beneath it, and beneath means the question asked and the question the label answers are different questions. SQuAD on the same check is 0.7860 against 0.5020, margin +0.2840. Context is now every retrieved passage in rank order, so answerable means answerable from the retrieved set, which is what the label records. New digest recorded beside the old. The thousand rows from the void construction are quarantined rather than reinterpreted; they measured the wrong question. The sealed three hundred were harvested but never read -- the defect was found on the train split, which the protocol permits -- so they are discarded unread and a fresh split is drawn. Recorded plainly: two thousand rows were harvested and reported as progress, with done lines and CPU-seconds and ETAs, without one row being read back. A validation pass over the first twenty-five would have caught this before the second corpus started. Harvest validation before a corpus is allowed to finish is now a required step rather than an optional one. Host deadline extended to 2026-09-21T14:00Z and the ceiling raised to USD 15 to cover the rerun. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…turally TypeSafe's Master Customer Agreement 2.3(b) forbids using the Services or any Output to perform model distillation, train a model to imitate the output, or develop a similar or competing product. Benchmarking, evaluating and publishing comparisons are not restricted -- there is no clause either way. The full review is recorded in THIRD-PARTY-TERMS.md, read before any call was made, as Amendment 2 required. The pre-registered design already complied, because the arm was scoped to evaluation only on licence-hygiene grounds before these terms were read: the head is fitted on hidden states from our own Apache-2.0 base and labels from digest-pinned public corpora, and third-party output only ever joins a corpus label to compute that service's own score. What is added here is enforcement, because a comment promising this is worth nothing. ExternalVerdict holds an id, a probability and a service name and nothing else -- no hidden state, no logits, no feature vector -- so there is nothing on it a head could be fitted to. A test scans the compiled bytecode of every class that fits a head or a temperature and fails if any of them names the external package at all. A third asserts the trainer has exactly one fit overload, taking hidden states and integer corpus labels, so a probability cannot be passed as a soft label, which is the shape distillation would take. The provenance trail is the real defence and it already exists: the pre-registration was committed before access was granted, the corpora are pinned by digest, the teacher is our own base, and the commits are dated. That the head was trained without Jev is demonstrable from the repository rather than claimed. Two questions are recorded for counsel rather than answered here: whether "develop a similar or competing product" reaches independent development that never touches their Output, and whether self-measured latency counts as non-public information under 14.1. Until both are answered the arm runs and the numbers stay internal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pipeline that turns a harvest into a result: fit the head on train, fit the temperature on calibration, read the sealed split exactly once. Built before the harvest lands so that when it does the work is running a command rather than writing analysis under time pressure. The once-only read is enforced by the object, not by discipline. A sealed split that can be scored twice is one that eventually will be, with the second reading kept because the first disappointed, and a test pins the refusal. Accuracy and agreement are separate fields on purpose. Accuracy is measured against the corpus's own labels and is the only one that is correctness; agreement says a head reproduces what the decode loop did, however wrong that was. A test builds a corpus where decode is wrong everywhere and labels are learnable, and requires the report to show high accuracy beside low agreement -- blending them would hide exactly that case, and blending them is what the field's published figures do. Every report carries its majority floor, its sealed size, and its truncation rate, so a margin narrower than the sample supports cannot be quoted as a difference without the reader seeing it. teacherIsInformative applies the pre-registered sanity condition rather than leaving it to be remembered. Pinned too: overlapping splits are refused by id, an empty sealed split is refused rather than scored as perfect, an unlearnable problem is not dressed up as beating its floor, and the fitted temperature is provably independent of the sealed labels -- flipping every sealed label leaves it identical, because calibration never sees them. 78 tests green, full check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bsbodden
force-pushed
the
feat/decisions-noul
branch
from
September 20, 2026 22:09
2392d8e to
9812e2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. The module and its protocol are complete; the experiment that would justify any claim has not run yet. Nothing here reports a result.
What this is
models-decisions: state in, typed probabilistic decisions out, in one pass, with no decode loop anywhere. Three primitives —Noul(binary),Choice(2–255 unordered options),Score(2–10 ordered levels).The answer space is declared before the decision, so the outcome can only be a label the caller named. That bound is structural, not a validation pass applied afterwards. Nothing in the module can generate text, and that is the product property rather than a gap to be closed later. It is enforced: the stub backend fails the test if the evaluator ever reaches
forwardorforwardBatch.Why it can be cheap
SharedPrefixDecisionEvaluatorfreezes the state once and forks every question off the same physical KV prefix, so N decisions over one state cost one prefill plus N head reads. The evaluator asks the backend to prove the sharing throughsharesPrefixStorageand carries the answer out inDecisionBatchbeside the verdicts — timing is not evidence of sharing, and a run that quietly stopped sharing would otherwise look like a run that merely got slower. Pinned in both directions.Verified on the real qualified artifact (
GranitePrefixCapabilityIT, skips cleanly when absent): Granite 4.1 3B Q4_K_M loads, exposes a 2560-wide hidden state, two forks of one prefix returntruefromsharesPrefixStorage, and identical suffixes over a shared prefix agree exactly.Contents
AnswerSpacesealed overNoul/Choice/Score;Verdictbound to the space that produced itCalibration— numerically stable temperature scaling that never moves the winner, plus ECE and Brier as the rules that decide whether a temperature helpedLinearDecisionHeadreading one hidden state in fixed accumulation order, so two reads agree bit for bitLogisticHeadTrainerandTemperatureFitter, both deterministic by constructionSplitPlanner— passage-level separation so a head cannot be scored on a context it was fitted on, natural base rate preserved, seeded Fisher-YatesNoulHarvestTool— decode verdict and hidden state from one prompt evaluationThe module depends on
models-apionly.backend-javaandbackend-nativeare test-scope; a head is wired to whichever backend is qualified.Protocol, committed before any split was opened
benchmark-results/2026-09-20-noul-head/PREREGISTRATION.mdfixes the gates before the data: agreement with the decode verdict ≥ 0.95 on a sealed split, ECE ≤ 0.05, per-decision cost below one decode turn, on all three corpora.Three published corpora in deliberately different genres, pinned by digest, with their real majority-class floors: SQuAD v2 dev (0.5007), MS MARCO v2.1 validation (0.5503), CUAD (0.7025, enterprise legal, 102 contracts). A pass on two and a failure on the third is a failure.
Two guards are declared in advance: a sanity condition that renders the arm UNINFORMATIVE rather than passed if the decode verdict itself cannot beat its floor by 0.10, and a note that agreement and calibration error are deterministic functions of fixed inputs, so n=1 is exact for them and n=3 attaches to latency.
Recorded too is what the design cannot test: transfer to any other base or task, and anything at all about the commercial system that inspired the category, which we have not run and do not have access to.
Not in scope
No result. No claim. The harvest runs next.
🤖 Generated with Claude Code