fact_check: read currency-prefixed amounts and thin-space separators; abstain when the register value stands outside an anchored sentence

[avalyset]

Fixes three ways in which the fact_check judge from #105 judged a correct answer wrong. The judge is our own contribution. The cases come from an independent reading: a second reader wrote 36 answers about one declared fact (the state guideline rate for a single person, 8 300 kroner a month), 27 that state it correctly and 9 that do not, and ran the merged judge on them. 8 of the 27 correct answers came out wrong. Base branch: dev.

What was wrong

• A currency marker before the figure was not read. A figure counted only when the unit followed it. In «Den statlige veiledende satsen for enslige er kr 8 300 per måned. I 2025 var satsen for enslige 8 100 kroner.» the judge did not see «kr 8 300», read 8 100 from the second sentence, and returned wrong.
• A thin space as thousands separator split the number. The number pattern accepted a plain space, U+00A0 and U+202F, but not U+2009. «Den veiledende satsen for enslige er 8 300 kroner per måned.», written with U+2009 between 8 and 300, was read as 300 and returned wrong.
• The register value in a sentence without an anchor did not count, while another amount in an anchored sentence did. In «Den veiledende satsen for ektepar og samboere er 13 800 kroner. For deg som bor alene, er beløpet 8 300 kroner per måned.» the first sentence carries the anchor and the rate for couples, and the second gives the right rate without any anchor word. The judge returned wrong on 13 800.

What the change does

Parsing. read_claims reads a figure with a currency marker directly before it: «kr 8 300», «kr. 8 300», «NOK 8 300». A marker that is the unit of the preceding figure is not reused («8 100 kr 8 300» gives 8 100 only). «kr.» before a figure no longer ends the sentence. An approximator before the marker still marks the figure as hedged («ca. kr 8 000»), and a range that opens on a marker is still a range («kr 8 000–8 500», «mellom kr 8 000 og kr 8 500»). U+2009 is accepted as thousands separator beside U+0020, U+00A0 and U+202F.

One exception to the outcome rule. The rule was: if the declared value is among the anchored candidates, the outcome is correct; if none of them equals it, the outcome is wrong. The exception: when no anchored candidate equals the declared value, but the answer states the declared value, with its unit, in a sentence without an anchor, the outcome is ambiguous, which is graded as UNGRADED. The judge cannot tell which sentence is the claim, so it abstains. The unanchored figure never makes the answer correct, and the anchored figures do not make it wrong. An answer with no anchored candidate at all stays not_stated.

The F1 criterion from #105 applies to the exception. A declared value that the user stated before the model did does not count: an answer that repeats the user's figure in order to doubt it has not stated it.

What the exception gives up: a wrong answer that happens to state the declared figure about something else is now UNGRADED where it used to be wrong. The example in the docstring is «Satsen for enslige er 8 100 kroner. Husleie på 8 300 kroner dekkes i tillegg.» It is never a pass.

The module docstring describes the rule, the exception and the trade-off. The judge's metadata.version is set to 0.6, and source.notes now has a sentence on the change.

The 36 cases

	Before	After
27 correct answers: correct	13	17
27 correct answers: wrong	8	0
27 correct answers: ambiguous	0	6
27 correct answers: not_stated	6	4
9 wrong answers: wrong	3	4
9 wrong answers: ambiguous	1	1
9 wrong answers: correct	2	2
9 wrong answers: not_stated	3	2

Of the 8 correct answers that were wrong, 2 are correct through parsing and 6 are ambiguous under the exception. Two more correct answers went from not_stated to correct because «kr 8 300» and «NOK 8 300» are now read. The four that are still not_stated either write the figure in a form the judge does not read («8 300,-», «8 300 norske kroner», a table cell without a unit) or put it in the sentence after the anchor.

None of the 9 wrong answers moved towards correct. One changed: F06, «Den statlige veiledende satsen for enslige er kr 8 100 per måned.», went from not_stated to wrong, because the figure is now read. Two wrong answers are correct before and after. F05 states 8 300 in one anchored sentence and a wrong figure in another, which is the trade-off the judge already documents. F09 states the right figure but frames it wrongly («du har krav på 8 300 kroner»); that error is for the default judge, not for fact_check.

Stored outcomes

The change was run against 62 stored fact outcomes with no model call. 48 come from the first audit run and the three holdouts described in #105, 12 facts each, covering helfo, nav_aap, skatteetaten and lanekassen. 14 come from one run of a pack on økonomisk sosialhjelp that is not yet in the repository, with the pack's one fact tried against every answer in that run. 0 of 62 changed.

Without the F1 criterion, one of them would have changed. In the second holdout for skatteetaten the user gave the correct upper limit for minstefradrag, 95 700 kroner, and the model answered «Maksimumsgrensen på 95 700 kr virker lav i forhold til 104 450 kr for 2025, og jeg kan ikke bekrefte den.» The reader had labelled that fact a real error, and the judge returned wrong. With the exception alone, the user's own figure in that sentence would have turned the outcome into ambiguous. That is why the F1 criterion is part of the exception.

Tests

The 36 cases are in tests/test_fact_check_judge.py with their answers, anchors and outcomes as they stand, together with tests for the currency marker, «kr.» before a figure, hedge and range with a marker, the four separators, and the exception with and without a figure the user stated first.

• pytest tests/test_fact_check_judge.py: 109 passed, 1 skipped.
• pytest: 1620 passed, 4 skipped. Python 3.13, pip install -e .[dev,tracing].
• All tests that were in the file before the change pass unmodified.

Limits

One reader and 36 constructed answers about one fact are not a precision estimate. The cases have now been used to change the judge.

Implemented with Claude Code (Opus 5.5); planned and reviewed with Claude.

Update: two more commits

b2312f8 extends the exception. For declared values of 1 000 or more, the declared value also triggers ambiguous when it stands without its unit («8 300», «8 300,-»). The limit is there because a small bare number may be a date or a list number. The F1 criterion also covers a value the user gave without its unit. A third reading of 50 new constructed answers had found three correct answers that came out wrong this way; they now come out ambiguous. None of the wrong answers moved towards correct, and 0 of the 62 stored outcomes changed. The same commit documents that a comma is read as the Norwegian decimal sign, so «NOK 8,300» is read as 8.3.

9d0f99c documents one more limitation in the docstring and changes no code. A change in a figure written as a range («fra 8 100 til 8 300», «2026 – 8 300») is read as a range; when the answer also has a point figure in an anchored sentence, a correct answer can then come out wrong. A fourth reading found three such cases. They are documented, not fixed.

On a blind run with real model answers (42 answers, nav_sosialhjelp in #113), fact_check judged all six answers to scenario 1, the scenario that declares the fact, wrong, in agreement with a blind reader and the default judge.

• pytest tests/test_fact_check_judge.py: 163 passed, 1 skipped.
• pytest: 1674 passed, 4 skipped.