Skip to content

A quote and its time words are found across a line break in the passage - #1081

Merged
WaylandYang merged 2 commits into
devfrom
fix/quotes-match-across-line-breaks
Oct 4, 2026
Merged

WaylandYang merged 2 commits into
devfrom
fix/quotes-match-across-line-breaks

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Why

Found in an end-to-end run on dev. An English board minute, as a PDF, lost every date: "will step down as Chief Technology Officer on December 31, 2024" became a statement with no time.

The PDF's text layer breaks lines at the margin, so the passage reads on December⏎31, 2024. The model copies the sentence back with a space. locate compared byte for byte, so the quote was not found in the chunk and the date was not found in the quote. The drops table said so: 5 × quote_not_in_chunk, 3 × time_not_in_quote for one page.

What changes

  • locate lets a run of whitespace match a run of whitespace. Everything else is compared as before, and the offsets it returns are still character positions in the passage.
  • locate_time finds the time words in the passage's own sentence rather than adding the position inside the copy, since the two can differ in how long the whitespace is (a break plus an indent against one space).

A word broken in the middle (Septem⏎ber) still does not match: that is a different word as far as the check can tell.

How it was checked

  • New unit test: a sentence and its date found across a break with an indent, the offsets pointing at the passage; the reverse case; a mid-word break and a changed date still refused.
  • End to end on a real server with deepseek-flash: a PDF whose lines break between words. Before: no dates and 8 drops. After: 2024-12-31, 2025-01-01 and 2024-09-01 on their statements and no drops.
  • cargo test -p utopia-server extraction_open, cargo clippy -p utopia-server --all-targets -- -D warnings, cargo fmt --all --check.

Before review

  • Every commit is signed off
  • fmt, clippy and the affected tests pass

🤖 Generated with Claude Code

WaylandYang and others added 2 commits October 4, 2026 21:01
A PDF's text layer breaks lines at the margin. The model copies a sentence
back with a space where the passage has the break, so the verbatim check
found neither the quote nor the date in it: the statement landed without a
span and without its time. A board minute with four dated sentences kept none
of the dates.

A run of whitespace now matches a run of whitespace; everything else is
compared as before, and offsets are still positions in the passage. A time
word is located in the passage's own sentence, since the copy and the passage
may differ in how long the whitespace is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit 256a388 into dev Oct 4, 2026
6 checks passed
@WaylandYang
WaylandYang deleted the fix/quotes-match-across-line-breaks branch October 4, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant