Repository navigation
A quote and its time words are found across a line break in the passage - #1081
Merged
Merged
Conversation
A PDF's text layer breaks lines at the margin. The model copies a sentence back with a space where the passage has the break, so the verbatim check found neither the quote nor the date in it: the statement landed without a span and without its time. A board minute with four dated sentences kept none of the dates. A run of whitespace now matches a run of whitespace; everything else is compared as before, and offsets are still positions in the passage. A time word is located in the passage's own sentence, since the copy and the passage may differ in how long the whitespace is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Found in an end-to-end run on dev. An English board minute, as a PDF, lost every date: "will step down as Chief Technology Officer on December 31, 2024" became a statement with no time.
The PDF's text layer breaks lines at the margin, so the passage reads
on December⏎31, 2024. The model copies the sentence back with a space.locatecompared byte for byte, so the quote was not found in the chunk and the date was not found in the quote. The drops table said so: 5 ×quote_not_in_chunk, 3 ×time_not_in_quotefor one page.What changes
locatelets a run of whitespace match a run of whitespace. Everything else is compared as before, and the offsets it returns are still character positions in the passage.locate_timefinds the time words in the passage's own sentence rather than adding the position inside the copy, since the two can differ in how long the whitespace is (a break plus an indent against one space).A word broken in the middle (
Septem⏎ber) still does not match: that is a different word as far as the check can tell.How it was checked
2024-12-31,2025-01-01and2024-09-01on their statements and no drops.cargo test -p utopia-server extraction_open,cargo clippy -p utopia-server --all-targets -- -D warnings,cargo fmt --all --check.Before review
🤖 Generated with Claude Code