Remove covered derived intervals after propagation completes - #1073
Conversation
Brings main's merge commit of deeplethe#1069 into dev so dev is up to date with main. Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Merge main into dev
Signed-off-by: Floating-Y <118035379+Floating-Y@users.noreply.github.com>
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks @Floating-Y. This is the cleanup the #1040 review left open, and doing it after propagation rather than during it is the right place: a narrower output with a shorter proof still has to stay in the adjacency lists while chains are being extended, which your last test pins.
Beyond the 102 tests, I compared dev with this branch on 400 deterministic random graphs (transitive, sub-property and inverse predicates, bounded, half-open and open intervals):
| dev | this branch | |
|---|---|---|
| Triples derived | 19,028 | 19,028, the same set |
| Outputs | 34,581 | 25,523 |
| Outputs covered by another output of the same triple | 9,058 | 0 |
| Intervals on dev not covered by an output here | 0 |
So it removes exactly the covered outputs, a quarter of them on these graphs, and loses no interval. The store's derivation tests pass against a migrated database here as well.
One thing to know, not to change: a removed narrow output has already taken a slot of the per-predicate cap, so a predicate can report as capped with fewer than 20,000 outputs left. That errs on the side of saying so. Landing it.
Signed-off-by: Wayland Yang <wayland0916@gmail.com> deeplethe#1073 was opened against main and merged there. This brings it into dev.
Why
When a narrow derivation arrives before a wider derivation of the same triple, both remain in the final output. A sub-property derivation of
A q D [10,20)followed by a transitive derivation over[0,100)reproduces this issue.This follows up on the residual issue recorded in the #1034 closing comment and the cleanup suggested in the #1040 review.
What changes
After propagation completes, group outputs by
(predicate, subject, object)and reusespan_containsto remove intervals covered by another single output. Equal intervals retain one output; incomparable intervals remain separate without merging or union-based coverage. Existing unbounded interval semantics are preserved.The surviving
Derivedretains its complete ordered premises,via, andrule. The reproduction now returns only[0,100), supported by premises102, 103, 104, withvia = qandrule = Transitive.Cleanup preserves #1051's propagation behavior, including narrower but shorter intermediate proofs needed to finish within
MAX_DEPTH. Public interfaces, depth limits, and output-cap semantics remain unchanged.Seven regression tests cover arrival order, input permutations and UUID ordering, interval boundaries, proof preservation, equal-output cleanup, and the narrower-shorter-proof case.
How it was checked
cargo test --locked -p utopia-reason: 102 passed.cargo fmt --all --check: passed.cargo clippy --workspace --all-targets -- -D warnings: passed.cargo test --workspace: 1,426 passed, 0 failed, 6 ignored.cargo build --workspace: passed.web/, frozen-lockfile installation, 228 tests, and the build passed.Workspace tests used a dedicated migrated PostgreSQL 16/pgvector database with
UTOPIA_TEST_REQUIRE_DB=1andUTOPIA_TEST_REQUIRE_PDFTOTEXT=1. An initial run encountered Windows proxy interference; the full suite passed after rerunning with loopback addresses in the test process'sNO_PROXY.Ignored tests, unconfigured external-service integration tests, and the opt-in benchmark were not executed.
Before review
git commit -s)cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warningsandcargo test --workspacepass