Conversation
`filter_by=A && B` costs what its wider side costs, whatever the conjunction matches. `compute_iterators()` materialises both subtrees and intersects the two results, so a filter that narrows to a hundred documents still pays for the leaf that matches two million. The estimate the node makes at construction time is right -- `min(left, right)` really is the size of the result -- but a small result says nothing about the cost of computing it, and here that cost is bounded by the larger side. An `&&` yields at most as many ids as its narrower side. When the other side is at least `AND_PROBE_RATIO` times wider, materialise the narrow side alone and ask the wide side about each of the ids it yields, through the `is_valid(id)` that `and_filter_iterators()` already uses to advance a lazy subtree. The ids come out in the same order, so the node's result, its validity, its `approx_filter_ids_length` and everything downstream of them are unchanged. It is a different plan for the same query. The plan applies where the wide side is still lazy when its parent `&&` is initialised: any string leaf above `string_filter_ids_threshold`, in every configuration, and an integer or float leaf when `enable_lazy_filter` is on. A range-index, geo or `id:*` leaf has materialised itself inside its own `init` before the `&&` node exists; probing it is still correct, and each probe is then a binary search, but it cannot give back a cost that was paid one level down. It is skipped, and both sides materialised as before, when either subtree filters on a referenced collection -- intersecting drops an id whose reference results have nothing in common, and probing has no such step -- when the node is an object filter root, and when the wide side is an operator rather than a leaf. Timeouts keep working the way they did: both children get a copy of the budget before anything is computed, the counted check inside `is_valid` cuts the probe loop short, and the forced check after it covers a loop too short to reach the clock. A -1 from `is_valid` is not read as a timeout -- in a lopsided `&&` it almost always means the wide side ran out of ids, which is an ordinary complete result -- so `validity` is what distinguishes the two. Measured on release builds, 2M documents, one selective and one broad string value, single request: the conjunction goes from 92 ms to 1 ms while its selective leaf alone costs 1 ms and its broad leaf 137 ms, from either operand order, returning the same 95 documents. A numeric wide side under the default configuration is unchanged at 37 ms, which is the scope limit above. Tests ----- `compute_iterators()` deletes both subtrees before it returns, so nothing left on the node says which plan computed it. `computed_by_probe` does, and `_get_computed_by_probe()` exposes it the way the other `_get_*` helpers do, so a test fails if a later change quietly goes back to materialising both sides. `AndProbeSelectiveSide` builds a collection whose fields differ in selectivity by a known factor and asserts both plans return the same ids in the same order for the reported shape and either operand order, for a wide side that runs out of ids first, for a `!=` wide side, for three leaves, for an operator on either side, and for a side that matches nothing or everything. `AndProbeNumericSide` does the same for a numeric wide side, lazy and eager, the eager case documenting the scope limit. `AndProbeRatioBoundary` pins the gate at 39 and 40 ids against a narrow side of 5. `AndProbeTimeout` covers a probe loop cut short by the counted check in `is_valid`, one too short to reach the clock and left to the forced check, and the two cases without a budget that must not report a timeout -- including the one where the wide side runs out of ids, which returns the same -1 a timeout does. `AndProbeKeepsReferences` joins a collection and asserts the node keeps intersecting and its reference results survive. End to end, `SelectiveAndDoesNotMaterializeWideSide` sizes its fixture around the two thresholds a test build inverts: 25 documents on the narrow side, over the 20 that make `Index::search` compute the iterator, against 500 on the wide one. Benchmark --------- The corpus the benchmark already downloads has the shape a conjunction needs to be expensive. `release_group_types:[Album,Single,Compilation]` matches 960,372 of the million songs and `primary_artist_name:Nirvana` matches 431, so the `&&` of the two returns 431 documents and, before this change, cost what the 960,372 cost: 31 ms before, 1 ms after, the same 431 documents. The scenario was chosen by measuring the alternatives. Against the same narrow side the gain tracks the width of the wide side: `release_decade:2000s` (631,584) gives 13 -> 2 ms, `release_group_types:Album` (913,296) gives 20 -> 1 ms, and the three release types above give 31 -> 1 ms. Going narrower than a few hundred documents on the selective side drives the result below the millisecond the API reports, which would leave the harness computing a percentage change against zero, so the narrow side stops there. `filter_complex` already covers this shape by accident -- `&&` and `||` are left-associative, so it parses as `(genres:Rock && primary_artist_name:Queen) || primary_artist_name:Led Zeppelin`, and that inner conjunction is 960 against 214,468. It went from 8 ms to 2 ms on the same corpus. But it measures the plan only incidentally, mixed in with an `||` and a second artist, so a regression there would be hard to read. The new scenario isolates it. The threshold matches `filter_simple`, the closest scenario in absolute cost: the `milliseconds` ceilings are p95 under 50 and 100 virtual users, and the k6 stack needs Docker, InfluxDB and Grafana, so this was not measured under load. The `percentage` check is what makes the scenario a regression guard regardless. `BenchmarkConfigSchema` derives its keys from `searchScenarios` and refines that every one of them has a threshold, so the scenario and its threshold cannot be separated.
Member
|
My concern with the 1:8 ratio heuristic is that it also selects probing when both sides are large, e.g. 1M matches AND 9M matches means up to 1M lookups, which could be slower than a sequential intersection, especially if the larger side is already materialized. There is an earlier PR by @tharropoulos in #2981 that gets this aspect right by requiring the smaller side’s estimated count to be below COMPUTE_FILTER_ITERATOR_THRESHOLD (25,000). Could we retain #3049’s implementation and safeguards, but adopt that cap alongside the 8× ratio, and skip probing when the larger side is already materialized Checking the actual smaller-side count after computing it would also protect against underestimated counts. Please also benchmark 1M versus 9M against the existing intersection path, with both a lazy string broad side and an already-materialized numeric broad side (enable_lazy_filter=false). Clustered and scattered candidate IDs, with latency and peak-memory measurements, would help establish where probing pays off. |
Contributor
|
I can port the changes from here onto my PR |
Contributor
|
Benchmarked inside a container on Ubuntu 20.04, aarch64, using production For 1M candidates versus 9M broad matches, every plan returned the same 900,000 IDs.
For smaller candidate sets against the same 9M-match lazy string side:
Probing the already-materialized numeric side was ~25–26% slower at 1M candidates, without meaningful peak-memory savings. The revised implementation avoids that regression. Below the cap, lazy-string probing reduces latency and saves approximately 140 MiB of peak RSS. Probing is still faster at 1M for the string workload, so 25,000 can be tweaked, or even be passed as a server parameter instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
filter_by=A && Bcosts what its wider side costs, regardless of what the conjunction matches.filter_result_iterator_t::compute_iterators()materialises both subtrees and intersects the two id arrays, so a filter that narrows to a hundred documents still pays for the leaf that matches two million.The node's estimate is already correct —
min(left, right)really is the size of the result — but a small result says nothing about the cost of computing it, and that cost is bounded by the larger side.An
&&yields at most as many ids as its narrower side. When the other side is at leastAND_PROBE_RATIO(8) times wider, this materialises the narrow side only and asks the wide side about each id it yields, via the existingis_valid(id)— the same calland_filter_iterators()already uses to advance a lazy subtree. Same ids, same order, sameapprox_filter_ids_length. It is a different plan for the same query, not a change in semantics. No new flag.Scope
Pays off where the wide side is still lazy when its parent
&&is initialised:string_filter_ids_threshold, in every configuration;enable_lazy_filteris on.A range-index, geo or
id:*leaf has already materialised inside its owninitbefore the&&node exists. Probing it is still correct — each probe is then a binary search — but it cannot give back a cost paid one level down.Falls back to intersecting both sides when either subtree filters on a referenced collection (intersecting drops an id whose reference results don't intersect; probing has no such step), when the node is an object filter root, and when the wide side is an operator rather than a leaf.
Timeouts are unchanged: both children get a copy of the budget before anything is computed, the counted check inside
is_validcuts the probe loop short, and a forced check after the loop covers a loop too short to reach the clock. A-1fromis_validis not read as a timeout — in a lopsided&&it almost always means the wide side ran out of ids, which is an ordinary complete result — sovaliditydistinguishes the two.Benchmarks
Apple M1 Max, 10 cores, 64 GB, macOS 26.6.2. Release builds (
bazel build //:typesense-server,-O2).origin/v31vs this branch, same on-disk index, single request, no concurrency.foundis identical in every row.musicbrainz-1M-songs— the corpusbenchmark/already downloadsfilter_selective_and(added here)filter_complex(existing)filter_simple(existing, single leaf — control)release_group_types:[Album,Single,Compilation]alonefilter_complexalready exercised this path by accident:&&and||are left-associative, so it parses as(genres:Rock && primary_artist_name:Queen) || primary_artist_name:Led Zeppelin, and that inner conjunction is 960 against 214,468.Synthetic, 2M documents
Two string fields: one whose value matches 100 documents (the selective side), one whose value matches 1,900,000 (the broad side).
filter_by&&broad&&selectivefilter_byThe conjunction reaches its selective leaf from either operand order. The broad leaf's 137 → 145 ms is run-to-run noise measured minutes apart; a single leaf never reaches the operator branch that changed. The numeric wide side is unchanged, which is the scope limit above, not a broken probe.
Tests
compute_iterators()deletes both subtrees before returning, so a privatecomputed_by_probewith a_get_computed_by_probe()accessor is what lets a test assert which plan ran.AndProbeSelectiveSide— result equivalence for both operand orders, a wide side that runs out of ids first, a!=wide side, three leaves, an operator on either side, and a side matching nothing or everything.AndProbeNumericSide— numeric wide side, lazy and eager; the eager case documents the scope limit.AndProbeRatioBoundary— 39 vs 40 ids against a narrow side of 5.AndProbeTimeout— a probe loop cut short by the counted check, one too short to reach the clock and left to the forced check, and two cases without a budget that must not report a timeout, including the one where the wide side runs out of ids and returns the same-1a timeout does.AndProbeKeepsReferences— a joined collection still intersects and keeps its reference results.CollectionFilteringTest.SelectiveAndDoesNotMaterializeWideSide— end to end, sized around the thresholds a test build inverts.Full suite run before and after: the same 21 tests fail on this machine either way (macOS cgroup parsing, ONNX model tests, NL-search model manager, metrics quantiles), so no regression.
FilterTest.ObjectFitlterIteratorandFilterTest.FilterReferencespass unchanged.