diff --git a/book/src/drive/document-ranked-trees.md b/book/src/drive/document-ranked-trees.md index 07db94bad79..6197c622352 100644 --- a/book/src/drive/document-ranked-trees.md +++ b/book/src/drive/document-ranked-trees.md @@ -91,7 +91,7 @@ One asymmetry is worth knowing when authoring: **the meta-schema demands the lit Two structural rules, both enforced at contract-parse time in rs-dpp: -- **Single-property indexes only.** `ranked aggregates are only supported on single-property indexes in this protocol version`. Two reasons, both relaxable at a future protocol version. First, a compound index whose *prefix* level also terminates an aggregating index would need its ranked terminal tree wrapped in a `NonCounted` / `NotSummed` shell so it contributes zero to the parent's aggregate — and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard for this is `INDEXED_INNER_UNWRAPPABLE`.) Second, the ranked query surface has no equality-prefix routing: with more than one property there would be a prefix to fix before ranking, and nothing to express it with. +- **No aggregating index on a compound ranked index's full prefix.** Ranked flags are allowed on compound indexes, with **per-prefix** semantics: a ranked `[identityId, class]` puts the indexed tree at each prefix value's terminal `class` property-name level — one ordered secondary per `identityId`, each ranking only that identity's `class` groups. There is deliberately no global cross-prefix ordering; the query surfaces require every leading property to be pinned by an equality `where` clause. The one shape that stays impossible — and is rejected per document type, where all indexes are visible (`validate_no_ranked_prefix_overlap`) — is a countable/summable index terminating at exactly the compound's leading prefix: its aggregating value trees would demand the `NonCounted` / `NotSummed` shell around the ranked terminal tree, and the storage layer structurally rejects any wrapper around an indexed tree, because the wrapper would neutralise the very aggregates the secondaries order by. (Drive's fail-closed guard behind the parse-time check is `INDEXED_INNER_UNWRAPPABLE`.) Only the exact `n-1` prefix conflicts: an aggregating index at a shorter prefix wraps a plain intermediate tree, and one extending past the ranked terminal lives inside its value trees — both supported. - **Non-unique indexes only.** `ranked aggregates are not supported on unique indexes: each group of a unique index contains at most one document, so there is nothing meaningful to rank`. Contested indexes are covered transitively — a contested index is unique by construction, so it hits the same check rather than needing its own. ### Version Gate @@ -231,9 +231,9 @@ Every ranked read — and, on the prove path, every ranked proof — is issued a / // e.g. b"restaurantId" ``` -The children of that tree are the *groups*: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a `string` property, its UTF-8 bytes — e.g. `b"alpha"`). The secondary entries a top-k read returns are keyed by those same group keys. A compound index `[a, b]` inserts ` / ` between the doctype and the terminal `` level — which is exactly the shape ranked indexes don't support yet. +The children of that tree are the *groups*: one value tree per distinct value of the last index property, keyed by the raw index-key bytes of that value (for a `string` property, its UTF-8 bytes — e.g. `b"alpha"`). The secondary entries a top-k read returns are keyed by those same group keys. A compound index `[a, b]` inserts ` / ` between the doctype and the terminal `` level — the value segment comes from the request's equality `where` pin on `a`, encoded with the same `serialize_value_for_key` the write path used to key that prefix's value tree, so the walk lands on **that prefix's own** indexed tree and secondary. -Prover and verifier build this path through the same function, `DriveDocumentRankedQuery::indexed_property_name_tree_path`, which is why they agree on the root hash by construction. +Prover and verifier build this path through the same function, `DriveDocumentRankedQuery::indexed_property_name_tree_path` (with the pinned prefix values encoded by the shared resolver, `resolve_ranked_query_for_mode`), which is why they agree on the root hash by construction. ## Write-Path Cost: The Grove v4 Cleanup Gates @@ -265,7 +265,7 @@ A demoted `CountSumTree` value tree contributes its `(count, sum)` to a ranked i - each group's value tree demotes from `ProvableCountProvableSumTree` to `CountSumTree`; - the `chefId` continuation inside it goes in `Element::NonCounted`, contributing zero to the group's count and sum. -The one place the two changes genuinely collide is the case the single-property rule already forbids: a ranked *terminal* level sitting inside an aggregating value tree would need a wrapper, and an indexed tree can never be wrapped. That is the `INDEXED_INNER_UNWRAPPABLE` guard, and it fails closed. +The one place the two changes genuinely collide is the case the prefix-overlap rule already forbids at contract-parse time: a ranked *terminal* level sitting inside an aggregating value tree would need a wrapper, and an indexed tree can never be wrapped. That is the `INDEXED_INNER_UNWRAPPABLE` guard, and it fails closed. ## Storage-Layout Invariants @@ -337,11 +337,11 @@ Note that the fixture puts each shape on its **own document type**. That's not a | You want | Set | |---|---| -| Top / bottom K groups by document count | `rankedCountable: true` on a single-property, non-unique index that already has `countable` + `rangeCountable: true` | +| Top / bottom K groups by document count | `rankedCountable: true` on a non-unique index that already has `countable` + `rangeCountable: true` | | Top / bottom K groups by sum of a property | `rankedSummable: true` on an index with `summable: ""` + `rangeSummable: true` | | Top / bottom K groups by average of a property | `rankedAverageable: true` on an index with `averageable: ""` + `rangeAverageable: true` (or the count+sum longhand) | | Two rankings on one index (e.g. by count *and* by average) | Both keywords. The tree is a PCPSIT carrying both axes in its TLV; you pay one secondary Merk per axis on every write. | -| A ranking filtered by another property (`top 5 restaurants in London`) | Not available. Ranked indexes are single-property and ranked queries take no `where` clause — the secondary is sorted by aggregate, not by group key, so it cannot express a filtered subset. Model the filter as part of the grouping property, or rank client-side over a range query. | +| A ranking filtered by another property (`top 5 restaurants in London`) | A **compound ranked index** with the filter property leading: `[city, restaurantId]` with the ranked flags. Each city gets its own secondary; the query pins the prefix with an equality `where` (`WHERE city == "London" GROUP BY restaurantId ORDER BY DESC LIMIT 5`). Only equality pins — a range or `IN` on the prefix is rejected, and there is no cross-prefix (global) ordering on a compound ranked index. | | A ranking on a unique or contested index | Not available, and not meaningful: every group holds at most one document. | | Range aggregates without ranking (the 4.0 surface) | Just the `range*` flags. Ranking is strictly additive — adding it never changes what a range query returns. | | Nothing ranking-aware (default) | Don't set any `ranked*` flag. The terminal property-name tree keeps the type its range flags give it. | diff --git a/packages/dapi-grpc/clients/platform/v0/objective-c/Platform.pbobjc.h b/packages/dapi-grpc/clients/platform/v0/objective-c/Platform.pbobjc.h index 295993f3f5e..c4e66dadcb9 100644 --- a/packages/dapi-grpc/clients/platform/v0/objective-c/Platform.pbobjc.h +++ b/packages/dapi-grpc/clients/platform/v0/objective-c/Platform.pbobjc.h @@ -2941,16 +2941,16 @@ typedef GPB_ENUM(GetDocumentsRequest_GetDocumentsRequestV1_Start_OneOfCase) { * - a is the In field AND b is the range field, in that order → existing compound distinct shape; entries carry both `in_key` (= a's value) and `key` (= b's value). * * `select=, group_by=[p], order_by=[]` (protocol v14+) — **ranked mode**: - * - exactly one `group_by` property, exactly one `order_by` clause naming the select's aggregate (`f` for `SUM(f)` / `AVG(f)`, the `$count` sentinel for `COUNT(*)`), a `limit` in `1 ..= 100`, an optional `offset`, and no `where` / `having` / `start_at`, on an index declaring the matching `rankedCountable` / `rankedSummable` / `rankedAverageable` axis → ranked executor, answered in `ResultData.ranked`. + * - exactly one `group_by` property, exactly one `order_by` clause naming the select's aggregate (`f` for `SUM(f)` / `AVG(f)`, the `$count` sentinel for `COUNT(*)`), a `limit` in `1 ..= 100`, an optional `offset`, and no `having` / `start_at`, on an index declaring the matching `rankedCountable` / `rankedSummable` / `rankedAverageable` axis → ranked executor, answered in `ResultData.ranked`. On a single-property ranked index no `where` is accepted; on a compound ranked index every leading index property must be pinned with an `EQUAL` where clause (one per property, `group_by` names the trailing property), selecting which prefix's ranking is read. * - `DESC` is the "top n" reading (walk the axis from the largest aggregate down), `ASC` the "bottom n" reading. Worked example: `SELECT AVG(grade) GROUP BY restaurantId ORDER BY grade DESC LIMIT 1 OFFSET 4` is the 5th-best restaurant. * * `select=, group_by=[p], having=[ ]` (protocol v14+) — **having-range mode**: - * - exactly one `group_by` property, exactly one `having` clause whose aggregate is the select's aggregate, an operator describing one contiguous range (`EQUAL`, `GREATER_THAN[_OR_EQUALS]`, `LESS_THAN[_OR_EQUALS]`, `BETWEEN*`; `NOT_EQUAL` / `IN` rejected), a `limit` in `1 ..= 100`, an optional `order_by` naming the same aggregate (walk direction; ascending by default), and no `where` / `offset` / `start_at` / `start_after`, on an index declaring the matching ranked axis → having-range executor, answered in `ResultData.ranked`. + * - exactly one `group_by` property, exactly one `having` clause whose aggregate is the select's aggregate, an operator describing one contiguous range (`EQUAL`, `GREATER_THAN[_OR_EQUALS]`, `LESS_THAN[_OR_EQUALS]`, `BETWEEN*`; `NOT_EQUAL` / `IN` rejected), a `limit` in `1 ..= 100`, an optional `order_by` naming the same aggregate (walk direction; ascending by default), and no `offset` / `start_at` / `start_after`, on an index declaring the matching ranked axis → having-range executor, answered in `ResultData.ranked`. `where` follows the same rule as ranked mode: none on a single-property ranked index; exactly one `EQUAL` pin per leading index property on a compound ranked index. * - no offset or cursor pagination: a page cut at `limit` continues only by tightening the bound past the last *distinct* aggregate value seen; a cut inside a tie (several groups sharing the boundary aggregate) cannot be continued, so size `limit` above the widest expected tie. * * **Rejected shapes** (return `Unsupported`): - * - any non-empty `having` on protocol v13 and earlier; at v14+, any `having` shape outside having-range mode above (multiple clauses, an aggregate other than the select's, `NOT_EQUAL` / `IN`, or a carried `where` / `offset` / cursor). - * - at v14+: a ranked-shaped request carrying a `where` clause, a `start_at` / `start_after` cursor, more than one `order_by`, or an `order_by` naming anything but the selected aggregate. + * - any non-empty `having` on protocol v13 and earlier; at v14+, any `having` shape outside having-range mode above (multiple clauses, an aggregate other than the select's, `NOT_EQUAL` / `IN`, a `where` shape other than the compound-index equality pins above, or a carried `offset` / cursor). + * - at v14+: a ranked-shaped request carrying a `where` shape other than the compound-index equality pins above (a non-`EQUAL` operator, a repeated or non-leading property, or a missing pin), a `start_at` / `start_after` cursor, more than one `order_by`, or an `order_by` naming anything but the selected aggregate. * - `select=DOCUMENTS` with non-empty `group_by`. * - `select=COUNT` with `group_by` on a field that is not constrained by an `In` or range where clause. * - `select=COUNT` with `group_by.len() > 2`. diff --git a/packages/dapi-grpc/protos/platform/v0/platform.proto b/packages/dapi-grpc/protos/platform/v0/platform.proto index bfa0b5cf68b..937962817c3 100644 --- a/packages/dapi-grpc/protos/platform/v0/platform.proto +++ b/packages/dapi-grpc/protos/platform/v0/platform.proto @@ -893,16 +893,16 @@ message GetDocumentsRequest { // - a is the In field AND b is the range field, in that order → existing compound distinct shape; entries carry both `in_key` (= a's value) and `key` (= b's value). // // `select=, group_by=[p], order_by=[]` (protocol v14+) — **ranked mode**: - // - exactly one `group_by` property, exactly one `order_by` clause naming the select's aggregate (`f` for `SUM(f)` / `AVG(f)`, the `$count` sentinel for `COUNT(*)`), a `limit` in `1 ..= 100`, an optional `offset`, and no `where` / `having` / `start_at`, on an index declaring the matching `rankedCountable` / `rankedSummable` / `rankedAverageable` axis → ranked executor, answered in `ResultData.ranked`. + // - exactly one `group_by` property, exactly one `order_by` clause naming the select's aggregate (`f` for `SUM(f)` / `AVG(f)`, the `$count` sentinel for `COUNT(*)`), a `limit` in `1 ..= 100`, an optional `offset`, and no `having` / `start_at`, on an index declaring the matching `rankedCountable` / `rankedSummable` / `rankedAverageable` axis → ranked executor, answered in `ResultData.ranked`. On a single-property ranked index no `where` is accepted; on a compound ranked index every leading index property must be pinned with an `EQUAL` where clause (one per property, `group_by` names the trailing property), selecting which prefix's ranking is read. // - `DESC` is the "top n" reading (walk the axis from the largest aggregate down), `ASC` the "bottom n" reading. Worked example: `SELECT AVG(grade) GROUP BY restaurantId ORDER BY grade DESC LIMIT 1 OFFSET 4` is the 5th-best restaurant. // // `select=, group_by=[p], having=[ ]` (protocol v14+) — **having-range mode**: - // - exactly one `group_by` property, exactly one `having` clause whose aggregate is the select's aggregate, an operator describing one contiguous range (`EQUAL`, `GREATER_THAN[_OR_EQUALS]`, `LESS_THAN[_OR_EQUALS]`, `BETWEEN*`; `NOT_EQUAL` / `IN` rejected), a `limit` in `1 ..= 100`, an optional `order_by` naming the same aggregate (walk direction; ascending by default), and no `where` / `offset` / `start_at` / `start_after`, on an index declaring the matching ranked axis → having-range executor, answered in `ResultData.ranked`. + // - exactly one `group_by` property, exactly one `having` clause whose aggregate is the select's aggregate, an operator describing one contiguous range (`EQUAL`, `GREATER_THAN[_OR_EQUALS]`, `LESS_THAN[_OR_EQUALS]`, `BETWEEN*`; `NOT_EQUAL` / `IN` rejected), a `limit` in `1 ..= 100`, an optional `order_by` naming the same aggregate (walk direction; ascending by default), and no `offset` / `start_at` / `start_after`, on an index declaring the matching ranked axis → having-range executor, answered in `ResultData.ranked`. `where` follows the same rule as ranked mode: none on a single-property ranked index; exactly one `EQUAL` pin per leading index property on a compound ranked index. // - no offset or cursor pagination: a page cut at `limit` continues only by tightening the bound past the last *distinct* aggregate value seen; a cut inside a tie (several groups sharing the boundary aggregate) cannot be continued, so size `limit` above the widest expected tie. // // **Rejected shapes** (return `Unsupported`): - // - any non-empty `having` on protocol v13 and earlier; at v14+, any `having` shape outside having-range mode above (multiple clauses, an aggregate other than the select's, `NOT_EQUAL` / `IN`, or a carried `where` / `offset` / cursor). - // - at v14+: a ranked-shaped request carrying a `where` clause, a `start_at` / `start_after` cursor, more than one `order_by`, or an `order_by` naming anything but the selected aggregate. + // - any non-empty `having` on protocol v13 and earlier; at v14+, any `having` shape outside having-range mode above (multiple clauses, an aggregate other than the select's, `NOT_EQUAL` / `IN`, a `where` shape other than the compound-index equality pins above, or a carried `offset` / cursor). + // - at v14+: a ranked-shaped request carrying a `where` shape other than the compound-index equality pins above (a non-`EQUAL` operator, a repeated or non-leading property, or a missing pin), a `start_at` / `start_after` cursor, more than one `order_by`, or an `order_by` naming anything but the selected aggregate. // - `select=DOCUMENTS` with non-empty `group_by`. // - `select=COUNT` with `group_by` on a field that is not constrained by an `In` or range where clause. // - `select=COUNT` with `group_by.len() > 2`. diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs index d54aac7b187..c334f4b586f 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs @@ -129,6 +129,27 @@ pub(super) fn no_ranked_index_key_length_check( Ok(()) } +/// RANKED: the cross-index structural check a generation runs over a +/// document type's parsed indices, before the merged index tree is built. +/// +/// Generation 3's implementation rejects the compound-ranked prefix-overlap +/// shape the storage layer cannot lay out; earlier generations pass +/// [`no_ranked_index_structure_check`] — their grammar rejects the +/// `ranked*` keywords, so no index they parse can carry a ranking axis. +/// Unlike [`RankedIndexKeyLengthCheck`] this runs on **every** parse path, +/// not only under `full_validation`: a contract admitted through a +/// non-validating parse would brick the first document insert. +pub(super) type RankedIndexStructureCheck = + fn(&BTreeMap) -> Result<(), ProtocolError>; + +/// The [`RankedIndexStructureCheck`] for a generation that has no ranking +/// axes to constrain. +pub(super) fn no_ranked_index_structure_check( + _indices: &BTreeMap, +) -> Result<(), ProtocolError> { + Ok(()) +} + /// Everything the shared parsing steps need to know about *which* generation is /// running them. /// @@ -169,6 +190,8 @@ pub(super) struct ParserGeneration { pub admit_ranked: bool, /// See [`RankedIndexKeyLengthCheck`]. pub ranked_index_key_length_check: RankedIndexKeyLengthCheck, + /// See [`RankedIndexStructureCheck`]. + pub ranked_index_structure_check: RankedIndexStructureCheck, } /// Reject a document type whose name is not a non-empty ASCII @@ -947,6 +970,12 @@ fn parse_indices( .transpose()? .unwrap_or_default(); + // Cross-index structural check owned by the generation, exactly like + // the per-property key-length check above: generations whose index + // grammar rejects the `ranked*` keywords pass the no-op, so the shared + // core never branches on a version. + (ctx.generation.ranked_index_structure_check)(&indices)?; + let index_structure = IndexLevel::try_from_indices(indices.values(), ctx.name, ctx.platform_version)?; diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v1/mod.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v1/mod.rs index 41a56c52db9..86107721e1f 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v1/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v1/mod.rs @@ -99,6 +99,7 @@ impl DocumentTypeV1 { // therefore has no ranked key ceiling to enforce. admit_ranked: false, ranked_index_key_length_check: common::no_ranked_index_key_length_check, + ranked_index_structure_check: common::no_ranked_index_structure_check, }, platform_version, ) diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs index 30ae2b94db9..1f449468155 100644 --- a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/mod.rs @@ -34,6 +34,9 @@ use crate::consensus::basic::data_contract::InvalidIndexedPropertyConstraintErro use super::common; +mod ranked_prefix_overlap; +use ranked_prefix_overlap::validate_no_ranked_prefix_overlap; + /// grovedb's ceiling on the key of an entry stored directly under an /// *indexed* tree's primary when the tree carries only the Count and/or Sum /// axes (`grovedb::operations::indexed_tree::MAX_CIDX_ITEM_KEY_LEN`). @@ -110,6 +113,15 @@ fn validate_ranked_index_property_key_length( property_type: &DocumentPropertyType, platform_version: &PlatformVersion, ) -> Result<(), ProtocolError> { + // Only the **terminal** property's encoded value becomes an + // indexed-tree item key, mirrored into the ordered secondary behind + // the sort key — that is where the tightened ceiling comes from. + // Leading properties of a compound ranked index are ordinary grovedb + // path segments, bound by the generic limits checked after this. + if index.properties.last().map(|p| p.name.as_str()) != Some(index_property_name) { + return Ok(()); + } + let Some(limit) = ranked_index_key_length_limit(index) else { return Ok(()); }; @@ -246,6 +258,7 @@ fn try_from_schema_generation_3( // RANKED: the constants that make this generation 3. admit_ranked: true, ranked_index_key_length_check: RANKED_INDEX_KEY_LENGTH_CHECK, + ranked_index_structure_check: validate_no_ranked_prefix_overlap, }, platform_version, )?; @@ -872,6 +885,122 @@ mod tests { ); } + /// A compound `[region, restaurantId]` avg-ranked index over the same + /// doctype shape as [`ranked_bound_schema`], with independent control + /// of the leading and terminal string properties' `maxLength`. + fn compound_ranked_bound_schema(leading: Value, terminal: Value) -> Value { + let mut index_entry: Vec<(Value, Value)> = vec![ + ( + Value::Text("name".to_string()), + Value::Text("byRegionRestaurant".to_string()), + ), + ( + Value::Text("properties".to_string()), + Value::Array(vec![ + Value::Map(vec![( + Value::Text("region".to_string()), + Value::Text("asc".to_string()), + )]), + Value::Map(vec![( + Value::Text("restaurantId".to_string()), + Value::Text("asc".to_string()), + )]), + ]), + ), + ]; + index_entry.extend( + avg_ranked_extras() + .into_iter() + .map(|(key, value)| (Value::Text(key.to_string()), value)), + ); + + Value::Map(vec![ + ( + Value::Text("type".to_string()), + Value::Text("object".to_string()), + ), + ( + Value::Text("properties".to_string()), + Value::Map(vec![ + (Value::Text("region".to_string()), leading), + (Value::Text("restaurantId".to_string()), terminal), + ( + Value::Text("grade".to_string()), + platform_value!({ + "type": "integer", + "minimum": 0, + "maximum": 100, + "position": 2, + }), + ), + ]), + ), + ( + Value::Text("required".to_string()), + Value::Array(vec![ + Value::Text("region".to_string()), + Value::Text("restaurantId".to_string()), + Value::Text("grade".to_string()), + ]), + ), + ( + Value::Text("additionalProperties".to_string()), + Value::Bool(false), + ), + ( + Value::Text("indices".to_string()), + Value::Array(vec![Value::Map(index_entry)]), + ), + ]) + } + + fn string_property_at(max_length: u32, position: u32) -> Value { + platform_value!({ + "type": "string", + "maxLength": max_length, + "position": position, + }) + } + + /// The ranked ceiling binds only the **terminal** property — the one + /// whose encoded value becomes the indexed tree's item key. A leading + /// prefix property is an ordinary grovedb path segment: the generic + /// 63-character indexed-string limit is what binds it, not the + /// 59-character avg-ranked ceiling. + #[test] + fn ranked_ceiling_binds_the_terminal_property_not_the_leading_prefix() { + // Leading at 63 (over the 59 ranked bound, at the generic bound) + // with a terminal that fits the ranked bound: accepted. + parse_bound(compound_ranked_bound_schema( + string_property_at(63, 0), + string_property_at(59, 1), + )) + .expect("a 63-character leading prefix is a path segment, not an item key"); + + // The same 60-character string the single-property tests reject is + // still rejected when it is the terminal property of a compound + // ranked index. + let error = parse_bound(compound_ranked_bound_schema( + string_property_at(30, 0), + string_property_at(60, 1), + )) + .expect_err("the terminal property keeps the 239-byte avg ceiling"); + let msg = format!("{error:?}"); + assert!( + msg.contains("maxLength") && msg.contains("59") && msg.contains("239"), + "the error must still name the ranked bound for the terminal; got {msg}" + ); + + // And the generic indexed-string limit still binds the leading + // property: 64 characters is over the 63-character cap whether or + // not the index is ranked. + parse_bound(compound_ranked_bound_schema( + string_property_at(64, 0), + string_property_at(59, 1), + )) + .expect_err("the generic 63-character limit still binds the leading prefix"); + } + /// Avg is strictly tighter than Count/Sum, and an index carrying Avg /// *alongside* the other axes has to satisfy the tightest of them: the /// 60-character string that a count-only ranked index accepts is refused @@ -982,4 +1111,192 @@ mod tests { parse_bound(ranked_bound_schema(string_property(63), range_only)) .expect("a range-averageable index without a ranking axis keeps the generic limit"); } + + // ------------------------------------------------------------------- + // Compound ranked indexes (per-prefix semantics) and the + // prefix-overlap conflict + // ------------------------------------------------------------------- + + /// One extra single-property index for [`compound_ranked_schema`]: + /// `(name, property, keys)`. + type ExtraIndexSpec<'a> = (&'a str, &'a str, Vec<(&'a str, Value)>); + + /// A doctype with a compound ranked index `[region, restaurantId]` + /// (avg axis on `grade`), plus optional extra single-property + /// indexes to provoke — or fail to provoke — the prefix-overlap + /// conflict. + fn compound_ranked_schema(extra_indexes: Vec) -> Value { + let compound_entry: Vec<(Value, Value)> = vec![ + ( + Value::Text("name".to_string()), + Value::Text("byRegionRestaurant".to_string()), + ), + ( + Value::Text("properties".to_string()), + Value::Array(vec![ + Value::Map(vec![( + Value::Text("region".to_string()), + Value::Text("asc".to_string()), + )]), + Value::Map(vec![( + Value::Text("restaurantId".to_string()), + Value::Text("asc".to_string()), + )]), + ]), + ), + ( + Value::Text("averageable".to_string()), + Value::Text("grade".to_string()), + ), + ( + Value::Text("rangeAverageable".to_string()), + Value::Bool(true), + ), + ( + Value::Text("rankedAverageable".to_string()), + Value::Bool(true), + ), + ]; + let mut indices = vec![Value::Map(compound_entry)]; + for (name, property, keys) in extra_indexes { + let mut entry: Vec<(Value, Value)> = vec![ + ( + Value::Text("name".to_string()), + Value::Text(name.to_string()), + ), + ( + Value::Text("properties".to_string()), + Value::Array(vec![Value::Map(vec![( + Value::Text(property.to_string()), + Value::Text("asc".to_string()), + )])]), + ), + ]; + entry.extend( + keys.into_iter() + .map(|(key, value)| (Value::Text(key.to_string()), value)), + ); + indices.push(Value::Map(entry)); + } + + Value::Map(vec![ + ( + Value::Text("type".to_string()), + Value::Text("object".to_string()), + ), + ( + Value::Text("properties".to_string()), + platform_value!({ + "region": { + "type": "string", + "maxLength": 32, + "position": 0, + }, + "restaurantId": { + "type": "string", + "maxLength": 32, + "position": 1, + }, + "grade": { + "type": "integer", + "minimum": 0, + "maximum": 100, + "position": 2, + }, + }), + ), + ( + Value::Text("required".to_string()), + Value::Array(vec![ + Value::Text("region".to_string()), + Value::Text("restaurantId".to_string()), + Value::Text("grade".to_string()), + ]), + ), + ( + Value::Text("additionalProperties".to_string()), + Value::Bool(false), + ), + (Value::Text("indices".to_string()), Value::Array(indices)), + ]) + } + + /// A compound ranked index is accepted at PV14 with per-prefix + /// semantics — on both the validating and the structural parse + /// paths — and the ranked flags land on the index alongside the + /// range axes they require. + #[test] + fn compound_ranked_index_accepted_at_pv14() { + for full_validation in [true, false] { + let v2 = parse_with(compound_ranked_schema(vec![]), pv14(), full_validation) + .unwrap_or_else(|e| { + panic!( + "a compound ranked index must parse \ + (full_validation: {full_validation}): {e}" + ) + }); + let index = v2 + .indices + .get("byRegionRestaurant") + .expect("index parsed under its name"); + assert!(index.ranked_averageable); + assert!(index.range_countable && index.range_summable); + assert_eq!(index.properties.len(), 2); + assert_eq!(index.properties[1].name, "restaurantId"); + } + } + + /// The one structurally impossible shape: a countable/summable + /// index terminating at the compound ranked index's full leading + /// prefix. The ranked terminal tree would sit inside aggregating + /// value trees and need the NonCounted/NotSummed shell the storage + /// layer rejects for indexed trees — so the contract is refused at + /// parse time, on the validating AND the structural path (a + /// contract smuggled through check_tx would brick document inserts). + #[test] + fn compound_ranked_with_aggregating_prefix_index_rejected() { + let schema = compound_ranked_schema(vec![( + "byRegion", + "region", + vec![("countable", Value::Text("countable".to_string()))], + )]); + for full_validation in [true, false] { + let error = parse_with(schema.clone(), pv14(), full_validation).expect_err( + "an aggregating index on the ranked compound's full prefix must be rejected", + ); + let message = format!("{error:?}"); + assert!( + message.contains("byRegionRestaurant") + && message.contains("byRegion") + && message.contains("NonCounted"), + "the rejection must name both indexes and the structural conflict \ + (full_validation: {full_validation}); got {message}" + ); + } + } + + /// Only the **exact** leading prefix conflicts: an aggregating + /// index over a different property — same arity as the prefix, but + /// not the prefix — coexists with the compound ranked index, as + /// does a plain (non-aggregating) index on the prefix property. + #[test] + fn compound_ranked_with_non_conflicting_indexes_accepted() { + // Countable over the *trailing* property's own single-property + // index: terminates at [restaurantId], not at the ranked + // index's [region] prefix. + let aggregating_elsewhere = compound_ranked_schema(vec![( + "byRestaurant", + "restaurantId", + vec![("countable", Value::Text("countable".to_string()))], + )]); + parse_with(aggregating_elsewhere, pv14(), true) + .expect("an aggregating index off the prefix must not conflict"); + + // A plain index on the prefix property: terminates at [region] + // but carries no aggregates, so its value trees stay normal and + // no wrapper shell is ever needed. + let plain_prefix = compound_ranked_schema(vec![("byRegion", "region", vec![])]); + parse_with(plain_prefix, pv14(), true) + .expect("a non-aggregating index on the prefix must not conflict"); + } } diff --git a/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/ranked_prefix_overlap.rs b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/ranked_prefix_overlap.rs new file mode 100644 index 00000000000..ff182ae52fa --- /dev/null +++ b/packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/v3/ranked_prefix_overlap.rs @@ -0,0 +1,90 @@ +//! The generation-3 [`super::common::RankedIndexStructureCheck`]: the cross-index +//! structural rule the ranked grammar adds on top of the shared parse core. +//! +//! Its own file for the same reason the mode-detection versions have their +//! own files: the rule is one frozen unit of generation-3 grammar, and a +//! later generation that needs a different structural rule supplies its own +//! callback instead of editing this one. + +use crate::data_contract::document_type::class_methods::consensus_or_protocol_data_contract_error; +use crate::data_contract::document_type::index::Index; +use crate::data_contract::errors::DataContractError; +use crate::ProtocolError; +use std::collections::BTreeMap; + +/// Rejects the one compound-ranked shape the storage layer cannot lay +/// out: a compound ranked index whose **full leading prefix** also +/// terminates a separate countable and/or summable index. +/// +/// A ranked flag on a compound index `[p1, …, pn]` puts an indexed tree +/// at each prefix's terminal `pn` property-name level — inside the value +/// trees of the `[p1, …, pn-1]` level. When another countable/summable +/// index terminates at exactly that prefix, those value trees are +/// aggregating (`CountTree` / `SumTree` / …), and every continuation +/// subtree inside them must be wrapped in a `NonCounted` / `NotSummed` +/// shell so its contents don't pollute the prefix index's aggregates. +/// grovedb structurally rejects that shell around an indexed tree — the +/// wrapper would neutralize the very aggregates the ranking indexes — +/// so the write path fails closed at document insert. Rejecting the +/// contract here surfaces the conflict at registration instead. +/// +/// Only the **exact** `n-1` prefix conflicts. An aggregating index +/// terminating at a shorter prefix wraps a plain intermediate +/// property-name tree (fine), and one extending *past* the ranked +/// terminal lives inside the indexed tree's value trees, which the +/// storage layer supports (see rs-drive's +/// `ranked_terminator_with_a_compound_continuation_gets_both_treatments`). +/// +/// Property comparison is by name, positionally: the merged index-level +/// tree keys sub-levels by property name in declaration order, so +/// `[a, b]` and `[b, a]` never share a level and cannot conflict. +/// +/// Unconditional (not gated on `full_validation`): the same structural +/// impossibility must reject the contract on every parse path — a +/// contract admitted through a non-validating parse would brick the +/// first document insert under the ranked index. +pub(super) fn validate_no_ranked_prefix_overlap( + indices: &BTreeMap, +) -> Result<(), ProtocolError> { + for ranked in indices.values() { + let is_ranked = + ranked.ranked_countable || ranked.ranked_summable || ranked.ranked_averageable; + if !is_ranked || ranked.properties.len() < 2 { + continue; + } + let prefix = &ranked.properties[..ranked.properties.len() - 1]; + for other in indices.values() { + if other.name == ranked.name { + continue; + } + let terminates_at_prefix = other.properties.len() == prefix.len() + && other + .properties + .iter() + .zip(prefix.iter()) + .all(|(a, b)| a.name == b.name); + let aggregates = other.countable.is_countable() || other.summable.is_some(); + if terminates_at_prefix && aggregates { + return Err(consensus_or_protocol_data_contract_error( + DataContractError::InvalidContractStructure(format!( + "compound ranked index `{}` conflicts with index `{}`: the ranked \ + index's leading prefix [{}] also terminates a countable/summable \ + index, so the ranked terminal tree would sit inside aggregating \ + value trees and need a NonCounted/NotSummed shell — which the \ + storage layer rejects for indexed trees because the wrapper would \ + neutralize the aggregates the ranking indexes. Drop the ranked \ + flags, or drop the aggregate flags from the prefix index", + ranked.name, + other.name, + prefix + .iter() + .map(|p| p.name.as_str()) + .collect::>() + .join(", "), + )), + )); + } + } + } + Ok(()) +} diff --git a/packages/rs-dpp/src/data_contract/document_type/index/mod.rs b/packages/rs-dpp/src/data_contract/document_type/index/mod.rs index edc39f7c934..2f099c93014 100644 --- a/packages/rs-dpp/src/data_contract/document_type/index/mod.rs +++ b/packages/rs-dpp/src/data_contract/document_type/index/mod.rs @@ -1181,22 +1181,21 @@ impl Index { )); } - // Ranked aggregates are restricted to single-property indexes in - // this protocol version. Two reasons: a compound index whose prefix - // level also terminates an aggregating index would need its ranked - // terminal tree wrapped in a NonCounted/NotSummed shell, which the - // storage layer structurally rejects for indexed trees (the wrapper - // would neutralize the very aggregates the ranking indexes); and the - // ranked query surface deliberately has no equality-prefix routing - // yet. Both are relaxable at a future protocol version. - if (ranked_countable || ranked_summable || ranked_averageable) && index_properties.len() > 1 - { - return Err(DataContractError::InvalidContractStructure( - "ranked aggregates are only supported on single-property \ - indexes in this protocol version" - .to_string(), - )); - } + // Ranked aggregates are allowed on compound indexes, with + // **per-prefix** semantics: the ranked flags land on the index's + // TERMINAL property-name level (see `IndexLevelTypeInfo`), so a + // ranked `[identityId, class]` maintains one ordered secondary per + // `identityId` value — each ordering that identity's `class` groups + // by the group aggregate. There is deliberately no global + // cross-prefix ordering; the query surfaces require every leading + // property to be pinned by an equality `where` clause. + // + // One compound shape stays structurally impossible and is rejected + // per document type (all indexes are needed to see it): a compound + // ranked index whose full leading prefix also terminates a separate + // countable/summable index. That check lives with the document + // type's index collection — see `validate_no_ranked_prefix_overlap` + // in `try_from_schema::common`. // `nullSearchable: false` suppresses the terminal reference for a // document that leaves the indexed property out — but the document @@ -2263,12 +2262,17 @@ mod tests { assert!(!index.ranked_averageable); } - /// Ranked flags on compound indexes are a v1 scope restriction (wrapper - /// conflict under aggregating prefixes + no equality-prefix query - /// routing), rejected at parse time so the limitation is visible at - /// registration rather than at document-insert or query time. + /// Ranked flags on compound indexes are accepted with per-prefix + /// semantics: the ranking axes attach to the terminal property-name + /// level, one ordered secondary per prefix value. The flag-dependency + /// rules (`ranked*` requires the matching `range*`) apply exactly as + /// on single-property indexes; the one structurally impossible shape + /// (a countable/summable index terminating at the compound's full + /// leading prefix) is a cross-index condition rejected where all the + /// document type's indexes are known — see + /// `validate_no_ranked_prefix_overlap` in `try_from_schema::common`. #[test] - fn test_index_try_from_ranked_on_compound_index_rejected() { + fn test_index_try_from_ranked_on_compound_index_accepted() { let mut index_map = ranked_index_map(vec![ ("averageable", Value::Text("score".to_string())), ("rangeAverageable", Value::Bool(true)), @@ -2285,15 +2289,37 @@ mod tests { Value::Text("asc".to_string()), )]), ]); + let index = Index::try_from_value_map(index_map.as_slice(), true) + .expect("ranked flags on a compound index must be accepted"); + assert!(index.ranked_averageable); + assert_eq!(index.properties.len(), 2); + assert_eq!(index.properties[1].name, "score"); + } + + /// The `ranked* requires range*` dependency is enforced on compound + /// indexes exactly as on single-property ones. + #[test] + fn test_index_try_from_compound_ranked_without_range_flags_rejected() { + let mut index_map = ranked_index_map(vec![("rankedCountable", Value::Bool(true))]); + index_map[0].1 = Value::Array(vec![ + Value::Map(vec![( + Value::Text("region".to_string()), + Value::Text("asc".to_string()), + )]), + Value::Map(vec![( + Value::Text("score".to_string()), + Value::Text("asc".to_string()), + )]), + ]); let result = Index::try_from_value_map(index_map.as_slice(), true); assert!( result.is_err(), - "ranked flags on a compound index must be rejected" + "rankedCountable without rangeCountable must be rejected on a compound index too" ); let msg = format!("{:?}", result.unwrap_err()); assert!( - msg.contains("single-property"), - "error must explain the single-property restriction; got {msg}" + msg.contains("rankedCountable") && msg.contains("rangeCountable"), + "error must name both flags; got {msg}" ); } diff --git a/packages/rs-drive-abci/src/query/document_query/v1/tests.rs b/packages/rs-drive-abci/src/query/document_query/v1/tests.rs index 206e421a6ab..475419587a2 100644 --- a/packages/rs-drive-abci/src/query/document_query/v1/tests.rs +++ b/packages/rs-drive-abci/src/query/document_query/v1/tests.rs @@ -3208,6 +3208,11 @@ mod having_range_tests { GROUP_PROPERTY, PROTOCOL_VERSION_V13, }; use super::*; + use crate::rpc::core::MockCoreRPCLike; + use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; + use dpp::document::{Document, DocumentV0Setters}; + use dpp::tests::json_document::json_document_to_contract; + use std::collections::BTreeMap; /// The canonical having-range request: one aggregate select, one /// `group_by`, one `having` clause on the selected aggregate, a @@ -3380,11 +3385,14 @@ mod having_range_tests { } } - /// `GROUP BY restaurantId, guests HAVING …` — the routing layer - /// sends any grouped single-clause having down the having path - /// (it owns *where* the request goes, not the grammar), and - /// drive's mode detection rejects the compound grouping: ranked - /// axes live on single-property indexes. + /// `GROUP BY restaurantId, guests HAVING …` — an **unpinned** + /// two-field grouping — is still rejected: the routing layer sends + /// any grouped single-clause having down the having path (it owns + /// *where* the request goes, not the grammar), and drive's mode + /// detection rejects the compound grouping, steering the caller to + /// the served pinned form (`WHERE = X GROUP BY + /// ` — exercised end to end in + /// [`pinned_prefix_having_is_served_end_to_end`]). #[test] fn compound_group_by_is_rejected_on_the_having_path() { let (platform, state, version) = setup_platform(None, Network::Testnet, None); @@ -3409,14 +3417,187 @@ mod having_range_tests { match ranked_error(&platform, &state, request, version) { QueryError::Query(QuerySyntaxError::InvalidParameter(message)) => { assert!( - message.contains("exactly one `group_by` property"), - "the rejection must say the surface is single-property, got: {message}" + message.contains("exactly one `group_by` property") + && message.contains("equality `where` clause"), + "the rejection must steer to the pinned-prefix form, got: {message}" ); } other => panic!("expected InvalidParameter, got {other:?}"), } } + /// Shared with rs-drive's `pinned_prefix` suites — the compound + /// ranked index `[identityId, class]` with the Avg axis on `grade`. + const GRADES_COMPOUND_CONTRACT_PATH: &str = + "../rs-drive/tests/supporting_files/contract/grades/grades-compound-ranked-contract.json"; + + fn register_grades_compound( + platform: &Platform, + platform_version: &PlatformVersion, + ) -> dpp::prelude::DataContract { + let contract = + json_document_to_contract(GRADES_COMPOUND_CONTRACT_PATH, false, platform_version) + .expect("expected to parse the compound ranked grades contract"); + store_data_contract(platform, &contract, platform_version); + contract + } + + fn insert_grade_docs( + platform: &Platform, + contract: &dpp::prelude::DataContract, + first_seed: u64, + rows: &[([u8; 32], &str, i64)], + platform_version: &PlatformVersion, + ) { + let document_type = contract + .document_type_for_name("grade") + .expect("grade doctype exists"); + for (i, (identity, class, grade)) in rows.iter().enumerate() { + let mut document: Document = document_type + .random_document(Some(first_seed + i as u64), platform_version) + .expect("random document"); + let mut properties = BTreeMap::new(); + properties.insert("identityId".to_string(), Value::Identifier(*identity)); + properties.insert("class".to_string(), Value::Text(class.to_string())); + properties.insert("grade".to_string(), Value::I64(*grade)); + document.set_properties(properties); + store_document( + platform, + contract, + document_type, + &document, + platform_version, + ); + } + } + + /// The pinned-prefix form end to end on the wire: `WHERE identityId + /// = X GROUP BY class HAVING AVG(grade) > 80 LIMIT 10` routes to + /// the same having executor, descends to X's terminal `class` tree, + /// and answers with `ResultData.ranked` (`skipped` unset) — with + /// per-prefix isolation visible in the entries. Value-level + /// behaviour (bounds, proofs, tamper) is pinned in rs-drive's + /// `pinned_prefix` suite; this pins the wire encoding and routing. + #[test] + fn pinned_prefix_having_is_served_end_to_end() { + let (platform, state, version) = setup_platform(None, Network::Testnet, None); + let contract = register_grades_compound(&platform, version); + let identity_x = [1u8; 32]; + let identity_y = [2u8; 32]; + insert_grade_docs( + &platform, + &contract, + 20_000, + &[ + (identity_x, "math", 80), + (identity_x, "math", 80), + (identity_x, "english", 80), + (identity_x, "english", 81), + (identity_x, "art", 85), + (identity_x, "art", 95), + // Y's science (avg 95) would qualify for X's bound too + // if the prefixes shared a secondary. + (identity_y, "science", 95), + (identity_y, "science", 95), + ], + version, + ); + + let mut request = having_request( + &contract, + "grade", + select(v1_select::Function::Avg, "grade"), + hc( + having_aggregate::Function::Avg, + "grade", + having_clause::Operator::GreaterThan, + Value::U64(80), + ), + Vec::new(), + Some(10), + false, + ); + request.group_by = vec!["class".to_string()]; + request.where_clauses = vec![wc( + "identityId", + ProtoWhereOperator::Equal, + Value::Bytes(identity_x.to_vec()), + )]; + + let page = ranked_page(&platform, &state, request.clone(), version); + assert_eq!( + page.skipped, None, + "a having-range page must leave the rank-based `skipped` field unset" + ); + assert_eq!( + group_keys(&page.entries), + vec!["english", "art"], + "ascending average order over X's own classes: english (80.5) before art (90); \ + math sits exactly at the threshold and Y's science must not leak in" + ); + + // The proved variant answers with a Proof payload. + request.prove = true; + let result = platform + .query_documents_v1(request, &state, version) + .expect("query call should not error at the transport layer"); + assert!( + result.errors.is_empty(), + "expected no validation errors, got {:?}", + result.errors + ); + match result.data { + Some(GetDocumentsResponseV1 { + result: Some(get_documents_response_v1::Result::Proof(_)), + metadata: Some(_), + }) => {} + other => panic!("expected a Proof result, got {:?}", other), + } + } + + /// The pinned-prefix shape is still a HAVING request, and protocol + /// version 13's query table (v0 helper) has no having path: the + /// same request a v14 node serves is refused with the blanket + /// rejection before any contract fetch. + #[test] + fn pinned_prefix_having_still_rejected_at_protocol_version_13() { + let (platform, state, version) = + setup_platform(None, Network::Testnet, Some(PROTOCOL_VERSION_V13)); + + let request = GetDocumentsRequestV1 { + data_contract_id: vec![0u8; 32], + document_type: "grade".to_string(), + where_clauses: vec![wc( + "identityId", + ProtoWhereOperator::Equal, + Value::Bytes(vec![1u8; 32]), + )], + order_by: Vec::new(), + limit: Some(10), + start: None, + prove: false, + selects: select(v1_select::Function::Avg, "grade"), + group_by: vec!["class".to_string()], + having: vec![hc( + having_aggregate::Function::Avg, + "grade", + having_clause::Operator::GreaterThan, + Value::U64(80), + )], + offset: None, + }; + + match ranked_error(&platform, &state, request, version) { + QueryError::Query(QuerySyntaxError::Unsupported(message)) => { + assert!( + message.contains("HAVING clause") && message.contains("not yet implemented"), + "expected v13's blanket rejection, got: {message}" + ); + } + other => panic!("expected Unsupported, got {other:?}"), + } + } + /// `OFFSET` stays ranked-only, and the having-range route gets its /// own rejection: the legacy message recommends `start_after` / /// `start_at`, which this surface also rejects, so the having @@ -3810,6 +3991,7 @@ mod having_trust_boundary { .expect("grade doctype exists") .indexes(), &mode.group_by_property, + &[], mode.bounds.axis(), &mode.aggregate_field, ) @@ -3822,6 +4004,7 @@ mod having_trust_boundary { document_type_name: "grade".to_string(), index, bounds: mode.bounds, + equality_prefix_values: Vec::new(), descending: mode.descending, limit: mode.limit, } diff --git a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/batched_group_drain.rs b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/batched_group_drain.rs index f178e6bf7da..d81d73c4267 100644 --- a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/batched_group_drain.rs +++ b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/batched_group_drain.rs @@ -369,11 +369,13 @@ fn verified_page( index: find_ranked_index_for_axis( indexes, GROUP_PROPERTY, + &[], axis.ranked, axis.aggregate_field(), ) .expect("the fixture declares this axis"), axis: axis.ranked, + equality_prefix_values: Vec::new(), descending: !ascending, k: 100, offset: 0, diff --git a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/ranked_index_e2e_tests.rs b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/ranked_index_e2e_tests.rs index 36ad01f54e9..c4e30fd9927 100644 --- a/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/ranked_index_e2e_tests.rs +++ b/packages/rs-drive/src/drive/contract/insert/insert_contract/v0/tests/ranked_index_e2e_tests.rs @@ -761,13 +761,11 @@ fn try_build_dish_contract( /// The write path's terminal-level resolver has to pick the indexed variant /// for a **compound** ranked index too — that level lives one step below the -/// doctype tree and is materialized lazily by the document index walker, not -/// at contract registration. -/// -/// Compound ranked indexes are rejected at contract-parse time (rs-dpp, -/// single-property restriction), so the index is built as a struct literal -/// here: this pins that the drive dispatch is already correct for the day a -/// future protocol version relaxes the grammar. +/// doctype tree and is materialized lazily, per prefix value, by the +/// document index walker rather than at contract registration. This pins +/// the resolver's half of that contract; the end-to-end half (documents +/// inserted, per-prefix secondaries read and proved) lives in the query +/// suites' `pinned_prefix` modules. #[test] fn compound_ranked_index_resolves_its_terminal_level_to_an_indexed_tree() { use crate::drive::document::ranked_index_tree_type::property_name_tree_type_and_ranked_axes; @@ -829,23 +827,24 @@ fn compound_ranked_index_resolves_its_terminal_level_to_an_indexed_tree() { assert_eq!(terminal_axes, vec![IndexAxis::Avg]); } -/// Compound ranked indexes are rejected at contract-parse time (rs-dpp's -/// single-property restriction) — a v1 scope choice: a ranked terminal -/// level under an aggregating prefix would need a NonCounted/NotSummed -/// wrapper, which grovedb structurally rejects for indexed trees, and the -/// ranked query surface has no equality-prefix routing yet. (GroveDB's -/// merged PR #657 does support creating and populating an indexed tree in -/// one batch, so lazy terminal creation itself is no longer a blocker — -/// the restriction is relaxable at a future protocol version.) +/// A compound ranked index parses on its own (per-prefix semantics — +/// the terminal level's indexed tree is materialized lazily per prefix +/// by the document walker; grovedb's PR #657 supports creating and +/// populating an indexed tree in one batch). What stays rejected, at +/// contract-parse time, is the one structurally impossible pairing: a +/// countable/summable index terminating at the compound's full leading +/// prefix, whose aggregating value trees would demand the +/// NonCounted/NotSummed wrapper grovedb structurally rejects for +/// indexed trees. /// -/// Storage-level backstop behind this gate: both wrapper dispatchers in -/// `fees/op.rs` fail closed (`DriveError::NotSupported`) for a ranked -/// terminal level inside an aggregating value tree — the frozen v0 -/// diagonal and the v14 zero-contribution matrix alike. The latter -/// matters since the v14 shared-prefix fix: its unwrapped fallback for -/// non-sum children of sum-only parents would otherwise have accepted an -/// indexed continuation, quietly creating a ranked tree that neither the -/// grammar nor the ranked query picker supports. +/// Storage-level backstop behind that parse-time gate: both wrapper +/// dispatchers in `fees/op.rs` fail closed (`DriveError::NotSupported`) +/// for a ranked terminal level inside an aggregating value tree — the +/// frozen v0 diagonal and the v14 zero-contribution matrix alike. The +/// latter matters since the v14 shared-prefix fix: its unwrapped +/// fallback for non-sum children of sum-only parents would otherwise +/// have accepted an indexed continuation, quietly creating a ranked +/// tree the picker never resolves. /// /// Note that a ranked index *sharing* its property with a compound index /// — `[a]` ranked next to `[a, b]` — is a different (and, since v14, @@ -853,16 +852,21 @@ fn compound_ranked_index_resolves_its_terminal_level_to_an_indexed_tree() { /// one, and the continuation hangs *below* it. See /// `ranked_index_ranks_correctly_next_to_a_compound_index_sharing_its_property`. #[test] -fn compound_ranked_index_contract_is_rejected_at_parse_time() { - for standalone_prefix_index in [false, true] { - let error = try_build_dish_contract(standalone_prefix_index) - .expect_err("a compound ranked index must be rejected at contract-parse time"); - let message = error.to_string(); - assert!( - message.contains("single-property"), - "expected the single-property restriction, got: {message}" - ); - } +fn compound_ranked_index_contract_parses_unless_its_prefix_aggregates() { + try_build_dish_contract(false) + .expect("a compound ranked index with no aggregating prefix index must parse"); + + let error = try_build_dish_contract(true).expect_err( + "a countable index terminating at the ranked compound's prefix must be rejected", + ); + let message = error.to_string(); + assert!( + message.contains("byRestaurantCourse") + && message.contains("byRestaurant") + && message.contains("NonCounted"), + "expected the prefix-overlap rejection naming both indexes and the structural \ + conflict, got: {message}" + ); } /// Build a `visit` contract whose single-property `rankedCountable` index @@ -1437,8 +1441,9 @@ fn verified_ranked_avg_page( document_type, contract_id: contract.id().to_buffer(), document_type_name: "review".to_string(), - index: find_ranked_index_for_axis(indexes, GROUP_PROPERTY, RankedAxis::Avg, "grade") + index: find_ranked_index_for_axis(indexes, GROUP_PROPERTY, &[], RankedAxis::Avg, "grade") .expect("the fixture declares rankedAverageable on grade"), + equality_prefix_values: vec![], axis: RankedAxis::Avg, descending: true, k: limit as u16, diff --git a/packages/rs-drive/src/fees/op.rs b/packages/rs-drive/src/fees/op.rs index 736441c13ca..6dcc2fc1cba 100644 --- a/packages/rs-drive/src/fees/op.rs +++ b/packages/rs-drive/src/fees/op.rs @@ -224,18 +224,22 @@ pub enum LowLevelDriveOperation { /// secondary root keys derived from it — is committed. A wrapped indexed tree /// would have nowhere to hang its secondaries. /// -/// Reachable shape: a ranked index whose terminal property-name tree sits -/// inside a value tree that itself aggregates, i.e. a compound ranked index -/// `[a, b]` on a doctype that ALSO declares an aggregating index terminating -/// at `[a]`. Failing closed here is deliberate — the alternative is silently -/// writing a non-indexed tree and having ranked queries return nothing. +/// The shape that would reach this — a ranked index whose terminal +/// property-name tree sits inside a value tree that itself aggregates, i.e. a +/// compound ranked index `[a, b]` on a doctype that ALSO declares an +/// aggregating index terminating at `[a]` — is rejected at contract-parse +/// time (`validate_no_ranked_prefix_overlap` in rs-dpp), so this is the +/// fail-closed backstop behind that check. Failing closed here is deliberate +/// — the alternative is silently writing a non-indexed tree and having +/// ranked queries return nothing. const INDEXED_INNER_UNWRAPPABLE: &str = "an indexed tree cannot be wrapped in NonCounted / NotSummed / NotCountedOrSummed: the \ wrapper suppresses the subtree's contribution to its parent's aggregate, but an indexed \ primary commits its aggregate (and the derived secondary root keys) through that very \ parent element. A ranked index's terminal property-name tree therefore cannot live inside \ - an aggregating value tree — i.e. a ranked compound index [a, b] is unsupported when the \ - same doctype also declares a countable/summable index terminating at [a]."; + an aggregating value tree — i.e. a ranked compound index [a, b] cannot coexist with a \ + countable/summable index terminating at [a]; contracts declaring that pair are rejected \ + at parse time."; impl LowLevelDriveOperation { /// Returns a list of the costs of the Drive operations. diff --git a/packages/rs-drive/src/query/drive_document_having_query/drive_dispatcher.rs b/packages/rs-drive/src/query/drive_document_having_query/drive_dispatcher.rs index 8c5b35da0c6..b484460081b 100644 --- a/packages/rs-drive/src/query/drive_document_having_query/drive_dispatcher.rs +++ b/packages/rs-drive/src/query/drive_document_having_query/drive_dispatcher.rs @@ -21,12 +21,11 @@ use grovedb::TransactionArg; /// wire-decoding + contract lookup — the same construction pattern as /// [`super::super::drive_document_ranked_query::DocumentRankedRequest`]. /// -/// `where_clauses`, `offset` and `start_at` are carried even though a -/// having-range request must leave all of them empty: drive owns the -/// rejection, so the contract is enforced identically no matter which -/// upstream path built the request. See -/// [`super::mode_detection::detect_having_mode_v0`] for why each is -/// refused rather than ignored. +/// `offset` and `start_at` are carried even though a having-range +/// request must leave both empty: drive owns the rejection, so the +/// contract is enforced identically no matter which upstream path built +/// the request. See [`super::mode_detection::detect_having_mode_v0`] +/// for why each is refused rather than ignored. pub struct DocumentHavingRequest<'a> { /// Live contract (already loaded by the handler). pub contract: &'a DataContract, @@ -44,7 +43,9 @@ pub struct DocumentHavingRequest<'a> { /// The `ORDER BY` clauses. Empty (ascending default) or exactly /// one, naming the selected aggregate. pub order_by: &'a [OrderClause], - /// Structured `where` clauses. Must be empty. + /// Structured `where` clauses. Empty for the single-property form; + /// equality pins on the covering compound index's leading + /// properties for the pinned-prefix form. pub where_clauses: &'a [WhereClause], /// Request `limit`. **Required**; `1 ..= MAX_HAVING_LIMIT`. pub limit: Option, diff --git a/packages/rs-drive/src/query/drive_document_having_query/executors.rs b/packages/rs-drive/src/query/drive_document_having_query/executors.rs index 97793ee923f..88347611e90 100644 --- a/packages/rs-drive/src/query/drive_document_having_query/executors.rs +++ b/packages/rs-drive/src/query/drive_document_having_query/executors.rs @@ -1,71 +1,21 @@ -//! Per-mode having-range executors on `impl Drive`, plus the shared -//! mode-to-query resolution. The dispatcher +//! Per-mode having-range executors on `impl Drive`. The dispatcher //! ([`super::drive_dispatcher`]) picks between the two executors on the //! request's `prove` flag. //! -//! Index resolution reuses the ranked surface's covering-index picker -//! ([`find_ranked_index_for_axis`]) — both surfaces read the same -//! indexed tree, and sharing the picker is what guarantees a proof and -//! an unproven read are about the same subtree. +//! Mode-to-query resolution — covering-index pick + equality-pin +//! encoding — is [`resolve_having_query_for_mode`], shared with the +//! SDK's proof helpers: both surfaces and both sides read the same +//! indexed tree, and sharing the resolution is what guarantees a proof +//! and an unproven read are about the same subtree. -use super::super::drive_document_ranked_query::index_picker::find_ranked_index_for_axis; use super::super::drive_document_ranked_query::RankedEntry; -use super::{DocumentHavingMode, DriveDocumentHavingQuery}; +use super::{resolve_having_query_for_mode, DocumentHavingMode}; use crate::drive::Drive; -use crate::error::query::QuerySyntaxError; use crate::error::Error; use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; -use dpp::data_contract::document_type::{DocumentTypeRef, Index}; +use dpp::data_contract::document_type::DocumentTypeRef; use dpp::version::PlatformVersion; use grovedb::TransactionArg; -use std::collections::BTreeMap; - -/// Resolve a validated [`DocumentHavingMode`] against a document type's -/// indexes into the executable [`DriveDocumentHavingQuery`]. -/// -/// `indexes` is threaded in separately for the same lifetime reason as -/// the ranked resolver: the returned query's `&'a Index` must outlive -/// this frame. Callers pass `document_type.indexes()`. -pub(super) fn having_query_for_mode<'a>( - contract_id: [u8; 32], - document_type: DocumentTypeRef<'a>, - document_type_name: String, - indexes: &'a BTreeMap, - mode: &DocumentHavingMode, -) -> Result, Error> { - let axis = mode.bounds.axis(); - let index = find_ranked_index_for_axis( - indexes, - &mode.group_by_property, - axis, - &mode.aggregate_field, - ) - .ok_or_else(|| { - Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty(format!( - "no ranked index covers `group_by = [{}]` on the {:?} axis: a `having` bound \ - is served from that axis's pre-sorted secondary, so the document type needs \ - a single-property index on `{}` declaring `{}`{}", - mode.group_by_property, - axis, - mode.group_by_property, - axis.required_index_keyword(), - if mode.aggregate_field.is_empty() { - String::new() - } else { - format!(" with `summable: \"{}\"`", mode.aggregate_field) - } - ))) - })?; - Ok(DriveDocumentHavingQuery { - document_type, - contract_id, - document_type_name, - index, - bounds: mode.bounds, - descending: mode.descending, - limit: mode.limit, - }) -} impl Drive { /// One page of groups matching a having bound, read without a proof. @@ -79,12 +29,13 @@ impl Drive { platform_version: &PlatformVersion, ) -> Result, Error> { let indexes = document_type.indexes(); - let having_query = having_query_for_mode( + let having_query = resolve_having_query_for_mode( contract_id, document_type, document_type_name, indexes, mode, + platform_version, )?; having_query.execute_range_no_proof(self, transaction, platform_version) } @@ -105,12 +56,13 @@ impl Drive { platform_version: &PlatformVersion, ) -> Result, Error> { let indexes = document_type.indexes(); - let having_query = having_query_for_mode( + let having_query = resolve_having_query_for_mode( contract_id, document_type, document_type_name, indexes, mode, + platform_version, )?; having_query.execute_range_with_proof(self, transaction, platform_version) } diff --git a/packages/rs-drive/src/query/drive_document_having_query/mod.rs b/packages/rs-drive/src/query/drive_document_having_query/mod.rs index 1b554131b35..f0418fc798a 100644 --- a/packages/rs-drive/src/query/drive_document_having_query/mod.rs +++ b/packages/rs-drive/src/query/drive_document_having_query/mod.rs @@ -64,12 +64,24 @@ #[cfg(any(feature = "server", feature = "verify"))] use dpp::data_contract::document_type::{DocumentTypeRef, Index}; +#[cfg(any(feature = "server", feature = "verify"))] +use dpp::platform_value::Value; +#[cfg(any(feature = "server", feature = "verify"))] +use dpp::version::PlatformVersion; +#[cfg(any(feature = "server", feature = "verify"))] +use std::collections::BTreeMap; +#[cfg(any(feature = "server", feature = "verify"))] +use super::drive_document_ranked_query::index_picker::{ + encode_equality_prefix_values, find_ranked_index_for_axis, no_covering_index_message, +}; #[cfg(any(feature = "server", feature = "verify"))] use super::drive_document_ranked_query::{ path::indexed_property_name_tree_path_for_index, RankedAxis, }; #[cfg(any(feature = "server", feature = "verify"))] +use crate::error::query::QuerySyntaxError; +#[cfg(any(feature = "server", feature = "verify"))] use crate::error::Error; #[cfg(any(feature = "server", feature = "verify"))] use grovedb::element::indexed::{encode_avg_sort_key, encode_count_sort_key, encode_sum_sort_key}; @@ -210,7 +222,10 @@ impl AxisRangeBounds { /// /// Produced by [`mode_detection::detect_having_mode`]. Parallels /// [`super::drive_document_ranked_query::DocumentRankedMode`]. -#[derive(Debug, Clone, PartialEq, Eq)] +/// +/// Not `Eq`: the equality pins carry [`Value`]s, whose float variant +/// keeps the type at `PartialEq`. +#[derive(Debug, Clone, PartialEq)] #[cfg(any(feature = "server", feature = "verify"))] pub struct DocumentHavingMode { /// Inclusive bounds on the aggregate, in the axis's own domain. @@ -223,11 +238,16 @@ pub struct DocumentHavingMode { /// `1 ..= MAX_HAVING_LIMIT`, required. pub limit: u16, /// The single `GROUP BY` property; must be the covering ranked - /// index's only property. + /// index's **last** property. pub group_by_property: String, /// The field the aggregate applies to. Empty for `COUNT(*)`; the /// index's `summable` property for `SUM` / `AVG`. pub aggregate_field: String, + /// The equality `where` pins, `(property, value)` per clause — + /// exactly one per leading property of the covering compound index, + /// in request order (the resolver re-orders them into index order + /// when it encodes the path). Empty for the single-property form. + pub equality_pins: Vec<(String, Value)>, } /// A resolved having-range query. Shared by the prover and the verifier — @@ -245,10 +265,17 @@ pub struct DriveDocumentHavingQuery<'a> { pub contract_id: [u8; 32], /// The document type name — a path segment. pub document_type_name: String, - /// The covering ranked index. Single-property by construction; its - /// one property is both the `GROUP BY` property and the last path - /// segment. + /// The covering ranked index. Its **last** property is the `GROUP + /// BY` property and the final path segment; any leading properties + /// are pinned by [`Self::equality_prefix_values`]. pub index: &'a Index, + /// Encoded index-key bytes of each leading index property's pinned + /// value, in index-property order — empty for a single-property + /// index. Part of the prover/verifier agreement exactly as on the + /// ranked surface: the segments feed straight into the shared path + /// builder. Produced by + /// [`super::drive_document_ranked_query::index_picker::encode_equality_prefix_values`]. + pub equality_prefix_values: Vec>, /// Inclusive bounds on the aggregate. Carry the axis; the index must /// declare the matching `ranked_*` flag. pub bounds: AxisRangeBounds, @@ -271,7 +298,8 @@ pub struct DriveDocumentHavingQuery<'a> { #[cfg(any(feature = "server", feature = "verify"))] impl DriveDocumentHavingQuery<'_> { /// Path of the terminal property-name tree the axis secondary hangs - /// off — identical to the ranked surface's path, because both read + /// off — identical to the ranked surface's path (including the + /// pinned-prefix segments of a compound index), because both read /// the same indexed tree. See /// [`DriveDocumentRankedQuery::indexed_property_name_tree_path`](super::drive_document_ranked_query::DriveDocumentRankedQuery::indexed_property_name_tree_path). pub fn indexed_property_name_tree_path(&self) -> Result>, Error> { @@ -279,6 +307,69 @@ impl DriveDocumentHavingQuery<'_> { &self.contract_id, &self.document_type_name, self.index, + &self.equality_prefix_values, ) } } + +/// Resolve a validated [`DocumentHavingMode`] against a document type's +/// indexes into the executable [`DriveDocumentHavingQuery`]: pick the +/// covering index (shared with the ranked surface — both read the same +/// indexed tree), encode the equality pins into prefix-value path +/// segments, and assemble the query. +/// +/// The **one** resolution path for the having surface, mirroring +/// [`super::drive_document_ranked_query::index_picker::resolve_ranked_query_for_mode`]: +/// the server's executors and the SDK's proof helpers both call it, so a +/// proof and an unproven read (and the client's verification) are about +/// the same subtree by construction. +/// +/// `indexes` is threaded in separately for the same lifetime reason as +/// the ranked resolver: the returned query's `&'a Index` must outlive +/// this frame. Callers pass `document_type.indexes()`. +#[cfg(any(feature = "server", feature = "verify"))] +pub fn resolve_having_query_for_mode<'a>( + contract_id: [u8; 32], + document_type: DocumentTypeRef<'a>, + document_type_name: String, + indexes: &'a BTreeMap, + mode: &DocumentHavingMode, + platform_version: &PlatformVersion, +) -> Result, Error> { + let axis = mode.bounds.axis(); + let pin_fields: Vec = mode + .equality_pins + .iter() + .map(|(field, _)| field.clone()) + .collect(); + let index = find_ranked_index_for_axis( + indexes, + &mode.group_by_property, + &pin_fields, + axis, + &mode.aggregate_field, + ) + .ok_or_else(|| { + Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty( + no_covering_index_message( + "having-range", + axis, + &mode.group_by_property, + &mode.equality_pins, + &mode.aggregate_field, + ), + )) + })?; + let equality_prefix_values = + encode_equality_prefix_values(document_type, index, &mode.equality_pins, platform_version)?; + Ok(DriveDocumentHavingQuery { + document_type, + contract_id, + document_type_name, + index, + equality_prefix_values, + bounds: mode.bounds, + descending: mode.descending, + limit: mode.limit, + }) +} diff --git a/packages/rs-drive/src/query/drive_document_having_query/mode_detection/v0/mod.rs b/packages/rs-drive/src/query/drive_document_having_query/mode_detection/v0/mod.rs index d8ff3d4f4b4..9cefa259130 100644 --- a/packages/rs-drive/src/query/drive_document_having_query/mode_detection/v0/mod.rs +++ b/packages/rs-drive/src/query/drive_document_having_query/mode_detection/v0/mod.rs @@ -6,7 +6,9 @@ use super::{AxisRangeBounds, DocumentHavingMode, MAX_HAVING_LIMIT}; use crate::error::query::QuerySyntaxError; use crate::error::Error; -use crate::query::drive_document_ranked_query::mode_detection::ranked_order_key; +use crate::query::drive_document_ranked_query::mode_detection::{ + equality_pins_from_where_clauses, ranked_order_key, +}; use crate::query::drive_document_ranked_query::{RankedAxis, RankedPaginationInputs}; use crate::query::having::{ HavingAggregateFunction, HavingClause, HavingOperator, HavingRightOperand, @@ -26,8 +28,11 @@ use grovedb::element::indexed::AVG_FIXED_POINT_SCALE; /// SELECT AVG(f) GROUP BY p HAVING AVG(f) [ORDER BY f [ASC|DESC]] LIMIT n /// ``` /// -/// with no `WHERE`, no `OFFSET`, no `START AT` / `START AFTER`, exactly -/// one `GROUP BY` property, exactly one `HAVING` clause whose aggregate +/// with no `OFFSET`, no `START AT` / `START AFTER`, exactly one +/// `GROUP BY` property, `WHERE` clauses (when present) that are +/// equality pins on distinct properties — one per leading property of a +/// covering compound ranked index, selecting which prefix's secondary +/// the bound reads — exactly one `HAVING` clause whose aggregate /// **is the selected aggregate** (same function, same field), an operator /// from the contiguous-range family (`=`, `>`, `>=`, `<`, `<=`, and the /// four `BETWEEN*` variants — `!=` and `IN` describe non-contiguous @@ -70,15 +75,17 @@ pub fn detect_having_mode_v0( ) -> Result { // ---- GROUP BY: exactly one property ---------------------------- // - // Same contract as the ranked surface: ranked indexes are - // single-property, and the sole property is what the secondary's - // group keys are. + // Same contract as the ranked surface: the one property is the + // covering index's LAST property, whose distinct values are the + // secondary's group keys. A compound ranked index filters each + // prefix's groups separately — its leading properties are pinned by + // equality `where` clauses, never grouped over. if group_by.len() != 1 { return Err(Error::Query(QuerySyntaxError::InvalidParameter(format!( - "having-range queries require exactly one `group_by` property (the ranked \ - index's only property); got {}. Ranked indexes are single-property — compound \ - ranked indexes are rejected at contract-parse time — so there is no compound \ - grouping to filter over.", + "having-range queries require exactly one `group_by` property (the covering \ + ranked index's trailing property); got {}. A compound ranked index bounds \ + each prefix's groups separately — pin every leading index property with an \ + equality `where` clause and `group_by` the trailing property.", group_by.len() )))); } @@ -230,21 +237,14 @@ pub fn detect_having_mode_v0( } }; - // ---- WHERE: must be absent -------------------------------------- + // ---- WHERE: equality pins on the compound prefix ------------------ // - // Identical rationale to the ranked surface: single-property - // indexes have no equality prefix to narrow, and the secondary is - // ordered by aggregate, not by group key. - if !where_clauses.is_empty() { - return Err(Error::Query( - QuerySyntaxError::InvalidWhereClauseComponents( - "having-range queries do not accept `where` clauses: ranked indexes are \ - single-property, so there is no equality prefix to narrow, and the axis \ - secondary is ordered by aggregate rather than by group key — it cannot \ - bound a filtered subset. Bound the whole index, or add a narrower index.", - ), - )); - } + // Identical contract to the ranked surface: empty for the + // single-property form; for a compound ranked index, one equality + // pin per leading property selects which prefix's secondary the + // bound reads. Shape-only here; the index picker enforces the + // exact-cover rule. + let equality_pins = equality_pins_from_where_clauses(where_clauses)?; // ---- LIMIT: required, 1 ..= MAX_HAVING_LIMIT --------------------- // @@ -312,6 +312,7 @@ pub fn detect_having_mode_v0( limit, group_by_property, aggregate_field, + equality_pins, }) } diff --git a/packages/rs-drive/src/query/drive_document_having_query/tests.rs b/packages/rs-drive/src/query/drive_document_having_query/tests.rs index 924fcafc275..9bfd3277323 100644 --- a/packages/rs-drive/src/query/drive_document_having_query/tests.rs +++ b/packages/rs-drive/src/query/drive_document_having_query/tests.rs @@ -310,11 +310,13 @@ mod grammar { assert!(result.is_err(), "cross-aggregate having must fail"); } - /// `GROUP BY identityId, class HAVING AVG(grade) > 80` — compound - /// grouping — is rejected: ranked axes live on single-property - /// indexes (a contract declaring a ranked flag on a compound index - /// is already rejected at contract-parse time), so there is no - /// compound grouping for a bound to filter over. + /// `GROUP BY identityId, class HAVING AVG(grade) > 80` — an + /// **unpinned** two-field grouping — is still rejected: a compound + /// ranked index bounds each prefix's groups separately, so the + /// served form pins the leading property with an equality `where` + /// and groups over the trailing one (`WHERE identityId = X GROUP BY + /// class …` — exercised end to end in the `pinned_prefix` suite). + /// The rejection must steer the caller to that form. #[test] fn compound_group_by_is_rejected() { let result = detect_having_mode_v0( @@ -330,10 +332,12 @@ mod grammar { &[], pagination(10), ); - let error = result.expect_err("compound group_by must fail"); + let error = result.expect_err("unpinned compound group_by must fail"); + let message = format!("{error}"); assert!( - format!("{error}").contains("exactly one `group_by` property"), - "the rejection must say the surface is single-property, got: {error}" + message.contains("exactly one `group_by` property") + && message.contains("equality `where` clause"), + "the rejection must steer to the pinned-prefix form, got: {error}" ); } @@ -571,7 +575,7 @@ mod execution { use super::clause; use crate::drive::Drive; use crate::error::Error; - use crate::query::drive_document_ranked_query::index_picker::find_ranked_index_for_axis; + use crate::query::drive_document_having_query::resolve_having_query_for_mode; use crate::query::drive_document_ranked_query::{ RankedEntry, RankedEntryValue, RankedPaginationInputs, RANKED_COUNT_ORDER_KEY, }; @@ -818,24 +822,17 @@ mod execution { .get(case.document_type_name) .expect("doctype exists") .indexes(); - let index = find_ranked_index_for_axis( - indexes, - &mode.group_by_property, - mode.bounds.axis(), - &mode.aggregate_field, - ) - .expect("the fixture declares the axis"); - DriveDocumentHavingQuery { - document_type: contract + resolve_having_query_for_mode( + contract.id_ref().to_buffer(), + contract .document_type_for_name(case.document_type_name) .expect("doctype exists"), - contract_id: contract.id_ref().to_buffer(), - document_type_name: case.document_type_name.to_string(), - index, - bounds: mode.bounds, - descending: mode.descending, - limit: mode.limit, - } + case.document_type_name.to_string(), + indexes, + &mode, + platform_version(), + ) + .expect("the fixture declares the axis") } fn grovedb_root_hash(drive: &Drive) -> [u8; 32] { @@ -1266,7 +1263,7 @@ mod identifier_group_keys { use super::super::DriveDocumentHavingQuery; use super::clause; use crate::drive::Drive; - use crate::query::drive_document_ranked_query::index_picker::find_ranked_index_for_axis; + use crate::query::drive_document_having_query::resolve_having_query_for_mode; use crate::query::drive_document_ranked_query::{ RankedEntry, RankedEntryValue, RankedPaginationInputs, }; @@ -1423,24 +1420,17 @@ mod identifier_group_keys { .get(DOCUMENT_TYPE) .expect("grade doctype exists") .indexes(); - let index = find_ranked_index_for_axis( - indexes, - &mode.group_by_property, - mode.bounds.axis(), - &mode.aggregate_field, - ) - .expect("the fixture declares the avg axis"); - DriveDocumentHavingQuery { - document_type: contract + resolve_having_query_for_mode( + contract.id_ref().to_buffer(), + contract .document_type_for_name(DOCUMENT_TYPE) .expect("grade doctype exists"), - contract_id: contract.id_ref().to_buffer(), - document_type_name: DOCUMENT_TYPE.to_string(), - index, - bounds: mode.bounds, - descending: mode.descending, - limit: mode.limit, - } + DOCUMENT_TYPE.to_string(), + indexes, + &mode, + platform_version(), + ) + .expect("the fixture declares the avg axis") } fn assert_proof_round_trips( @@ -1537,3 +1527,585 @@ mod identifier_group_keys { assert_proof_round_trips(&drive, &contract, &descending, &flipped); } } + +mod pinned_prefix { + //! `SELECT AVG(grade) FROM grades WHERE identityId = X GROUP BY + //! class HAVING AVG(grade) > 80` — the compound ranked index + //! `[identityId, class]` with its leading property pinned by an + //! equality `where` clause. Per-prefix semantics: each identity's + //! terminal `class` property-name tree is its own indexed tree, so + //! the bound reads (and proves) only the pinned identity's class + //! groups. Documents are inserted through the real write path, so + //! these tests also pin that the document walkers create and + //! maintain the per-prefix secondaries for compound ranked indexes. + + use super::super::drive_dispatcher::{DocumentHavingRequest, DocumentHavingResponse}; + use super::super::mode_detection::detect_having_mode; + use super::super::DriveDocumentHavingQuery; + use super::clause; + use crate::drive::Drive; + use crate::error::query::QuerySyntaxError; + use crate::error::Error; + use crate::query::drive_document_having_query::resolve_having_query_for_mode; + use crate::query::drive_document_ranked_query::{ + RankedEntry, RankedEntryValue, RankedPaginationInputs, + }; + use crate::query::having::{HavingAggregateFunction, HavingOperator}; + use crate::query::projection::SelectProjection; + use crate::query::{OrderClause, WhereClause, WhereOperator}; + use crate::util::object_size_info::DocumentInfo::DocumentRefInfo; + use crate::util::object_size_info::{DocumentAndContractInfo, OwnedDocumentInfo}; + use crate::util::storage_flags::StorageFlags; + use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; + use dpp::block::block_info::BlockInfo; + use dpp::data_contract::accessors::v0::DataContractV0Getters; + use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; + use dpp::data_contract::document_type::random_document::CreateRandomDocument; + use dpp::document::{Document, DocumentV0Setters}; + use dpp::platform_value::Value; + use dpp::prelude::DataContract; + use dpp::tests::json_document::json_document_to_contract; + use dpp::version::PlatformVersion; + use grovedb::element::indexed::compute_avg_fixed_point; + use std::collections::BTreeMap; + + const PREFIX_PROPERTY: &str = "identityId"; + const GROUP_PROPERTY: &str = "class"; + const DOCUMENT_TYPE: &str = "grade"; + + fn platform_version() -> &'static PlatformVersion { + PlatformVersion::latest() + } + + fn setup_grades_compound_ranked() -> (Drive, DataContract) { + let drive = setup_drive_with_initial_state_structure(None); + let pv = platform_version(); + let contract = json_document_to_contract( + "tests/supporting_files/contract/grades/grades-compound-ranked-contract.json", + false, + pv, + ) + .expect("expected to parse the compound ranked grades contract"); + drive + .apply_contract( + &contract, + BlockInfo::default(), + true, + StorageFlags::optional_default_as_cow(), + None, + pv, + ) + .expect("expected to apply the compound ranked grades contract"); + (drive, contract) + } + + fn insert_grades( + drive: &Drive, + contract: &DataContract, + first_seed: u64, + rows: &[([u8; 32], &str, i64)], + ) { + let pv = platform_version(); + let document_type = contract + .document_type_for_name(DOCUMENT_TYPE) + .expect("grade doctype exists"); + for (i, (identity, class, grade)) in rows.iter().enumerate() { + let mut doc: Document = document_type + .random_document(Some(first_seed + i as u64), pv) + .expect("random document"); + let mut props = BTreeMap::new(); + props.insert(PREFIX_PROPERTY.to_string(), Value::Identifier(*identity)); + props.insert(GROUP_PROPERTY.to_string(), Value::Text(class.to_string())); + props.insert("grade".to_string(), Value::I64(*grade)); + doc.set_properties(props); + drive + .add_document_for_contract( + DocumentAndContractInfo { + owned_document_info: OwnedDocumentInfo { + document_info: DocumentRefInfo((&doc, None)), + owner_id: None, + }, + contract, + document_type, + }, + false, + BlockInfo::default(), + true, + None, + pv, + None, + ) + .expect("expected to insert a grade document"); + } + } + + fn pin(identity: [u8; 32]) -> Vec { + vec![WhereClause { + field: PREFIX_PROPERTY.to_string(), + operator: WhereOperator::Equal, + value: Value::Identifier(identity), + }] + } + + fn run( + drive: &Drive, + contract: &DataContract, + where_clauses: &[WhereClause], + order_by: &[OrderClause], + prove: bool, + ) -> Result { + let group_by = vec![GROUP_PROPERTY.to_string()]; + let having = vec![clause( + HavingAggregateFunction::Avg, + "grade", + HavingOperator::GreaterThan, + Value::U64(80), + )]; + drive.execute_document_having_request( + DocumentHavingRequest { + contract, + document_type: contract + .document_type_for_name(DOCUMENT_TYPE) + .expect("grade doctype exists"), + group_by: &group_by, + select: SelectProjection::avg("grade"), + having: &having, + order_by, + where_clauses, + limit: Some(10), + offset: None, + has_start_at: false, + prove, + }, + None, + platform_version(), + ) + } + + fn entries_of(response: DocumentHavingResponse) -> Vec { + match response { + DocumentHavingResponse::Entries(entries) => entries, + DocumentHavingResponse::Proof(_) => panic!("expected entries, got a proof"), + } + } + + fn client_side_query<'a>( + contract: &'a DataContract, + where_clauses: &[WhereClause], + order_by: &[OrderClause], + ) -> DriveDocumentHavingQuery<'a> { + let group_by = vec![GROUP_PROPERTY.to_string()]; + let having = vec![clause( + HavingAggregateFunction::Avg, + "grade", + HavingOperator::GreaterThan, + Value::U64(80), + )]; + let mode = detect_having_mode( + &SelectProjection::avg("grade"), + &group_by, + &having, + order_by, + where_clauses, + RankedPaginationInputs { + limit: Some(10), + offset: None, + has_start_at: false, + }, + platform_version(), + ) + .expect("the case is well-formed"); + let indexes = contract + .document_types() + .get(DOCUMENT_TYPE) + .expect("grade doctype exists") + .indexes(); + resolve_having_query_for_mode( + contract.id_ref().to_buffer(), + contract + .document_type_for_name(DOCUMENT_TYPE) + .expect("grade doctype exists"), + DOCUMENT_TYPE.to_string(), + indexes, + &mode, + platform_version(), + ) + .expect("the fixture's compound index covers the pinned request") + } + + fn assert_proof_round_trips( + drive: &Drive, + contract: &DataContract, + where_clauses: &[WhereClause], + order_by: &[OrderClause], + expected: &[RankedEntry], + ) { + let proof = match run(drive, contract, where_clauses, order_by, true) + .expect("prove must succeed") + { + DocumentHavingResponse::Proof(proof) => proof, + DocumentHavingResponse::Entries(_) => panic!("expected a proof, got entries"), + }; + let (root_hash, verified) = client_side_query(contract, where_clauses, order_by) + .verify_having_range_proof(&proof, platform_version()) + .expect("the proof must verify"); + assert_eq!( + verified, expected, + "verified entries must equal what the unproven read returned" + ); + assert_eq!( + root_hash, + drive + .grove + .root_hash(None, &platform_version().drive.grove_version) + .unwrap() + .expect("root hash must be readable"), + "the proof must reconstruct the live grovedb root hash" + ); + } + + const IDENTITY_X: [u8; 32] = [1u8; 32]; + const IDENTITY_Y: [u8; 32] = [2u8; 32]; + + /// The shared dataset: two identities whose class groups overlap by + /// name, so cross-prefix leakage is visible by construction — + /// `math` fails X's bound (avg 80, strict `>`) but passes Y's (avg + /// 92.5), and `science` exists only under Y. + fn insert_two_identities(drive: &Drive, contract: &DataContract) { + insert_grades( + drive, + contract, + 1000, + &[ + // Identity X: math avg 80 (at threshold, excluded under >), + // english avg 80.5 (fractional, just above), art avg 90, + // history avg 70. + (IDENTITY_X, "math", 80), + (IDENTITY_X, "math", 80), + (IDENTITY_X, "english", 80), + (IDENTITY_X, "english", 81), + (IDENTITY_X, "art", 85), + (IDENTITY_X, "art", 95), + (IDENTITY_X, "history", 60), + (IDENTITY_X, "history", 80), + // Identity Y: math avg 92.5, science avg 95 — both would + // qualify for X's bound too if prefixes leaked. + (IDENTITY_Y, "math", 90), + (IDENTITY_Y, "math", 95), + (IDENTITY_Y, "science", 95), + (IDENTITY_Y, "science", 95), + ], + ); + } + + /// Exact-threshold exclusion, fractional inclusion, byte-exact + /// string group keys, both walk directions, proof round-trips — + /// and isolation: identity Y's qualifying classes never appear in + /// X's result, even where the class *name* collides. + #[test] + fn avg_threshold_over_pinned_prefix_reads_and_proves() { + let (drive, contract) = setup_grades_compound_ranked(); + insert_two_identities(&drive, &contract); + + // Pinned to X, ascending: english (80.5) then art (90). math sits + // exactly at the threshold and stays out under `>`; Y's math + // (92.5) and science (95) must not leak in. + let x_pin = pin(IDENTITY_X); + let entries = + entries_of(run(&drive, &contract, &x_pin, &[], false).expect("read succeeds")); + assert_eq!( + entries, + vec![ + RankedEntry { + key: b"english".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(161, 2)), + }, + RankedEntry { + key: b"art".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(180, 2)), + }, + ], + "ascending walk over X's classes: 80.5 then 90, keyed by raw utf-8 class bytes" + ); + assert_proof_round_trips(&drive, &contract, &x_pin, &[], &entries); + + // Same pin, descending: art then english, with its own proof. + let descending = vec![OrderClause { + field: "grade".to_string(), + ascending: false, + }]; + let flipped = + entries_of(run(&drive, &contract, &x_pin, &descending, false).expect("read succeeds")); + assert_eq!( + flipped.iter().map(|e| &e.key).collect::>(), + vec![&b"art".to_vec(), &b"english".to_vec()], + "descending walk: 90 then 80.5" + ); + assert_proof_round_trips(&drive, &contract, &x_pin, &descending, &flipped); + + // Pinned to Y: math qualifies *here* (92.5) even though the same + // class name failed X's bound — the two prefixes rank separately. + let y_pin = pin(IDENTITY_Y); + let y_entries = + entries_of(run(&drive, &contract, &y_pin, &[], false).expect("read succeeds")); + assert_eq!( + y_entries, + vec![ + RankedEntry { + key: b"math".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(185, 2)), + }, + RankedEntry { + key: b"science".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(190, 2)), + }, + ], + "Y's own classes: math 92.5 then science 95" + ); + assert_proof_round_trips(&drive, &contract, &y_pin, &[], &y_entries); + } + + /// An unpinned prefix cannot be served: with `group_by = class` and + /// no `where`, no single-property ranked index on `class` exists, + /// and the compound index's per-prefix secondaries have no global + /// ordering to read. The rejection names the missing coverage. + #[test] + fn unpinned_prefix_is_rejected() { + let (drive, contract) = setup_grades_compound_ranked(); + insert_two_identities(&drive, &contract); + + let error = run(&drive, &contract, &[], &[], false) + .expect_err("an unpinned compound prefix must not resolve"); + match error { + Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty(message)) => { + assert!( + message.contains("no ranked index covers") + && message.contains("single-property index on `class`"), + "the rejection must explain the missing coverage, got: {message}" + ); + } + other => panic!("expected a no-covering-index rejection, got {other:?}"), + } + } + + /// `IN` on the prefix is rejected at detection with the + /// not-yet-supported message (v1 pins are equality-only), and a pin + /// on a property that is not the index's leading property fails + /// resolution. + #[test] + fn in_prefix_and_wrong_pins_are_rejected() { + let (drive, contract) = setup_grades_compound_ranked(); + insert_two_identities(&drive, &contract); + + let in_clause = vec![WhereClause { + field: PREFIX_PROPERTY.to_string(), + operator: WhereOperator::In, + value: Value::Array(vec![ + Value::Identifier(IDENTITY_X), + Value::Identifier(IDENTITY_Y), + ]), + }]; + let error = run(&drive, &contract, &in_clause, &[], false) + .expect_err("IN prefixes are not yet supported"); + match error { + Error::Query(QuerySyntaxError::Unsupported(message)) => { + assert!( + message.contains("IN") && message.contains("not yet supported"), + "the IN rejection must say it is a not-yet capability, got: {message}" + ); + } + other => panic!("expected Unsupported for an IN prefix, got {other:?}"), + } + + // A pin on the wrong property: `grade` is not the index's + // leading property, so nothing covers [grade, class]. + let wrong_pin = vec![WhereClause { + field: "grade".to_string(), + operator: WhereOperator::Equal, + value: Value::I64(80), + }]; + let error = run(&drive, &contract, &wrong_pin, &[], false) + .expect_err("a pin on a non-leading property must not resolve"); + assert!( + matches!( + error, + Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty(_)) + ), + "expected a no-covering-index rejection, got {error:?}" + ); + } + + /// A pin on an identity that never inserted a document addresses a + /// prefix value tree that does not exist. The read and the prover + /// both surface an error rather than fabricating an empty page — + /// same contract as the empty-secondary limitation on the + /// single-property surface (the abci layer maps these to a + /// client-visible rejection). + #[test] + fn unknown_prefix_value_errors_rather_than_fabricating_an_empty_page() { + let (drive, contract) = setup_grades_compound_ranked(); + insert_two_identities(&drive, &contract); + + let unknown = pin([9u8; 32]); + assert!( + run(&drive, &contract, &unknown, &[], false).is_err(), + "reading a never-written prefix value tree must error" + ); + assert!( + run(&drive, &contract, &unknown, &[], true).is_err(), + "proving a never-written prefix value tree must error" + ); + } + + /// A **null** pin addresses the empty-segment prefix the write path + /// creates for an absent optional leading property — the having + /// bound reads (and proves) the tagless subtree, and tagged rows + /// stay in their own prefix. Sibling of the ranked suite's + /// `a_null_pin_addresses_the_absent_value_prefix`. + #[test] + fn a_null_pin_bounds_the_absent_value_prefix() { + const TAGGED_DOCTYPE: &str = "taggedGrade"; + let (drive, contract) = setup_grades_compound_ranked(); + let pv = platform_version(); + let document_type = contract + .document_type_for_name(TAGGED_DOCTYPE) + .expect("taggedGrade doctype exists"); + + for (i, (tag, class, grade)) in [ + (None, "math", 80i64), + (None, "math", 90), + (None, "science", 60), + (Some("honors"), "science", 100), + ] + .iter() + .enumerate() + { + let mut doc: Document = document_type + .random_document(Some(8000 + i as u64), pv) + .expect("random document"); + let mut props = BTreeMap::new(); + if let Some(tag) = tag { + props.insert("tag".to_string(), Value::Text(tag.to_string())); + } + props.insert(GROUP_PROPERTY.to_string(), Value::Text(class.to_string())); + props.insert("grade".to_string(), Value::I64(*grade)); + doc.set_properties(props); + drive + .add_document_for_contract( + DocumentAndContractInfo { + owned_document_info: OwnedDocumentInfo { + document_info: DocumentRefInfo((&doc, None)), + owner_id: None, + }, + contract: &contract, + document_type, + }, + false, + BlockInfo::default(), + true, + None, + pv, + None, + ) + .expect("expected to insert a tagged grade document"); + } + + let null_pin = vec![WhereClause { + field: "tag".to_string(), + operator: WhereOperator::Equal, + value: Value::Null, + }]; + let group_by = vec![GROUP_PROPERTY.to_string()]; + let having = vec![clause( + HavingAggregateFunction::Avg, + "grade", + HavingOperator::GreaterThan, + Value::U64(70), + )]; + let request = |prove: bool| DocumentHavingRequest { + contract: &contract, + document_type, + group_by: &group_by, + select: SelectProjection::avg("grade"), + having: &having, + order_by: &[], + where_clauses: &null_pin, + limit: Some(10), + offset: None, + has_start_at: false, + prove, + }; + + // Only the tagless math group (avg 85) clears the bound: the + // tagless science (60) misses it, and the tagged science (100) + // lives under its own prefix. + let entries = match drive + .execute_document_having_request(request(false), None, pv) + .expect("the null-pinned read succeeds") + { + DocumentHavingResponse::Entries(entries) => entries, + DocumentHavingResponse::Proof(_) => panic!("expected entries, got a proof"), + }; + assert_eq!( + entries, + vec![RankedEntry { + key: b"math".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(170, 2)), + }], + "the tagless bound: math 85 only — the honors science row \ + must not leak into the null prefix" + ); + + let proof = match drive + .execute_document_having_request(request(true), None, pv) + .expect("the null-pinned prove succeeds") + { + DocumentHavingResponse::Proof(proof) => proof, + DocumentHavingResponse::Entries(_) => panic!("expected a proof, got entries"), + }; + let mode = detect_having_mode( + &SelectProjection::avg("grade"), + &group_by, + &having, + &[], + &null_pin, + RankedPaginationInputs { + limit: Some(10), + offset: None, + has_start_at: false, + }, + pv, + ) + .expect("the null-pinned case is well-formed"); + let query = resolve_having_query_for_mode( + contract.id_ref().to_buffer(), + document_type, + TAGGED_DOCTYPE.to_string(), + contract + .document_types() + .get(TAGGED_DOCTYPE) + .expect("taggedGrade doctype exists") + .indexes(), + &mode, + pv, + ) + .expect("the compound index covers the null-pinned request"); + assert_eq!( + query.equality_prefix_values, + vec![Vec::::new()], + "a null pin must encode as the write path's empty segment" + ); + let (root_hash, verified) = query + .verify_having_range_proof(&proof, pv) + .expect("the null-pinned proof must verify"); + assert_eq!(verified, entries); + assert_eq!( + root_hash, + drive + .grove + .root_hash(None, &pv.drive.grove_version) + .unwrap() + .expect("root hash must be readable"), + ); + } +} diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/drive_dispatcher.rs b/packages/rs-drive/src/query/drive_document_ranked_query/drive_dispatcher.rs index 1783abbc925..23bd902fe6e 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/drive_dispatcher.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/drive_dispatcher.rs @@ -37,19 +37,18 @@ use grovedb::TransactionArg; /// canonicalized or rewritten the way count's where-clauses are, so /// taking ownership would just force the handler into a clone. /// -/// `where_clauses`, `having` and `start_at` are carried even though a -/// ranked request must leave all of them empty: drive owns the -/// rejection, so the contract is enforced identically no matter which -/// upstream path built the request. See -/// [`super::mode_detection::detect_ranked_mode_v0`] for why each is -/// refused rather than ignored. +/// `having` and `start_at` are carried even though a ranked request +/// must leave both empty: drive owns the rejection, so the contract is +/// enforced identically no matter which upstream path built the +/// request. See [`super::mode_detection::detect_ranked_mode_v0`] for +/// why each is refused rather than ignored. pub struct DocumentRankedRequest<'a> { /// Live contract (already loaded by the handler). pub contract: &'a DataContract, /// Resolved document type within `contract`. pub document_type: DocumentTypeRef<'a>, - /// The single `GROUP BY` property. Must be the ranked index's only - /// property. + /// The single `GROUP BY` property. Must be the covering ranked + /// index's trailing property. pub group_by: &'a [String], /// The projection being ranked: `COUNT(*)`, `SUM(field)` or /// `AVG(field)`. @@ -61,7 +60,9 @@ pub struct DocumentRankedRequest<'a> { /// aggregate (`$count` for `COUNT(*)`, otherwise the select's /// field); its direction is the ranking direction. pub order_by: &'a [OrderClause], - /// Structured `where` clauses. Must be empty. + /// Structured `where` clauses. Empty for the single-property form; + /// equality pins on the covering compound index's leading + /// properties for the pinned-prefix form. pub where_clauses: &'a [WhereClause], /// Request `limit` — the ranking's `k`. **Required**; there is no /// server default a verifying client could reproduce. diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/executors/mod.rs b/packages/rs-drive/src/query/drive_document_ranked_query/executors/mod.rs index b48bebc6560..56f31682687 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/executors/mod.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/executors/mod.rs @@ -15,55 +15,10 @@ pub mod top_k_no_proof; pub mod top_k_proof; -use super::index_picker::find_ranked_index_for_mode; -use super::{DocumentRankedMode, DriveDocumentRankedQuery}; -use crate::error::query::QuerySyntaxError; -use crate::error::Error; -use dpp::data_contract::document_type::{DocumentTypeRef, Index}; -use std::collections::BTreeMap; - -/// Resolve a validated [`DocumentRankedMode`] against a document type's -/// indexes into the executable [`DriveDocumentRankedQuery`]. -/// -/// `indexes` is threaded in separately rather than read off -/// `document_type` here because -/// [`DocumentTypeV0Getters::indexes`] borrows its receiver — taking the -/// map from the caller lets the returned query's `&'a Index` outlive this -/// frame. Callers pass `document_type.indexes()`. -/// -/// The only failure is "no index covers this", which is reported with the -/// exact contract keyword the index is missing so the caller can act on -/// it without reading the schema spec. -pub(super) fn ranked_query_for_mode<'a>( - contract_id: [u8; 32], - document_type: DocumentTypeRef<'a>, - document_type_name: String, - indexes: &'a BTreeMap, - mode: &DocumentRankedMode, -) -> Result, Error> { - let index = find_ranked_index_for_mode(indexes, mode).ok_or_else(|| { - Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty(format!( - "no ranked index covers `group_by = [{}]` on the {:?} axis: the document type \ - needs a single-property index on `{}` declaring `{}`{}", - mode.group_by_property, - mode.axis, - mode.group_by_property, - mode.axis.required_index_keyword(), - if mode.aggregate_field.is_empty() { - String::new() - } else { - format!(" with `summable: \"{}\"`", mode.aggregate_field) - } - ))) - })?; - Ok(DriveDocumentRankedQuery { - document_type, - contract_id, - document_type_name, - index, - axis: mode.axis, - descending: mode.descending, - k: mode.k, - offset: mode.offset, - }) -} +// Resolution — covering-index pick + equality-pin encoding — is shared +// with the SDK's proof helpers through +// [`super::index_picker::resolve_ranked_query_for_mode`]: both sides +// must land on the same index and the same prefix segments, or a client +// would verify a proof about a different subtree than the one an +// unproven read returned. +pub(super) use super::index_picker::resolve_ranked_query_for_mode as ranked_query_for_mode; diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_no_proof.rs b/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_no_proof.rs index 1b7b2b9d200..281fdfe2c2a 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_no_proof.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_no_proof.rs @@ -31,6 +31,7 @@ impl Drive { document_type_name, indexes, mode, + platform_version, )?; ranked_query.execute_top_k_no_proof(self, transaction, platform_version) } diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_proof.rs b/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_proof.rs index 478f498e05d..f276d7c0204 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_proof.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/executors/top_k_proof.rs @@ -35,6 +35,7 @@ impl Drive { document_type_name, indexes, mode, + platform_version, )?; ranked_query.execute_top_k_with_proof(self, transaction, platform_version) } diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/index_picker.rs b/packages/rs-drive/src/query/drive_document_ranked_query/index_picker.rs index 251979e2b87..5f5408fbf30 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/index_picker.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/index_picker.rs @@ -1,24 +1,35 @@ -//! Covering-index picker for the ranked query. +//! Covering-index picker for the ranked query, plus the shared +//! prefix-value encoding. //! -//! Pure function on the document type's index map plus the -//! `(group property, axis, aggregate field)` triple +//! Pure functions on the document type's index map plus the +//! `(group property, equality pins, axis, aggregate field)` tuple //! [`super::mode_detection`] resolved. No Drive, no proof — the server -//! and the SDK verifier both call it so they land on the same index (and -//! therefore the same grove path) for the same request. +//! and the SDK verifier both call these so they land on the same index +//! (and therefore the same grove path) for the same request. -use super::{DocumentRankedMode, RankedAxis}; -use dpp::data_contract::document_type::Index; +use super::{DocumentRankedMode, DriveDocumentRankedQuery, RankedAxis}; +use crate::error::query::QuerySyntaxError; +use crate::error::Error; +use dpp::data_contract::document_type::methods::DocumentTypeV0Methods; +use dpp::data_contract::document_type::{DocumentTypeRef, Index}; +use dpp::platform_value::Value; +use dpp::version::PlatformVersion; use std::collections::BTreeMap; /// Find the index that can serve `axis` ranking grouped by -/// `group_by_property`, aggregating `aggregate_field`. +/// `group_by_property` with the given equality pins, aggregating +/// `aggregate_field`. /// /// An index qualifies when **all** of: /// -/// - it has exactly one property, and that property is -/// `group_by_property` — the ranked secondary's group keys are the -/// values of an index's *last* property, and rs-dpp only allows -/// single-property ranked indexes, so "last" and "only" coincide; +/// - it has exactly one more property than there are pins, and its +/// **last** property is `group_by_property` — the ranked secondary's +/// group keys are the values of an index's last property; +/// - every **leading** property is pinned: each appears (by name) among +/// `equality_pin_fields`. Lengths matching plus the pins being +/// distinct (enforced upstream by +/// [`super::mode_detection::equality_pins_from_where_clauses`]) makes +/// this set equality, so no pin is left over either; /// - it declares the ranking keyword for `axis` /// ([`RankedAxis::required_index_keyword`]); /// - for [`RankedAxis::Sum`] / [`RankedAxis::Avg`], its `summable` @@ -27,17 +38,25 @@ use std::collections::BTreeMap; /// group's count), so summing a *different* field than the one the /// index accumulates would silently answer about the wrong property. /// +/// With no pins this degenerates to the original single-property rule. +/// A partial pin (some but not all leading properties) matches nothing — +/// the per-prefix secondary lives under one value tree per leading +/// property, so there is no subtree an unpinned prefix could address — +/// and callers turn the `None` into a loud +/// [`crate::error::query::QuerySyntaxError`] naming what is missing. +/// /// Returns `None` when nothing qualifies; callers turn that into /// [`crate::error::query::QuerySyntaxError::WhereClauseOnNonIndexedProperty`] /// with a message naming the missing keyword. /// -/// At most one index can qualify for a given `(group property, axis, -/// field)` triple — rs-dpp rejects two indexes over the same property set -/// on one document type — so "first match wins" is not a tie-break in -/// practice. Should that ever change, the `BTreeMap` iteration order -/// (index name, ascending) keeps the choice deterministic, which is what -/// prover/verifier agreement actually requires: both sides run this same -/// function over the same contract and must land on the same grove path. +/// At most one index can qualify for a given `(group property, pins, +/// axis, field)` tuple — rs-dpp rejects two indexes over the same +/// property set on one document type — so "first match wins" is not a +/// tie-break in practice. Should that ever change, the `BTreeMap` +/// iteration order (index name, ascending) keeps the choice +/// deterministic, which is what prover/verifier agreement actually +/// requires: both sides run this same function over the same contract +/// and must land on the same grove path. /// /// Note that axis availability is decided from the index's `ranked_*` /// flags, **not** from the element variant the write path laid down: a @@ -47,12 +66,23 @@ use std::collections::BTreeMap; pub fn find_ranked_index_for_axis<'b>( indexes: &'b BTreeMap, group_by_property: &str, + equality_pin_fields: &[String], axis: RankedAxis, aggregate_field: &str, ) -> Option<&'b Index> { indexes.values().find(|index| { - // Single-property, and that property is the grouping property. - if index.properties.len() != 1 || index.properties[0].name != group_by_property { + // Trailing property is the grouping property; every leading + // property is pinned exactly once (length equality + distinct + // pins ⇒ set equality). + let Some((terminal, leading)) = index.properties.split_last() else { + return false; + }; + if terminal.name != group_by_property + || leading.len() != equality_pin_fields.len() + || !leading + .iter() + .all(|property| equality_pin_fields.iter().any(|f| f == &property.name)) + { return false; } match axis { @@ -73,10 +103,176 @@ pub fn find_ranked_index_for_mode<'b>( indexes: &'b BTreeMap, mode: &DocumentRankedMode, ) -> Option<&'b Index> { + let pin_fields: Vec = mode + .equality_pins + .iter() + .map(|(field, _)| field.clone()) + .collect(); find_ranked_index_for_axis( indexes, &mode.group_by_property, + &pin_fields, mode.axis, &mode.aggregate_field, ) } + +/// Resolve a validated [`DocumentRankedMode`] against a document type's +/// indexes into the executable [`DriveDocumentRankedQuery`]: pick the +/// covering index, encode the equality pins into prefix-value path +/// segments, and assemble the query. +/// +/// This is the **one** resolution path — the server's executors and the +/// SDK's proof helpers both call it, which is what guarantees a proof +/// and an unproven read (and the client's verification) are about the +/// same subtree. +/// +/// `indexes` is threaded in separately rather than read off +/// `document_type` here because +/// [`DocumentTypeV0Getters::indexes`](dpp::data_contract::document_type::accessors::DocumentTypeV0Getters::indexes) +/// borrows its receiver — taking the map from the caller lets the +/// returned query's `&'a Index` outlive this frame. Callers pass +/// `document_type.indexes()`. +/// +/// The main failure is "no index covers this", reported with the exact +/// contract keyword (and, for pinned requests, the exact index shape) +/// the request needs, so the caller can act on it without reading the +/// schema spec. +pub fn resolve_ranked_query_for_mode<'a>( + contract_id: [u8; 32], + document_type: DocumentTypeRef<'a>, + document_type_name: String, + indexes: &'a BTreeMap, + mode: &DocumentRankedMode, + platform_version: &PlatformVersion, +) -> Result, Error> { + let index = find_ranked_index_for_mode(indexes, mode).ok_or_else(|| { + Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty( + no_covering_index_message( + "ranked", + mode.axis, + &mode.group_by_property, + &mode.equality_pins, + &mode.aggregate_field, + ), + )) + })?; + let equality_prefix_values = + encode_equality_prefix_values(document_type, index, &mode.equality_pins, platform_version)?; + Ok(DriveDocumentRankedQuery { + document_type, + contract_id, + document_type_name, + index, + equality_prefix_values, + axis: mode.axis, + descending: mode.descending, + k: mode.k, + offset: mode.offset, + }) +} + +/// The "no index covers this request" rejection text, shared by the +/// ranked and having-range resolutions (and the SDK's mirrors of them) +/// so a rejected request reads identically everywhere. Names the exact +/// index the request needs: property list (pins first, in request +/// order, then the grouping property), ranking keyword, and `summable` +/// field where applicable. +pub fn no_covering_index_message( + surface: &str, + axis: RankedAxis, + group_by_property: &str, + equality_pins: &[(String, Value)], + aggregate_field: &str, +) -> String { + let pin_fields = || { + equality_pins + .iter() + .map(|(field, _)| field.as_str()) + .collect::>() + .join(", ") + }; + let index_shape = if equality_pins.is_empty() { + format!("a single-property index on `{group_by_property}`") + } else { + format!( + "a compound index on [{}, {group_by_property}] (every leading property pinned \ + by an equality `where` clause, the trailing property grouped over)", + pin_fields() + ) + }; + format!( + "no ranked index covers `group_by = [{group_by_property}]`{} on the {axis:?} axis \ + for this {surface} query: the document type needs {index_shape} declaring `{}`{}", + if equality_pins.is_empty() { + String::new() + } else { + format!(" with equality pins on [{}]", pin_fields()) + }, + axis.required_index_keyword(), + if aggregate_field.is_empty() { + String::new() + } else { + format!(" with `summable: \"{aggregate_field}\"`") + } + ) +} + +/// Encode the equality pins into the grove path's prefix-value +/// segments: for each **leading** property of `index`, in index order, +/// the pinned value's index-key bytes +/// (`DocumentType::serialize_value_for_key` — the same encoding the +/// write path used to key that prefix's value tree). +/// +/// This is part of the prover/verifier agreement: server executors and +/// the SDK's proof helpers both come through here, so a pinned value +/// can only ever name one subtree, identically on both sides. +/// +/// `index` must have been picked by [`find_ranked_index_for_axis`] +/// against these same pins — every leading property is then guaranteed +/// a pin. A value the property's type cannot encode (a string against +/// an integer property, an out-of-range integer) is a caller error, +/// reported as a query-syntax rejection naming the property. +pub fn encode_equality_prefix_values( + document_type: DocumentTypeRef, + index: &Index, + equality_pins: &[(String, Value)], + platform_version: &PlatformVersion, +) -> Result>, Error> { + let leading = &index.properties[..index.properties.len().saturating_sub(1)]; + leading + .iter() + .map(|property| { + let (_, value) = equality_pins + .iter() + .find(|(field, _)| field == &property.name) + .ok_or_else(|| { + Error::Query(QuerySyntaxError::InvalidWhereClauseComponents( + "internal resolution mismatch: the picked compound ranked index has \ + a leading property with no equality pin — the index picker and the \ + prefix encoder disagreed on the pins", + )) + })?; + // A null pin addresses the subtree the write walkers create + // for an **absent** value: they encode it as + // `get_raw_for_document_type(..).unwrap_or_default()` — an + // empty path segment — for user and system properties alike. + // Null must short-circuit here because the system-property + // encoders (`$updatedAt`, `$creatorId`, …) reject null before + // any encoding happens, which would make the stored + // empty-segment prefix unaddressable. + if value.is_null() { + return Ok(Vec::new()); + } + document_type + .serialize_value_for_key(&property.name, value, platform_version) + .map_err(|e| { + Error::Query(QuerySyntaxError::InvalidParameter(format!( + "the equality pin on `{}` does not encode as that property's \ + index key: {e}", + property.name + ))) + }) + }) + .collect() +} diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/mod.rs b/packages/rs-drive/src/query/drive_document_ranked_query/mod.rs index 70cc2dd2339..fed2b5dffbd 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/mod.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/mod.rs @@ -38,15 +38,20 @@ //! secondary Merk, keyed by `(sort_key ‖ group_key)`. Three consequences //! shape the API: //! -//! 1. **No `where` clauses.** Ranked indexes are single-property only -//! (compound ones are rejected at contract-parse time in rs-dpp — -//! their terminal level is created lazily by the same batch that -//! populates it, which grovedb rejects for indexed trees). With one -//! property there is no equality prefix to narrow, and a `where` on -//! the ranked property itself would ask for a *filtered* ranking, -//! which the secondary cannot express — it is sorted by aggregate, -//! not by group key. Non-empty `where` is therefore rejected rather -//! than silently ignored. +//! 1. **`where` clauses are equality pins on a compound prefix — or +//! absent.** A single-property ranked index has no prefix to narrow, +//! so its requests carry no `where`. A compound ranked index +//! `[p1, …, pn]` maintains one secondary **per prefix value** +//! (per-prefix semantics: each terminal `pn` property-name tree, +//! inside the `[p1, …, pn-1]` value trees, is its own indexed tree +//! — grovedb creates and populates it in the same document batch), +//! so a request must pin every leading property with an equality +//! clause to name which prefix's secondary the walk reads. A `where` +//! on the grouped (terminal) property itself would ask for a +//! *filtered* ranking, which no secondary can express — it is sorted +//! by aggregate, not by group key — and is rejected rather than +//! silently ignored, as is any non-equality prefix clause (`IN` +//! included: one walk per element is a future multi-`IN` capability). //! 2. **`limit` is mandatory, `offset` is free, `start_at` is refused.** //! `limit` is the `k` of the walk and the ranked surface has no //! server default for it, so it must be supplied. `offset` is the @@ -63,6 +68,8 @@ #[cfg(any(feature = "server", feature = "verify"))] use dpp::data_contract::document_type::{DocumentTypeRef, Index}; +#[cfg(any(feature = "server", feature = "verify"))] +use dpp::platform_value::Value; /// The fixed-point scale grovedb's Avg axis sorts by: /// `avg_fixed_point = floor(sum * RANKED_AVG_SCALE / count)` with @@ -268,10 +275,19 @@ pub struct DriveDocumentRankedQuery<'a> { pub contract_id: [u8; 32], /// The document type name — a path segment. pub document_type_name: String, - /// The covering ranked index. Single-property by construction; its - /// one property is both the `GROUP BY` property and the last path - /// segment. + /// The covering ranked index. Its **last** property is the `GROUP + /// BY` property and the final path segment; any leading properties + /// are pinned by [`Self::equality_prefix_values`]. pub index: &'a Index, + /// Encoded index-key bytes of each leading index property's pinned + /// value, in index-property order — empty for a single-property + /// index. Together with `index` these determine the grove path + /// (each leading property contributes a name segment and a value + /// segment), so they are as much a part of the prover/verifier + /// agreement as the path builder itself. Produced by + /// [`index_picker::encode_equality_prefix_values`] from the + /// request's equality `where` pins. + pub equality_prefix_values: Vec>, /// Which aggregate the groups are ranked by. Must be covered by /// `index`'s matching `ranked_*` flag. pub axis: RankedAxis, @@ -371,8 +387,11 @@ pub struct RankedPaginationInputs { /// the versioned classification of a request — but carries data rather /// than being a bare discriminant, because the ranked surface has exactly /// one executor pair (no-proof / proof) and all of its variation is in -/// these five values. -#[derive(Debug, Clone, PartialEq, Eq)] +/// these values. +/// +/// Not `Eq`: the equality pins carry [`Value`]s, whose float variant +/// keeps the type at `PartialEq`. +#[derive(Debug, Clone, PartialEq)] #[cfg(any(feature = "server", feature = "verify"))] pub struct DocumentRankedMode { /// The ranking axis, from the `SELECT` function. @@ -383,11 +402,19 @@ pub struct DocumentRankedMode { pub k: u16, /// Ranks to skip — the `OFFSET`, `0` when unset. pub offset: u32, - /// The single `GROUP BY` property; must be the ranked index's only - /// property. + /// The single `GROUP BY` property; must be the covering ranked + /// index's **last** property. pub group_by_property: String, /// The field the aggregate applies to. Empty for /// [`RankedAxis::Count`] (`COUNT(*)`); the index's `summable` /// property for [`RankedAxis::Sum`] / [`RankedAxis::Avg`]. pub aggregate_field: String, + /// The equality `where` pins, `(property, value)` per clause — + /// exactly one per leading property of the covering compound index, + /// in whatever order the request supplied them (the resolver + /// re-orders them into index-property order when it encodes the + /// path). Empty for the single-property form. Shape-validated only: + /// the index-aware checks (does a compound index exist whose leading + /// properties these pin?) live in [`index_picker`]. + pub equality_pins: Vec<(String, Value)>, } diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/mod.rs b/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/mod.rs index b661ad7cfac..17b2d46b5f5 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/mod.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/mod.rs @@ -88,4 +88,4 @@ pub fn ranked_order_key(select: &SelectProjection) -> &str { mod v0; // Re-exported so the dispatcher's callers (`drive_dispatcher`, the // test suites) keep addressing the frozen grammar by its old path. -pub use v0::detect_ranked_mode_v0; +pub use v0::{detect_ranked_mode_v0, equality_pins_from_where_clauses}; diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/v0/mod.rs b/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/v0/mod.rs index 6dfa811229c..8a96f033b47 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/v0/mod.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/mode_detection/v0/mod.rs @@ -12,22 +12,95 @@ use crate::error::query::QuerySyntaxError; use crate::error::Error; use crate::query::having::HavingClause; use crate::query::projection::{SelectFunction, SelectProjection}; -use crate::query::{OrderClause, WhereClause}; +use crate::query::{OrderClause, WhereClause, WhereOperator}; +use dpp::platform_value::Value; + +/// Translate a request's `where` clauses into equality pins — +/// `(property, value)` pairs, one per clause — for the ranked and +/// having-range surfaces. +/// +/// Both surfaces read a compound index's per-prefix secondary by +/// descending through one prefix value tree per **leading** index +/// property, and only an equality clause names a single value tree to +/// descend into. So the grammar is: every `where` clause must be an +/// equality (`==`), each on a distinct property. `IN` is rejected +/// separately from the other operators because it *will* eventually be +/// serviceable (one branch per element, once multi-`IN` branching lands +/// on the document query surface) — the message says so — while a range +/// operator on a prefix property can never pin a single subtree. +/// +/// Shape-only, like everything in this module: whether the pinned +/// properties are exactly the leading properties of a covering compound +/// index is the index picker's call. +pub fn equality_pins_from_where_clauses( + where_clauses: &[WhereClause], +) -> Result, Error> { + let mut pins: Vec<(String, Value)> = Vec::with_capacity(where_clauses.len()); + for clause in where_clauses { + match clause.operator { + WhereOperator::Equal => {} + WhereOperator::In => { + return Err(Error::Query(QuerySyntaxError::Unsupported(format!( + "`{} IN …` is not yet supported on a ranked / having-range query's \ + prefix properties: each `IN` element names a different prefix value \ + tree, so serving it means one secondary walk per element and a merged \ + result — a future capability layered on multi-`IN` branching. Pin \ + each leading index property with `==`, or issue one request per \ + value.", + clause.field + )))); + } + _ => { + return Err(Error::Query( + QuerySyntaxError::InvalidWhereClauseComponents( + "a ranked / having-range query's `where` clauses must pin the covering \ + compound index's leading properties with `==`: the per-prefix \ + secondary lives under one prefix value tree per leading property, and \ + only an equality names a single value tree to descend into — a range \ + operator cannot pin a prefix", + ), + )); + } + } + if clause.field.is_empty() { + return Err(Error::Query( + QuerySyntaxError::InvalidWhereClauseComponents( + "a ranked / having-range query's `where` clause names an empty property", + ), + )); + } + if pins.iter().any(|(field, _)| field == &clause.field) { + return Err(Error::Query( + QuerySyntaxError::InvalidWhereClauseComponents( + "a ranked / having-range query pins the same property twice: each leading \ + index property takes exactly one equality pin", + ), + )); + } + pins.push((clause.field.clone(), clause.value.clone())); + } + Ok(pins) +} /// v0 of the ranked request grammar. /// /// Accepts exactly: /// /// ```text -/// SELECT COUNT(*) GROUP BY p ORDER BY $count [ASC|DESC] LIMIT n [OFFSET m] -/// SELECT SUM(f) GROUP BY p ORDER BY f [ASC|DESC] LIMIT n [OFFSET m] -/// SELECT AVG(f) GROUP BY p ORDER BY f [ASC|DESC] LIMIT n [OFFSET m] +/// SELECT COUNT(*) [WHERE q1 = v1 [AND …]] GROUP BY p ORDER BY $count [ASC|DESC] LIMIT n [OFFSET m] +/// SELECT SUM(f) [WHERE q1 = v1 [AND …]] GROUP BY p ORDER BY f [ASC|DESC] LIMIT n [OFFSET m] +/// SELECT AVG(f) [WHERE q1 = v1 [AND …]] GROUP BY p ORDER BY f [ASC|DESC] LIMIT n [OFFSET m] /// ``` /// -/// with no `WHERE`, no `HAVING`, no `START AT` / `START AFTER`, exactly -/// one `GROUP BY` property, exactly one `ORDER BY` clause naming the +/// with no `HAVING`, no `START AT` / `START AFTER`, exactly one +/// `GROUP BY` property, exactly one `ORDER BY` clause naming the /// selected aggregate, `1 ≤ n ≤` [`MAX_RANKED_LIMIT`], and any -/// `m ≥ 0`. +/// `m ≥ 0`. `WHERE` clauses, when present, must be **equality pins** on +/// distinct properties — one per leading property of a covering +/// compound ranked index (see +/// [`equality_pins_from_where_clauses`]); the ranking then reads that +/// pinned prefix's own secondary. With no `where` the covering index is +/// single-property, exactly as before. /// /// `DESC` walks the axis from the largest aggregate down (the "top n" /// reading), `ASC` from the smallest up (the "bottom n" reading). @@ -63,16 +136,19 @@ pub fn detect_ranked_mode_v0( ) -> Result { // ---- GROUP BY: exactly one property ---------------------------- // - // Ranked indexes are single-property (rs-dpp rejects compound ones - // at parse time), and the sole property is what the secondary's - // group keys are. Zero group_by would ask to rank a single global - // aggregate against itself; two or more would need a compound index. + // The one property is the covering ranked index's LAST property — + // the level whose distinct values are the secondary's group keys. + // Zero group_by would ask to rank a single global aggregate against + // itself. Two or more is rejected because a compound ranked index + // ranks per prefix, not across a compound grouping: its leading + // properties are pinned by equality `where` clauses, and only the + // trailing property is grouped over. if group_by.len() != 1 { return Err(Error::Query(QuerySyntaxError::InvalidParameter(format!( - "ranked queries require exactly one `group_by` property (the ranked index's \ - only property); got {}. Ranked indexes are single-property — compound ranked \ - indexes are rejected at contract-parse time — so there is no compound \ - grouping to rank over.", + "ranked queries require exactly one `group_by` property (the covering ranked \ + index's trailing property); got {}. A compound ranked index ranks each \ + prefix's groups separately — pin every leading index property with an \ + equality `where` clause and `group_by` the trailing property.", group_by.len() )))); } @@ -178,24 +254,16 @@ pub fn detect_ranked_mode_v0( )))); } - // ---- WHERE: must be absent -------------------------------------- + // ---- WHERE: equality pins on the compound prefix ------------------ // - // Rejected rather than ignored. A single-property ranked index has no - // equality prefix a clause could narrow, and a clause on the ranked - // property itself asks for a ranking over a filtered subset — which - // the secondary cannot answer, because it is ordered by aggregate, - // not by group key. Silently dropping the filter would return the - // global ranking under the guise of a filtered one. - if !where_clauses.is_empty() { - return Err(Error::Query( - QuerySyntaxError::InvalidWhereClauseComponents( - "ranked queries do not accept `where` clauses: ranked indexes are \ - single-property, so there is no equality prefix to narrow, and the axis \ - secondary is ordered by aggregate rather than by group key — it cannot \ - rank a filtered subset. Rank the whole index, or add a narrower index.", - ), - )); - } + // Empty for the single-property form. For a compound ranked index, + // each `where` clause must pin one leading index property with `==` + // — that is what selects which prefix's secondary the walk reads + // (per-prefix semantics: there is no global cross-prefix ordering to + // serve). Anything other than a distinct-property equality is + // rejected loudly here; whether the pinned set matches a covering + // index's leading properties exactly is the index picker's call. + let equality_pins = equality_pins_from_where_clauses(where_clauses)?; // ---- LIMIT: required, 1 ..= MAX_RANKED_LIMIT --------------------- // @@ -260,5 +328,6 @@ pub fn detect_ranked_mode_v0( offset, group_by_property, aggregate_field, + equality_pins, }) } diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/path.rs b/packages/rs-drive/src/query/drive_document_ranked_query/path.rs index a66c13b08c6..ed3dccf0620 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/path.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/path.rs @@ -19,32 +19,51 @@ use crate::error::drive::DriveError; use crate::error::Error; use dpp::data_contract::document_type::Index; -/// Path of a single-property index's terminal property-name tree — -/// shared by the ranked and having-range query surfaces, which read -/// the same indexed tree. See -/// [`DriveDocumentRankedQuery::indexed_property_name_tree_path`] for -/// the segment layout and the single-property requirement. +/// Path of an index's terminal property-name tree — shared by the +/// ranked and having-range query surfaces, which read the same indexed +/// tree. See [`DriveDocumentRankedQuery::indexed_property_name_tree_path`] +/// for the segment layout. +/// +/// `equality_prefix_values` carries the **encoded index-key bytes** of +/// each leading property's pinned value, in index-property order — one +/// per property before the terminal one. Empty for a single-property +/// index. The arity must match exactly: a compound index's terminal +/// tree sits under one prefix value tree per leading property, and only +/// an equality `where` clause can name those values, so a missing or +/// surplus value means the caller resolved the wrong index — a typed +/// error, not a guess. pub(crate) fn indexed_property_name_tree_path_for_index( contract_id: &[u8; 32], document_type_name: &str, index: &Index, + equality_prefix_values: &[Vec], ) -> Result>, Error> { - let [property] = index.properties.as_slice() else { + let Some((terminal_property, leading_properties)) = index.properties.split_last() else { return Err(Error::Drive(DriveError::NotSupported( - "ranked and having-range queries require a single-property index: the \ - axis secondary lives on the index's terminal property-name tree, and \ - for a compound index that tree sits under a prefix value tree whose \ - value only a `where` clause could name — but these queries accept no \ - `where` clauses", + "ranked and having-range queries require an index with at least one \ + property", ))); }; - Ok(vec![ - vec![RootTree::DataContractDocuments as u8], - contract_id.to_vec(), - vec![1u8], - document_type_name.as_bytes().to_vec(), - property.name.as_bytes().to_vec(), - ]) + if leading_properties.len() != equality_prefix_values.len() { + return Err(Error::Drive(DriveError::NotSupported( + "ranked and having-range queries over a compound index require exactly one \ + encoded equality value per leading index property: the axis secondary lives \ + on the index's terminal property-name tree, which for a compound index sits \ + under one prefix value tree per leading property, and only an equality \ + `where` clause can name those values", + ))); + } + let mut path = Vec::with_capacity(5 + 2 * leading_properties.len()); + path.push(vec![RootTree::DataContractDocuments as u8]); + path.push(contract_id.to_vec()); + path.push(vec![1u8]); + path.push(document_type_name.as_bytes().to_vec()); + for (property, value) in leading_properties.iter().zip(equality_prefix_values) { + path.push(property.name.as_bytes().to_vec()); + path.push(value.clone()); + } + path.push(terminal_property.name.as_bytes().to_vec()); + Ok(path) } impl DriveDocumentRankedQuery<'_> { @@ -52,6 +71,8 @@ impl DriveDocumentRankedQuery<'_> { /// whose primary holds one value tree per group and whose per-axis /// secondaries hold the ranking. /// + /// For a single-property index: + /// /// ```text /// [ RootTree::DataContractDocuments as u8 ] // 0x01 /// / @@ -60,23 +81,32 @@ impl DriveDocumentRankedQuery<'_> { /// / /// ``` /// - /// The children of that tree are the groups, keyed by the raw - /// index-key bytes of the property value — the same bytes that come - /// back as [`super::RankedEntry::key`]. + /// For a compound index `[p1, …, pn]`, each leading property + /// contributes two segments — its name and the **encoded index-key + /// bytes of its pinned value** (from + /// [`Self::equality_prefix_values`]) — and the terminal property + /// name closes the path: + /// + /// ```text + /// … / / / … / + /// ``` + /// + /// so the ranking read lands on **that prefix's** indexed tree: the + /// per-prefix secondary orders only the pinned prefix's groups. + /// + /// The children of the terminal tree are the groups, keyed by the + /// raw index-key bytes of the terminal property value — the same + /// bytes that come back as [`super::RankedEntry::key`]. /// - /// Errors when the index is not single-property. A compound ranked - /// index would terminate one level *below* a prefix value tree, so - /// its path would need the prefix property's value — which only a - /// `where` clause could supply, and ranked queries take none. rs-dpp - /// rejects compound ranked indexes at contract-parse time, so this is - /// a fail-closed backstop for the day that grammar relaxes: better a - /// typed error than a path pointing at a prefix level whose element - /// is not an indexed tree at all. + /// Errors when the number of encoded prefix values does not match + /// the index's leading-property count — the fail-closed backstop + /// for a caller that resolved the query against the wrong index. pub fn indexed_property_name_tree_path(&self) -> Result>, Error> { indexed_property_name_tree_path_for_index( &self.contract_id, &self.document_type_name, self.index, + &self.equality_prefix_values, ) } } diff --git a/packages/rs-drive/src/query/drive_document_ranked_query/tests.rs b/packages/rs-drive/src/query/drive_document_ranked_query/tests.rs index 07bc14ccb23..4a477c98195 100644 --- a/packages/rs-drive/src/query/drive_document_ranked_query/tests.rs +++ b/packages/rs-drive/src/query/drive_document_ranked_query/tests.rs @@ -31,7 +31,7 @@ //! `adjustment` exists for the write-path suite's signed-average coverage //! (`delta` admits negatives, `grade` does not) and is unused here. -use super::index_picker::find_ranked_index_for_axis; +use super::index_picker::{find_ranked_index_for_axis, resolve_ranked_query_for_mode}; use super::mode_detection::{detect_ranked_mode, detect_ranked_mode_v0}; use super::*; use crate::drive::Drive; @@ -490,26 +490,104 @@ fn sum_and_avg_selects_require_a_field() { } } -/// A `where` clause is refused, not ignored. The axis secondary is -/// ordered by aggregate, not by group key, so it cannot rank a filtered -/// subset — silently dropping the filter would answer the unfiltered -/// question under the guise of the filtered one. +/// `where` clauses are equality pins on a compound index's leading +/// properties. Detection is shape-only: an equality clause becomes a +/// pin (whether a compound index actually covers it is the resolver's +/// call — see `pins_without_a_covering_compound_index_are_rejected`); +/// anything that is not a distinct-property equality is refused +/// loudly, `IN` with its own not-yet message because it will become +/// serviceable once multi-`IN` branching lands. #[test] -fn where_clauses_are_rejected() { - let where_clauses = vec![WhereClause { - field: GROUP_PROPERTY.to_string(), +fn where_clauses_resolve_to_equality_pins_and_reject_everything_else() { + // Equality pin: accepted at detection, carried in the mode. + let pinned = vec![WhereClause { + field: "chefId".to_string(), operator: WhereOperator::Equal, value: Value::Text("alpha".to_string()), }]; + let mode = detect_ranked_mode_v0( + &SelectProjection::avg("grade"), + &group_by(), + &[], + &order_by("grade", false), + &pinned, + page(Some(2), None), + ) + .expect("an equality pin is a well-formed prefix pin"); + assert_eq!( + mode.equality_pins, + vec![("chefId".to_string(), Value::Text("alpha".to_string()))] + ); + + // A range operator can never pin a single prefix value tree. + let ranged = vec![WhereClause { + field: "chefId".to_string(), + operator: WhereOperator::GreaterThan, + value: Value::Text("alpha".to_string()), + }]; let error = detect_ranked_mode_v0( &SelectProjection::avg("grade"), &group_by(), &[], &order_by("grade", false), - &where_clauses, + &ranged, page(Some(2), None), ) - .expect_err("ranked queries take no where clauses"); + .expect_err("a range operator cannot pin a prefix"); + assert!(matches!( + error, + Error::Query(QuerySyntaxError::InvalidWhereClauseComponents(_)) + )); + + // `IN` is rejected with its own message: v1 pins are equality-only, + // multi-`IN` branching is a future capability. + let in_clause = vec![WhereClause { + field: "chefId".to_string(), + operator: WhereOperator::In, + value: Value::Array(vec![Value::Text("alpha".to_string())]), + }]; + let error = detect_ranked_mode_v0( + &SelectProjection::avg("grade"), + &group_by(), + &[], + &order_by("grade", false), + &in_clause, + page(Some(2), None), + ) + .expect_err("IN prefixes are not yet supported"); + match error { + Error::Query(QuerySyntaxError::Unsupported(message)) => { + assert!( + message.contains("IN") && message.contains("not yet supported"), + "the IN rejection must say it is a not-yet capability, got: {message}" + ); + } + other => panic!("expected Unsupported for an IN prefix, got {other:?}"), + } + + // The same property pinned twice is a caller error, not a silent + // last-write-wins. + let duplicated = vec![ + WhereClause { + field: "chefId".to_string(), + operator: WhereOperator::Equal, + value: Value::Text("alpha".to_string()), + }, + WhereClause { + field: "chefId".to_string(), + operator: WhereOperator::Equal, + value: Value::Text("beta".to_string()), + }, + ]; + let error = detect_ranked_mode_v0( + &SelectProjection::avg("grade"), + &group_by(), + &[], + &order_by("grade", false), + &duplicated, + page(Some(2), None), + ) + .expect_err("duplicate pins on one property must fail"); assert!(matches!( error, Error::Query(QuerySyntaxError::InvalidWhereClauseComponents(_)) @@ -633,12 +711,12 @@ fn picker_requires_the_index_to_declare_the_requested_axis() { let indexes = index_map(vec![index]); assert!( - find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, RankedAxis::Count, "").is_some(), + find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, &[], RankedAxis::Count, "").is_some(), "the declared axis resolves" ); for (axis, field) in [(RankedAxis::Sum, "grade"), (RankedAxis::Avg, "grade")] { assert!( - find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, axis, field).is_none(), + find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, &[], axis, field).is_none(), "{axis:?} is not declared even though the index is summable and the stored \ element could host that secondary" ); @@ -657,11 +735,11 @@ fn picker_requires_the_select_field_to_be_the_indexed_summable() { for axis in [RankedAxis::Sum, RankedAxis::Avg] { assert!( - find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, axis, "grade").is_some(), + find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, &[], axis, "grade").is_some(), "{axis:?} on the indexed summable resolves" ); assert!( - find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, axis, "tipAmount").is_none(), + find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, &[], axis, "tipAmount").is_none(), "{axis:?} on a different field must not resolve" ); } @@ -676,7 +754,7 @@ fn picker_rejects_an_unknown_group_property() { let indexes = index_map(vec![index]); assert!( - find_ranked_index_for_axis(&indexes, "chefId", RankedAxis::Avg, "grade").is_none(), + find_ranked_index_for_axis(&indexes, "chefId", &[], RankedAxis::Avg, "grade").is_none(), "no index groups by `chefId`" ); } @@ -697,7 +775,8 @@ fn picker_rejects_compound_indexes() { let indexes = index_map(vec![index]); assert!( - find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, RankedAxis::Avg, "price").is_none() + find_ranked_index_for_axis(&indexes, GROUP_PROPERTY, &[], RankedAxis::Avg, "price") + .is_none() ); } @@ -943,25 +1022,17 @@ fn client_side_query<'a>( .get(case.document_type_name) .expect("doctype exists") .indexes(); - let index = find_ranked_index_for_axis( - indexes, - &mode.group_by_property, - mode.axis, - &mode.aggregate_field, - ) - .expect("the fixture declares the axis"); - DriveDocumentRankedQuery { - document_type: contract + resolve_ranked_query_for_mode( + contract.id_ref().to_buffer(), + contract .document_type_for_name(case.document_type_name) .expect("doctype exists"), - contract_id: contract.id_ref().to_buffer(), - document_type_name: case.document_type_name.to_string(), - index, - axis: mode.axis, - descending: mode.descending, - k: mode.k, - offset: mode.offset, - } + case.document_type_name.to_string(), + indexes, + &mode, + platform_version(), + ) + .expect("the fixture declares the axis") } fn grovedb_root_hash(drive: &Drive) -> [u8; 32] { @@ -1958,13 +2029,13 @@ fn a_doctype_with_several_indexes_ranks_each_group_property_on_its_own_index() { ); assert_eq!(client_side_query(&contract, &by_chef).index.name, "byChef"); assert_eq!( - find_ranked_index_for_axis(indexes, GROUP_PROPERTY, RankedAxis::Avg, "grade") + find_ranked_index_for_axis(indexes, GROUP_PROPERTY, &[], RankedAxis::Avg, "grade") .expect("the Avg ranking resolves") .name, "byRestaurant", ); assert_eq!( - find_ranked_index_for_axis(indexes, CHEF_PROPERTY, RankedAxis::Count, "") + find_ranked_index_for_axis(indexes, CHEF_PROPERTY, &[], RankedAxis::Count, "") .expect("the Count ranking resolves") .name, "byChef", @@ -2015,7 +2086,7 @@ fn a_doctype_with_several_indexes_ranks_each_group_property_on_its_own_index() { // ranking over chefs has no index — and `byChefRestaurant` must not // be press-ganged into serving it. assert!( - find_ranked_index_for_axis(indexes, CHEF_PROPERTY, RankedAxis::Avg, "grade").is_none(), + find_ranked_index_for_axis(indexes, CHEF_PROPERTY, &[], RankedAxis::Avg, "grade").is_none(), "`byChefRestaurant` leads with chefId but is compound and unranked" ); let avg_by_chef = RankedCase { @@ -2033,3 +2104,459 @@ fn a_doctype_with_several_indexes_ranks_each_group_property_on_its_own_index() { "the error must name the missing keyword, got: {error}" ); } + +mod pinned_prefix { + //! `SELECT AVG(grade) FROM grades WHERE identityId = X GROUP BY + //! class ORDER BY AVG(grade) DESC LIMIT k` — the ranked top-k + //! surface over a compound ranked index `[identityId, class]` with + //! its leading property pinned. Per-prefix semantics: the walk + //! reads (and proves) the pinned identity's own secondary, so two + //! identities' class rankings never mix. The having-path sibling + //! (`drive_document_having_query::tests::pinned_prefix`) shares the + //! fixture and pins the write path; this module pins the rank walk + //! and its paginated proof. + + use super::super::drive_dispatcher::{DocumentRankedRequest, DocumentRankedResponse}; + use super::super::index_picker::resolve_ranked_query_for_mode; + use super::super::mode_detection::detect_ranked_mode; + use super::super::{DriveDocumentRankedQuery, RankedEntry, RankedEntryValue}; + use crate::drive::Drive; + use crate::error::query::QuerySyntaxError; + use crate::error::Error; + use crate::query::drive_document_ranked_query::RankedPaginationInputs; + use crate::query::projection::SelectProjection; + use crate::query::{OrderClause, WhereClause, WhereOperator}; + use crate::util::object_size_info::DocumentInfo::DocumentRefInfo; + use crate::util::object_size_info::{DocumentAndContractInfo, OwnedDocumentInfo}; + use crate::util::storage_flags::StorageFlags; + use crate::util::test_helpers::setup::setup_drive_with_initial_state_structure; + use dpp::block::block_info::BlockInfo; + use dpp::data_contract::accessors::v0::DataContractV0Getters; + use dpp::data_contract::document_type::accessors::DocumentTypeV0Getters; + use dpp::data_contract::document_type::random_document::CreateRandomDocument; + use dpp::document::{Document, DocumentV0Setters}; + use dpp::platform_value::Value; + use dpp::prelude::DataContract; + use dpp::tests::json_document::json_document_to_contract; + use dpp::version::PlatformVersion; + use grovedb::element::indexed::compute_avg_fixed_point; + use std::collections::BTreeMap; + + const PREFIX_PROPERTY: &str = "identityId"; + const CLASS_PROPERTY: &str = "class"; + const DOCUMENT_TYPE: &str = "grade"; + const IDENTITY_X: [u8; 32] = [1u8; 32]; + const IDENTITY_Y: [u8; 32] = [2u8; 32]; + + fn platform_version() -> &'static PlatformVersion { + PlatformVersion::latest() + } + + fn setup_grades_compound_ranked() -> (Drive, DataContract) { + let drive = setup_drive_with_initial_state_structure(None); + let pv = platform_version(); + let contract = json_document_to_contract( + "tests/supporting_files/contract/grades/grades-compound-ranked-contract.json", + false, + pv, + ) + .expect("expected to parse the compound ranked grades contract"); + drive + .apply_contract( + &contract, + BlockInfo::default(), + true, + StorageFlags::optional_default_as_cow(), + None, + pv, + ) + .expect("expected to apply the compound ranked grades contract"); + (drive, contract) + } + + fn insert_grades(drive: &Drive, contract: &DataContract, rows: &[([u8; 32], &str, i64)]) { + let pv = platform_version(); + let document_type = contract + .document_type_for_name(DOCUMENT_TYPE) + .expect("grade doctype exists"); + for (i, (identity, class, grade)) in rows.iter().enumerate() { + let mut doc: Document = document_type + .random_document(Some(4000 + i as u64), pv) + .expect("random document"); + let mut props = BTreeMap::new(); + props.insert(PREFIX_PROPERTY.to_string(), Value::Identifier(*identity)); + props.insert(CLASS_PROPERTY.to_string(), Value::Text(class.to_string())); + props.insert("grade".to_string(), Value::I64(*grade)); + doc.set_properties(props); + drive + .add_document_for_contract( + DocumentAndContractInfo { + owned_document_info: OwnedDocumentInfo { + document_info: DocumentRefInfo((&doc, None)), + owner_id: None, + }, + contract, + document_type, + }, + false, + BlockInfo::default(), + true, + None, + pv, + None, + ) + .expect("expected to insert a grade document"); + } + } + + fn pin(identity: [u8; 32]) -> Vec { + vec![WhereClause { + field: PREFIX_PROPERTY.to_string(), + operator: WhereOperator::Equal, + value: Value::Identifier(identity), + }] + } + + fn run( + drive: &Drive, + contract: &DataContract, + where_clauses: &[WhereClause], + limit: u32, + prove: bool, + ) -> Result { + let group_by = vec![CLASS_PROPERTY.to_string()]; + let order_by = vec![OrderClause { + field: "grade".to_string(), + ascending: false, + }]; + drive.execute_document_ranked_request( + DocumentRankedRequest { + contract, + document_type: contract + .document_type_for_name(DOCUMENT_TYPE) + .expect("grade doctype exists"), + group_by: &group_by, + select: SelectProjection::avg("grade"), + having: &[], + order_by: &order_by, + where_clauses, + limit: Some(limit), + offset: None, + has_start_at: false, + prove, + }, + None, + platform_version(), + ) + } + + fn client_side_query<'a>( + contract: &'a DataContract, + where_clauses: &[WhereClause], + limit: u32, + ) -> DriveDocumentRankedQuery<'a> { + let group_by = vec![CLASS_PROPERTY.to_string()]; + let order_by = vec![OrderClause { + field: "grade".to_string(), + ascending: false, + }]; + let mode = detect_ranked_mode( + &SelectProjection::avg("grade"), + &group_by, + &[], + &order_by, + where_clauses, + RankedPaginationInputs { + limit: Some(limit), + offset: None, + has_start_at: false, + }, + platform_version(), + ) + .expect("the case is well-formed"); + let indexes = contract + .document_types() + .get(DOCUMENT_TYPE) + .expect("grade doctype exists") + .indexes(); + resolve_ranked_query_for_mode( + contract.id_ref().to_buffer(), + contract + .document_type_for_name(DOCUMENT_TYPE) + .expect("grade doctype exists"), + DOCUMENT_TYPE.to_string(), + indexes, + &mode, + platform_version(), + ) + .expect("the fixture's compound index covers the pinned request") + } + + /// Top-k per pinned prefix, with the paginated proof round-tripped + /// against the live root hash — and isolation between prefixes: + /// X's and Y's class rankings come off different secondaries. + #[test] + fn top_k_over_pinned_prefix_reads_and_proves() { + let (drive, contract) = setup_grades_compound_ranked(); + insert_grades( + &drive, + &contract, + &[ + // X: art 90, english 80.5, math 80, history 70. + (IDENTITY_X, "math", 80), + (IDENTITY_X, "math", 80), + (IDENTITY_X, "english", 80), + (IDENTITY_X, "english", 81), + (IDENTITY_X, "art", 85), + (IDENTITY_X, "art", 95), + (IDENTITY_X, "history", 60), + (IDENTITY_X, "history", 80), + // Y: science 95, math 92.5. + (IDENTITY_Y, "math", 90), + (IDENTITY_Y, "math", 95), + (IDENTITY_Y, "science", 95), + (IDENTITY_Y, "science", 95), + ], + ); + + // X's top 2 by average, best first: art (90) then english (80.5). + // Y's science (95) and math (92.5) would both outrank them if the + // prefixes shared a secondary. + let x_pin = pin(IDENTITY_X); + let page = match run(&drive, &contract, &x_pin, 2, false).expect("read succeeds") { + DocumentRankedResponse::Entries(page) => page, + DocumentRankedResponse::Proof(_) => panic!("expected entries, got a proof"), + }; + assert_eq!(page.skipped, 0); + assert_eq!( + page.entries, + vec![ + RankedEntry { + key: b"art".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(180, 2)), + }, + RankedEntry { + key: b"english".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(161, 2)), + }, + ], + "X's own top 2: art 90 then english 80.5" + ); + + let proof = match run(&drive, &contract, &x_pin, 2, true).expect("prove succeeds") { + DocumentRankedResponse::Proof(proof) => proof, + DocumentRankedResponse::Entries(_) => panic!("expected a proof, got entries"), + }; + let (root_hash, verified) = client_side_query(&contract, &x_pin, 2) + .verify_ranked_top_k_proof(&proof, platform_version()) + .expect("the proof must verify"); + assert_eq!( + verified.entries, page.entries, + "verified entries must equal the unproven read" + ); + assert_eq!( + root_hash, + drive + .grove + .root_hash(None, &platform_version().drive.grove_version) + .unwrap() + .expect("root hash must be readable"), + "the proof must reconstruct the live grovedb root hash" + ); + + // Y's own ranking, proving the prefixes are separate secondaries. + let y_pin = pin(IDENTITY_Y); + let y_page = match run(&drive, &contract, &y_pin, 2, false).expect("read succeeds") { + DocumentRankedResponse::Entries(page) => page, + DocumentRankedResponse::Proof(_) => panic!("expected entries, got a proof"), + }; + assert_eq!( + y_page + .entries + .iter() + .map(|e| e.key.as_slice()) + .collect::>(), + vec![b"science".as_slice(), b"math".as_slice()], + "Y's own top 2: science 95 then math 92.5" + ); + } + + /// A **null** pin addresses the prefix subtree the write path + /// creates for an *absent* optional leading property: the walkers + /// encode a missing value as an empty path segment + /// (`get_raw_for_document_type(..).unwrap_or_default()`), so + /// `WHERE tag == null` must resolve to that empty segment — read, + /// proof, and client-side verification all reconstructing the same + /// stored path. Documents that *do* carry a tag live under their + /// own prefix and must not leak into the null prefix's ranking. + #[test] + fn a_null_pin_addresses_the_absent_value_prefix() { + const TAGGED_DOCTYPE: &str = "taggedGrade"; + let (drive, contract) = setup_grades_compound_ranked(); + let pv = platform_version(); + let document_type = contract + .document_type_for_name(TAGGED_DOCTYPE) + .expect("taggedGrade doctype exists"); + + // Tagless rows land under the empty-segment prefix; the + // "honors" row must stay in its own prefix. + for (i, (tag, class, grade)) in [ + (None, "math", 80i64), + (None, "math", 90), + (None, "science", 60), + (Some("honors"), "math", 100), + ] + .iter() + .enumerate() + { + let mut doc: Document = document_type + .random_document(Some(7000 + i as u64), pv) + .expect("random document"); + let mut props = BTreeMap::new(); + if let Some(tag) = tag { + props.insert("tag".to_string(), Value::Text(tag.to_string())); + } + props.insert(CLASS_PROPERTY.to_string(), Value::Text(class.to_string())); + props.insert("grade".to_string(), Value::I64(*grade)); + doc.set_properties(props); + drive + .add_document_for_contract( + DocumentAndContractInfo { + owned_document_info: OwnedDocumentInfo { + document_info: DocumentRefInfo((&doc, None)), + owner_id: None, + }, + contract: &contract, + document_type, + }, + false, + BlockInfo::default(), + true, + None, + pv, + None, + ) + .expect("expected to insert a tagged grade document"); + } + + let null_pin = vec![WhereClause { + field: "tag".to_string(), + operator: WhereOperator::Equal, + value: Value::Null, + }]; + let group_by = vec![CLASS_PROPERTY.to_string()]; + let order_by = vec![OrderClause { + field: "grade".to_string(), + ascending: false, + }]; + let request = |prove: bool| DocumentRankedRequest { + contract: &contract, + document_type, + group_by: &group_by, + select: SelectProjection::avg("grade"), + having: &[], + order_by: &order_by, + where_clauses: &null_pin, + limit: Some(2), + offset: None, + has_start_at: false, + prove, + }; + + let page = match drive + .execute_document_ranked_request(request(false), None, pv) + .expect("the null-pinned read succeeds") + { + DocumentRankedResponse::Entries(page) => page, + DocumentRankedResponse::Proof(_) => panic!("expected entries, got a proof"), + }; + assert_eq!( + page.entries, + vec![ + RankedEntry { + key: b"math".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(170, 2)), + }, + RankedEntry { + key: b"science".to_vec(), + value: RankedEntryValue::AvgFixedPoint(compute_avg_fixed_point(60, 1)), + }, + ], + "the tagless ranking: math 85 then science 60 — the honors row \ + (math 100) must not leak into the null prefix" + ); + + // Proof round trip through the shared resolver, so the verifier + // reconstructs the same empty-segment path the prover walked. + let proof = match drive + .execute_document_ranked_request(request(true), None, pv) + .expect("the null-pinned prove succeeds") + { + DocumentRankedResponse::Proof(proof) => proof, + DocumentRankedResponse::Entries(_) => panic!("expected a proof, got entries"), + }; + let mode = detect_ranked_mode( + &SelectProjection::avg("grade"), + &group_by, + &[], + &order_by, + &null_pin, + RankedPaginationInputs { + limit: Some(2), + offset: None, + has_start_at: false, + }, + pv, + ) + .expect("the null-pinned case is well-formed"); + let query = resolve_ranked_query_for_mode( + contract.id_ref().to_buffer(), + document_type, + TAGGED_DOCTYPE.to_string(), + contract + .document_types() + .get(TAGGED_DOCTYPE) + .expect("taggedGrade doctype exists") + .indexes(), + &mode, + pv, + ) + .expect("the compound index covers the null-pinned request"); + assert_eq!( + query.equality_prefix_values, + vec![Vec::::new()], + "a null pin must encode as the write path's empty segment" + ); + let (root_hash, verified) = query + .verify_ranked_top_k_proof(&proof, pv) + .expect("the null-pinned proof must verify"); + assert_eq!(verified.entries, page.entries); + assert_eq!( + root_hash, + drive + .grove + .root_hash(None, &pv.drive.grove_version) + .unwrap() + .expect("root hash must be readable"), + ); + } + + /// An unpinned request over the compound-only contract has no + /// covering index — there is no global cross-prefix ordering to + /// serve, so the rejection names the missing coverage. + #[test] + fn unpinned_prefix_is_rejected() { + let (drive, contract) = setup_grades_compound_ranked(); + insert_grades(&drive, &contract, &[(IDENTITY_X, "math", 80)]); + + let error = run(&drive, &contract, &[], 2, false) + .expect_err("an unpinned compound prefix must not resolve"); + assert!( + matches!( + error, + Error::Query(QuerySyntaxError::WhereClauseOnNonIndexedProperty(_)) + ), + "expected a no-covering-index rejection, got {error:?}" + ); + } +} diff --git a/packages/rs-drive/tests/supporting_files/contract/grades/grades-compound-ranked-contract.json b/packages/rs-drive/tests/supporting_files/contract/grades/grades-compound-ranked-contract.json new file mode 100644 index 00000000000..df4179096ce --- /dev/null +++ b/packages/rs-drive/tests/supporting_files/contract/grades/grades-compound-ranked-contract.json @@ -0,0 +1,100 @@ +{ + "$formatVersion": "0", + "id": "AgradesC5w7Y9R4nDqJk2vHpL3uM6tF1xE8cA2bN7zXq", + "ownerId": "7m6mTfWqkrCnvLLPK3eqxQM2x2RDpYV6dsAyhVKsAEAQ", + "version": 1, + "documentSchemas": { + "grade": { + "type": "object", + "documentsMutable": false, + "canBeDeleted": false, + "indices": [ + { + "name": "byIdentityAndClass", + "properties": [ + { + "identityId": "asc" + }, + { + "class": "asc" + } + ], + "averageable": "grade", + "rangeAverageable": true, + "rankedAverageable": true + } + ], + "properties": { + "identityId": { + "type": "array", + "byteArray": true, + "minItems": 32, + "maxItems": 32, + "position": 0, + "contentMediaType": "application/x.dash.dpp.identifier" + }, + "class": { + "type": "string", + "maxLength": 32, + "position": 1 + }, + "grade": { + "type": "integer", + "minimum": 0, + "maximum": 100, + "position": 2 + } + }, + "required": [ + "identityId", + "class", + "grade" + ], + "additionalProperties": false + }, + "taggedGrade": { + "type": "object", + "documentsMutable": false, + "canBeDeleted": false, + "indices": [ + { + "name": "byTagAndClass", + "properties": [ + { + "tag": "asc" + }, + { + "class": "asc" + } + ], + "averageable": "grade", + "rangeAverageable": true, + "rankedAverageable": true + } + ], + "properties": { + "tag": { + "type": "string", + "maxLength": 32, + "position": 0 + }, + "class": { + "type": "string", + "maxLength": 32, + "position": 1 + }, + "grade": { + "type": "integer", + "minimum": 0, + "maximum": 100, + "position": 2 + } + }, + "required": [ + "class", + "grade" + ], + "additionalProperties": false + } + } +} \ No newline at end of file diff --git a/packages/rs-platform-version/src/version/drive_abci_versions/drive_abci_query_versions/v3.rs b/packages/rs-platform-version/src/version/drive_abci_versions/drive_abci_query_versions/v3.rs index 74c2e14564a..6b0098a97c4 100644 --- a/packages/rs-platform-version/src/version/drive_abci_versions/drive_abci_query_versions/v3.rs +++ b/packages/rs-platform-version/src/version/drive_abci_versions/drive_abci_query_versions/v3.rs @@ -16,9 +16,11 @@ use crate::version::drive_abci_versions::drive_abci_query_versions::{ /// secondary. Everything else — including multi-clause `having` and /// `having` on a select with no ranked axis — keeps the v1 behavior. /// -/// Same mixed-network rationale as the v1 → v2 flip: earlier protocol -/// versions keep the v2 table and keep rejecting the shape, so nodes -/// agree until the upgrade carries. The wire surface is unchanged — +/// Mixed-network safety comes from the shipped tables: protocol +/// versions 1–11 select `DRIVE_ABCI_QUERY_VERSIONS_V0`, and versions +/// 12–13 select `DRIVE_ABCI_QUERY_VERSIONS_V1`. Both use helper +/// version 0 and reject ranked and `HAVING` shapes, so nodes agree +/// until the PV14 upgrade carries. The wire surface is unchanged — /// `GetDocumentsRequestV1.having` has been wire-stable since the v1 /// document query, and the response reuses the additive /// `ResultData.ranked` entries shape (with `skipped` unset, since a diff --git a/packages/rs-sdk/src/platform/documents/document_having_entries.rs b/packages/rs-sdk/src/platform/documents/document_having_entries.rs index d8349ca5ef5..6b6522e00b0 100644 --- a/packages/rs-sdk/src/platform/documents/document_having_entries.rs +++ b/packages/rs-sdk/src/platform/documents/document_having_entries.rs @@ -22,17 +22,20 @@ //! a contiguous-range operator (`=`, `>`, `>=`, `<`, `<=`, `BETWEEN*` — //! `!=` and `IN` are rejected), and a `LIMIT`. `ORDER BY` is optional: //! omitted means ascending by the aggregate; naming the selected -//! aggregate sets the direction. No `where`, no `offset`, no -//! `start_at`. +//! aggregate sets the direction. `where` clauses are equality pins on a +//! covering compound ranked index's leading properties (one per leading +//! property, selecting which prefix's groups the bound reads) — absent +//! for a single-property index. No `offset`, no `start_at`. //! //! ## Contract prerequisites //! //! Same as the ranked surface: the index must opt in with //! `rankedCountable` / `rankedSummable` / `rankedAverageable` -//! (meta-schema v3, **protocol version 14+**), and ranked indexes are -//! single-property. Against a pre-v14 node the request is refused with -//! "HAVING clause is not yet implemented" — the intended activation -//! gate. +//! (meta-schema v3, **protocol version 14+**). The index may be +//! single-property (`group_by` its property, no `where`) or compound +//! (`group_by` its trailing property, equality-pin every leading one). +//! Against a pre-v14 node the request is refused with "HAVING clause +//! is not yet implemented" — the intended activation gate. //! //! ## Reading the result //! @@ -307,9 +310,10 @@ mod tests { } } - /// The generic FromProof guard in drive-proof-verifier must not be - /// reachable from the SDK path: this impl (on `DocumentQuery`) is - /// the one `fetch` resolves, and it runs the real verification. + /// HAVING limits are a hard inclusive range, `1..=100`: `0` (the + /// unset sentinel) and anything above `MAX_HAVING_LIMIT` are + /// rejected client side rather than clamped, because the limit is + /// echoed in the proof envelope and re-checked by the verifier. #[test] fn limit_is_required_and_capped_client_side() { for limit in [0u32, 101] { diff --git a/packages/rs-sdk/src/platform/documents/document_ranked_entries.rs b/packages/rs-sdk/src/platform/documents/document_ranked_entries.rs index b0cd0d41f95..f5a80ee3a59 100644 --- a/packages/rs-sdk/src/platform/documents/document_ranked_entries.rs +++ b/packages/rs-sdk/src/platform/documents/document_ranked_entries.rs @@ -18,7 +18,10 @@ //! //! Exactly one aggregate `select`, exactly one `group_by` property, //! exactly one `ORDER BY` clause naming that select's aggregate, and a -//! `LIMIT` — plus an optional `OFFSET`. No `where`, no `having`, no +//! `LIMIT` — plus an optional `OFFSET`. `where` clauses are equality +//! pins on a covering compound ranked index's leading properties (one +//! per leading property, selecting which prefix's own ranking the walk +//! reads) — absent for a single-property index. No `having`, no //! `start_at`: each of those is rejected rather than ignored, on both //! sides, because a ranked walk cannot honour them and silently //! answering a different question is worse than an error. @@ -32,8 +35,10 @@ //! ## Contract prerequisites //! //! The index must opt in with `rankedCountable` / `rankedSummable` / -//! `rankedAverageable` (meta-schema v3, **protocol version 14+**), and -//! ranked indexes are single-property. Against a protocol-version-13 +//! `rankedAverageable` (meta-schema v3, **protocol version 14+**). The +//! index may be single-property (`group_by` its property, no `where`) +//! or compound (`group_by` its trailing property, equality-pin every +//! leading one). Against a protocol-version-13 //! node the request is refused — v13's query table has no ranked path //! and rejects the ordering as `Unsupported`. That is the intended //! activation gate, not a bug: a v13 node and a v14 node must disagree diff --git a/packages/rs-sdk/src/platform/documents/having_proof_helpers.rs b/packages/rs-sdk/src/platform/documents/having_proof_helpers.rs index 2958f7ab0e7..d989c2ab254 100644 --- a/packages/rs-sdk/src/platform/documents/having_proof_helpers.rs +++ b/packages/rs-sdk/src/platform/documents/having_proof_helpers.rs @@ -4,9 +4,10 @@ //! caller-built [`DocumentQuery`] plus the node's response into a //! verified entry list. The routing decisions — which axis, which //! inclusive bounds the operator translates to, which direction, which -//! index covers them — are **not** re-derived here. They come from -//! rs-drive's own [`detect_having_mode`] and -//! [`find_ranked_index_for_axis`], the same two functions the server +//! index covers them, which prefix-value segments a pinned compound +//! request descends through — are **not** re-derived here. They come +//! from rs-drive's own [`detect_having_mode`] and +//! [`resolve_having_query_for_mode`], the same two functions the server //! calls, so client and server land on the same grove path and the same //! bounds by construction rather than by two copies of a grammar //! agreeing. The bounds matter doubly here: the verifier rebuilds the @@ -25,10 +26,8 @@ use dpp::{ data_contract::document_type::accessors::DocumentTypeV0Getters, }; use drive::query::drive_document_having_query::mode_detection::detect_having_mode; -use drive::query::drive_document_ranked_query::index_picker::find_ranked_index_for_axis; -use drive::query::{ - DocumentHavingMode, DriveDocumentHavingQuery, RankedEntry, RankedPaginationInputs, -}; +use drive::query::drive_document_having_query::resolve_having_query_for_mode; +use drive::query::{DocumentHavingMode, RankedEntry, RankedPaginationInputs}; use drive_proof_verifier::verify_having_range_proof; /// Validate that the caller-built [`DocumentQuery`] really describes a @@ -71,8 +70,9 @@ pub(super) fn assert_having_shape( query is `.with_select()`, `.with_group_by()`, \ `.with_having()` and `.with_limit(n)`, optionally \ - `.order_by_selected_aggregate()`, with no where clauses, no offset \ - and no start_at." + `.order_by_selected_aggregate()`, with no offset and no start_at; \ + where clauses, when present, must be equality pins on the covering compound \ + index's leading properties." ), }) } @@ -110,41 +110,29 @@ pub(super) fn verify_having_query( .or(Err(drive_proof_verifier::Error::EmptyResponseMetadata))?; let mode = assert_having_shape(&request, platform_version)?; - let axis = mode.bounds.axis(); - // Pick the index the prover picked — rs-drive's own picker, shared - // with the ranked surface because both read the same indexed tree. - let index = find_ranked_index_for_axis( + // Resolve the query exactly as the prover did — rs-drive's own + // resolution (covering-index pick, shared with the ranked surface + // because both read the same indexed tree, plus the equality-pin + // encoding into prefix path segments for pinned compound requests). + let having_query = resolve_having_query_for_mode( + request.data_contract.id().to_buffer(), + document_type, + request.document_type_name.clone(), document_type.indexes(), - &mode.group_by_property, - axis, - &mode.aggregate_field, + &mode, + platform_version, ) - .ok_or_else(|| drive_proof_verifier::Error::RequestError { + .map_err(|e| drive_proof_verifier::Error::RequestError { error: format!( - "no index on document type `{}` can serve a `{:?}` having bound grouped on \ - `{}`: a having-range query needs a single-property index over `{}` declaring \ - `{}` (and, for SUM / AVG, `summable: \"{}\"`). Ranked indexes are opt-in \ - contract grammar (meta-schema v3, protocol version 14+).", + "document type `{}` cannot serve this having-range query: {e}. Ranked indexes \ + are opt-in contract grammar (meta-schema v3, protocol version 14+); a pinned \ + (compound-index) bound additionally needs every leading index property pinned \ + by an equality where clause.", request.document_type_name, - axis, - mode.group_by_property, - mode.group_by_property, - axis.required_index_keyword(), - mode.aggregate_field, ), })?; - let having_query = DriveDocumentHavingQuery { - document_type, - contract_id: request.data_contract.id().to_buffer(), - document_type_name: request.document_type_name.clone(), - index, - bounds: mode.bounds, - descending: mode.descending, - limit: mode.limit, - }; - // Binds the reconstructed grovedb root hash to the quorum-signed // app hash before returning — see the module docs. let (root_hash, entries) = diff --git a/packages/rs-sdk/src/platform/documents/ranked_proof_helpers.rs b/packages/rs-sdk/src/platform/documents/ranked_proof_helpers.rs index 9211efbce89..1aa2fb817eb 100644 --- a/packages/rs-sdk/src/platform/documents/ranked_proof_helpers.rs +++ b/packages/rs-sdk/src/platform/documents/ranked_proof_helpers.rs @@ -4,9 +4,10 @@ //! caller-built [`DocumentQuery`] plus the node's response into a //! verified [`RankedPage`]. The routing decisions — which ranking axis, //! which direction, how many groups, how many ranks to skip, which -//! index covers them — are **not** re-derived here. They come from -//! rs-drive's own [`detect_ranked_mode`] and -//! [`find_ranked_index_for_axis`], the same two functions the server +//! index covers them, which prefix-value segments a pinned compound +//! request descends through — are **not** re-derived here. They come +//! from rs-drive's own [`detect_ranked_mode`] and +//! [`resolve_ranked_query_for_mode`], the same two functions the server //! calls, so client and server land on the same grove path and the same //! `(axis, k, descending, offset)` tuple by construction rather than by //! two copies of a grammar agreeing. @@ -27,11 +28,9 @@ use dpp::{ data_contract::accessors::v0::DataContractV0Getters, data_contract::document_type::accessors::DocumentTypeV0Getters, }; -use drive::query::drive_document_ranked_query::index_picker::find_ranked_index_for_axis; +use drive::query::drive_document_ranked_query::index_picker::resolve_ranked_query_for_mode; use drive::query::drive_document_ranked_query::mode_detection::detect_ranked_mode; -use drive::query::{ - DocumentRankedMode, DriveDocumentRankedQuery, RankedPage, RankedPaginationInputs, -}; +use drive::query::{DocumentRankedMode, RankedPage, RankedPaginationInputs}; use drive_proof_verifier::verify_ranked_top_k_proof; /// Validate that the caller-built [`DocumentQuery`] really describes a @@ -88,8 +87,9 @@ pub(super) fn assert_ranked_shape( "this DocumentQuery is not a well-formed ranked query: {e}. A ranked query is \ `.with_select()`, `.with_group_by()`, \ `.order_by_selected_aggregate()` and `.with_limit(n)`, \ - optionally `.with_offset(m)`, with no where clauses, no having and no \ - start_at." + optionally `.with_offset(m)`, with no having and no start_at; where clauses, \ + when present, must be equality pins on the covering compound index's leading \ + properties." ), }) } @@ -139,45 +139,31 @@ pub(super) fn verify_ranked_query( let mode = assert_ranked_shape(&request, platform_version)?; - // Pick the index the prover picked. Availability is decided from - // the index's `ranked_*` flags, not from the stored element - // variant, and this is rs-drive's own picker — a second - // implementation here could choose a different index on a - // contract with several candidates and then verify a proof of the - // wrong subtree's ranking (or, more likely, fail to verify at all - // with a confusing merk error). - let index = find_ranked_index_for_axis( + // Resolve the query exactly as the prover did — rs-drive's own + // resolution (covering-index pick + equality-pin encoding into + // prefix path segments). A second implementation here could choose + // a different index on a contract with several candidates, or + // encode a pinned prefix value differently, and then verify a proof + // of the wrong subtree's ranking (or, more likely, fail to verify + // at all with a confusing merk error). + let ranked_query = resolve_ranked_query_for_mode( + request.data_contract.id().to_buffer(), + document_type, + request.document_type_name.clone(), document_type.indexes(), - &mode.group_by_property, - mode.axis, - &mode.aggregate_field, + &mode, + platform_version, ) - .ok_or_else(|| drive_proof_verifier::Error::RequestError { + .map_err(|e| drive_proof_verifier::Error::RequestError { error: format!( - "no index on document type `{}` can rank by `{:?}` grouped on `{}`: a ranked \ - query needs a single-property index over `{}` declaring `{}` (and, for SUM / \ - AVG, `summable: \"{}\"`). Ranked indexes are opt-in contract grammar \ - (meta-schema v3, protocol version 14+).", + "document type `{}` cannot serve this ranked query: {e}. Ranked indexes are \ + opt-in contract grammar (meta-schema v3, protocol version 14+); a pinned \ + (compound-index) ranking additionally needs every leading index property \ + pinned by an equality where clause.", request.document_type_name, - mode.axis, - mode.group_by_property, - mode.group_by_property, - mode.axis.required_index_keyword(), - mode.aggregate_field, ), })?; - let ranked_query = DriveDocumentRankedQuery { - document_type, - contract_id: request.data_contract.id().to_buffer(), - document_type_name: request.document_type_name.clone(), - index, - axis: mode.axis, - descending: mode.descending, - k: mode.k, - offset: mode.offset, - }; - // Binds the reconstructed grovedb root hash to the quorum-signed // app hash before returning — see the module docs. let (root_hash, page) =