Skip to content

Repository files navigation

Vector Gateway Interface

vgi-matchrecognize

CI Latest release MIT licence Minimum supported Rust version 1.97 Supported platforms

SQL:2016 MATCH_RECOGNIZE row pattern matching for DuckDB.

Oracle, Trino, Snowflake, Redshift, BigQuery and Flink all ship MATCH_RECOGNIZE. DuckDB does not — so finding "three failed logins then a success", or "a price dip followed by a recovery", means hand-rolling it out of LAG/LEAD chains, gap-and-island tricks, recursive CTEs and correlated subqueries: slow to run, miserable to read, and usually subtly wrong on ties and NULLs.

This is a VGI worker that gives DuckDB the standard surface as one table-in / table-out function. It buffers a relation, partitions it, sorts each partition, runs a regular expression over rows, and emits either one summary row per match or every matched row.

Local compute only: no network, no secrets, no credentials. The engine — pattern compiler, Pratt expression parser, bind-time type inference, backtracking matcher — is hand-rolled in the Arrow-free mr-core crate. (Input is spooled to a temporary file while it buffers; see Operating notes.)

Quick start

INSTALL vgi FROM community;
LOAD vgi;

-- The worker is a local binary. No secret needed: pure compute, no egress.
ATTACH 'mr' AS mr (TYPE vgi, LOCATION '/path/to/vgi-matchrecognize-worker');

Find each V-shaped dip — a falling run, then a rising run — per symbol. This runs as-is:

SELECT *
FROM mr.match_recognize(
       (SELECT * FROM (VALUES
          ('ACME', 1, 10), ('ACME', 2,  8), ('ACME', 3,  6),
          ('ACME', 4,  9), ('ACME', 5, 11), ('ACME', 6,  7)
        ) AS t(symbol, ts, price)),
       partition_by := ['symbol'],
       order_by     := ['ts'],
       pattern      := 'START DOWN+ UP+',
       define       := MAP {
         'DOWN': 'price < PREV(price)',
         'UP':   'price > PREV(price)'
       },
       measures     := MAP {
         'match_no':  'MATCH_NUMBER()',
         'start_ts':  'FIRST(START.ts)',
         'bottom_ts': 'LAST(DOWN.ts)',
         'end_ts':    'LAST(UP.ts)',
         'drawdown':  'FIRST(START.price) - LAST(DOWN.price)'
       });
┌─────────┬──────────┬──────────┬───────────┬────────┬──────────┐
│ symbol  │ match_no │ start_ts │ bottom_ts │ end_ts │ drawdown │
│ varchar │  int64   │  int64   │   int64   │ int64  │  int64   │
├─────────┼──────────┼──────────┼───────────┼────────┼──────────┤
│ ACME    │        1 │        1 │         3 │      5 │        4 │
└─────────┴──────────┴──────────┴───────────┴────────┴──────────┘

START has no predicate, so it matches any row; DOWN+ then takes the falling run and UP+ the recovery. The dip at ts = 6 starts no match because nothing rises after it.

The function

Everything except the relation is a named, bind-time constant. Functions live in catalog mr, schema mainmr.match_recognize(…) and mr.main.match_recognize(…) both resolve.

mr.match_recognize(
    (<relation>),                -- positional: the input relation, buffered whole
    partition_by  := ['col', …], -- VARCHAR[]  (default [] = one global partition)
    include       := ['col', …], -- VARCHAR[]  input columns carried through as-is
    order_by      := ['col', …], -- VARCHAR[]  (required; 'col DESC', 'col NULLS FIRST')
    pattern       := '…',        -- the row pattern: a regex over variables
    define        := MAP {…},    -- VAR      -> boolean predicate
    subset        := MAP {…},    -- union var -> list of members    (SQL:2016 SUBSET)
    measures      := MAP {…},    -- out column -> expression
    rows          := 'one'|'all',-- ONE (default) | ALL ROWS PER MATCH
    empty_matches := 'show'|'omit', -- SHOW (default) | OMIT EMPTY MATCHES
    after         := '…',        -- AFTER MATCH SKIP mode (default 'past last row')
    step_budget   := 5000000     -- backtracking guard; omit to scale with partition size
) -> TABLE

define, subset and measures: a MAP, a STRUCT, or JSON

Each of the three maps a name to something, so write it as a MAP and let DuckDB check the syntax:

define   := MAP {'DOWN': $$price < PREV(price)$$}
subset   := MAP {'U': ['A', 'B']}
measures := MAP {'match_no': 'MATCH_NUMBER()', 'low': 'LAST(DOWN.price)'}

A STRUCT{'DOWN': $$…$$} — works the same way, and so does the JSON string the map is equivalent to, which is the easier form for a tool assembling a query:

define := '{"DOWN": "price < PREV(price)"}'

Key order is the output column order for measures, in every form.

Use $$…$$ for predicates containing string literals. This is the part that actually removes escaping, and it is independent of which form you choose — a MAP's values are still SQL string literals, so 'outcome = ''fail''' needs its quotes doubled just as it would inside JSON. Dollar-quoting does not:

define := MAP {'FAIL': $$outcome = 'fail'$$}          -- clear
define := $${"FAIL": "outcome = 'fail'"}$$            -- also clear
define := MAP {'FAIL': 'outcome = ''fail'''}          -- same thing, harder to read

The list form of measures, which pins an output type, is a list of structs: [{'as': 'ratio', 'expr': '…', 'type': 'DECIMAL(18,6)'}, …]. DuckDB unifies the struct fields across the list, so an entry that should infer its type writes 'type': NULL.

order_by is the matching order, not the output order

order_by is required, and it is not presentation — it defines the sequence the pattern is matched against, so it decides which rows match at all. The same three rows, one pattern:

-- order_by := ['ts']       -> DOWN+ finds a falling run: 2 rows, prices 8 then 6
-- order_by := ['ts DESC']  -> the same rows now rise, so DOWN+ finds nothing
pattern := 'DOWN+', define := '{"DOWN": "price < PREV(price)"}'

PREV, NEXT, FIRST, LAST and the very idea of "a run" are defined relative to it, which is why SQL:2016 makes ORDER BY mandatory inside MATCH_RECOGNIZE too.

It is also not interchangeable with an ORDER BY in the input subquery. That one is a performance hint (see Operating notes) which we cannot rely on, because nothing guarantees the order a table function receives its rows in; order_by is the declaration of what the pattern means. Write both if you want the speedup, and expect them to name the same columns.

As for the output: within a partition, rows come out in order_by order. Across partitions the order is whatever the input happened to arrive in, and is not guaranteed — add your own ORDER BY to the outer query if you need the result ordered.

Also: mr.explain_pattern(p) -> VARCHAR pretty-prints a compiled pattern (handy for checking greediness, no data access), and mr.main.after_match_skip_modes is a browsable list of the after modes. The worker's build version is the catalog's implementation_version in vgi_catalogs().

Patterns

A regular expression over pattern variables, not characters:

Construct Syntax Meaning
Variable A, DOWN, START one row satisfying define["A"] (any row if undefined)
Concatenation A B C A then B then C
Alternation A | B A or B — the left branch is preferred
Quantifiers * + ? {n} {n,} {n,m} {,m} {,} 0+, 1+, 0/1, exactly n, n+, n..m, 0..m, 0+
Reluctant A+?, A*?, A{2,}? match as few rows as possible
Grouping (A B)+ quantify or alternate a sub-pattern
Anchors ^A, A$ partition start / end
Permutation PERMUTE(A, B, C) A, B and C in any order — the alternation of every permutation, preferring the order written

Unquoted variable names are case-insensitive and canonicalize to upper case. A double-quoted name is case-sensitive, so "b" and b are different variables (the latter being B), and CLASSIFIER() reports the canonical spelling.

DEFINE and MEASURES

Both clauses share one expression language. define decides whether a row can bind a variable; measures computes the output columns.

you can write
Columns price, A.price, "My Col"
Navigation PREV/NEXT/FIRST/LAST(expr[, n]); LAG/LEAD are accepted as Snowflake spells PREV/NEXT
Aggregates SUM COUNT AVG MIN MAX ARRAY_AGG ARBITRARY, plus COUNT(*), COUNT(), COUNT(A.*)
Match info CLASSIFIER([label]), MATCH_NUMBER(), MATCH_SEQUENCE_NUMBER() (the row's 1-based position in its match)
Horizon RUNNING / FINAL
Operators arithmetic, comparison, AND/OR/NOT, IS [NOT] NULL, BETWEEN, IN, ||, CAST/::
Scalars abs ceil floor round sqrt lower upper trim ltrim rtrim length coalesce nullif greatest least

DEFINE predicates are always RUNNING: they see only the rows matched so far, which is what lets a predicate refer back to the match in progress.

Both clauses are type-checked at bind, before any data is read: a predicate must be BOOLEAN, and every column and pattern variable it names must exist. A predicate that could never be true — {"B": "price"}, or {"B": "sym > 3"} comparing a VARCHAR with an integer — is an error rather than an empty result, and the error names the key it came from:

match_recognize type-inference error: define['B']: cannot compare VARCHAR with BIGINT

A bare A.price means LAST(A.price) under the prevailing RUNNING/FINAL horizon — the last row bound to A, or NULL if A has not bound one yet. So match-dependent predicates read the way you would write them: "B": "price > A.price" compares each candidate B against A's row. Qualified physical navigation anchors on the variable too: PREV(A.price, n) steps back n rows from A's last row (the standard's PREV(LAST(A.price), n)), while unqualified PREV(price) steps back from the current row.

array_agg(expr) collects matched values in match order as a DuckDB list (BIGINT[], VARCHAR[], …). Over an empty match it is an empty list, not NULL — so a RUNNING array_agg grows row by row.

subset := MAP {'U': ['A', 'B']} declares SQL:2016 union variables. U then stands for any of its members wherever a pattern variable may appear: U.price, COUNT(U.*), SUM(U.price), CLASSIFIER(U), after := 'to last U'. A union variable may not have its own DEFINE predicate.

Anything else: compose around the call

The expression language is a deliberate subset, and it does not need to be complete — DuckDB's full library is available on both sides of the call:

-- row-local scalar: compute it in the input subquery
(SELECT *, lower(event) AS ev FROM t)   →  define := '{"A": "ev = ''view''"}'

-- post-processing a measure: do it in the outer SELECT
SELECT CAST(lower(cls) || '_label' AS VARCHAR(7))
FROM mr.match_recognize(…, measures := '{"cls": "LAST(CLASSIFIER())"}')

What composition cannot reach: an unsupported scalar inside a DEFINE predicate (the predicate feeds back into matching, so it has to run in the matcher), an unsupported aggregate over match state (it depends on which rows are bound, so neither side of the call can see it), and subqueries in either clause — a standalone worker receives Arrow batches and has no catalog to resolve them against.

What comes out

The output schema is fixed at bind time, before any data flows.

  • rows := 'one' — the partition_by columns, the include columns, then one column per measure.
  • rows := 'all' — the partition_by columns, the include columns, the order_by columns, match_number BIGINT and classifier VARCHAR (automatic unless a measure of that name shadows them), then one column per measure.

Deviation from Trino/Oracle. Under ALL ROWS PER MATCH they emit every input column and no automatic match_number/classifier. Name the ones you want in includeinclude := ['value'] — which both buffers and emits them; a measure of the same expression works too. Everything else stays unbuffered, which is what keeps an unread column from costing anything.

include carries an input column through with its own name and type: under rows := 'all' it is the value on each matched row, under rows := 'one' the value on the match's first row (where the partition keys are read from). A column that is already emitted — a partition key, or an order key under rows := 'all' — is not repeated.

Measure types are inferred from the input schema: MATCH_NUMBER() and COUNT(…)BIGINT; CLASSIFIER()VARCHAR; FIRST/LAST/PREV/NEXT/MIN/MAX → that column's type; SUM widens (integer → HUGEINT, float → DOUBLE); AVGDOUBLE; ARRAY_AGG → a list of the argument's type; arithmetic → the widened numeric type; comparison and logic → BOOLEAN; ||VARCHAR. When inference cannot decide — a measure that resolves to an untyped NULL, say — use the array form and pin the type:

UBIGINT keeps its own type rather than widening to BIGINT, so a value above i64::MAX round-trips intact. Because u64 and i64 contain neither the other, mixing them widens to HUGEINT (u + bigint, and -u), while SUM(u)HUGEINT and AVG(u)DOUBLE as for any integer. One limitation: the expression lexer parses an integer literal as BIGINT, so a constant above i64::MAX has to be written CAST('18446744073709551615' AS UBIGINT).

[
  { "as": "ratio", "expr": "SUM(A.qty) / SUM(B.qty)", "type": "DECIMAL(18,6)" },
  { "as": "label", "expr": "CLASSIFIER()" }
]

Semantics worth knowing

MATCH_NUMBER() counts within a partition. It restarts at 1 for every partition, as SQL:2016 specifies — it is not a global row number.

Empty matches are real matches. A pattern that legitimately matches zero rows at some position — B* where the row cannot bind B — produces one output row positioned on the row it sits on, and consumes a match number. Its measures see a match with nothing bound: CLASSIFIER() and the navigation functions are NULL, COUNT(*) is 0, array_agg is empty. This surprises people, and it is what every conforming engine does. empty_matches := 'omit' drops those rows under rows := 'all'; rows := 'one' always reports them.

AFTER MATCH SKIP decides where the next search begins:

after Next search resumes at notes
'past last row' the row after the match non-overlapping (default)
'to next row' the row after the match start overlapping matches
'to first <VAR>' the first row bound to VAR VAR may be a union variable
'to last <VAR>' the last row bound to VAR VAR may be a union variable

A no-progress safeguard advances the cursor by at least one row per iteration, so the scan always terminates regardless of the skip mode. mr.main.after_match_skip_modes lists the same thing from SQL.

Recipes

Sessionization — split a stream where the gap exceeds 30 minutes

SELECT user_id, session_no, session_start, session_end, n_events
FROM mr.match_recognize(
       (SELECT user_id, ts FROM clicks),
       partition_by := ['user_id'],
       order_by     := ['ts'],
       pattern      := 'A B*',
       define       := '{ "B": "ts <= PREV(ts) + INTERVAL 1800 SECOND" }',
       measures     := '{
         "session_no":    "MATCH_NUMBER()",
         "session_start": "FIRST(ts)",
         "session_end":   "LAST(ts)",
         "n_events":      "COUNT(*)"
       }');

A takes any row; B* extends the session for as long as each event is within 30 minutes of the previous one. The next session starts at the first row that is not.

Brute force then breach — three or more failures, then a success

SELECT *
FROM mr.match_recognize(
       (SELECT user_id, ts, outcome FROM auth_events),
       partition_by := ['user_id'],
       order_by     := ['ts'],
       pattern      := 'FAIL{3,} OK',
       define       := MAP {
         'FAIL': $$outcome = 'fail'$$,
         'OK':   $$outcome = 'success'$$
       },
       measures     := MAP {
         'var':        'CLASSIFIER()',
         'n_fails':    'FINAL COUNT(FAIL.*)',
         'first_fail': 'FIRST(FAIL.ts)',
         'breach_ts':  'LAST(OK.ts)'
       },
       rows := 'all');   -- every event in the burst, tagged by classifier

Funnel — view → click → purchase, in order, per user

SELECT *
FROM mr.match_recognize(
       (SELECT user_id, ts, event FROM events),
       partition_by := ['user_id'],
       order_by     := ['ts'],
       pattern      := 'V+ C+ P',
       define       := '{
         "V": "event = ''view''",
         "C": "event = ''click''",
         "P": "event = ''purchase''"
       }',
       measures     := '{
         "views":    "COUNT(V.*)",
         "clicks":   "COUNT(C.*)",
         "elapsed":  "LAST(P.ts) - FIRST(V.ts)",
         "sequence": "array_agg(CLASSIFIER())"
       }');

Operating notes

Running it as a container

The image serves the same worker over a network transport, for when DuckDB should not (or cannot) spawn a local binary — a shared worker, or a Fly.io-style deploy:

docker run --rm -p 8000:8000 ghcr.io/query-farm/vgi-matchrecognize      # HTTP, /health
docker run --rm -p 8001:8001 ghcr.io/query-farm/vgi-matchrecognize tcp  # raw Arrow-IPC
ATTACH 'mr' AS mr (TYPE vgi, LOCATION 'http://localhost:8000');
ATTACH 'mr' AS mr (TYPE vgi, LOCATION 'tcp://localhost:8001');

Two things follow from this being a buffering function, and neither applies to the stateless workers in this family:

  • It needs temp space. The spool is roughly 24 bytes per row of the columns the pattern reads, and a sharded run peaks at about 1.5x that. The image declares no volume, so it lands in the container's writable layer — mount one at /tmp for inputs that will not fit there.

  • Every phase of a query must reach the same spool. The buffering phase and the producing phase are separate worker processes, and the spool belongs to whichever container wrote it. Over HTTP or TCP one container serves both, so this is only a question of sticky routing (or a single replica) behind a load balancer. Over stdio it bites immediately: the extension spawns a pool of workers, so each spawn is its own container, and they share nothing unless you give them one volume to share —

    docker volume create mr-spool
    # LOCATION for the ATTACH below:
    docker run -i --rm -v mr-spool:/tmp ghcr.io/query-farm/vgi-matchrecognize stdio

    Getting it wrong fails loudly rather than quietly: the sink-count guard raises an error instead of returning a short result. (For on-host use, the release binary is simpler than a container in stdio mode, and has none of this.)

Buffering and memory

Row pattern matching is intrinsically a whole-partition operation, so the function buffers its input before it matches anything. Batches are spooled to disk as Arrow IPC — one append-only file per sink thread under $TMPDIR, mode 0700 — not to process memory.

  • Unused columns are free. Only the columns the pattern reads — the partition and order keys plus everything define/measures reference — are buffered; the rest are projected away first. Since buffering volume dominates runtime, selecting fewer columns in the input subquery is the cheapest speedup available.
  • Matching runs on several threads. Partitions are independent, so a chunk of them is matched concurrently and the results are emitted in partition order — byte-identical to matching one at a time. VGI_MR_MATCH_THREADS pins the count; 1 forces the serial path.
  • Memory is bounded, and tunable. If the buffered input exceeds VGI_MR_FINALIZE_MEMORY_BYTES (default 128 MB), it is split by partition key into shards and each is matched separately, so peak memory tracks a shard rather than the relation. Lowering the budget trades time for memory: the split is a second pass over the input. Two things cannot be divided that way — a query with no partition_by is a single partition by definition, and one enormous partition is irreducible, because a match may span all of it.
  • Output streams, including inside a match. Rows are emitted in ~8k-row batches and the producer stops filling one the moment it is full, even in the middle of a long match — so a single partition emitting millions of rows costs one batch of memory, not a partition's worth. DuckDB can also stop pulling, and a LIMIT then skips the matching it does not need.
  • Pre-sorting the input still helps a little. Ours is a single-threaded sort over the buffered rows; DuckDB's is parallel and vectorized. Adding ORDER BY to the input subquery — (SELECT … FROM t ORDER BY user_id, ts) — lets our sort see an already-ordered run, which is markedly cheaper than sorting a scrambled one. We sort regardless, so this is purely a speedup, never a correctness dependency.

Matching cannot be made fully streaming: a match may span an entire partition, so the partition must be complete first, and DuckDB does not deliver input clustered by partition key. docs/perf-baseline.md is the measured cost of every phase.

Leave VGI_WORKER_SHARED_STORAGE alone. The spool holds the buffered rows, but the SDK store still holds the small control records the finalize phase replays, and the two phases may run in different worker processes — so memory is refused at bind, while sqlite (the default) and fs are both fine. Whatever goes wrong there — a truncated spool file, two phases not sharing state — is an error that names what went missing, never a short answer.

Disk use is about 24 bytes per buffered row — and roughly twice that while a split is in progress, since the shards are written before the buffered input can be released. Size $TMPDIR accordingly if you lower the memory budget. Spool files are deleted as soon as the finalize phase has read them. A query killed before that leaves the worker no hook at all, so directories older than VGI_BUFFERING_STORE_TTL_SECS (default 24h, the same knob the SDK's own store uses) are swept when a worker process first spools.

Not hanging, not crashing

An ambiguous pattern can backtrack catastrophically, so the matcher runs on a per-partition step budget and returns a clean error rather than running forever. It never panics or aborts: property tests drive arbitrary patterns over arbitrary row tables.

The budget scales with the partition (128 steps per row, floor 5,000,000), because a constant is the wrong shape — catastrophic backtracking is super-linear in partition size, so one number cannot both catch a bad pattern on a small partition and let an ordinary linear match finish on a large one. Pin it with step_budget := <n> for a hard ceiling.

A long match is linear, not quadratic. The cost of a row does not depend on how long the match containing it is, so a qualified reference (A.price), a running aggregate and FIRST/LAST navigation are all as cheap inside a million-row match as inside a five-row one. This is worth stating because the obvious implementation of each is not: docs/perf-baseline.md has the measurements.

Backtracking uses an explicit heap stack rather than host recursion, so one match may span millions of rows: an A B* sessionization whose partition is a single long session is bounded by the budget, not by the OS stack. Deeply nested pattern/define/measures input is refused past 128 levels for the same reason.

Conformance

Checked against three other implementations, all inside the end-to-end suite:

  • Trino — 133 assertions ported from its MATCH_RECOGNIZE test suite (test/sql/trino_conformance.test). Of the 151 cases expressible on this surface, none produces a wrong answer; the remaining 18 error cleanly on features we do not implement (subqueries in DEFINE/MEASURES, two-argument aggregates such as max_by, a few scalar functions, and the ALL ROWS column layout above). test/trino/README.md has the tally and how to regenerate it.
  • Apache Flink — the portable cases from its MatchRecognizeITCase (test/sql/flink_conformance.test), which add logical offsets at large indices, aggregates over expressions in DEFINE, NULL handling, and multi-key ordering with mixed directions. We agree with Flink on all of them but one, where we report an extra match that SQL:2016 requires and Flink's non-backtracking NFA misses; that case carries the analysis inline. Flink's own documentation lists the standard features it omits, several of which we support (pattern groups, alternation, anchors, SUBSET, physical PREV/NEXT, ALL ROWS PER MATCH).
  • Snowflake — its documented worked example reproduced exactly, plus coverage of the features reading those docs turned up (test/sql/snowflake_conformance.test). Three of them are now supported: PERMUTE, MATCH_SEQUENCE_NUMBER(), and LAG/LEAD as spellings of PREV/NEXT.

Still unimplemented: pattern exclusion {- … -}, WITH UNMATCHED ROWS, and some exotic temporal/INTERVAL type-lattice corners (route those through the explicit type override).

Build & test

cargo test --workspace     # mr-core unit + property tests, mr-worker integration
cargo build --release      # the worker binary
./run_tests.sh             # haybarn SQLLogic end-to-end suite
./test/trino/port.sh       # re-port Trino's suite (needs a Trino checkout)

Links

Get it

  • Latest release — one .tar.gz per platform (linux_amd64, linux_arm64, osx_amd64, osx_arm64, windows_amd64, plus wasm), each with a SHA256 and a keyless cosign bundle next to it.
  • Container imageghcr.io/query-farm/vgi-matchrecognize, multi-arch, for the HTTP/TCP transports (from v0.2.1).
  • The vgi extension — the DuckDB community extension that ATTACH … (TYPE vgi, …) comes from.

This project

MATCH_RECOGNIZE in other engines

Useful when porting a query in or out — and the sources the conformance suites in test/sql are ported from:

License

MIT — Copyright 2026 Query Farm LLC.

About

Row pattern matching in DuckDB with SQL — SQL:2016 MATCH_RECOGNIZE (PARTITION/ORDER/PATTERN/DEFINE/MEASURES) for sequences: V-shapes, funnels, sessionization, event patterns. A VGI worker.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages