SQL:2016 MATCH_RECOGNIZE row pattern matching for DuckDB.
Oracle, Trino, Snowflake, Redshift, BigQuery and Flink all ship MATCH_RECOGNIZE.
DuckDB does not — so finding "three failed logins then a success", or "a price dip
followed by a recovery", means hand-rolling it out of LAG/LEAD chains,
gap-and-island tricks, recursive CTEs and correlated subqueries: slow to run,
miserable to read, and usually subtly wrong on ties and NULLs.
This is a VGI worker that gives DuckDB the standard surface as one table-in / table-out function. It buffers a relation, partitions it, sorts each partition, runs a regular expression over rows, and emits either one summary row per match or every matched row.
Local compute only: no network, no secrets, no credentials. The engine —
pattern compiler, Pratt expression parser, bind-time type inference, backtracking
matcher — is hand-rolled in the Arrow-free mr-core crate. (Input is spooled to
a temporary file while it buffers; see Operating notes.)
INSTALL vgi FROM community;
LOAD vgi;
-- The worker is a local binary. No secret needed: pure compute, no egress.
ATTACH 'mr' AS mr (TYPE vgi, LOCATION '/path/to/vgi-matchrecognize-worker');Find each V-shaped dip — a falling run, then a rising run — per symbol. This runs as-is:
SELECT *
FROM mr.match_recognize(
(SELECT * FROM (VALUES
('ACME', 1, 10), ('ACME', 2, 8), ('ACME', 3, 6),
('ACME', 4, 9), ('ACME', 5, 11), ('ACME', 6, 7)
) AS t(symbol, ts, price)),
partition_by := ['symbol'],
order_by := ['ts'],
pattern := 'START DOWN+ UP+',
define := MAP {
'DOWN': 'price < PREV(price)',
'UP': 'price > PREV(price)'
},
measures := MAP {
'match_no': 'MATCH_NUMBER()',
'start_ts': 'FIRST(START.ts)',
'bottom_ts': 'LAST(DOWN.ts)',
'end_ts': 'LAST(UP.ts)',
'drawdown': 'FIRST(START.price) - LAST(DOWN.price)'
});┌─────────┬──────────┬──────────┬───────────┬────────┬──────────┐
│ symbol │ match_no │ start_ts │ bottom_ts │ end_ts │ drawdown │
│ varchar │ int64 │ int64 │ int64 │ int64 │ int64 │
├─────────┼──────────┼──────────┼───────────┼────────┼──────────┤
│ ACME │ 1 │ 1 │ 3 │ 5 │ 4 │
└─────────┴──────────┴──────────┴───────────┴────────┴──────────┘
START has no predicate, so it matches any row; DOWN+ then takes the falling
run and UP+ the recovery. The dip at ts = 6 starts no match because nothing
rises after it.
Everything except the relation is a named, bind-time constant. Functions live in
catalog mr, schema main — mr.match_recognize(…) and
mr.main.match_recognize(…) both resolve.
mr.match_recognize(
(<relation>), -- positional: the input relation, buffered whole
partition_by := ['col', …], -- VARCHAR[] (default [] = one global partition)
include := ['col', …], -- VARCHAR[] input columns carried through as-is
order_by := ['col', …], -- VARCHAR[] (required; 'col DESC', 'col NULLS FIRST')
pattern := '…', -- the row pattern: a regex over variables
define := MAP {…}, -- VAR -> boolean predicate
subset := MAP {…}, -- union var -> list of members (SQL:2016 SUBSET)
measures := MAP {…}, -- out column -> expression
rows := 'one'|'all',-- ONE (default) | ALL ROWS PER MATCH
empty_matches := 'show'|'omit', -- SHOW (default) | OMIT EMPTY MATCHES
after := '…', -- AFTER MATCH SKIP mode (default 'past last row')
step_budget := 5000000 -- backtracking guard; omit to scale with partition size
) -> TABLE
Each of the three maps a name to something, so write it as a MAP and let DuckDB
check the syntax:
define := MAP {'DOWN': $$price < PREV(price)$$}
subset := MAP {'U': ['A', 'B']}
measures := MAP {'match_no': 'MATCH_NUMBER()', 'low': 'LAST(DOWN.price)'}A STRUCT — {'DOWN': $$…$$} — works the same way, and so does the JSON string the
map is equivalent to, which is the easier form for a tool assembling a query:
define := '{"DOWN": "price < PREV(price)"}'Key order is the output column order for measures, in every form.
Use $$…$$ for predicates containing string literals. This is the part that
actually removes escaping, and it is independent of which form you choose — a MAP's
values are still SQL string literals, so 'outcome = ''fail''' needs its quotes
doubled just as it would inside JSON. Dollar-quoting does not:
define := MAP {'FAIL': $$outcome = 'fail'$$} -- clear
define := $${"FAIL": "outcome = 'fail'"}$$ -- also clear
define := MAP {'FAIL': 'outcome = ''fail'''} -- same thing, harder to readThe list form of measures, which pins an output type, is a list of structs:
[{'as': 'ratio', 'expr': '…', 'type': 'DECIMAL(18,6)'}, …]. DuckDB unifies the
struct fields across the list, so an entry that should infer its type writes
'type': NULL.
order_by is required, and it is not presentation — it defines the sequence the
pattern is matched against, so it decides which rows match at all. The same three
rows, one pattern:
-- order_by := ['ts'] -> DOWN+ finds a falling run: 2 rows, prices 8 then 6
-- order_by := ['ts DESC'] -> the same rows now rise, so DOWN+ finds nothing
pattern := 'DOWN+', define := '{"DOWN": "price < PREV(price)"}'PREV, NEXT, FIRST, LAST and the very idea of "a run" are defined relative to
it, which is why SQL:2016 makes ORDER BY mandatory inside MATCH_RECOGNIZE too.
It is also not interchangeable with an ORDER BY in the input subquery. That
one is a performance hint (see Operating notes) which we cannot
rely on, because nothing guarantees the order a table function receives its rows in;
order_by is the declaration of what the pattern means. Write both if you want the
speedup, and expect them to name the same columns.
As for the output: within a partition, rows come out in order_by order. Across
partitions the order is whatever the input happened to arrive in, and is not
guaranteed — add your own ORDER BY to the outer query if you need the result
ordered.
Also: mr.explain_pattern(p) -> VARCHAR pretty-prints a compiled pattern (handy
for checking greediness, no data access), and mr.main.after_match_skip_modes is a
browsable list of the after modes. The worker's build version is the catalog's
implementation_version in vgi_catalogs().
A regular expression over pattern variables, not characters:
| Construct | Syntax | Meaning |
|---|---|---|
| Variable | A, DOWN, START |
one row satisfying define["A"] (any row if undefined) |
| Concatenation | A B C |
A then B then C |
| Alternation | A | B |
A or B — the left branch is preferred |
| Quantifiers | * + ? {n} {n,} {n,m} {,m} {,} |
0+, 1+, 0/1, exactly n, n+, n..m, 0..m, 0+ |
| Reluctant | A+?, A*?, A{2,}? |
match as few rows as possible |
| Grouping | (A B)+ |
quantify or alternate a sub-pattern |
| Anchors | ^A, A$ |
partition start / end |
| Permutation | PERMUTE(A, B, C) |
A, B and C in any order — the alternation of every permutation, preferring the order written |
Unquoted variable names are case-insensitive and canonicalize to upper case. A
double-quoted name is case-sensitive, so "b" and b are different variables
(the latter being B), and CLASSIFIER() reports the canonical spelling.
Both clauses share one expression language. define decides whether a row can
bind a variable; measures computes the output columns.
| you can write | |
|---|---|
| Columns | price, A.price, "My Col" |
| Navigation | PREV/NEXT/FIRST/LAST(expr[, n]); LAG/LEAD are accepted as Snowflake spells PREV/NEXT |
| Aggregates | SUM COUNT AVG MIN MAX ARRAY_AGG ARBITRARY, plus COUNT(*), COUNT(), COUNT(A.*) |
| Match info | CLASSIFIER([label]), MATCH_NUMBER(), MATCH_SEQUENCE_NUMBER() (the row's 1-based position in its match) |
| Horizon | RUNNING / FINAL |
| Operators | arithmetic, comparison, AND/OR/NOT, IS [NOT] NULL, BETWEEN, IN, ||, CAST/:: |
| Scalars | abs ceil floor round sqrt lower upper trim ltrim rtrim length coalesce nullif greatest least |
DEFINE predicates are always RUNNING: they see only the rows matched so far, which is what lets a predicate refer back to the match in progress.
Both clauses are type-checked at bind, before any data is read: a predicate
must be BOOLEAN, and every column and pattern variable it names must exist. A
predicate that could never be true — {"B": "price"}, or {"B": "sym > 3"}
comparing a VARCHAR with an integer — is an error rather than an empty result,
and the error names the key it came from:
match_recognize type-inference error: define['B']: cannot compare VARCHAR with BIGINT
A bare A.price means LAST(A.price) under the prevailing RUNNING/FINAL
horizon — the last row bound to A, or NULL if A has not bound one yet. So
match-dependent predicates read the way you would write them:
"B": "price > A.price" compares each candidate B against A's row. Qualified
physical navigation anchors on the variable too: PREV(A.price, n) steps back n
rows from A's last row (the standard's PREV(LAST(A.price), n)), while
unqualified PREV(price) steps back from the current row.
array_agg(expr) collects matched values in match order as a DuckDB list
(BIGINT[], VARCHAR[], …). Over an empty match it is an empty list, not NULL —
so a RUNNING array_agg grows row by row.
subset := MAP {'U': ['A', 'B']} declares SQL:2016 union variables. U then
stands for any of its members wherever a pattern variable may appear: U.price,
COUNT(U.*), SUM(U.price), CLASSIFIER(U), after := 'to last U'. A union
variable may not have its own DEFINE predicate.
The expression language is a deliberate subset, and it does not need to be complete — DuckDB's full library is available on both sides of the call:
-- row-local scalar: compute it in the input subquery
(SELECT *, lower(event) AS ev FROM t) → define := '{"A": "ev = ''view''"}'
-- post-processing a measure: do it in the outer SELECT
SELECT CAST(lower(cls) || '_label' AS VARCHAR(7))
FROM mr.match_recognize(…, measures := '{"cls": "LAST(CLASSIFIER())"}')What composition cannot reach: an unsupported scalar inside a DEFINE predicate (the predicate feeds back into matching, so it has to run in the matcher), an unsupported aggregate over match state (it depends on which rows are bound, so neither side of the call can see it), and subqueries in either clause — a standalone worker receives Arrow batches and has no catalog to resolve them against.
The output schema is fixed at bind time, before any data flows.
rows := 'one'— thepartition_bycolumns, theincludecolumns, then one column per measure.rows := 'all'— thepartition_bycolumns, theincludecolumns, theorder_bycolumns,match_number BIGINTandclassifier VARCHAR(automatic unless a measure of that name shadows them), then one column per measure.
Deviation from Trino/Oracle. Under
ALL ROWS PER MATCHthey emit every input column and no automaticmatch_number/classifier. Name the ones you want ininclude—include := ['value']— which both buffers and emits them; a measure of the same expression works too. Everything else stays unbuffered, which is what keeps an unread column from costing anything.
include carries an input column through with its own name and type: under
rows := 'all' it is the value on each matched row, under rows := 'one' the value
on the match's first row (where the partition keys are read from). A column that is
already emitted — a partition key, or an order key under rows := 'all' — is not
repeated.
Measure types are inferred from the input schema: MATCH_NUMBER() and
COUNT(…) → BIGINT; CLASSIFIER() → VARCHAR;
FIRST/LAST/PREV/NEXT/MIN/MAX → that column's type; SUM widens
(integer → HUGEINT, float → DOUBLE); AVG → DOUBLE; ARRAY_AGG → a list of
the argument's type; arithmetic → the widened numeric type; comparison and logic →
BOOLEAN; || → VARCHAR. When inference cannot decide — a measure that resolves
to an untyped NULL, say — use the array form and pin the type:
UBIGINT keeps its own type rather than widening to BIGINT, so a value above
i64::MAX round-trips intact. Because u64 and i64 contain neither the
other, mixing them widens to HUGEINT (u + bigint, and -u), while
SUM(u) → HUGEINT and AVG(u) → DOUBLE as for any integer. One
limitation: the expression lexer parses an integer literal as BIGINT, so a
constant above i64::MAX has to be written
CAST('18446744073709551615' AS UBIGINT).
[
{ "as": "ratio", "expr": "SUM(A.qty) / SUM(B.qty)", "type": "DECIMAL(18,6)" },
{ "as": "label", "expr": "CLASSIFIER()" }
]MATCH_NUMBER() counts within a partition. It restarts at 1 for every
partition, as SQL:2016 specifies — it is not a global row number.
Empty matches are real matches. A pattern that legitimately matches zero rows
at some position — B* where the row cannot bind B — produces one output row
positioned on the row it sits on, and consumes a match number. Its measures see a
match with nothing bound: CLASSIFIER() and the navigation functions are NULL,
COUNT(*) is 0, array_agg is empty. This surprises people, and it is what every
conforming engine does. empty_matches := 'omit' drops those rows under
rows := 'all'; rows := 'one' always reports them.
AFTER MATCH SKIP decides where the next search begins:
after |
Next search resumes at | notes |
|---|---|---|
'past last row' |
the row after the match | non-overlapping (default) |
'to next row' |
the row after the match start | overlapping matches |
'to first <VAR>' |
the first row bound to VAR | VAR may be a union variable |
'to last <VAR>' |
the last row bound to VAR | VAR may be a union variable |
A no-progress safeguard advances the cursor by at least one row per iteration, so
the scan always terminates regardless of the skip mode. mr.main.after_match_skip_modes
lists the same thing from SQL.
SELECT user_id, session_no, session_start, session_end, n_events
FROM mr.match_recognize(
(SELECT user_id, ts FROM clicks),
partition_by := ['user_id'],
order_by := ['ts'],
pattern := 'A B*',
define := '{ "B": "ts <= PREV(ts) + INTERVAL 1800 SECOND" }',
measures := '{
"session_no": "MATCH_NUMBER()",
"session_start": "FIRST(ts)",
"session_end": "LAST(ts)",
"n_events": "COUNT(*)"
}');A takes any row; B* extends the session for as long as each event is within
30 minutes of the previous one. The next session starts at the first row that is
not.
SELECT *
FROM mr.match_recognize(
(SELECT user_id, ts, outcome FROM auth_events),
partition_by := ['user_id'],
order_by := ['ts'],
pattern := 'FAIL{3,} OK',
define := MAP {
'FAIL': $$outcome = 'fail'$$,
'OK': $$outcome = 'success'$$
},
measures := MAP {
'var': 'CLASSIFIER()',
'n_fails': 'FINAL COUNT(FAIL.*)',
'first_fail': 'FIRST(FAIL.ts)',
'breach_ts': 'LAST(OK.ts)'
},
rows := 'all'); -- every event in the burst, tagged by classifierSELECT *
FROM mr.match_recognize(
(SELECT user_id, ts, event FROM events),
partition_by := ['user_id'],
order_by := ['ts'],
pattern := 'V+ C+ P',
define := '{
"V": "event = ''view''",
"C": "event = ''click''",
"P": "event = ''purchase''"
}',
measures := '{
"views": "COUNT(V.*)",
"clicks": "COUNT(C.*)",
"elapsed": "LAST(P.ts) - FIRST(V.ts)",
"sequence": "array_agg(CLASSIFIER())"
}');The image serves the same worker over a network transport, for when DuckDB should not (or cannot) spawn a local binary — a shared worker, or a Fly.io-style deploy:
docker run --rm -p 8000:8000 ghcr.io/query-farm/vgi-matchrecognize # HTTP, /health
docker run --rm -p 8001:8001 ghcr.io/query-farm/vgi-matchrecognize tcp # raw Arrow-IPCATTACH 'mr' AS mr (TYPE vgi, LOCATION 'http://localhost:8000');
ATTACH 'mr' AS mr (TYPE vgi, LOCATION 'tcp://localhost:8001');Two things follow from this being a buffering function, and neither applies to the stateless workers in this family:
-
It needs temp space. The spool is roughly 24 bytes per row of the columns the pattern reads, and a sharded run peaks at about 1.5x that. The image declares no volume, so it lands in the container's writable layer — mount one at
/tmpfor inputs that will not fit there. -
Every phase of a query must reach the same spool. The buffering phase and the producing phase are separate worker processes, and the spool belongs to whichever container wrote it. Over HTTP or TCP one container serves both, so this is only a question of sticky routing (or a single replica) behind a load balancer. Over stdio it bites immediately: the extension spawns a pool of workers, so each spawn is its own container, and they share nothing unless you give them one volume to share —
docker volume create mr-spool # LOCATION for the ATTACH below: docker run -i --rm -v mr-spool:/tmp ghcr.io/query-farm/vgi-matchrecognize stdioGetting it wrong fails loudly rather than quietly: the sink-count guard raises an error instead of returning a short result. (For on-host use, the release binary is simpler than a container in stdio mode, and has none of this.)
Row pattern matching is intrinsically a whole-partition operation, so the function
buffers its input before it matches anything. Batches are spooled to disk as Arrow
IPC — one append-only file per sink thread under $TMPDIR, mode 0700 — not to
process memory.
- Unused columns are free. Only the columns the pattern reads — the partition
and order keys plus everything
define/measuresreference — are buffered; the rest are projected away first. Since buffering volume dominates runtime, selecting fewer columns in the input subquery is the cheapest speedup available. - Matching runs on several threads. Partitions are independent, so a chunk of
them is matched concurrently and the results are emitted in partition order —
byte-identical to matching one at a time.
VGI_MR_MATCH_THREADSpins the count; 1 forces the serial path. - Memory is bounded, and tunable. If the buffered input exceeds
VGI_MR_FINALIZE_MEMORY_BYTES(default 128 MB), it is split by partition key into shards and each is matched separately, so peak memory tracks a shard rather than the relation. Lowering the budget trades time for memory: the split is a second pass over the input. Two things cannot be divided that way — a query with nopartition_byis a single partition by definition, and one enormous partition is irreducible, because a match may span all of it. - Output streams, including inside a match. Rows are emitted in ~8k-row batches
and the producer stops filling one the moment it is full, even in the middle of a
long match — so a single partition emitting millions of rows costs one batch of
memory, not a partition's worth. DuckDB can also stop pulling, and a
LIMITthen skips the matching it does not need. - Pre-sorting the input still helps a little. Ours is a single-threaded sort over
the buffered rows; DuckDB's is parallel and vectorized. Adding
ORDER BYto the input subquery —(SELECT … FROM t ORDER BY user_id, ts)— lets our sort see an already-ordered run, which is markedly cheaper than sorting a scrambled one. We sort regardless, so this is purely a speedup, never a correctness dependency.
Matching cannot be made fully streaming: a match may span an entire partition, so the partition must be complete first, and DuckDB does not deliver input clustered by partition key. docs/perf-baseline.md is the measured cost of every phase.
Leave VGI_WORKER_SHARED_STORAGE alone. The spool holds the buffered rows, but the
SDK store still holds the small control records the finalize phase replays, and the two
phases may run in different worker processes — so memory is refused at bind, while
sqlite (the default) and fs are both fine. Whatever goes wrong there — a truncated
spool file, two phases not sharing state — is an error that names what went missing,
never a short answer.
Disk use is about 24 bytes per buffered row — and roughly twice that while a split is
in progress, since the shards are written before the buffered input can be released.
Size $TMPDIR accordingly if you lower the memory budget. Spool files are deleted as
soon as the finalize phase has read them. A query killed
before that leaves the worker no hook at all, so directories older than
VGI_BUFFERING_STORE_TTL_SECS (default 24h, the same knob the SDK's own store uses)
are swept when a worker process first spools.
An ambiguous pattern can backtrack catastrophically, so the matcher runs on a per-partition step budget and returns a clean error rather than running forever. It never panics or aborts: property tests drive arbitrary patterns over arbitrary row tables.
The budget scales with the partition (128 steps per row, floor 5,000,000),
because a constant is the wrong shape — catastrophic backtracking is super-linear in
partition size, so one number cannot both catch a bad pattern on a small partition
and let an ordinary linear match finish on a large one. Pin it with
step_budget := <n> for a hard ceiling.
A long match is linear, not quadratic. The cost of a row does not depend on how
long the match containing it is, so a qualified reference (A.price), a running
aggregate and FIRST/LAST navigation are all as cheap inside a million-row match as
inside a five-row one. This is worth stating because the obvious implementation of
each is not: docs/perf-baseline.md has the measurements.
Backtracking uses an explicit heap stack rather than host recursion, so one match
may span millions of rows: an A B* sessionization whose partition is a single long
session is bounded by the budget, not by the OS stack. Deeply nested
pattern/define/measures input is refused past 128 levels for the same reason.
Checked against three other implementations, all inside the end-to-end suite:
- Trino — 133 assertions ported from its
MATCH_RECOGNIZEtest suite (test/sql/trino_conformance.test). Of the 151 cases expressible on this surface, none produces a wrong answer; the remaining 18 error cleanly on features we do not implement (subqueries inDEFINE/MEASURES, two-argument aggregates such asmax_by, a few scalar functions, and theALL ROWScolumn layout above).test/trino/README.mdhas the tally and how to regenerate it. - Apache Flink — the portable cases from its
MatchRecognizeITCase(test/sql/flink_conformance.test), which add logical offsets at large indices, aggregates over expressions inDEFINE, NULL handling, and multi-key ordering with mixed directions. We agree with Flink on all of them but one, where we report an extra match that SQL:2016 requires and Flink's non-backtracking NFA misses; that case carries the analysis inline. Flink's own documentation lists the standard features it omits, several of which we support (pattern groups, alternation, anchors,SUBSET, physicalPREV/NEXT,ALL ROWS PER MATCH). - Snowflake — its documented worked example reproduced exactly, plus coverage of
the features reading those docs turned up (
test/sql/snowflake_conformance.test). Three of them are now supported:PERMUTE,MATCH_SEQUENCE_NUMBER(), andLAG/LEADas spellings ofPREV/NEXT.
Still unimplemented: pattern exclusion {- … -}, WITH UNMATCHED ROWS, and some
exotic temporal/INTERVAL type-lattice corners (route those through the explicit
type override).
cargo test --workspace # mr-core unit + property tests, mr-worker integration
cargo build --release # the worker binary
./run_tests.sh # haybarn SQLLogic end-to-end suite
./test/trino/port.sh # re-port Trino's suite (needs a Trino checkout)Get it
- Latest release —
one
.tar.gzper platform (linux_amd64,linux_arm64,osx_amd64,osx_arm64,windows_amd64, plus wasm), each with a SHA256 and a keyless cosign bundle next to it. - Container image
—
ghcr.io/query-farm/vgi-matchrecognize, multi-arch, for the HTTP/TCP transports (from v0.2.1). - The
vgiextension — the DuckDB community extension thatATTACH … (TYPE vgi, …)comes from.
This project
- CHANGELOG.md — what changed, and why.
- docs/perf-baseline.md — measured cost of every phase, peak memory by query shape, and what the tuning knobs are actually worth.
- Issues — bugs and feature requests.
- Query Farm — who builds this, and the rest of VGI.
- vgi-actions — the shared release and image-publishing workflows this repo calls.
- haybarn — the DuckDB distribution the end-to-end suite runs on (haybarn-unittest, haybarn-cli).
MATCH_RECOGNIZE in other engines
Useful when porting a query in or out — and the sources the conformance suites in
test/sql are ported from:
MIT — Copyright 2026 Query Farm LLC.
