feat(sep-xxxx): Conformance suite for triggers and events

[panyam]

Draft, and it must not merge as it stands. MCP Events has no SEP number, so every check id here carries a placeholder sep-9999- prefix. Renaming it is a merge blocker rather than a follow-up: plan.modelcontextprotocol.io builds from main, so a draft is invisible to it, but a merge would publish SEP 9999 as though it were real. Opening it now to get the extraction and the scenario shape reviewed while that number is settled. See The number question below, which is the one decision I would most like input on.

What changes

Adds a requirement-traceability yaml for MCP Events and the first two of five server scenarios, scoring an implementation against the design sketch that merged on main of modelcontextprotocol/experimental-ext-triggers-events on 2026-09-08. events-discovery grades the capability declaration, events/list and the error-code contract. events-poll grades poll delivery, the EventOccurrence shape and cursor lifecycle. Together they emit 45 of the 131 declared rows; push and webhook follow in later phases and report untested until then.

Provenance

panyam offered an Events conformance suite in #triggers-events-wg on 2026-08-28 and Peter Alexander accepted on 2026-08-31 ("No objections, that would be great!"). This is the first installment. It follows the SEP-2640 suite (#330) in shape: extract the requirements into a traceability yaml first, then write scenarios that emit one check per row.

Per AGENTS.md this would normally start as an issue. The WG conversation is that discussion, and the extraction is easier to argue about as a diff than as prose, so it is here as a draft instead. Happy to split it into an issue if you would rather.

Prerequisite knowledge

• Events design sketch — the document every row is extracted from. The Summary and Cursor Lifecycle sections carry most of what events-poll tests; Subscribing and Event Delivery is the densest normative section and is mostly still ahead of us.
• AGENTS.md § Scenario design and § Check conventions — why this is two scenarios with 45 checks rather than 45 scenarios, and why a missing prerequisite fails rather than skips (Standardize how scenarios report setup/execution failures #248).
• src/traceability/ — how a declared row becomes tested or untested. This is what makes declaring all 131 rows up front the right move rather than an overreach.
• src/seps/sep-2640.yaml — the closest precedent for the yaml's shape and for how much provenance belongs in its header.

The number question

This is the part I would most like a decision on, because it is the only thing blocking merge.

SEP numbers are PR numbers in modelcontextprotocol/modelcontextprotocol, and Events has no PR there — the work lives in its own repository. Both traceability gates are numeric: the directory filter at src/traceability/index.ts:174 (/^sep-\d+\.yaml$/) and the check-id filter at line 41 (/^sep-\d+-/). A file named events.yaml is silently dropped from the manifest rather than rejected, and src/new-sep/index.ts:163 refuses any argument that is not a positive integer. So there is no honest name that works today.

Three ways out, and I have no strong preference between the first two:

1. Reserve a number. Cleanest, and it makes this mergeable as-is after a mechanical rename.
2. Let the traceability layer carry pre-SEP extensions. Relax both regexes to accept a non-numeric key, so work like this can be tracked before it has a number. Wider change, but Events will not be the last extension in this position.
3. Keep it unnumbered and drop the yaml. Loses the traceability artifact, which is most of the value.

I used 9999 to unblock the extraction, deliberately far from the live range. The rename touches the filename, the sep: field, every sep-9999- id in the yaml, and the same ids under src/scenarios/server/events/ — mechanical, but wide enough that it is worth doing once, in the right direction.

Reviewer's guide

A restaurant can cook a dish that is not on the menu. The kitchen has the ingredients, the line knows the recipe, and if you somehow know to ask you get it — but nobody reading the menu ever orders it, so from the outside it may as well not exist. That is the state this suite found its first implementation in: events/list answers, all three delivery modes work, and nothing appears under capabilities.events, which is the menu a spec-following client reads before deciding what to ask for. This matters for how the suite is shaped, because the obvious design misses it. A conformance harness normally treats an undeclared optional capability as not-applicable and skips, which is right for a restaurant that genuinely does not serve the dish and wrong for one that serves it off-menu: the skip reports a clean run over a surface no client can reach. So the capability gate here asks before it skips — it calls events/list, and only a server that both declares nothing and implements nothing gets the skip. The same instinct runs through the rest of the diff. Wherever a check cannot be exercised, the scenario names the missing prerequisite and fails, because the one thing worse than a red conformance run is a green one that tested nothing.

Read in this order:

1. src/seps/sep-9999.yaml — start here, header first. It covers the placeholder, how 144 keyword occurrences became 131 declared checks and 30 excluded rows, and which rows this phase emits. The rows themselves are skimmable once the header lands.
2. src/scenarios/server/events/helpers.ts — the shared vocabulary. Note that EVENTS_EXTENSION_ID is only a suite-selection key. Events declares its capability top-level, not under capabilities.extensions, unlike every other extension here, so that constant is never a path into the capability object.
3. src/scenarios/server/events/discovery.ts — the off-menu gate is at the top of checks(). Everything after it grades whatever the server actually serves, field by field.
4. src/scenarios/server/events/poll.ts — the interesting part is quietAdvanceChecks: two polls with the first response's cursor fed into the second, which is what makes cursor advancement gradeable against a server where nothing is happening. That is the state a conformance fixture is usually in.
5. src/scenarios/server/events/negative.test.ts — acceptance for all of the above. Every case pairs a conformant fixture with one broken in exactly one way.
6. src/scenarios/index.ts, src/types.ts — registration and the extension id. Mechanical.

How it works

discovery:
  declared, value <- capabilities.events
  if not declared:
      probe <- events/list
      if probe is -32601:  skip everything          # genuinely does not do events
      else:                fail the declaration check, keep grading
  grade events/list, then each descriptor field across all descriptors,
  reporting the first offender per field so check ids stay stable regardless
  of how large a catalog the server serves
  probe one unknown event name -> grades NotFound and the server-range rule

poll:
  pick the first descriptor advertising "poll"; none -> untestable, not skip
  p1 <- poll(cursor=null)        # bootstrap: must replay nothing
  p2 <- poll(cursor=p1.cursor)   # quiet period: a cursor must still come back
  p3 <- poll(no cursor field)    # absent must be accepted as null
  p4 <- poll(maxEvents=1)        # batch must be capped
  p5 <- poll(cursor=p1.cursor, maxAgeMs=300000)
  p6 <- poll(cursor=null, maxAgeMs=300000)   # must be ignored, replay nothing
  grade EventOccurrence only if some response actually carried an event

The truncated rows that need a stale cursor are reported untestable rather than skipped. Cursors are opaque by definition, so the harness cannot mint one old enough to fall outside a retention window, and saying so is more useful than a silent pass.

Decision log

• All 131 rows are declared now, though only 45 are emitted. An undeclared requirement is invisible; an undeclared-and-unemitted one is indistinguishable from a requirement nobody noticed. Declaring the full set makes the 86 outstanding rows surface as untested, which is the manifest working rather than a gap in this file.
• Pure MAY and OPTIONAL sentences get no row. Five initially got one and were demoted to excluded: rather than deleted, so the keyword sweep stays auditable against the source.
• Client, host, receiver and SDK-guidance obligations are excluded rather than declared. Roughly a fifth of the normative sentences in the document address the client or the callback endpoint. Declaring them would inflate the denominator with rows no server-side scenario could ever emit.
• 86 of the 131 rows quote no RFC 2119 keyword. They come from the EventOccurrence field table, the error-code table and the descriptor fields, which the sketch states declaratively. They are normative and directly testable. Since "severity follows the keyword" is undefined for them, the yaml header states the rule used: a field the document marks required, or a wire fact a client cannot work without, is FAILURE; anything else drawn from an example payload is WARNING.
• Two scenarios now, five eventually, rather than one big one. A server may implement any non-empty subset of the three delivery modes, so poll-only and push-only servers are both conformant and grading them together would misreport each. Webhook splits further, because its delivery half needs a callback URL the server under test can reach over https — which localhost cannot satisfy, since the same SSRF rules the suite checks require a conformant server to refuse it.
• Negative tests use in-process mock servers rather than a new example server. Matching src/scenarios/server/skills/negative.test.ts. An events-capable fixture in examples/servers/typescript/ is real work and belongs with the push and webhook scenarios, which genuinely need one.
• The sketch is not a spec diff, so new-sep's PR lookup does not apply and specPathToUrl (src/new-sep/index.ts:19) hard-requires a docs/specification/draft/ prefix that does not exist here. Scaffolded with --spec-url. There are no section anchors to cite, so rows carry no per-row url and spec_url names the document itself.

Risk / blast radius

• Affects: additive. Two new scenarios, one new yaml, one new entry in EXTENSION_IDS. No existing scenario, suite list or check id changes.
• Not selected by default. source is { extensionId }, so matchesSpecVersion returns false for every --spec-version and these never join a dated run. They are also in the pending list. Reaching them takes --scenario events-* or --suite all.
• Tests covering this: src/scenarios/server/events/negative.test.ts, 29 cases. Full suite 651 pass across 48 files.
• Be paranoid about: the capability gate. It is the one place this suite deliberately departs from how every other extension scenario here behaves, and it is load-bearing for the headline finding. If you think an undeclared-but-serving server should skip, say so — that choice is the difference between a green run and a red one. There is an inline comment on it.
• src/seps/traceability.json will drift once these ids are emitted in a real run. Per AGENTS.md the traceability workflow refreshes it by PR and it is not a gate, so this leaves it alone.

Before / after

There was no Events suite before, so what matters is what an implementation looks like when scored. Run against mcpkit's examples/events/kitchen-sink at its main:

events-discovery   Passed: 8/11, 3 failed, 1 warnings
events-poll        Passed: 18/28, 10 failed, 5 warnings

Six divergences, each observed against the running server rather than inferred from source:

Check	What the server does
sep-9999-capability-events-object	answers events/list while declaring no capabilities.events
sep-9999-descriptor-input-schema	no descriptor carries inputSchema
sep-9999-descriptor-delivery-subset	events.topology sends delivery: null where an array is required
sep-9999-poll-events-array	omits events instead of returning an empty array
sep-9999-poll-mode-unsupported	answers events/poll for a type advertising only push/webhook
sep-9999-poll-next-poll-ms	sends nextPollSeconds, the name spec commit 197c32b4 retired

Five of the six were not previously tracked by that implementation. They are here as evidence the checks catch something rather than as the point of the PR — the SEP-2640 experience was that running a suite against an implementation nobody wrote it for is what finds the suite's own bugs, and a second implementation is next.

flowchart TD
    A[server under test] --> B{capabilities.events declared?}
    B -->|yes| G[grade everything]
    B -->|no| C{events/list answers?}
    C -->|"-32601"| D[SKIP all: does not do events]
    C -->|returns a catalog| E[FAIL the declaration check]
    E --> G
    G --> H{check exercisable?}
    H -->|yes| I[SUCCESS or FAILURE]
    H -->|no| J[untestable: FAIL and name the prerequisite]

Loading

Out of scope

• events-push, events-webhook, events-webhook-delivery — later phases. Their 86 rows are declared and report untested.
• A --header option on the server runner. Needed to score a second implementation that sits behind OAuth, which the runner currently cannot reach at all. Its own PR.
• Renaming sep-9999-* once the number question above is settled. Merge blocker.

[pkg-pr-new]

Open in StackBlitz

npx https://pkg.pr.new/@modelcontextprotocol/conformance@504

commit: cb0f5af

[panyam]

Flagging this deliberately, because it is the one place this suite departs from how every other extension scenario in the repo behaves, and the departure is what turns the run against mcpkit red rather than green.

Everywhere else here, an optional capability the server never declared is a clean SKIP. That is correct when the server genuinely does not implement the feature. It is wrong when the server implements it and never says so: mcpkit answers events/list, serves all three delivery modes, and declares nothing under capabilities.events, so a client that reads capabilities to decide what to call never reaches any of it. A SKIP reports that as a clean run over a surface no client can use.

So the gate asks before it skips. One extra events/list separates "does not do events" (-32601, skip everything) from "does events off-menu" (returns a catalog, fail the declaration check and keep grading the rest).

The precedent is declaredSkillsCapability in src/scenarios/server/skills/helpers.ts, split out from skillsCapability for the same reason: a malformed declaration was being folded into "not declared" and skipping the whole suite, which "reads as a clean run against a server that is plainly wrong". This applies that argument to an absent declaration rather than a malformed one.

The cost is one extra request against servers that do not implement events at all. If you would rather pay nothing there and accept the false SKIP, say so, but the finding seems worth the request.