AI-837: connector definitions API

[kanat]

T15 of AI-816: AI-837. The connector catalog over the connector_definitions table from #716: list and search the built-ins and the app's own, read one, add a custom MCP server.

How it works

flowchart LR
    B[App backend] -->|GET /v1/agents/connectors q limit cursor| L[listConnectors]
    B -->|GET /v1/agents/connectors/id| G[getConnector]
    B -->|POST /v1/agents/connectors| C[createConnector]
    L --> S[(connector_definitions newest revision per id)]
    G --> S
    C --> V{customManifest}
    V -->|"id not custom_, scheme not registered, operator client, bad scope, unknown field"| E["400 error"]
    V -->|egress.ValidatePublicHTTPSURL refuses| E
    V -->|ok| W[store.CreateConnectorDefinition next revision or same]
    W --> S
    S --> O[connectorDefinitionOf non-secret fields only]

Loading

flowchart TD
    R[POST body] --> P{"Huma schema: id matches ^custom_, name 1-120, scopes RFC 6749, no unknown field"}
    P -->|no| X[400]
    P -->|yes| Q{every scheme in Options.Connectors.Schemes}
    Q -->|no| X
    Q -->|yes| K{client.policy has operator}
    K -->|yes| X
    K -->|no| M{core.Manifest.Validate}
    M -->|no| X
    M -->|yes| N{"egress: public https, resolves to public IPs only"}
    N -->|no| X
    N -->|yes| Y["stored, 200"]

Loading

Operations

Method	Path	Client-accessible	Why
GET	/v1/agents/connectors (listConnectors)	No	The catalog is what the app's backend and dashboard pick from when they set connectors up. An end user is sent to a consent by that backend (T17), not shown a catalog. Opening it later is a one-line change
GET	/v1/agents/connectors/{id} (getConnector)	No	Same reader as the list. Another app's custom id answers 404
POST	/v1/agents/connectors (createConnector)	No	Writes app configuration

The posture is enforced by the existing middleware reading the spec. TestPostureSuite covers the three, and each also has its own assertPosture(serverOnly, …) test.

Exposed vs withheld

Built field by field in connectorDefinitionOf (internal/api/connectors.go:334). A field added to core.Manifest later stays hidden until someone adds it there.

Exposed	Withheld	Why withheld
id, revision, name, category, description, custom, created_at	endpoints, vars	Built-ins: templates the router resolves itself. Custom: an MCP URL can carry a credential in its path (some hosted MCP services embed a key there, unverified). Egress refuses a query string, so the path is the only place left for one
schemes	authorize_params, token_params	Wire details of the OAuth exchange
inputs (name, enum, pattern, default)	capture, identity	How the router recognises an account
scopes (the list)	scopes.separator, send_on_refresh, step_up_union	How scopes go on the wire
client.policy, client.auth_method, client.alg	client.env	The name of the operator's environment variables for its client id and secret
	refresh, rate_limit, sources, hooks	Runtime behaviour of the router

TestWhatTheRouterReadsToConnectIsNeverShown reads Slack's built-in, whose YAML has all of these. It checks that none of the withheld keys is in the JSON, that client has no env, and that SLACK, mcp.slack.com, slack.com/api and $.team.id appear nowhere in the body. TestACustomConnectorsEndpointIsNeverShown checks the same for a custom connector's endpoint.

Custom MCP definition

Rule	Where	Test
id ^custom_[a-z][a-z0-9_]{0,56}$, so it cannot shadow a built-in (built-ins never take the prefix, store.builtinManifests) → 400 {"error": "validation failed: expected string to be custom_ then …"}	Huma pattern, connectors.go:119	TestAnIdThatWouldShadowABuiltInIsRefused (also checks Slack is untouched)
endpoint is public https with no userinfo, query or fragment, and resolves only to public IPs	egress.ValidatePublicHTTPSURL, connectors.go:326	TestAnEndpointOnAPrivateNetworkIsRefused (127.0.0.1, 10.0.0.7, 169.254.169.254, ::1), TestAnEndpointThatIsNotPlainHTTPSIsRefused. Only IP literals, so no test resolves a name
every scheme is in the injected registry	Options.Connectors core.Registry (server.go:172), connectors.go:285	TestASchemeThisDeploymentDoesNotHaveIsRefused
client.policy cannot be operator (no operator client exists for an app's own server, and a custom definition cannot set client.env)	connectors.go:301	TestAnOperatorClientIsRefusedForACustomConnector
unknown fields (hooks, endpoints, client.env) refused	additionalProperties: false	TestAFieldTheRequestDoesNotHaveIsRefused
same body again keeps the same revision, a changed one is the next revision	store.saveRevision (#716)	TestSendingTheSameConnectorAgainChangesNothingAndAChangeIsTheNextRevision
tenant isolation: another app neither lists nor reads it, and can use the same id without touching it	store customer_id IN ('', ?)	TestAnotherAppsCustomConnectorIsNeitherListedNorRead, TestAnotherAppMayUseTheSameCustomIdWithoutTouchingThisOne

The registry is a field of the server, not a global. cmd/router is not wired: no scheme exists on accelerate yet (T9 adds oauth2_code). Until it is, every createConnector is a 400 naming the scheme, while list and get work. The tests register a name-only namedScheme("oauth2_code") (stubs_test.go) through a new RouterSuite.connectors field.

Paging

As the pagination skill says, the store change stays inside ListConnectorDefinitions (store/connectors.go:164).

• Order: built-ins, then the app's own, each by id (the order feat(connectors): connector definitions table seeded from built-in manifests (AI-833) #716 had).
• Cursor: opaque base64url of {c: custom, id}. It holds custom rather than the customer id, so it carries nothing about whose it was. Next page is (customer_id <> '', id) > (c, id).
• limit + 1 rows for has_more, no count. A bad cursor is a 400.
• q is a case-insensitive substring of id name category description, as the prototype's catalog search did (internal/api/connectors.go:72 on codex/connector-support). It is matched on the newest revision only, in an outer query over DISTINCT ON, so an old name does not match. %, _ and \ are escaped.

Tests: TestPagingWalksTheWholeListWithoutRepeatingOrSkipping (limit 2 to the end equals one page), TestTheListIsTheBuiltInsThenTheAppsOwnEachById, TestTheSearchReadsTheNewestRevisionOnly, TestASearchForAWildcardMatchesOnlyItself, TestACursorThisListDidNotHandOutIsRefused.

Values and their sources

Value	Source
name ≤ 120, category ≤ 80, description ≤ 1000	The prototype's CreateConnectorDefinitionRequest (api/openapi.yaml on codex/connector-support at cf62af0)
id ^custom_[a-z][a-z0-9_]{0,56}$ (64 max)	The prototype's customConnectorIDPattern (internal/api/connectors.go:55 there) without -, which core.Manifest.Validate refuses in an id
scope ^[!#-\[\]-~]+$	RFC 6749 §3.3 scope-token
alg RS256, PS256	core.assertionAlgs
q ≤ 120	Longest name, unverified as the right bound for search
page 25 default / 200 max	Copied from the session list (store/sessions.go:16-19), unverified for a catalog
endpoint ≤ 2048	unverified: the prototype set none
scopes ≤ 100 items	unverified: well over Slack's 29 (providers/slack.yaml)

Tests run

• ROUTER_POSTGRES_DSN=…/<db>_test ROUTER_REDIS_ADDR=… go test -tags integration ./internal/api: all suites pass, including the 27 in ConnectorsSuite and TestPostureSuite/TestTheCommittedSpecIsRenderedFromTheOperations. The same command passes for ./internal/store (TestStoreSuite) and ./cmd/router (TestOpenStoreSuite).
• go vet ./... and go test ./... in acceleration/ pass. go generate ., go build ./... and go vet ./... in sdks/go pass.
• Mutation checks, each reverted afterwards:

Mutation	Test that fails
Drop the custom_ pattern on id	TestAnIdThatWouldShadowABuiltInIsRefused: 500, not 400 (the store refuses it as an unexpected error)
Add env to ConnectorClient and copy client.env	TestWhatTheRouterReadsToConnectIsNeverShown
Store list ignores the customer	TestAnotherAppsCustomConnectorIsNeitherListedNorRead

After review

Review comment	Change	Test (fails with the change undone)
created_at of the create differed from the read	saveRevision inserts RETURNING created_at (6607b953)	TestACustomMCPConnectorIsStoredAndReadBack compares with s.Equal. It is //go:build integration, and CI's go job runs go test ./... without the tag (.github/workflows/ci.yml:82), so no CI job runs it; on macOS time.Now() has microseconds only, so it passes there either way. The guard is a manual integration run on Linux
Egress errors told an unresolvable name from a private one	Every endpoint refusal is one message (e1d93fce)	TestAnEndpointThatDoesNotResolveIsRefusedLikeAPrivateOne (.invalid, RFC 6761 §6.4, vs 10.0.0.7)
An oauth2_code definition without a client policy was stored	Refused: pickClient in schemes/oauth2code/client.go fails with ErrNoClient on an empty policy	TestAnOAuthConnectorWithoutAClientPolicyIsRefused
Repeated scope or client owner was stored	uniqueItems:"true" on both	TestARepeatedScopeOrClientOwnerIsRefused
Search matched across two fields (q=k s → slack)	Each field matched on its own	TestTheSearchMatchesTheIdNameCategoryOrDescriptionIgnoringCase (CRM Ticketing spans name and category)

Open

• The JS SDK and dashboard types are not regenerated, as AGENTS.md «SDK changes» asks (Go first). The js CI job checks npm run types -- --check, so it will report the spec as ahead, as it did on feat(conversation): ask a person before a tool call runs, on its ai_tool_call step #725. A note for the other SDKs is at the bottom of .claude/skills/sdk/SKILL.md.
• Go has the regenerated client only (ListConnectors, GetConnector, CreateConnector), no client.Connectors() resource methods yet.
• POST answers 200 both when it stores a new revision and when the newest already said the same. The store does not say which happened.
• Running the API suites against a _test database that a prototype-branch run had migrated fails: connector_definitions already exists there with the prototype's columns. A fresh _test database passes.
• No CHANGELOG entry: connectors cannot be used end to end until a scheme is registered (T9) and connections exist (T16).

🤖 Generated with Claude Code

[kanat]

One nit, inline. Otherwise this is an approve: the catalog, the custom MCP checks, and the paging held up.

TestConnectorsSuite passed on a fresh database. The js failure is the generated types lagging the spec; generated.sh reports that drift as inherited from accelerate.

[kanat]

createConnector returns the time.Now() that saveRevision stamps at acceleration/internal/store/connectors.go:242. getConnector returns the microseconds Postgres stored. This allows the two to differ by up to a millisecond.

RETURNING on that insert would make the create response the stored row. The insert itself is outside this diff.

[kanat]

Fixed in 6607b95: the insert in saveRevision now has .Returning("created_at"), so the create answers with the value Postgres stored (microseconds), the same one a later read returns. The test now compares the create answer and the read with s.Equal, without the millisecond tolerance.

Mutation check: without Returning the test fails on Linux (the answer to the create is the stored row) and passes with it. On macOS it passes either way, because time.Now() there only has microsecond resolution (Nanosecond() % 1000 is always 0 on darwin, non-zero on Linux in golang:1.27-bookworm), so CI on Linux is what catches a regression.

[kanat]

Reviewed at 6607b953. CI: js fails, the rest pass. The js failure is the generated types lagging the spec, and generated.sh reports that drift (sdks/js/src/generated/api.ts) as inherited from accelerate at 96a5904, so it is not from this PR.

I ran go test -count=1 -tags integration ./internal/api -run 'TestConnectorsSuite|TestPostureSuite' and ./internal/store -run TestStoreSuite against a fresh pr727_test database. Both pass, and all 24 ConnectorsSuite tests ran.

Two inline comments: one Question about how the created_at fix is guarded, one Nit on search.

Nit (description): the «Open» bullet saying created_at comes back to the nanosecond and that "the fix belongs in store.saveRevision, which this PR does not touch" is stale. 6607b95 adds .Returning("created_at") there, and TestACustomMCPConnectorIsStoredAndReadBack now uses s.Equal.

Verdict: approve once it leaves draft (I'm posting this as a comment because it is your own PR). Neither finding blocks.

Checked and sound:

• Paging: the cursor predicate (customer_id <> '', id) > (c, id) sits inside the DISTINCT ON and matches the outer ORDER BY cd.customer_id, cd.id. It is safe because only '' and the caller's id are in scope.
• Text search runs on the newest revision only, and %, _ and \ are escaped.
• connectorDefinitionOf copies fields one by one.
• A whitespace-only name is trimmed and then refused by Manifest.Validate (name: is empty).
• schemes: null is refused by Validate (schemes: is empty).
• None of the three operations is x-client-accessible.
• The sdk skill note is at the bottom.

[kanat]

Question: the earlier thread says CI on Linux is what catches a regression of the Returning("created_at") fix. The go job in .github/workflows/ci.yml:82 runs go test ./... without -tags integration, and this file is //go:build integration. So no CI job runs this assertion, and on macOS it passes with or without the fix. Today the regression is caught only when someone runs the integration suite on Linux by hand. Does that guard count as enough, or should the PR description say so instead of naming CI?

[kanat]

Right: connectors_test.go is //go:build integration and the go job runs go test ./... without the tag (.github/workflows/ci.yml:82), so no CI job runs this assertion, and on macOS it passes with or without the fix. My earlier reply was wrong to name CI. The PR description now says the guard is a manual integration run on Linux (section «After review»). Making it a CI guard means running the integration suites in CI, which is outside this PR.

[kanat]

Nit: because the fields are joined with a space, a query can match across the boundary between two fields. q=k s matches slack, whose joined text is slack Slack …. I checked it on the seeded built-ins:

SELECT id, concat_ws(' ', id, name, category, description) ILIKE '%k s%'
FROM (SELECT DISTINCT ON (customer_id, id) * FROM connector_definitions
      WHERE customer_id = '' ORDER BY customer_id, id, revision DESC) cd;
-- linear|f
-- slack|t

If matching only within one field is what's intended, cd.id ILIKE ?0 OR cd.name ILIKE ?0 OR … gives that. If it doesn't matter for a picker, ignore this.

[kanat]

Fixed in e1d93fc: each field is matched on its own, (cd.id ILIKE ?0 OR cd.name ILIKE ?0 OR cd.category ILIKE ?0 OR cd.description ILIKE ?0). TestTheSearchMatchesTheIdNameCategoryOrDescriptionIgnoringCase now also asserts CRM Ticketing (end of the name, start of the category) matches nothing; with the concat_ws form back it fails.

[kanat]

Reviewed at 6607b953 in ../Vision-Agents-pr-727. CI: js fails, the rest pass. generated.sh reports the sdks/js/src/generated/api.ts drift as inherited from accelerate at 96a5904, so it is not from this PR.

Ran on a fresh database: go test -count=1 -tags integration ./internal/api -run 'TestConnectorsSuite|TestPostureSuite' passes, and so does ./internal/store -run TestStoreSuite. On the first run against a database that did not exist yet, TestConnectorsSuite failed in testDatabase (router_suite_test.go:175): two parallel suites raced on CREATE DATABASE (duplicate key value violates unique constraint "pg_database_datname_index"). That helper comes from #698 on the base, not from this PR, and a rerun passes.

Three inline comments: one Should fix (the egress error tells the caller whether an internal hostname exists), one Question, one Nit.

The two unresolved threads still apply at this head:

• The CI Question on connectors_test.go:102: .github/workflows/ci.yml:82 runs go test ./... with no -tags integration, so CI does not run this file.
• The concat_ws Nit on store/connectors.go:184.

The description's «Open» bullet saying that created_at comes back to the nanosecond, and that the fix "belongs in store.saveRevision, which this PR does not touch", is still stale. 6607b95 made that fix.

Verdict: approve once it leaves draft. Nothing here blocks: no scheme is registered on accelerate, so no custom connector can be created or connected yet. The egress message is worth fixing before T9 registers one.

Checked and sound:

• None of the three operations is x-client-accessible.
• connectorDefinitionOf copies fields one by one.
• The custom endpoint is checked again at dial time: egress.NewClient → dialPublic, egress/public.go:248. A DNS rebind after create does not get past it.
• The cursor predicate sits inside the DISTINCT ON and matches the outer order.
• A {placeholder} in a custom endpoint is refused by checkTemplate, because a custom definition declares no inputs.
• The sdk skill note is at the bottom.

[kanat]

Should fix: this passes the egress error to the caller word for word. The caller can tell a name that does not resolve from one that resolves to a private address:

• egress/public.go:296: egress: endpoint host could not be resolved
• egress/public.go:300: egress: endpoint host resolved to a non-public address

On the hosted router, any app with a server-side token can send https://<guess>/mcp and learn which internal names the router's resolver knows, such as cluster service names. This handler is the only caller of ValidatePublicHTTPSURL (grep -rn ValidatePublicHTTPSURL internal), so this PR is the first to put that difference on the API.

One message for both resolution failures closes it, for example endpoint: host does not resolve to a public address. The syntax errors can keep their wording. Resolution time still differs between the two cases, but the text is the cheap part to close.

Not run: no test resolves a name, as the description says.

[kanat]

Fixed in e1d93fc: every egress refusal now answers endpoint must be a public https URL without userinfo, query or fragment, with a comment on why. TestAnEndpointThatDoesNotResolveIsRefusedLikeAPrivateOne posts https://router-probe.invalid/mcp (.invalid never resolves, RFC 6761 §6.4) and https://10.0.0.7/mcp and requires the two error bodies to be equal; passing the egress error through again makes it fail.

[kanat]

Question: when client is omitted, the definition is stored with an empty client.policy, which ConnectorClient.Policy documents as "Empty when the connector needs none". The only scheme a custom definition can name is oauth2_code, and an OAuth code exchange needs a client. If T9's oauth2_code cannot connect without a client owner, createConnector answers 200 for a definition no connection can use. This can't be settled until T9 says what oauth2_code needs. The fix would be either Manifest.Validate requiring a policy for a scheme that needs a client, or this handler refusing an empty client while oauth2_code is the only scheme.

[kanat]

Settled now that #730 is merged: pickClient in internal/connectors/schemes/oauth2code/client.go tries only the owners the policy names and returns ErrNoClient when it names none. Fixed in e1d93fc: a custom definition naming oauth2_code without client.policy is a 400 (client.policy is required with oauth2_code …). Test: TestAnOAuthConnectorWithoutAClientPolicyIsRefused; it fails with the check removed. The test helper now sends policy: [dcr] by default.

[kanat]

Nit: a repeated scope, such as ["crm.read", "crm.read"], is stored as sent, and so is a repeated client.policy owner such as ["dcr", "dcr"] (line 65). Manifest.Validate refuses a repeated scheme ("%q is listed twice", core/manifest.go) but checks neither of these. uniqueItems:"true" on Scopes and on ConnectorClient.Policy would refuse them in the schema, like the other rules here. Not run.

[kanat]

Fixed in e1d93fc: uniqueItems:"true" on CustomConnectorRequest.Scopes and ConnectorClient.Policy; openapi.yaml regenerated (Go SDK unchanged). TestARepeatedScopeOrClientOwnerIsRefused sends ["crm.read","crm.read"] and ["dcr","dcr"] and expects 400 for each; without the tags it fails.