feat(core)!: discriminated target union on relationship associations

.changeset/lazy-hounds-search.md

@@ -0,0 +1,21 @@
+---
+'@haverstack/record-adapter-sqlite': patch
+---
+
+Association query filters read through their indexes
+
+`tags`, `hasAttachment`, `attachmentFileId` and `relatedTo` were correlated
+`EXISTS` subqueries, which make SQLite scan every record and probe the association
+primary key for each. Phrased as semi-joins, the planner drives from the association
+side instead — reading the matching rows through `idx_assoc_kind_label`,
+`idx_assoc_kind_file_id`, `idx_file_refs_file_id` or `idx_assoc_related`, then looking
+up those records. The cost of an association filter becomes proportional to how many
+records match it rather than to how many the stack holds, so the gain grows with
+selectivity: the more precisely you ask, the more you save.
+
+Measured on 20k records with 4k associations, none of the four now needs a full table
+scan. `attachmentFileId` benefits most, at 8.8ms to 0.14ms, because SQLite resolves
+its two-sided condition as a multi-index OR across both indexes rather than scanning
+once and probing twice.
+
+Results are unchanged; this is the same set of records, found a different way.

.changeset/quiet-pandas-tickle.md

@@ -0,0 +1,51 @@
+---
+'@haverstack/core': minor
+'@haverstack/record-adapter-sqlite': minor
+'@haverstack/conformance-fixtures': minor
+'@haverstack/adapter-api': minor
+---
+
+Relationship associations carry a discriminated `target` instead of a bare `recordId`
+
+A relationship's target now names which identifier space its value belongs to:
+`{ scope: 'record', recordId, stackUrl? }` for a Record here or in another stack,
+`{ scope: 'entity', entityId }` for a DID, and `{ scope: 'external', ns, id }` for
+anything outside the stack — an ATProto post, an ActivityPub actor, an email address,
+a URL. Core expresses the reference and never dereferences it, so no protocol is
+privileged.
+
+The `entity` arm closes a gap in the identity model rather than only enabling external
+references: group rosters stored member DIDs in a field typed `RecordId`, and the
+permission path compared the two as plain strings. A roster entry carrying a `record`
+target now confers nothing, even when its value equals a member's DID.
+
+`RecordFilter.relatedTo` moves with it. It names a label, a target, or both, and each
+is a pattern: a bare `label` matches every target under it, and an external target with
+no `id` matches a whole namespace. A `record` target with no `stackUrl` matches only
+local targets — absence names this stack rather than acting as a wildcard. Label-only
+and namespace-wide queries were not expressible before. "Carries any relationship at
+all" is deliberately not expressible, in line with `tags` and `hasAttachment`, which
+have no match-any form either.
+
+Reference-creation gating now applies only to a relationship naming a Record in this
+stack; the other arms name nothing core can resolve, so there is no access for the
+gate to protect. The SQLite association table gains `related_scope`, `related_ns` and
+`related_stack` columns, all part of the primary key — so two copies of one record on
+two networks are two associations rather than a silent no-op. Existing stack files
+predate those columns and must be recreated.
+
+Over the wire, the relationship filter's scope is implied by which parameters appear
+(`relatedTo`/`relatedToStack`, `relatedToEntity`, or `relatedToNs`/`relatedToId`), and
+a request mixing scopes is rejected with 400. At least one is always present, so the
+filter cannot encode to an empty query string and widen the query it meant to narrow.
+
+A target names exactly one thing, exactly one way, and both halves are enforced at
+runtime rather than only by the type — a target reaching a server in a request body,
+or a filter decoded from query parameters, is a plain object the type never saw. A
+`scope` outside the three, or an empty string where a target names something, is
+rejected with `StackValidationError`; a `relatedTo` naming neither a label nor a target
+is rejected with `StackQueryError` instead of matching every Record carrying a
+relationship. This stack is named by omitting `stackUrl`, never by sending an empty
+one: storage, association identity and the filter all read absent and empty as this
+stack, and reference-creation gating now reads them that way too, so both spellings of
+a local Record require read access to it.

AGENTS.md

@@ -52,6 +52,7 @@ Design lives in [`docs/spec.md`](./docs/spec.md) and [`docs/spec/`](./docs/spec/
 - A change to observable behavior (API contract, permission rule, wire shape, error mapping) **updates the spec in the same change**.
 - Section names are load-bearing — code comments reference them as `docs/spec/<file>.md § Section`. Verify a section exists before linking to it, and update inbound references when renaming a heading.
 - Needing a long comment to explain a rule that isn't in the spec means the spec is missing a section. Add it.
+- **Spec prose states the system as it is, never how it got there.** The no-previous-implementations rule above applies to prose: "is in core now", "this used to be flat", "when #16 lands" date the document and describe a system the reader can't see. Say which layer carries what instead. This covers `docs/commons/` too — a design guide, not a changelog.
 
 ## No backward compatibility
 

CONTRIBUTING.md

@@ -187,6 +187,23 @@ A change to observable behavior — an API contract, a permission rule, a wire s
 
 When you link to a spec section from a comment, use the `docs/spec/<file>.md § Section` form and make sure the section actually exists. Section names are load-bearing; renaming a heading means updating the references to it.
 
+**Prose describes the system as it is, not how it got there.** The [no-references-to-previous-implementations](#comments) rule is not only about code comments — it is the same rule, and it bites harder in the spec, because a spec is read by people deciding what to build against. "Relationship targets are in core now", "`relatedTo` used to require a `recordId`", "reconciliation is expected when #15 lands" all date the document and describe a system the reader cannot see. They also rot silently: the sentence stays true-sounding long after the change it refers to stopped being news.
+
+```md
+<!-- Don't: -->
+
+The reference fabric this rests on is in core now. What remains outstanding is
+the bridge.
+
+<!-- Do: -->
+
+Core carries the reference fabric: relationship targets that name a record in
+someone else's stack, and a `relatedTo` filter that queries them. A bridge
+carries its own translation table.
+```
+
+This applies to the Schema Commons ([`docs/commons/`](./docs/commons/)) as much as to `docs/spec/` — a design guide describing what a type means, not a record of how the type arrived at that meaning. Issue numbers in prose are usually the same mistake wearing a different hat: `#204` tells a reader nothing they can act on, and a document that sequences itself against open issues stops being true the moment one closes. Where the reasoning matters, state it.
+
 ---
 
 ## No backward compatibility yet

README.md

@@ -201,7 +201,14 @@ Tags, attachments, and relationships are unified under a single model:
 ```ts
 { kind: 'tag',          label: 'favourite' }
 { kind: 'attachment',   label: 'avatar',   fileId: '...' }
-{ kind: 'relationship', label: 'reply-to', recordId: '...' }
+{ kind: 'relationship', label: 'reply-to', target: { scope: 'record', recordId: '...' } }
+```
+
+A relationship's `target` says which identifier space its value lives in — a Record here or in another stack (`{ scope: 'record', recordId, stackUrl? }`), a "who" as a DID (`{ scope: 'entity', entityId }`), or something outside the stack entirely (`{ scope: 'external', ns, id }`). That last one is how a record points at an ATProto post, an ActivityPub actor, an email address or a plain URL: Haverstack expresses the reference and never dereferences it, so no protocol is privileged.
+
+```ts
+{ kind: 'relationship', label: 'syndicated-to',
+  target: { scope: 'external', ns: 'atproto', id: 'at://did:plc:abc/app.bsky.feed.post/3k4' } }
 ```
 
 ### Migrations

docs/commons/README.md

@@ -131,27 +131,48 @@ these.
 ## Cross-type conventions
 
 Some semantics belong to no single type. These association labels carry commons
-authority on **any** record, of any type:
+authority on **any** record, of any type. Relationship labels are orthogonal to the
+target arm they ride on: a label says what the reference _means_, a target says which
+identifier space it lives in, and every label below works with any arm that makes
+sense for it.
 
-- **`location`** — `{ kind: 'relationship', label: 'location', recordId: <place> }`
+- **`location`** — `{ kind: 'relationship', label: 'location', target: { scope: 'record', recordId: <place> } }`
   points at a [`place`](./place.md) record: the geotagged photo, the note written at a
   café, the check-in. Apps that understand places understand every located record for
   free, whatever its type.
 - **`embed`** — `{ kind: 'attachment', label: 'embed', fileId }` marks a file
   referenced from a record's body text (`note`, `article`, `page`, `message`). How the
   body refers to the embed is app territory in v1; a commons syntax is an expected
   follow-up proposal.
-- **`series`** — `{ kind: 'relationship', label: 'series', recordId }` groups records
+- **`series`** — `{ kind: 'relationship', label: 'series', target: { scope: 'record', recordId } }` groups records
   that are occurrences of one recurring thing (materialized [`event`](./event.md)
   occurrences are the motivating case). Reserved now so recurrence proposals build on
   it rather than around it.
-- **`author`** — `{ kind: 'relationship', label: 'author', recordId: <contact> }`
-  attributes any record to a person the stack knows, pointing at a
-  [`contact`](./contact.md) (or `_entity`) record. Complements, never replaces, a
-  displayed-byline string like `article.author`: the string is what the work says
-  about itself (a property, faithful to the work as published); the association is
-  what _you_ know about the world. Multiple `author` associations express
-  co-authorship. Prior art: ActivityStreams `attributedTo`.
+- **`author`** — attributes any record to a person the stack knows. Two forms, and the
+  target arm is what distinguishes them: an `entity` target
+  (`{ scope: 'entity', entityId: <DID> }`) names the identity itself, while a `record`
+  target pointing at a [`contact`](./contact.md) or `_entity` record names your card
+  about them. Complements, never replaces, a displayed-byline string like
+  `article.author`: the string is what the work says about itself (a property, faithful
+  to the work as published); the association is what _you_ know about the world.
+  Multiple `author` associations express co-authorship. Prior art: ActivityStreams
+  `attributedTo`.
+- **`alias`** — `{ kind: 'relationship', label: 'alias', target: { scope: 'external', ns, id } }`
+  on an `_entity` record records that this identity is also known by that identifier
+  elsewhere: `{ ns: 'atproto', id: 'did:plc:…' }`, `{ ns: 'activitypub', id: <actor URL> }`,
+  `{ ns: 'email', id: 'alice@example.com' }`. It is the resolution primitive — an
+  inbound record whose author arrives as a foreign identifier is matched back to a
+  known entity with one indexed `relatedTo` query — which is why it is an association
+  and not a content field: array fields are opaque to the query engine, so the same
+  list inside `_entity.content` would force a scan of every entity record. Machine
+  identifiers only; a person's own words about someone belong on a `contact`.
+- **`syndicated-to`** — `{ kind: 'relationship', label: 'syndicated-to', target: { scope: 'external', ns, id } }`
+  records that a copy of this record was published elsewhere. The canonical copy stays
+  in the stack; a bridge publishes and stamps the copy's address here. Asking a
+  namespace without an id ("everything already on ATProto") is the inventory query a
+  syndication tool runs. Where a copy has a lifecycle of its own — retraction state, a
+  remote content address, which account it went out from — that is a record of the
+  bridge's own type, related back to this one; the label alone carries only the link.
 
 Cross-type conventions are governed like fields: proposing one is proposing it for
 every record in every stack, so the bar is correspondingly higher.
@@ -233,9 +254,8 @@ concrete intended writer exists — the group cluster graduates when a group-too
 or demo is real, building on the grant/group primitives (`_group`, type-level grants)
 documented in the identity and access-control specs.
 
-Deliberately absent from the initial set: `post` (public social shapes are reconciled
-with the ATProto-compat RFC, #15 — `message` is the group-scoped shape, not the social
-one), recurrence rules (see `event`: occurrences are materialized in @1),
+Deliberately absent from the initial set: `post` (the broadcast contract — `message` is
+the group-scoped shape, not the social one), recurrence rules (see `event`: occurrences are materialized in @1),
 `file`/`document` (a first-class `file` type is expected to follow `photo`'s pattern;
 until a real writer needs it, a record plus attachment covers it), and `checkin`
 (subsumed by the `location` cross-type convention plus any record).

docs/commons/article.md

@@ -51,8 +51,9 @@ await stack.defineType('org.haverstack/article@1', 'Article', {
   the canonical location back to the record). `url` present therefore means "this work
   lives somewhere on the web" — symmetrically for authored and captured articles — and
   any app can render a "view published" link. Syndicated copies elsewhere never
-  overwrite it: canonical means canonical; other locations are sidecar territory (or
-  future syndication relationships, #15).
+  overwrite it: canonical means canonical, and a copy's location is a `syndicated-to`
+  relationship with an external target — see
+  [Cross-type conventions](./README.md#cross-type-conventions).
 - **Cover image**: attachment association with label `cover` (JSON Feed `image`,
   h-entry `u-featured`).
 - **Embedded media** in `text`: attachment associations with label `embed`, per the
@@ -101,9 +102,8 @@ keep the paired bookmark record described above.
 - Layout, theme, rendering hints — presentation is the publishing app's concern.
 - `slug` and site-structure fields — that is [`page`](./page.md) territory; an article
   is a work in a feed, a page is a node in a site tree.
-- Social/microblog `post` semantics (replies, reposts, mentions) — deferred to
-  reconciliation with the ATProto-compat RFC (#15). An article is a work; a post is an
-  utterance.
+- Social/microblog `post` semantics (replies, reposts, mentions) — a separate contract.
+  An article is a work; a post is an utterance.
 - Comments — already covered, no new type: a comment is a [`message`](./message.md)
   whose `parentId` is the article (sent into a shared space, moment-indexed,
   tamper-evident), while a reader's private marginalia is a `note` with an `about`

docs/commons/contact.md

@@ -45,9 +45,13 @@ await stack.defineType('org.haverstack/contact@1', 'Contact', {
 **Labels** (vCard's `TYPE` parameter, humanized): an open vocabulary of user-facing
 words. Well-known values: `"home"`, `"work"`, `"mobile"`. Unknown labels are displayed
 verbatim — they are the user's words ("boat phone" is legal and correct), not a
-machine namespace. Absent means unspecified. This is deliberately the same
-labeled-multi-value shape as #15's revised `externalIds` on `EntityContent`, differing
-only in that `label` is for humans where `ns` is for machines.
+machine namespace. Absent means unspecified.
+
+These are **directory data the user typed**, not machine identifiers. The resemblance
+to a relationship's `{ scope: 'external', ns, id }` target is superficial and the two
+do different jobs: an `alias` relationship is what an app resolves an inbound record's
+author _through_, while `contact.emails` is what a person reads. A contact that is also
+a known principal carries both — see the conventions below.
 
 Note that array fields are opaque to the query engine; apps filter contacts by `name`
 or via associations, not by address.
@@ -56,6 +60,10 @@ or via associations, not by address.
 
 - **Avatar**: attachment association with label `avatar` — the spec's own example of an
   attachment association, honored here.
+- **Alias identifiers**: a machine identifier for someone in another system —
+  `did:plc:…`, an ActivityPub actor URL, an npub — is an `alias` relationship with an
+  external target, on the `_entity` record rather than here. See
+  [Cross-type conventions](./README.md#cross-type-conventions).
 - **Contact ↔ entity linkage**: when a contact _is_ a known principal (they appear in
   grants, groups, or authorship), link the records with
   `{ kind: 'relationship', label: 'entity', recordId: <_entity record id> }` on the

docs/commons/message.md

@@ -12,9 +12,8 @@ concerns and have no representation here.
 
 **This is not the social post.** A `message` is addressed to a group by living in that
 group's stack; a social post is broadcast to the world under a public identity. The
-social shape (reposts, mentions, public replies) belongs to the ATProto-compat RFC
-(#15) and is still deferred; this type must not foreclose it, and reconciliation is
-expected when #15 lands.
+broadcast shape is its own type — see [Choosing a text type](./text-types.md) — and
+nothing here forecloses it: the two differ in contract, not in fields.
 
 Messages are **sent** — speech addressed to others, whose meaning is indexed to its
 moment and thread. For the boundary with `note` (kept) and `article` (published), see
@@ -84,7 +83,7 @@ discussion.
 
 ## Deliberately excluded
 
-- Social-post semantics — #15 territory (see above).
+- Social-post semantics — the broadcast contract's territory (see above).
 - Reactions — a future micro-proposal (likely tag associations by non-authors — which
   needs finer-grained association permissions than today's `update-own`/`update-any`
   grant actions provide).

docs/commons/place.md

@@ -45,7 +45,7 @@ await stack.defineType('org.haverstack/place@1', 'Place', {
   "I was here," addressed to no one — is a (possibly empty-text) `note` with a
   `location` association, its `createdAt` the check-in time; "I'm at the café, come
   join me" sent to a group is a `message` with one; the Foursquare-style public
-  check-in is the future broadcast `post` (#15) with one. The `location` association
+  check-in is the future broadcast `post` with one. The `location` association
   is the invariant; the contract varies with the act.
 - **Deduplication** is app-side: two records with the same coordinates are two records.
   An app importing venues should query by exact coordinates before creating.