Add unlisted records — enumeration without permissions

.changeset/unlisted-records.md

@@ -0,0 +1,35 @@
+---
+'@haverstack/core': minor
+'@haverstack/sqlite-shared': minor
+'@haverstack/record-adapter-sqlite': minor
+'@haverstack/adapter-local': minor
+'@haverstack/wire-types': minor
+'@haverstack/adapter-api': minor
+'@haverstack/conformance-fixtures': minor
+---
+
+Add an `unlisted` state for records — reachable by ID, absent from enumeration by default.
+
+`StackRecord.unlistedAt` is a native field, orthogonal to `permissions`: it says nothing
+about who may read a record, only whether it is enumerable. A record with `unlistedAt` set
+is reachable by `get()` for anyone who may already read it, and excluded from an unfiltered
+`query()` and the change feed by default — the same posture soft delete already has.
+
+- `stack.create(typeId, content, { unlisted: true })` creates a record already unlisted, so
+  there is no window where it exists and is briefly enumerable.
+- `stack.setUnlisted(id, unlisted)` toggles it on an existing record, gated exactly like
+  `setPermissions()` under `ScopedStack` — both decide who can discover a record, not merely
+  read one already found.
+- `RecordFilter.includeUnlisted` and `SubscribeOptions.includeUnlisted` opt a query or
+  subscription back in. Unlike `includeDeleted`, `includeUnlisted` is refused to everyone but
+  the stack owner acting alone under `ScopedStack` — enumeration standing rests on nothing but
+  ownership, so no grant or delegation carries it.
+- The change feed matches `query()`'s exclusion, with one exception: marking a record unlisted
+  emits a dedicated `unlist` op (kind `deleted`) so a subscriber that already knows the record
+  is told to drop it; relisting emits `list` (kind `changed`), an ordinary upsert like
+  `undelete`. Every other transition — created unlisted, an edit while already unlisted, a
+  purge of a record that was never listed — needs no special-casing, since it falls out of
+  checking the record's current state.
+
+See docs/spec/access-control.md § Unlisted records and docs/spec/events.md § The unlisted
+transition.

docs/spec.md

@@ -48,7 +48,7 @@ stack.timezone; // from adapter.timezone — string | undefined
 
 `LocalAdapter.initialize()` fails if the file already exists. `LocalAdapter.open()` fails if the file does not exist. This makes the distinction explicit and prevents silent config divergence.
 
-**`StackClient` is the passable interface.** Plugin and extension code that doesn't need to know the underlying backend should accept `StackClient` rather than the concrete `Stack` or `ScopedStack`. It covers the full record API (`create`, `get`, `query`, `update`, `delete`, `undelete`, `associate`, `dissociate`, `setPermissions`, `getVersions`, `getVersion`, `restoreVersion`, `getAttachment`, `putAttachment`, `deleteAttachment`, `collectAttachmentGarbage`) plus a `features` getter. Both `Stack` and `ScopedStack` implement it.
+**`StackClient` is the passable interface.** Plugin and extension code that doesn't need to know the underlying backend should accept `StackClient` rather than the concrete `Stack` or `ScopedStack`. It covers the full record API (`create`, `get`, `query`, `update`, `delete`, `undelete`, `associate`, `dissociate`, `setPermissions`, `setUnlisted`, `getVersions`, `getVersion`, `restoreVersion`, `getAttachment`, `putAttachment`, `deleteAttachment`, `collectAttachmentGarbage`) plus a `features` getter. Both `Stack` and `ScopedStack` implement it.
 
 ### The `_config` record
 

docs/spec/access-control.md

@@ -75,6 +75,75 @@ What this buys is an invariant the mutate gate already assumed: **anything that
 
 What is _not_ offered is blind mutation of an existing Record. `update()` is a merge patch defined over content the requester would not be able to see, `ifVersion` needs a version number that comes from a read, and the write bit's recoverability argument requires prior content by construction. A write-only surface is a total write; nothing in this model is one.
 
+## Unlisted records
+
+`unlistedAt` is a native field, orthogonal to `permissions`: it says nothing about who may read a Record, only whether it is enumerable. A Record with `unlistedAt` set is reachable by `get()` for anyone who may already read it, and absent from an unfiltered `query()` and the change feed by default — a signal for content that is genuinely public where the _location_ is what's being withheld (a bonus post for feed subscribers, a superseded page kept alive for old links), never a substitute for `permissions` on content that must stay unreadable.
+
+```ts
+type StackRecord = {
+  // ...
+  unlistedAt?: Date; // Present if withheld from enumeration
+};
+
+type RecordFilter = {
+  // ...
+  includeUnlisted?: boolean; // Unlisted Records are excluded by default
+};
+```
+
+**Stated plainly, because the name will otherwise over-promise:**
+
+> Unlisted withholds a Record from enumeration and announcement. It never withholds the Record. A requester who may read it and holds its ID gets it. A requester without the ID has no supported way to discover it.
+
+That sits inside the same threat model as [record IDs being guessable](#errors-and-information-exposure) — "refuse to confirm a candidate" is the existing posture, and unlisted is that posture applied to discovery rather than to a single ID.
+
+### Three tiers, not two
+
+The reach question access control usually asks is binary — enforced or not — but enumeration has a real middle tier, and `unlistedAt` occupies it rather than inventing a softer word for "advisory":
+
+|               | Behavior                                   | Occupants                            |
+| ------------- | ------------------------------------------ | ------------------------------------ |
+| **Enforced**  | Refused regardless of what the caller asks | `permissions`, grants                |
+| **Defaulted** | Refused unless the caller asks             | `deletedAt`, `unlistedAt`, `_config` |
+| **Advisory**  | Always returned; the consumer decides      | A tag convention                     |
+
+A consumer that has never heard of `unlistedAt` gets correct behavior by default — an unfiltered `query()` excludes it, the same posture as `deletedAt` and `_config`. That default is what makes the field real rather than a documentation-only convention: nothing about it depends on every consumer choosing to respect it.
+
+### Setting it
+
+```ts
+await stack.create(typeId, content, { unlisted: true }); // created already unlisted — no window where it's briefly enumerable
+await stack.setUnlisted(recordId, true); // withhold an existing Record
+await stack.setUnlisted(recordId, false); // relist it
+```
+
+`setUnlisted()` is gated exactly like [`setPermissions()`](#the-write-bit-a-recoverability-trust-model) under `ScopedStack` — owner-or-creator, asked of both identities under delegation — because both decide who or what can _discover_ a Record rather than merely read one already found. `_group` Records follow the same admin-or-owner rule `setPermissions()` uses there too. No-op if the Record is already in the requested state.
+
+### `includeUnlisted` is owner-only
+
+Unlike `includeDeleted` — which any `ScopedStack` requester may pass, since a soft-deleted Record's own `permissions` still gate whether they can see it — **`includeUnlisted` is refused to everyone but the owner acting alone**, on both `query()` and `subscribe()`:
+
+```ts
+stack.query({ filter: { includeUnlisted: true } }); // plain Stack: honored
+scoped.query({ filter: { includeUnlisted: true } }); // ScopedStack, non-owner: StackPermissionError
+```
+
+Enumeration standing rests on nothing but ownership. A grant conveys reach over specific Records or a type family; it says nothing about whether the requester should see the stack's _entire_ enumeration surface, unlisted Records included — so no grant, and no delegation, carries the flag. An owner principal acting for a visitor through the owner's own server does not lend that visitor the flag either, for the same reason delegation carries none of the [owner-acting-alone verbs](#delegation-principal-and-subject). The flag is refused outright rather than silently dropped: a caller that believes it captured the full enumeration and silently got the filtered one is worse off than one that was told no — the same reasoning [create-time `permissions`](#delegation-principal-and-subject) is refused under delegation rather than quietly stripped.
+
+### The feed matches `query()`
+
+An unlisted Record that emits a change event to a default subscriber is not unlisted — so `subscribe()`'s default exclusion and `includeUnlisted` opt-in mirror `query()`'s exactly, including the owner-only gate on the opt-in. The one wrinkle is the transition itself: marking a Record unlisted must still reach a subscriber who already knows it, so it can drop its copy, even though the Record's new state would otherwise fail that same exclusion. See [Change events § The unlisted transition](./events.md#the-unlisted-transition) for the full transition table and the `list`/`unlist` change ops.
+
+### What this is not
+
+**Not a fourth `Permission` variant.** `Permission` is a union over mutually exclusive answers to "who may read this"; `unlistedAt` is orthogonal to that question, not another answer to it, and composes with any permission tier — public-and-unlisted (a bonus post) and owner-only-and-unlisted are both coherent, meaning different things.
+
+**Not per-audience.** There is no `Listing[]` parallel to `Permission[]` — enumeration does not vary by who is asking, the way reach does. A record is unlisted for everyone or for no one; if a future need for audience-varying enumeration arises, that is new surface, not a reinterpretation of this field.
+
+**Not a query filter for the excluded half.** `RecordFilter` has no negation, so `includeUnlisted: true` returns _both_ listed and unlisted Records together — there is no "unlisted only" filter. A consumer that needs to tell them apart checks `unlistedAt` on the results it gets back.
+
+`ScopedStack.query()`'s `total` excludes unlisted Records from the count the same way it excludes everything else the requester can't see — a count that included what the exclusion just hid would leak the fact being withheld.
+
 ## Type-level grants
 
 A Grant authorises one or more Entities to perform specific actions on Records of a given Type, without touching individual records — a `read-any` grant on `comment@1` makes all comments of that type readable by the grantee without setting `permissions` on each one. Grants are modeled as Records of the built-in system type `_grant`, making them queryable, versioned, and subject to the same lifecycle as any other Record.
@@ -208,6 +277,7 @@ Read in the other direction, an **owner principal** acting for someone else —
 
 | Verb                                                          | Why delegation doesn't carry it                                                                                      |
 | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
+| `includeUnlisted` on `query()`/`subscribe()`                  | Enumeration standing rests on nothing but ownership — see [Unlisted records](#unlisted-records)                      |
 | Hard delete                                                   | Irreversible; the subject holds soft delete already                                                                  |
 | `deleteAttachment()`, `collectAttachmentGarbage()`            | Irreversible, and neither takes a Record to gate on                                                                  |
 | Unstripped snapshot `permissions`                             | Discloses the stack's sharing graph                                                                                  |

docs/spec/data-model.md

@@ -24,6 +24,7 @@ type StackRecord = {
   updatedBy?: string; // Who performed the most recent mutation. Unlike entityId, it moves with every write (see Authorship and attribution)
   updatedVia?: string; // The principal behind that mutation, when it isn't updatedBy
   deletedAt?: Date; // Present if soft-deleted
+  unlistedAt?: Date; // Present if withheld from enumeration — reachable by get(), absent from query()/the feed by default (see Access control)
   permissions?: Permission[]; // Access control (see Access control)
   associations?: Association[]; // Tags, attachments, relationships
 };
@@ -214,7 +215,7 @@ The migration registry is **per-stack-instance** — different stacks can be at
 - **`presentAt: 'latest'`** — an explicit opt-in on both `get()` and `query()` that applies the registered migration chain in memory before returning. Nothing is written to disk; this is a read-time convenience, never a persistence mechanism. Throws `StackMigrationError` when a matched Record's version can't be reconciled with what this app instance has registered (see stale-writer behavior below).
 - **`update()` never migrates.** It validates the merge-patched content against the Record's _own current_ stored Type — never the latest — and writes back at the same `typeId`. An unrelated content edit can never fold an invisible schema rewrite into the same version-history entry.
 - **Path composition** — migrations between adjacent versions are automatically chained (v1→v2→v3), so apps only ever register one step at a time.
-- **`migrateAll("com.example.myapp/note")`** eagerly commits all pending migrations for a type family in one deliberate pass — call it at app startup after registering migrations, or after a schema change. It sweeps soft-deleted Records unconditionally (`includeDeleted` is not a caller option in either direction — see [Deletion](./versioning.md#deletion)), validates each migrated result against the target Type's schema before writing, and aborts immediately on the first validation failure (a buggy migration function is a bug to surface, not to paper over by skipping the offending records) — anything already committed earlier in the pass stays committed. Previous content is snapshotted to version history before each write.
+- **`migrateAll("com.example.myapp/note")`** eagerly commits all pending migrations for a type family in one deliberate pass — call it at app startup after registering migrations, or after a schema change. It sweeps soft-deleted and unlisted Records unconditionally (`includeDeleted`/`includeUnlisted` are not caller options in either direction — see [Deletion](./versioning.md#deletion) and [Unlisted records](./access-control.md#unlisted-records)), validates each migrated result against the target Type's schema before writing, and aborts immediately on the first validation failure (a buggy migration function is a bug to surface, not to paper over by skipping the offending records) — anything already committed earlier in the pass stays committed. Previous content is snapshotted to version history before each write.
 - **`commitMigration(id, toTypeId, content)`** is the single-record counterpart, changing one Record's `typeId` and `content` together in one step. Unlike `migrateAll()`, `content` here is supplied by the caller rather than produced by a registered `Migration` function — the client-side app that owns `toTypeId` computes it, and the library validates it against `toTypeId`'s schema exactly as `create()`/`update()` validate against a schema. This is what backs the wire's `POST /records/:id/migrate` (see [Wire format](./wire-format.md#records)). Under `ScopedStack` it is **owner-acting-alone**, matching `migrateAll()`'s own absence from `StackClient` — no grant or record-level `write` substitutes for it (see [Access control](./access-control.md#type-level-grants)). Previous content and `typeId` are snapshotted to version history first, same as `migrateAll()`.
 
   Because `content` is a full replacement written under a new `typeId`, a migration commit is create-shaped at the destination and update-shaped over the Record as it stands, and owes both sets of integrity checks. DID bindings are held to immutability across the union of the two families' binding fields — a card can neither shed its `did` by migrating out of `_entity`/`_app` nor pick one up on the way in — and to uniqueness in the destination family (see [Identity § DID bindings](./identity.md#did-bindings)). An `_attachment@1` Record's `fileId`, `mimeType` and `size` stay immutable, and a Record arriving from outside that family is held to the same mimeType-establishment check `create()` applies. Migrating _into_ `_group` is refused outright: a group's `admin` roster entry is stamped at creation and a migration cannot stamp one, so it would produce a group nobody but the owner can manage — version-to-version migration within `_group` stays open and carries the existing roster with it.
@@ -303,6 +304,8 @@ By default, `query()` (like `get()`) returns Records exactly as stored — see [
 
 **`query()` never returns the `_config` record**, regardless of filter — it's addressable only by ID, via `get('_config')` or the adapter's own typed `ownerEntityId`/`timezone` properties (see [Stack initialization](../spec.md#stack-initialization)). This is the one exception to "adapters are storage engines, `Stack` is the invariant layer": the exclusion must live in the adapter's own query predicate (a `WHERE` clause, or the equivalent for an in-memory adapter) rather than be post-filtered by `Stack`, since post-filtering after the adapter applies `limit` would silently under-fill a page. Every adapter — including test doubles — implements this exclusion directly; it is not optional convention.
 
+**Unlisted Records are excluded by default too**, the same posture as soft-deleted ones: `includeUnlisted` opts a query back in, and — unlike `includeDeleted` — `ScopedStack` restricts that opt-in to the owner acting alone. See [Unlisted records](./access-control.md#unlisted-records).
+
 ### Sorting and pagination
 
 ```ts