docs: 1.11

contents/docs/connection.mdx

@@ -7,11 +7,13 @@ title: Connection Status
 Zero manages a persistent connection to `zero-cache` with the following lifecycle:
 
 <ImageLightbox
-  src="/images/connection-state/lifecycle.png"
+  src="/images/connection-state/lifecycle.svg"
   caption="Zero's connection lifecycle"
   invert="light"
 />
 
+Before any of this, Zero is [`initializing`](#initializing) while it loads local data. After that it moves to `connecting` and never goes back.
+
 ## Usage
 
 The current connection state is available in the `zero.connection.state` property. This is subscribable and also has reactive hooks for React and SolidJS:
@@ -40,6 +42,8 @@ function ConnectionStatus() {
   const state = useConnectionState()
 
   switch (state.name) {
+    case 'initializing':
+      return <div>Loading...</div>
     case 'connecting':
       return <div title={state.reason}>Connecting...</div>
     case 'connected':
@@ -64,6 +68,10 @@ function ConnectionStatus() {
 
   return (
     <Switch>
+      <Match when={state().name === 'initializing'}>
+        <div>Loading...</div>
+      </Match>
+
       <Match when={state().name === 'connecting'}>
         <div title={state().reason}>Connecting...</div>
       </Match>
@@ -91,6 +99,9 @@ function ConnectionStatus() {
 ```ts
 zero.connection.state.subscribe(state => {
   switch (state.name) {
+    case 'initializing':
+      console.log('Loading local data...')
+      break
     case 'connecting':
       console.log(`Connecting... ${state.reason}`)
       break
@@ -120,6 +131,7 @@ Zero [does not support offline writes](#why-zero-doesnt-support-offline-writes).
 
 | State          | Reads | Writes      |
 | -------------- | ----- | ----------- |
+| `initializing` | ✅    | ✅ (queued) |
 | `connecting`   | ✅    | ✅ (queued) |
 | `connected`    | ✅    | ✅          |
 | `disconnected` | ✅    | ❌          |
@@ -137,11 +149,23 @@ The easiest way to implement this is with a modal overlay that covers the entire
 
 ## Details
 
+### Initializing
+
+Zero starts in the `initializing` state, where it loads its local data. It hasn't contacted `zero-cache` yet, so the network plays no part. How long this takes depends on the device and on how much data is stored locally. On a slow device with a lot of data, it can take many seconds.
+
+When the data is loaded, Zero moves to `connecting`. This only happens once, right after Zero is created. Later reconnects go straight to `connecting`.
+
+While `initializing`, show a loading indicator rather than an offline or error message. If your app warns the user when Zero takes too long to connect, start that timer when Zero enters `connecting`.
+
+There is no timeout while `initializing`, and the time doesn't count against `disconnectTimeoutMs`. If the local data never finishes loading, Zero stays in `initializing`, so add your own timeout if you need one.
+
+Reads and writes are allowed. Queries return results once the local data is loaded, and writes are queued until Zero connects.
+
 ### Connecting
 
-Zero starts in the `connecting` state.
+Zero enters the `connecting` state after [`initializing`](#initializing).
 
-While `connecting`, Zero repeatedly tries to connect to `zero-cache`. After 1 minute of failed attempts, it transitions to `disconnected`. This timeout can be configured with the `disconnectTimeoutMs` constructor parameter:
+While `connecting`, Zero repeatedly tries to connect to `zero-cache`. After 1 minute of failed attempts, it transitions to `disconnected`. Time spent `initializing` does not count toward this minute. This timeout can be configured with the `disconnectTimeoutMs` constructor parameter:
 
 ```tsx
 const opts: ZeroOptions = {
@@ -177,7 +201,7 @@ Once Zero connects to `zero-cache` and syncs the first time, it transitions to t
 
 After the `disconnectTimeoutMs` elapses while in the `connecting` state, Zero transitions to `disconnected`. Zero also transitions to `disconnected` when the tab is hidden for `hiddenTabDisconnectDelay` (default 5 minutes).
 
-While `disconnected`, Zero continues to try to reconnect to `zero-cache` every 5 seconds.
+While `disconnected`, Zero continues to try to reconnect to `zero-cache` every 5 seconds. After a hidden-tab disconnect, Zero instead waits until the tab is visible again, then goes back to `connecting`.
 
 Reads are allowed while `disconnected`, but writes are rejected and return an offline error. See [Offline](#offline) for more information.
 
@@ -229,6 +253,60 @@ Most applications will never call `close()`, and even if they do, they should no
 
 Reads and writes are both rejected while Zero is in the `closed` state.
 
+## Storage Failures
+
+Zero stores synced data on the device, in IndexedDB in the browser or SQLite on [React Native](/docs/react-native). Sometimes that fails: the device runs out of space, or the database won't open.
+
+When it does, Zero calls `onStorageFailure` once with a `StorageFailureError`. Its `kind` says what went wrong:
+
+| `kind`        | Meaning                                 |
+| ------------- | --------------------------------------- |
+| `full`        | The device is out of space.             |
+| `cannot-open` | The database can't be opened.           |
+| `io-error`    | A read or write to the database failed. |
+
+In the browser, Zero reports a storage failure only when IndexedDB fails to open: `full` if the browser says the disk is full, otherwise `cannot-open`.
+
+Zero keeps working either way. If storage fails while Zero is starting, Zero runs in memory for the rest of the session, as if you had set `kvStore: 'mem'`, and nothing is saved on the device. If it fails later, Zero stops saving to the device. Queries and mutations keep working on the data in memory and mutations are still sent to `zero-cache`, but a read that needs data Zero hasn't loaded into memory can still fail.
+
+Zero only tries the device's storage again when the app creates a new `Zero` instance, for example after a restart.
+
+By default, Zero only logs the failure. Use `onStorageFailure` to tell the user what to do:
+
+```ts
+import type {ZeroOptions} from '@rocicorp/zero'
+
+const opts: ZeroOptions = {
+  // ...
+  onStorageFailure: failure => {
+    // showBanner stands in for your app's own UI.
+    if (failure.kind === 'full') {
+      showBanner(
+        'Your device is out of space. Free up some space, then restart the app.'
+      )
+    } else {
+      showBanner(
+        'Data could not be saved on this device. Restart the app to try again.'
+      )
+    }
+  }
+}
+```
+
+`StorageFailureError` is exported from `@rocicorp/zero`. Its `cause` is the original error from the browser or the SQLite driver. A custom `kvStore` can throw a `StorageFailureError` to get the same handling.
+
+### Checking the Store
+
+`zero.kvStore.kind` tells you which storage Zero is using: `'idb'`, `'mem'`, `'expo-sqlite'`, or `'op-sqlite'`. A custom store reports its own `kind`, if it sets one.
+
+```ts
+if (zero.kvStore.kind === 'mem') {
+  // Data is not saved on this device.
+}
+```
+
+Without IndexedDB, `kind` is `'mem'` from the start and `onStorageFailure` isn't called. When IndexedDB fails to open, `kind` changes to `'mem'` only when Zero first reads or writes the store, so check `kind` when you need it, not once at startup.
+
 ## Why Zero Doesn't Support Offline Writes
 
 Supporting offline writes in collaborative applications is inherently difficult, and no sync engine or CRDT algorithm can automatically solve it for you. Despite what their marketing says 😉.

contents/docs/debug/slow-queries.mdx

@@ -45,4 +45,4 @@ Fly.io provides physically attached SSDs, even for their smallest VMs. Deploying
 
 ## /statz
 
-`zero-cache` makes some internal health statistics available via the `/statz` endpoint of `zero-cache`. In order to access this, you must configure an [admin password](/docs/zero-cache-config#admin-password).
+`zero-cache` makes some internal health statistics available via the `/statz` endpoint of `zero-cache`. In order to access this, you must configure an [admin password](/docs/zero-cache-config#admin-password) or an [operator password](/docs/zero-cache-config#operator-password).

contents/docs/deprecated/ad-hoc-queries.mdx

@@ -23,6 +23,8 @@ const schema = createSchema({
 })
 ```
 
+Ad-hoc queries send their query ASTs directly to `zero-cache`, which rejects them by default. To allow them, also set [`ZERO_ALLOW_LEGACY_QUERIES`](/docs/zero-cache-config#allow-legacy-queries) to `true` on `zero-cache`.
+
 Once enabled, queries are available at `z.query.<tablename>`.
 
 ```tsx

contents/docs/mutators.mdx

@@ -864,6 +864,74 @@ If Zero receives HTTP 401 or 403, the client will enter the needs auth state and
 
 If you want a different behavior, it is possible to [implement the mutate endpoint](#custom-mutate-implementation) yourself and handle errors differently.
 
+### Retrying Mutators
+
+Some errors go away on a second try. At the `repeatable read` and `serializable` [isolation levels](/docs/server-zql#isolation-level), for example, Postgres can abort a transaction that conflicts with a concurrent one. It fails with a serialization failure, code `40001`.
+
+To have Zero run a server mutator again in a fresh transaction when it fails with an error like that, call `tx.retryOn()`:
+
+```ts
+function isSerializationFailure(error: unknown) {
+  return (
+    error instanceof Error &&
+    'code' in error &&
+    error.code === '40001'
+  )
+}
+
+const updateIssue = defineMutator(
+  z.object({id: z.string(), title: z.string()}),
+  async ({tx, args: {id, title}}) => {
+    if (tx.location === 'server') {
+      tx.retryOn(isSerializationFailure)
+    }
+    await tx.mutate.issue.update({id, title})
+  }
+)
+```
+
+`retryOn()` takes a predicate. It gets the error and the attempt number: `1` when deciding the first re-run, `2` for the second, and so on. Return `true` to run the mutator again. If it returns anything else or throws, Zero records the error and skips the mutation, as with any [other error](#handling-errors).
+
+The predicate sees errors the mutator throws, and also errors Postgres raises at `COMMIT`, which the mutator can't catch. Everything the failed run wrote is rolled back, and the re-run reads fresh data.
+
+The error is passed as it was thrown. node-postgres and postgres.js errors have the Postgres error code in `code`. Other libraries can wrap the error, so check what yours throws.
+
+The predicate can also return a promise. Zero waits for it, so you can use it to back off before the next run:
+
+```ts
+tx.retryOn(async (error, attempt) => {
+  if (!isSerializationFailure(error)) {
+    return false
+  }
+  // Wait 50ms, then 100ms, then 150ms, and so on.
+  await new Promise(resolve =>
+    setTimeout(resolve, attempt * 50)
+  )
+  return true
+})
+```
+
+`retryOn()` only applies to the current run. The re-run starts the mutator from the top and calls `retryOn()` again, so call it before anything that can fail. You can call it more than once, and Zero runs the mutator again if any predicate returns `true`.
+
+Zero runs a mutator at most 5 times per mutation in each request to your mutate endpoint. If the last run also fails, Zero records the error.
+
+The predicate runs after the transaction has ended, so it can't use `tx`: reading or writing through `tx` there throws.
+
+Only retry mutators that are safe to run more than once. Work outside the database, such as sending an email, isn't rolled back. `retryOn()` only exists on the server, when `tx.location === 'server'`.
+
+<Note
+  heading="Custom `Database` implementations"
+  slug="retry-custom-database"
+>
+  The built-in adapters and `ZQLDatabase` support
+  `retryOn()`. If you pass your own `Database` to
+  `handleMutateRequest`, it must pass the
+  `addRetryPredicate` it receives in
+  `TransactionProviderInput` on to the transaction it
+  creates with `CRUDMutatorFactory.createTransaction()`.
+  Otherwise `tx.retryOn()` throws.
+</Note>
+
 ### Custom Mutate URL
 
 By default, Zero sends mutations to the URL specified in the `ZERO_MUTATE_URL` parameter.
@@ -1064,14 +1132,55 @@ const read2 = await zero.run(
 
 If the client-side mutator fails, `.server` also resolves to an error result. Awaiting `.server` therefore covers both client- and server-side failures.
 
-<Note
-  heading="Returning data from mutators"
-  slug="returning-data"
->
-  There is not yet a way to return data from mutators in the
-  success case. [Let us know](https://discord.rocicorp.dev/)
-  if you need this.
-</Note>
+### Returning Data
+
+A mutator can return a JSON value. The caller gets it in the `data` field of the success result.
+
+The client and server runs can return different values. For example, the server can return a value that only the database knows:
+
+```ts
+const createIssue = defineMutator(
+  z.object({id: z.string(), title: z.string()}),
+  async ({tx, args: {id, title}}) => {
+    await tx.mutate.issue.insert({id, title})
+
+    if (tx.location === 'client') {
+      // The client doesn't know the short ID yet.
+      return {shortID: null}
+    }
+
+    // Postgres assigns shortID when the row is inserted.
+    const issue = await tx.run(
+      zql.issue.where('id', id).one()
+    )
+    return {shortID: issue?.shortID ?? null}
+  }
+)
+```
+
+The `.client` result's `data` is what the client run returned, and the `.server` result's `data` is what the server run returned:
+
+```ts
+const write = zero.mutate(
+  mutators.issue.create({
+    id: crypto.randomUUID(),
+    title: 'New title'
+  })
+)
+
+const serverRes = await write.server
+if (serverRes.type === 'success') {
+  // `data` is a plain JSON value, so check its shape.
+  const {shortID} = z
+    .object({shortID: z.number().nullable()})
+    .parse(serverRes.data)
+  console.log('Short ID:', shortID)
+}
+```
+
+`data` is typed as `ReadonlyJSONValue | undefined` rather than as the mutator's return type, so check its shape before you use it. A mutator that returns nothing gives `undefined`.
+
+The server's value only reaches the client if your [mutate handler](#implementing-the-endpoint) returns the mutator's result, as `return mutator.fn({args, tx})` does in the examples above. Zero also saves that value in your database with the mutation's result, so the client still gets it if the response from your mutate endpoint is lost. Nothing is saved for a mutator that returns nothing.
 
 ## Permissions