chore(guides): describe what the subscription lock guarantees once progress is fenced

guides/Architecture.md

@@ -13,3 +13,5 @@ The pooled connection uses `:exp` back-off. However the other connections use `:
 #### Why are these connections stopped on error?
 
 Related processes need to be notified when the connection exits or is restarted. The [postgrex](https://hexdocs.pm/postgrex/) library doesn't provide a way of being notified when the connection terminates. As an example, EventStore uses [Postgres' advisory locks](https://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS) to guarantee only one instance of a subscription runs, regardless of how many nodes are running (or whether they are clustered). These advisory locks are tied to a database connection and are released when the connection terminates. When this happens, EventStore attempts to reacquire the locks. The `EventStore.MonitoredServer` module provides this process monitoring and `after_exit` and `after_restart` callback functions.
+
+Losing the lock is detected after the fact, so progress writes are also guarded by a per-subscription lock generation recorded in the `subscriptions` table. The generation moves on with each lock acquisition and a checkpoint from a previous generation is rejected, which is how a node that no longer holds the lock finds out.

guides/Cluster.md

@@ -10,6 +10,10 @@ PostgreSQL's `LISTEN` / `NOTIFY` is used to pub/sub event notifications. A liste
 
 PostgreSQL's [advisory locks](https://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS) are used to limit each uniquely named subscription to run at most once. This prevents multiple instances of a subscription from running on different nodes. Advisory locks are faster than table locks, are stored in memory to avoid table bloat, and are automatically cleaned up by the server at the end of the session.
 
+The lock decides which node may record the subscription's progress, not which node may run its handler. A node only learns that its lock session ended after the fact, so for a short time two nodes can both process the same events. Each time the lock is acquired the subscription row records a new lock generation, and every checkpoint is written against the generation its owner holds. A node whose lock was taken over has its checkpoints rejected, drops the events it still had in flight, and tries to subscribe again later. The node that took over resumes from the last checkpoint committed before the hand-off, and a checkpoint can never move the subscription's position backwards.
+
+Subscribers must therefore be idempotent, as they are for any at-least-once delivery. This protection takes effect once every node runs a version that records lock generations; until then, nodes on an older version behave as before.
+
 ## Automatic cluster formation
 
 You can use [libcluster](https://github.com/bitwalker/libcluster) to automatically form clusters of Erlang nodes, with either static or dynamic node membership.

guides/Upgrades.md

@@ -2,6 +2,10 @@
 
 The [CHANGELOG](https://github.com/commanded/eventstore/blob/master/CHANGELOG.md) is used to indicate when a schema migration is required for a given version of the EventStore.
 
+## Migrate before rolling nodes
+
+Run the migration before starting nodes on the new version. A node on a version that expects a newer schema cannot acquire subscription locks against an unmigrated database and keeps retrying until the migration has run. Nodes on the previous version keep working against the migrated schema, so a rolling upgrade is safe once the migration is in place.
+
 ## Migrating a database with `mix`
 
 You can upgrade an existing EventStore database using the following mix task: