Skip to content

[vector store 8/10] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization - #1618

Draft
edwinyyyu wants to merge 30 commits into
MemMachine:mainfrom
edwinyyyu:feat/qdrant-collection-options-speedkick
Draft

edwinyyyu wants to merge 30 commits into
MemMachine:mainfrom
edwinyyyu:feat/qdrant-collection-options-speedkick

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Purpose of the change

QdrantConf gains hnsw_config, optimizers_config and quantization_config, plain mappings mirroring qdrant-client's HnswConfigDiff, OptimizersConfigDiff and QuantizationConfig, so qdrant-client stays optional for configuration parsing; the store's params validate them against qdrant's own models. They apply to the store's one collection (its registry is SQL tables since #1631). The sample configurations and the configuration docs' parameter table carry the three keys.

m must be 0 or unset: the collection is multi-tenant and disables the global graph in favor of per-partition payload indexing, so a deployment tunes payload_m, which defaults to 16 as before.

Stack

21 PRs on main: 7 independent ones directly on main, then the vector store chain stacked on them, then the SQLite store fixes stacked on the chain. A stacked PR's diff on GitHub is cumulative until the PRs under it merge.

# PR change
Independent PRs, each directly on main; review and merge in any order:
#1541 fix(event-backend): make expand_context return timeline-neighbor episodes
#1661 Overhaul segment store: shared tables with incarnation-scoped tenant keys (port of #1548)
#1460 Publish vector index files atomically (but not durably) (as merged into speedkick, #1588)
#1671 Remove custom sharding from the Qdrant store (port of #1654)
#1630 Bound every request to a remote vector store by a configured timeout
#1682 Hand the Qdrant vector store a metrics factory, so its tracker emits (port of #1532)
#1624 Make no memory request create a project
The vector store chain, stacked on the independent PRs, merged bottom-up:
[vector store 1/10] #1663 Answer with cosine scores and uuids, not vectors and stale properties, and require a vector on the record (port of #1598 and #1603)
[vector store 2/10] #1670 Remove per-project filterable properties (port of #1606)
[vector store 3/10] #1622 Create a session's storage with the session, never on a request
[vector store 4/10] #1625 Remove open-or-create and close from both stores
[vector store 5/10] #1626 Rename logical collection to partition, and open to get, on both stores
[vector store 6/10] #1627 Make a vector store one collection, with string-keyed partitions
[vector store 7/10] #1631 Mint an incarnation per partition life in a SQL-arbitrated registry; delete logically, reclaim by purge, and keep the tombstone until a clean sweep and a retention
[vector store 8/10] #1618 (this PR) Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization
The SQLite store fixes, stacked on [vector store 8/10]:
[sqlite store fixes 1/6] #1469 Never reuse a row id in SQLiteVectorStore (as merged into speedkick, #1589)
[sqlite store fixes 2/6] #1672 Own the search engine's concurrency in the store, not in each engine (port of #1612)
[sqlite store fixes 3/6] #1673 Serialize a partition's writes so the engine sees them in order (port of #1607)
[sqlite store fixes 4/6] #1674 Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)
[sqlite store fixes 5/6] #1675 Take SQLite's write lock at BEGIN, not at the first write (port of #1609)
[sqlite store fixes 6/6] #1676 Give every write a fresh row id, so a key names one version (port of #1610)
Parked on speedkick until #1684 (the port of #1597: EventMemory session, source and expansion) is stacked under them on main:
[vector store 9/10] #1628 Make a vector store filter only on the properties it declares
[vector store 10/10] #1616 Close the filter union, and make negation the complement on every backend

This PR's own change is its 2 commits, 75ece3878, c533194d2; the rest of its diff is the PRs under it. Stacked on #1631; #1469 is stacked on it.

Verification

On main ff58f38c (2026-09-17, after the review round of the same day). ruff check, ruff format --check and ty check clean at every commit of the chain and fixes and on every independent PR. pytest packages/server/server_tests -m 'not integration': green on each independent PR alone (1869–1926 passed, 3 skipped), 1957 passed on the seven together, 1905 passed on the chain's tip (c533194d) and 1929 passed / 3 skipped on the stack's tip (b21343e5); the vector store and resource manager suites at every commit. pytest packages/client/client_tests: 255 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

@edwinyyyu
edwinyyyu marked this pull request as draft September 14, 2026 20:03
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 4 times, most recently from 28884f8 to 6ef5caf Compare September 14, 2026 20:33
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from 6ef5caf to 1364e65 Compare September 14, 2026 21:40
@edwinyyyu edwinyyyu changed the title (Depends on #1606) Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 5 times, most recently from 556eb26 to 2a1da2c Compare September 14, 2026 23:16
@edwinyyyu edwinyyyu changed the title [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) [vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 14, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch 2 times, most recently from 556eb26 to 918e7cb Compare September 15, 2026 17:26
@edwinyyyu edwinyyyu changed the title [vector store 11/12] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) [vector store 12/13] Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization (speedkick) Sep 15, 2026
edwinyyyu and others added 29 commits September 17, 2026 16:40
…keys (fixes MemMachine#1544, MemMachine#1546, MemMachine#1549) (speedkick) (MemMachine#1548)

* Fix: Detach segment store partitions before dropping them, and lock partition metadata on delete

Deleting a partition on PostgreSQL dropped its child tables with CASCADE.
The foreign key from segment_store_dv_ln to segment_store_sg is declared on
the partitioned parents, so the CASCADE dropped the parent-level constraint
rather than only the part belonging to the deleted partition. After the
first partition deletion the store stopped enforcing the link for every
remaining partition, and ON DELETE CASCADE stopped removing derivative
links with it, so delete_segments left orphaned rows that
get_derivative_uuids_by_segment_uuids still returned. Detaching each child
before dropping it keeps the constraint and the cascade intact.

delete_partition also took only a row lock on the partition row. ROW SHARE
does not conflict with the SHARE ROW EXCLUSIVE table lock the create paths
take, so a concurrent create and delete could reach segment_store_sg and
segment_store_dv_ln in opposite order and deadlock; that reproduced in 4 of
7 sampled interleavings against PostgreSQL 16, and in 0 of 7 once delete
takes the same table lock first. The row lock stays, because it is what
makes delete wait for in-flight writers holding FOR SHARE on that row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpFwBnZ6SxdMe3pSuMTCHD

* fix: address partition child tables directly for segment DML

Fixes #1546. Parent-table queries carry the partition key as a bind
parameter; once asyncpg's prepared statement flips to a cached generic
plan (after five executions) PostgreSQL locks every child partition on
every execution before runtime pruning. With hundreds of partitions and
concurrent sessions this exhausts the lock table (searches fail 500
'out of shared memory') and saturates the database CPU with lock churn
and generic-plan startup.

The partition handle now maps the ORM entities onto its own child
tables (orm.aliased with adapt_on_names) and targets them for
insert/delete, so every plan references exactly one partition. SQLite
keeps the parent tables (it has no children).

Measured on a store with 316 partitions: max relation locks held by a
backend during a read loop drops 1276 -> 4; the 239 HTTP 500s in a
4-worker load test disappear; throughput at 128 concurrent requests
rises ~20-30% with PostgreSQL no longer pinned at its CPU cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Test: Pin the partition-delete lock ordering

The foreign-key half of this branch has a regression test; the lock
ordering did not. A deadlock test would be timing-dependent, so assert the
invariant the deadlock analysis rests on instead: delete_partition issues
LOCK TABLE segment_store_pt IN SHARE ROW EXCLUSIVE MODE, and issues it
before any DETACH or DROP of a child table. Without the lock the test fails
and reports the statement sequence it saw.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpFwBnZ6SxdMe3pSuMTCHD

* Style: Apply ruff format to the lock-ordering test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpFwBnZ6SxdMe3pSuMTCHD

* test: segment DML must address only the partition's child tables

Regression test for #1546. Captures the SQL the partition handle emits
across add / seed read / windowed read / filtered read / uuid maps /
delete and asserts no statement references the partitioned parents --
the deterministic observable of the generic-plan lock explosion (lock
counts would need timing-dependent pg_locks sampling). Fails against
the parent-table implementation, passes with per-partition DML.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: drop annotation-only AliasedClass and Table imports

This SQLAlchemy version exports no public AliasedClass name (only the
aliased() factory), so the attribute annotations forced an import from
sqlalchemy.orm.util. The annotations were documentation only; the
branch comment already records that the attributes hold either the ORM
class or its child-table alias.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: derive partition table names from one helper and the models

The parent names come from the models' __tablename__ and the child
naming pattern lives in _pg_child_table_name, used by child-table
creation, teardown, and the per-partition DML targets, so the three
sites cannot drift apart. Physical names are unchanged; the regression
tests keep literal names to pin the on-disk naming contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix: Drop detached children, and stop holding the store-wide lock while waiting for writers

Three follow-ups from review of this branch.

The child-table probe asked whether the table exists, but the statement it
guards is DETACH PARTITION, which requires that the table be attached. A
child left detached by manual maintenance or an interrupted DETACH
PARTITION CONCURRENTLY (that form is not transactional) passed the probe and
made DETACH raise "is not a partition of", rolling back the transaction, so
the partition became neither deletable nor recreatable -- a state the
CASCADE drop this branch replaced used to clean up. Probe pg_inherits for
attachment instead, in one round trip for both children, and drop an
unattached child directly.

delete_partition took the partitions-table lock before the row lock that
waits for in-flight writers, so a slow writer on one partition stalled
open_or_create_partition for every partition, which runs on the request
path. Take the row lock first; the table lock only has to be held across the
child DDL for the deadlock argument to hold. Re-measured: 4/7 sampled
interleavings deadlock with no table lock, 0/7 with either ordering.

Tests: the foreign key is now asserted to be enforced after a partition
delete, not only that the cascade fires -- the PR's measurements list those
as separate things the CASCADE drop broke. A second test leaves a child
detached and requires deletion to succeed and the key to be reusable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MpFwBnZ6SxdMe3pSuMTCHD

* fix: memoize partition entities per key; validate keys in the naming helper

Review follow-ups on the child-table change:

- The child Table objects and aliases are now built once per partition
  key (functools.cache) instead of per handle. SQLAlchemy's compiled-
  statement cache keys on the Table objects a statement references, so
  per-handle tables made every handle's statements recompile and
  polluted the cache for everything else (verified: cache keys differed
  across handles for the same partition; now identical).
- The tables carry columns only. The to_metadata copies dragged along
  foreign keys with unresolvable targets and duplicate index names --
  latent hazards for anything walking that MetaData.
- _pg_child_table_name validates the partition key itself, so every SQL
  string built from a child table name (including the DDL literals) is
  safe by construction rather than by call-site convention; the store's
  validator moved to module level beside it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: make partition key validation part of the segment store contract

Partition keys are embedded in native storage identifiers by any
implementation, so the alphabet/length rule is interface-level, not a
SQLAlchemy detail: validate_partition_key now lives in the package's
data_types (exported from the package), the SegmentStore.create_partition
docstring states the contract, and the SQLAlchemy store imports it.
Deliberately NOT unified with the vector store's identical identifier
rule: the repo-wide naming contract is not wired through yet, so the
convergence is treated as incidental for now.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: move partition key validation to segment_store/utils.py

Mirrors the vector store's layout (validate_identifier in
vector_store/utils.py); the interface docstring states the key rule
plainly instead of referencing a code path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: state partition key naming constraints in the VectorStore format

Same ABC-level 'Naming constraints:' block the vector store uses, no
method-level restatement, and the length limit is enforced and
documented in bytes, matching validate_identifier.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: state the key rule as the regex, not a prose fragment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: restore the ABC's original naming-constraints docstring

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: make validate_partition_key boolean, call sites raise

Mirrors the vector store's validate_identifier: the predicate returns
bool so callers can compose it, and each entry point raises its own
error in the vector store's message style.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Revert "refactor: make validate_partition_key boolean, call sites raise"

This reverts commit 88be86603c36e4b759da0f876968e6dd6db8d913.

* fix: bound the partition-entity cache with lru_cache(4096)

functools.cache grew ~29 KiB per distinct partition key (measured) for
the life of the process, including deleted partitions. The LRU cap
bounds it at ~115 MiB per worker; eviction is harmless since a rebuilt
entry is identical and only costs recompiling that partition's
statements once.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: drop point-in-time memory figures from the cache comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: address second review round

- Type the partition entities: _pg_partition_entities returns a
  NamedTuple with real field types instead of a positional tuple of
  object, which was adding 55 ty diagnostics (the CI static check would
  have failed) and forcing blind unpacking; callers and the memoization
  test use named fields, and the private _generate_cache_key assertion
  (redundant given object identity) is dropped.
- DROP TABLE gains IF EXISTS back, so an out-of-band drop landing
  between the state probe and the drop cannot leave a partition
  half-deleted.
- open_or_create_partition opens existing partitions without the
  store-wide management lock (double-checked: unlocked read, then lock
  and re-check only when creating), so request-path opens no longer
  serialize behind a concurrent deletion's DDL window; pinned by
  test_open_existing_partition_takes_no_management_lock.
- The engine's compiled-statement cache is raised from the default 500
  (per-partition statements would thrash it once enough partitions are
  live concurrently).
- Comments and the DML test docstring scope the lock claim honestly:
  PostgreSQL's FK integrity triggers still address the parents
  internally, costing a one-shot per-backend lock spike on
  writes/deletes when a trigger plan first goes generic (verified:
  ~10 locks steady, one spike at execution six, then back) -- tracked
  on #1546.
- The detach test cleans up its detached child unconditionally so a
  failure cannot poison the session-scoped container for later tests;
  the statement recorder is a shared fixture instead of copy-paste;
  the byte-length check encodes once.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: drop the typing casts from the partition entities

inspect(Model).columns yields real Column objects (the stubs type
__table__ as FromClause, which forced the cast), and SQLAlchemy's
typing convention represents an aliased entity as the mapped class
type, so the NamedTuple fields are type[SegmentRow] /
type[DerivativeLinkRow] and aliased() assigns without coercion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* design: shared segment store tables with incarnation-scoped keys

Design record for replacing the per-tenant partitioned layout with
shared tables on every dialect: the tenant registry carries an
incarnation, data rows are keyed by <logical_key>@<incarnation>,
deletion is an O(1) registry write plus a purge queue, and fencing
fails stale handles loudly. Records the measured comparison against
PARTITION OF and standalone-table layouts and the scaling requirements
(cheap tenant creation at 1e5-1e6 tenants, 1e4-1e7 rows per tenant)
that decided it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor!: shared segment store tables, incarnation fencing, O(1) delete

Implements design/segment_store_shared_tables.md. Fixes #1544, #1546,
and #1549 by construction:

- The ORM models are the physical schema on every dialect; PostgreSQL
  partitioning, per-tenant DDL, the detach machinery, the store-wide
  management lock, and the per-partition entity cache are all removed.
  No partitions means no generic-plan lock fan-out (client or
  RI-trigger) and no DDL for lifecycle deadlocks to live in -- the
  churn smoke that measured 41-83 deadlocks per 20s on every
  partitioned build measures zero, with 60x more write throughput.
- segment_store_pt becomes the tenant registry: partition_key +
  incarnation. Data rows are keyed by <logical_key>@<incarnation>, so
  a deleted-and-recreated tenant never sees its predecessor's rows.
- create_partition is a row insert (no DDL); delete_partition is O(1):
  FOR UPDATE on the registry row (drains writer pins), enqueue the
  physical key on segment_store_gc, delete the row.
  purge_deleted_partitions reclaims rows in chunked background batches.
- Writes pin the registry row FOR SHARE with an incarnation predicate;
  reads check it too: a stale handle raises
  SegmentStorePartitionStaleError on every dialect, SQLite included.
  Measured cost: one extra registry round trip per read operation.
- The segment table's FK to the registry is removed (registry and data
  rows are deliberately decoupled for O(1) deletion); the link-table FK
  and cascade remain.

Same-moment ABAB vs the partitioned build: ingest and windowed reads at
parity, lifecycle cycles 4-6x faster, tenant creation ~1000x cheaper
(row insert vs DDL).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: identify tenant data rows by incarnation UUID alone

Data rows drop the composite <logical_key>@<incarnation> string for a
bare incarnation UUID column: a data query cannot be constructed
without resolving the registry, so referencing the wrong tenant is
structurally impossible; index entries narrow from a 41-byte varchar
to a native 16-byte uuid; random UUIDs are globally unique across
nodes without coordination, so tenant moves between databases carry
rows verbatim; and collisions among incarnations with live traces are
rejected by constraints (unique on the registry, primary key on the
purge queue) instead of left to probability. The purge queue keeps the
logical key for forensics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: finish the physical-key -> incarnation wording in the design doc

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: fence by incarnation alone

The incarnation is unique-constrained, so it resolves the registry row
by itself; the logical-key predicate was a leftover from the composite
string design and contradicted the rule that the incarnation is the
handle's sole authority.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename the stale error to name the handle, not the partition

The handle is what is stale -- the partition is deleted -- and
SegmentStorePartitionHandleStaleError follows the existing noun+state
convention (ConfigMismatch, AlreadyExists).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: inline uuid4 for incarnation generation

new_incarnation() was a one-line wrapper adding indirection for no
behavior; the multi-node rationale lives in the design doc.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: normalize segment timestamps to UTC before persisting

Segment-store slice of #1462: SQLite's DateTime(timezone=True) discards
tzinfo and stores wall-clock fields verbatim, so a non-UTC timezone-aware
timestamp written without UTC normalization read back shifted by its
offset (13:30:45-08:00 came back as 05:30:45-08:00). The read path
already assumed UTC and reapplies the separately stored offset; only the
write was missing the conversion. PostgreSQL timestamptz stores a true
instant, so this is a no-op there.

Regression test parametrized over UTC/-08:00/+05:30 runs on both
backends; verified the non-UTC params fail without the fix and pass
with it (sqlite 54, pg 58).

The companion filter-bound normalization lives in shared
sql_filter_util.py (used by episode and cluster stores too) and stays
in #1462.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: normalize datetime filter bounds to UTC in SQL filter compilation

Second half of the #1462 segment-store slice: timestamp columns now hold
the UTC instant, so comparison bounds must be named in the same frame.
On SQLite an aware datetime bind is rendered as wall-clock digits with
tzinfo dropped and compared lexically, so `timestamp <=
2024-01-01T08:00+08:00` excluded a row stored at 00:00Z -- the same
instant. _normalize_column_value converts datetime values (Comparison
and In leaves) to UTC before binding; PostgreSQL compares timestamptz
by instant either way, so the two backends now agree.

The helper lives in the shared sql_filter_util because that is where
column leaves are compiled; other stores' write paths (episode, cluster)
are intentionally not touched here.

Regression test parametrized over the same instant named in +00:00,
+08:00, and -08:00, on both backends; verified the non-UTC bounds fail
without the fix and pass with it (sqlite 57, pg 61; full server suite
1880 passed, 3 skipped).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: promote purge_deleted_partitions to the SegmentStore ABC

Physical reclamation of deleted partitions is now an ABC capability:
callers schedule it however they want; the store never schedules it
itself. delete_partition's contract notes that reclamation may be
deferred. Implementations whose deletes reclaim physically implement
it as a no-op returning False.

Signature review against the prior purge iterations (#1199/#1205):
- The old three-step orphan-derivative API (get_orphaned / mark /
  purge) existed only because derivative purging interleaved with
  vector-collection deletes between steps; incarnation purge is fully
  internal to the store, so a single method suffices.
- The old scheduling knob (purge_interval loop in ExtraMemory) lived
  in the consumer -- preserved: no scheduling in the store.
- The bound is max_segments (domain unit; derivative links ride along
  uncounted) rather than max_batches, which presumed chunked-transaction
  implementations. batch_size stays as a keyword on the SQLAlchemy
  implementation only, as a transaction-size tuning knob.
- Returns bool ("reclaimable work may remain") instead of rows deleted:
  a row count cannot distinguish "drained" from "stopped at the bound"
  when a dead incarnation has zero data rows, and the scheduling caller
  needs exactly the more-work signal.

New test pins the bound and the completion signal on both dialects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: drop batch_size from purge_deleted_partitions

With max_segments as the caller's bound, a per-call batch_size is
redundant: bounded calls already cap every delete transaction at the
remaining budget, so the knob only governed the unbounded case --
where transaction sizing is engine policy, not caller policy. The
chunk is now an internal constant (_PURGE_CHUNK_SIZE); if a deployment
ever needs to tune it, it belongs in SQLAlchemySegmentStoreParams,
not per call. Tests exercise multi-chunk draining by patching the
constant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: purge is one atomic slice per call; fix open_or_create race

Purge contract resolved to atomic-slice-per-call: each
purge_deleted_partitions call is a single transaction that reclaims up
to max_segments rows and either commits that progress or nothing.
Draining a backlog is the caller's loop (call until False), so
reclamation never holds a long transaction, committed slices survive
interruption, and there is no internal chunking competing with the
caller's bound. max_segments=None means the store-chosen slice size
(_PURGE_SLICE_SEGMENTS), keeping engine-appropriate transaction sizing
out of callers' hands. Rationale over the alternatives: cross-call
atomicity is anti-useful for gc (a huge atomic purge is exactly the
long-transaction hazard, and an error would forfeit all progress),
while batch_size+max_batches exposes the store's transaction quantum
and bounds a call only as a product of two knobs.

Also fixes a TOCTOU in _open_or_create_partition caught by the new
lifecycle churn test: losing the insert race and then finding no row
(a concurrent delete removed the winner) raised RuntimeError; the
read-then-insert sequence now retries, since every retry implies
another actor changed the state. New deterministic fencing tests:
test_write_landing_during_delete_is_never_orphaned (the write pin means
rows can never land under an incarnation the purge queue no longer
tracks) and test_concurrent_remote_delete_yields_single_queue_entry
(the delete pin means racing deletions enqueue exactly once).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: lock-necessity suite verified by per-lock ablation in both generations

New test_segment_store_locking.py (PostgreSQL integration lane) pins
each locking property through the public API, using only surface shared
with the pre-overhaul partitioned store so the module runs against both
generations. All interleavings are staged event-driven: blocked-ness is
decided by observing pg_stat_activity lock waits, not elapsed time;
there are no grace sleeps, and paused writers are released in finally
blocks so a failing assertion cannot wedge fixture teardown. The lane
adds under a second of CI time.

Ablation matrix (each lock removed one at a time via source-patched
variant trees, PYTHONPATH-shadowed; old = pre-overhaul partitioned
store at 14b8f0a2~1):

- write pin ablated (either generation): write-pin test fails, plus
  the no-orphaned-writes fencing test on the new store.
- delete pin ablated (new store): churn, concurrent-delete, and
  single-queue-entry tests fail (double-enqueue IntegrityError).
- delete row pin ablated (old store): write-pin test fails (the delete
  no longer waits out the in-flight writer).
- ordered delete_segments row locks ablated (either generation): no
  test fails -- identical DELETE shapes lock rows in identical orders
  on PostgreSQL (sorted scalar-array probes, TID-ordered bitmap scans),
  so the AB/BA cycle needs plan divergence the store never produces.
  The overlap test is kept as a regression canary and documented as
  such; whether to keep the pre-lock itself is a separate decision.
- old store with ALL locks intact: churn and concurrent-delete tests
  fail with DeadlockDetectedError in the two cycle shapes documented
  on #1546 (delete-vs-delete lock upgrade over the table mutex;
  create-vs-delete DDL cycles through the shared parents). Those
  deadlocks are inherent to the partitioned layout -- the property the
  shared-table overhaul removes, and these tests now pin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: purge claims queue entries with SKIP LOCKED; correct lock-order rationale

Concurrent purgers were a real deadlock surface: two processes draining
the same dead incarnation delete overlapping row sets through unordered
scans. Claiming queue entries with FOR UPDATE SKIP LOCKED removes the
contention instead of ordering it -- racing purgers partition the queue,
and only the claiming call touches a dead incarnation's rows (writers
cannot; the fence pins live incarnations only), so reclamation is
deadlock-free by construction. This is the claiming half of the purger
scale-out design in the design doc; the ABC now states the contract
(concurrent calls, including cross-process, must neither error nor
deadlock).

Tests: test_purge_skips_entries_claimed_by_concurrent_purger stages a
purger from another process holding its claim uncommitted -- a
concurrent purge must skip the entry and complete without blocking;
verified to fail (blocks on the held queue row) with the claim ablated
and pass with it. test_concurrent_purges_reclaim_everything pins the
correctness property on both dialects: racing drain loops terminate
cleanly with full reclamation.

Also rewords the ordered-row-lock rationale in the locking suite: the
consistent acquisition order that makes the ablation unobservable is
current PostgreSQL executor behavior, not a guarantee any engine
documents -- the pre-lock imposes the order deliberately, and the canary
catches divergence if an engine or plan change ever produces it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: accept any task result type in _wait_until_blocked_or_done

The helper only observes done-ness; Task[None] rejected the purge
task (Task[bool]).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: reject minting an incarnation whose garbage is still awaiting purge

Data rows are keyed by incarnation alone, so a fresh mint colliding with
a dead-but-unpurged incarnation would adopt its garbage and then be
erased by the purger. The registry's unique constraint only guarded
collisions with live incarnations; the purge-queue case was guarded by
uuid randomness alone.

The mint (shared by create_partition and open_or_create_partition) now
re-checks the purge queue inside the insert transaction and re-mints on
collision. The check is race-free with the existing tables -- no ledger
table needed: it runs after the registry insert, so a concurrent
deletion moving a colliding row to the queue (the insert waited on its
uncommitted registry delete) is already visible, and no new queue entry
for the minted value can appear before commit because the only registry
row carrying it is uncommitted. The locking read sees latest-committed
state on dialects whose plain reads serve transaction-start snapshots;
SQLite serializes whole transactions. An incarnation value can therefore
never be reused while any trace of it remains within a database; across
databases, uniqueness still rests on random-uuid collision resistance.

test_incarnation_with_garbage_left_is_never_reused forces the collision
by stubbing the mint (both creation paths, both dialects); verified to
fail with the re-check ablated and pass with it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: one collision error for live and garbage incarnation mints

Both collision causes look the same to the mint's callers and share the
same remedy -- mint a fresh incarnation and retry -- so they now share
one error, with the cause classification (key taken vs incarnation
collision) resolved inside _insert_partition_row: an IntegrityError with
a committed row under the key means the key is taken
(SegmentStorePartitionAlreadyExistsError: open or delete it instead);
without one, the incarnation collided with a live row. Errors are typed
by the decision the caller makes, not by the failing constraint, and
both call sites shrink to one remedy branch per error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: fold locking tests into the segment store test file; drop "slice"

All SQLAlchemy segment store tests live in one file. The separate
locking module existed so the same tests could import against the
pre-overhaul partitioned store for the lock-ablation matrix; that
verification is done and recorded, so the split's constraint is spent.
The per-lock coverage map moves to a section comment.

Also replaces the "one slice per call" purge wording, which was
circular (a slice being defined as whatever one call does), with the
actual contract: each call reclaims at most max_segments segments --
in this store, one transaction that commits that progress or nothing --
and None means the store's default bound (_DEFAULT_PURGE_MAX_SEGMENTS,
renamed from _PURGE_SLICE_SEGMENTS).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: correct design-doc drift; drop dead _is_postgresql flag

Accuracy review of the design doc against the code:
- The create bullet said "one row insert"; the mint transaction also
  re-checks the purge queue.
- The locking model omitted the purger's SKIP LOCKED queue claims and
  the mint's collision-case queue pin; it now lists every row lock and
  why reclamation cannot contend with anything.
- The consequences section claimed the only remaining dialect split is
  the LATERAL-vs-loop read strategy; the PostgreSQL-only ordered row
  locks in delete_segments and SQLite's foreign-key pragma are splits
  too.

_is_postgresql was assigned and never read -- dead since the overhaul
removed the DDL branches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: make the default purge bound a store construction parameter

SQLAlchemySegmentStoreParams.default_purge_max_segments (default 10000)
replaces the module constant: each purge call is one transaction, so
the right default bound is dialect- and deployment-dependent, and the
construction parameter lets an application set it once instead of
every purge caller reading configuration to pass max_segments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: cover live-incarnation collision and purge-bound params wiring

Coverage audit of the recent additions found two unpinned paths:

- The live half of the mint's collision handling (registry unique
  violation classified by the key re-read, then re-mint) had no test --
  only the garbage half did. test_incarnation_colliding_with_live_
  partition_is_never_reused forces the collision on both creation paths
  and both dialects; verified to fail with the classification ablated
  (create_partition misreports AlreadyExists) and pass with it.
- The purge tests patched the store's default-bound attribute directly,
  leaving the SQLAlchemySegmentStoreParams.default_purge_max_segments
  wiring itself untested. test_default_purge_bound_comes_from_params
  constructs a store with a small configured bound and observes it
  govern an unbounded purge call.

Also converts the two override-method docstrings (delete_partition,
purge_deleted_partitions) to body comments: the contract lives on the
ABC; overrides keep only implementation mechanics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename _logical_partition_key to _partition_key

The "logical" qualifier contrasted with the physical partition key of
the composite-key era; data rows now carry no key at all, so there is
nothing physical to distinguish from.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename PurgeRow to PurgeQueueRow

The model classes are named for what a row represents (PartitionRow,
SegmentRow, DerivativeLinkRow); a segment_store_gc row is not a purge
but an entry of the purge queue.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: restore @staticmethod on _resolve_segment_field

It became an instance method when field resolution went through the
handle's per-partition aliased entities; the shared-table overhaul
resolves against the module-level SegmentRow again, leaving self
unused. Call sites return to the original class-qualified form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: pin the mint's insert-then-check statement ordering

The collision guard relies on checking the purge queue AFTER the
registry insert: under READ COMMITTED, the insert's unique-index wait
on a concurrent deletion's uncommitted registry delete is what forces
that deletion's queue entry to be committed -- and therefore visible to
the later check. Checked before the insert, the queue is read too
early and the mint commits a live partition whose incarnation is on
the purge queue, handing its rows to the purger.

Only a concurrent interleave distinguishes the orderings, so the
sequential collision tests cannot pin it: verified by swapping the two
statements -- the sequential tests all still pass (the opposite order
is correct for non-concurrent use), while the new
test_mint_detects_collision_with_concurrent_deletion fails (it is
incorrect for concurrent use). The test stages the interleave
deterministically: a raw-session deletion held uncommitted, the
colliding mint observed blocking on it via pg_stat_activity, then the
deletion committed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: plain "maximum number of segment rows purged per call" wording

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: match params docstring to the pydantic field description

Convention in the class: the Attributes entry carries the field
description plus the default; the field expresses the default via its
default attribute only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: purge claims queue entries one at a time

The claim SELECT had no limit: it materialized and row-locked every
unclaimed queue entry even when max_segments exhausted on the first
incarnation -- a mass-deletion backlog was fetched wholesale per call,
and the first purger claimed the entire queue, so concurrent purgers
skipped everything and exited instead of sharing the backlog.

Claims are now LIMIT 1 FOR UPDATE SKIP LOCKED, issued as the call
processes entries: a bounded call locks exactly what it works on.
Within the transaction each claimed entry is retired before the next
claim, so the call's own claims (which SKIP LOCKED does not skip)
cannot recur and the loop terminates.

test_purge_claims_queue_entries_incrementally pins the property via
recorded SQL: every queue claim carries LIMIT, and a call whose bound
exhausts on its first incarnation issues exactly one claim; verified
to fail against the previous claim-all form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: batch, not chunk, for the purge deletion unit

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: purge runs on an engine connection for typed rowcount

AsyncSession.execute has no DML overload -- it is typed Result[Any] for
every non-typed statement, so reading rowcount needed an isinstance
narrowing to CursorResult (whose unreachable else-branch would have
fabricated a zero count). AsyncConnection.execute is typed CursorResult
in every overload, and the purge transaction is pure Core DML with no
session features, so it now runs on self._engine.begin(): the library's
own annotations carry the type and the narrowing disappears.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: manual formatting

Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>

* fix: address code-review findings (round 3)

Six confirmed-or-verified defects from a second review session, each
fixed with a test verified to fail on the pre-fix code:

- SQLite write fence was a no-op: the driver defers BEGIN to the first
  data-modifying statement, so the fence SELECT ran outside the write
  transaction and a write racing a delete-plus-purge committed rows no
  queue entry tracked. The write fence (and deletion's row check) now
  issue a no-op registry UPDATE first, opening the write transaction so
  the check is transactional and racing deletions serialize.
- The shared column-leaf UTC normalization silently changed OTHER
  stores' datetime filters: their write paths still store wall clock,
  so on SQLite their filters stopped matching rows they had just
  written. Normalization is now an explicit compile_sql_filter opt-in
  (column_datetimes_are_utc) that only the segment store sets; other
  stores regain their previous behavior, and #1462 flips the opt-in
  for the stores whose write paths it fixes.
- Mint collision retries were unbounded: any persistent IntegrityError
  with the key absent became an infinite hot loop. Both creation paths
  cap consecutive collision retries (_MAX_MINT_ATTEMPTS) and re-raise
  the underlying error -- consecutive failures at that depth mean a
  permanent cause, not a race.
- purge_deleted_partitions accepted non-positive bounds and returned
  True unconditionally, spinning the documented drain loop; it now
  raises ValueError. Empty incarnations charge one segment of budget,
  so a backlog of empty tenants is bounded per call instead of drained
  in one unbounded transaction.
- open_or_create committed the registry row before materializing the
  payload codec, leaving an unopenable partition behind on codec
  failure; the codec is loaded before the insert again.
- validate_partition_key used re.match with $, accepting keys with a
  trailing newline; now re.fullmatch.

Also from the review: the ABC documents the stale-handle contract on
SegmentStorePartition and corrects purge's False semantics (work owned
by a concurrent purger is not counted); the blocked-or-done test helper
scopes pg_stat_activity to the current database.

Rejected findings, with grounds recorded in the PR discussion: the
read-fence round trip is the deliberate loud-fencing contract (#1549);
fence/live-check unification, the forensic enqueued_at column, and
FIFO claiming are declined as taste; the partition-key rule's overlap
with service_locator stays per the incidental-convergence ruling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: LongTermMemory erasure drains the purge queue inline

delete_partition's physical reclamation is deferred by design, but the
review found its one production caller now leaked: session deletion
previously removed data physically (DROP on PostgreSQL, cascade on
SQLite) and nothing anywhere called purge_deleted_partitions. The
erasure path drains the queue inline before returning, restoring
physical removal semantics; background scheduling remains available to
other callers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: UTC-normalize every SQL store; self-checking SQLite fence; FIFO purge

Four follow-ups to the review round, per direction:

- The datetime-normalization opt-in is gone: instead of scoping the
  shared compiler's UTC bound normalization to the segment store, every
  SQL store's write path is fixed honestly in this PR. #1462's episode
  and cluster fixes (created_at + start/end bounds; last_ts + pending
  created_at) are ported with their regression tests, and the compiler
  normalizes column datetime bounds unconditionally -- correct for all
  consumers, since the semantic-storage columns it also serves are
  server-generated UTC (func.now()). Fixes #1558 and #1559 here.

- The SQLite fence is one self-checking statement instead of a no-op
  UPDATE plus a SELECT: the proper primitive, BEGIN IMMEDIATE, is only
  expressible engine-wide in SQLAlchemy (it would put every read
  transaction behind the write lock), so the registry-row UPDATE
  acquires the same write lock scoped to the transaction, and its match
  count is the staleness check. Deletion opens its transaction the same
  way, with zero matches as the idempotent no-op case.

- The purge queue is FIFO: claims order by enqueued_at (indexed), so
  the oldest garbage is reclaimed first and the name is honest. Queue
  entries carry their own per-call bound
  (SQLAlchemySegmentStoreParams.purge_max_partitions, default 1000)
  instead of charging a fake segment of budget: their cost is round
  trips rather than row deletions, and empty partitions are cheap to
  mass-create-and-delete, normally or adversarially. Empty entries no
  longer consume max_segments.

- PostgreSQL-only concurrency coverage gains SQLite counterparts
  wherever the property exists on both dialects: lifecycle churn,
  racing deletions (plus a single-enqueue assertion), overlapping
  segment deletes now run on both; new SQLite tests pin the
  mint-vs-deletion collision race and O(1) deletion via recorded SQL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: precise rationale for the SQLite fence primitive

BEGIN IMMEDIATE is expressible per-transaction in principle, but only
atop engine-wide rewiring (isolation_level=None plus a begin-event
hook) that the store cannot apply to a caller-owned, possibly shared
engine; say that instead of "only expressible engine-wide".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: two-character index name tokens, matching the prior convention

pk_ev / pk_ts_ev_bk_ix / pk_su used two characters per indexed column;
in (incarnation) and ea (enqueued_at) follow, replacing inc and enq.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor!: purge takes no arguments; cap derivatives at ingestion

purge_deleted_partitions() -> bool. Callers cannot know
engine-appropriate transaction sizing -- the same argument that made
the default a construction parameter removes the per-call override:
the caller's whole protocol is "call until False", and every bound
(purge_max_segments, renamed from default_purge_max_segments;
purge_max_partitions) is implementation policy set once at
construction. Non-positive bounds are now impossible by pydantic
validation, superseding the runtime ValueError.

The derivative side is bounded where it is created, not where it is
reclaimed: purge keeps relying on the link-table ON DELETE CASCADE --
benchmarked against manual link deletion on the real schema and 50-68%
faster (1 link/segment: ~312k vs ~209k segs/s; 4 links: ~266k vs
~158k; the manual pattern's extra round trips and array shipping cost
more than the per-row indexed trigger probes) -- and ingestion rejects
more than max_derivatives_per_segment links per segment (default 100),
so one purge call's work is at most purge_max_segments segment rows
plus that many times the cap in link rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* revert: drop the store-level derivatives-per-segment cap

The cap rejected at add_segments time, when the caller has already
segmented and derived and can do nothing to obey it -- the bound on
link fan-out is ingestion-pipeline policy (deriver/segmenter design),
not a store contract. Performance also gives the cap no case: measured
across densities, cascade deletion saturates around 3M link rows/s
(380k segments/s at one link per segment, 302k at 4, 175k at 16, 46k
at 64 -- per-row cost FALLS with density, 1.3us/row at 1 link to
0.34us at 64), so a purge_max_segments=10000 call finishes in ~0.45s
even at 64 links per segment. The design doc records where the bound
lives and the measured sensitivity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: state the purge contract's promise to callers

The bounds are implementation policy; what the caller is promised is
that a purge call does not noticeably degrade concurrent request
serving. The design doc also records why a store-level link cap would
be unactionable (only the deployment's segmenter/deriver choice can
change the ingested shape, so a dedicated error type would have no
useful handler).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: rebalance the purge entry bound to measured cost

Retiring an empty queue entry measures ~0.95 ms through the store
(four round trips), roughly 200x a segment row at the measured purge
rate -- not the ~10x the old default implied. purge_max_partitions
drops from 1000 (a ~0.95 s transaction when saturated, 20x the row
bound's ~46 ms) to 50, putting a full-entry call and a full-row call
at comparable transaction duration. Backlog drain throughput is
unchanged (~1k entries/s regardless of slicing); only per-call
transaction length shrinks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: public SegmentStorePermanentError; power-of-ten entry bound

Mint-retry exhaustion raised "last_collision.__cause__ or
last_collision" -- expedient plumbing that leaked either the wrapped
SQLAlchemy IntegrityError or the private collision type to callers.
Per the error-design principle (type by the caller's decision), the
decision here is "retrying will not fix this; diagnose", so both
creation paths now raise the ABC-declared SegmentStorePermanentError
with the underlying error chained as the cause. The ABC documents it
on create_partition and open_or_create_partition.

purge_max_partitions defaults to 100 instead of 50: sibling fields of
one config keep to the same numeric family (powers of ten, alongside
purge_max_segments=10000); a saturated entry call (~95 ms measured)
and a saturated row call (~46 ms) stay within the same order of
transaction duration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename SegmentStorePermanentError to SegmentStoreRetriesExhaustedError

"Permanent" asserted a diagnosis the store cannot make -- sustained
adversarial churn could in principle clear on a later attempt. The
name now states only what happened (internal retries exhausted), with
the guidance phrased as likelihood: an immediate retry is unlikely to
succeed; diagnose the chained cause.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: drop illustrative examples from contract docstrings

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: bound open_or_create's lost-race arm with the same retry cap

The collision arm was capped but the AlreadyExists arm looped
unboundedly -- the reviewer's livelock finding. Both non-terminating
outcomes now count toward one retry budget, and exhausting it raises
SegmentStoreRetriesExhaustedError with the last error chained. With
this, every retry construct in the store is bounded: purge makes
guaranteed progress per call, deletion is a single idempotent
transaction, fences raise stale, and reads are single-pass -- the
creation paths were the only sites with retries to exhaust.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: attempts, not retries

SegmentStoreAttemptsExhaustedError, with the counter and docstrings
using the same word: "retry" is ambiguous between a re-attempt and the
whole attempt sequence, and _MAX_MINT_ATTEMPTS already counted
attempts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: attempts vocabulary in the mint-exhaustion message

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* nit: increase max mint attempts from 8 to 10

Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>

* nit: manual formatting

Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>

* review: fold the read fence into the data statement; harden creation, drain, purge

Third review round (15 findings; 12 acted on, 3 declined with grounds
in the PR body).

- Reads no longer issue a separate registry round trip: the liveness
  predicate rides in each data statement as an EXISTS conjunct (one
  statement, one snapshot -- a stale handle reads nothing), and the
  registry check is issued on its own only when a read returns no rows,
  to tell an empty partition from a stale handle. The write fence and
  the read check share one query builder (`_registry_row_query`) and
  one checker (`_ensure_partition_live(pin=...)`).
- `create_partition` materializes the payload codec before inserting,
  like `open_or_create_partition`; its mint loop uses the same
  attempt-counter idiom and message as the other path, which also
  removes the possibly-unbound `last_collision`.
- `drop_session_partition` nulls its handles before the inline drain,
  so a drain failure cannot leave them pointing at deleted resources;
  the drain's comment states exactly what it guarantees (the queue is
  global, the drain uncapped, and an entry a concurrent drain claimed
  is finished by that drain).
- The purge queue's enqueue stamp is the database clock (`now()`), so
  every server's entries order on one clock; the unreachable
  `remaining <= 0` guard is gone; the purge comment and design doc
  state SQLite's actual claiming behavior (plain read, serialized on
  the database write lock at the DELETE; duplicated round trips only).
- `startup()` refuses the old partitioned layout (registry without the
  incarnation column) with a directive to recreate the schema, instead
  of letting create_all leave the old tables in place for an opaque
  missing-column error later.
- Cluster store reads use the shared `ensure_tz_aware`; the private
  clone is deleted. Contract wording: "every data operation" raises
  the stale-handle error (the config property never did).

Tests: unloadable codec guard parametrized over both creation paths,
FIFO pinned with explicit stamps set against insertion order, the
database-clock stamp and the folded liveness check pinned via recorded
SQL, the startup probe on both dialects, and the LTM nulling order
under a failing drain. The codec and nulling tests were each verified
to fail with their fix ablated.

Read-path ABAB against the previous HEAD (interleaved rounds, medians):
seed context reads 1.17 vs 1.42 ms, event lookups 1.04 vs 1.61 ms,
derivative lookups 1.05 vs 1.33 ms (5 rounds), windowed context
expansion 8.74 vs 9.67 ms (8 rounds x 600 reps, paired median
-0.91 ms); reads that find nothing unchanged (two statements either
way). Server-side EXPLAIN ANALYZE: the EXISTS conjunct plans as a
one-time InitPlan (~3 us per statement); a windowed read's 3
statements execute in 0.069 ms vs the previous 4 statements' 0.067 ms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: purge bound scales with link fan-out; queue stamp is transaction time

Follow-up notes from the review session: the purge_max_segments
description says its derivative links cascade uncounted, so a call's
transaction also scales with the deployment's links per segment (the
promise in the ABC is kept by sizing this bound with that fan-out in
mind, which is the deployment's knob, not the caller's); the enqueue
stamp comment records that PostgreSQL's now() is transaction-start
time and that one deletion per transaction makes it one stamp per
entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: a deleted partition's handle is permanently invalid

"Obtain a fresh handle to continue" read as if deletion-and-recreation
were a routine flow; the contract is simply that deletion permanently
invalidates the handle, including against a later same-key creation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* revert: drop the old-layout startup probe

Handling pre-existing partitioned-layout deployments is out of scope
for the opt-in, pre-GA event backend; existing databases recreate
their schema, and startup stays a plain create_all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* nit: slim the purge claiming comment

The code comment keeps only the invariants the loop relies on; the
full rationale stays in design/segment_store_shared_tables.md, which
the comment now points at.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* nit: params docstring matches field descriptions, defaults at the end

The purge bounds' field descriptions carry the full text and the
docstring repeats them verbatim, with (default: N) moved to the end
of each description per the params convention.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: second-round fixes across locator, stores, and race tests

Second review round from the local review session (9 findings; 7
acted on here, the background-purger suggestion lands separately, and
the unbounded link-retire guard is kept with its tradeoff stated in a
comment -- bounding it would add a budget for a case that indicates a
broken schema).

- partition_key_for_session validated with a drifted private copy of
  the store's key contract: its re.match passed a trailing-newline
  session id through unhashed, and the store's re.fullmatch then
  hard-rejected it, failing session creation where hashing would have
  succeeded. The copy is deleted; the locator (and its tests) now call
  the store's own validate_partition_key, and the hash slice length
  comes from the now-public PARTITION_KEY_MAX_BYTES, so the two can
  never disagree again. Regression test verified to fail pre-fix.
- The SegmentStorePartition contract states that a call with empty
  input does no work and returns without checking the handle -- the
  empty-set guards return before any fence, which the docstring's
  "from then on" overstated.
- delete_partition on SQLite resolves the incarnation in the pin
  UPDATE itself via RETURNING; the locking select is PostgreSQL's path
  only, removing SQLite's extra round trip and its unreachable
  row-is-None branch.
- _open_or_create_partition loads the payload codec only on the create
  path (still before any registry write); opening an existing
  partition no longer materializes a codec it discards.
- Episode-store reads use ensure_tz_aware instead of an inline clone
  in the same file that imports it for writes.
- The purge's link-retire guard comment states it is normally a
  zero-row delete and unbounded only if referential integrity was
  actually broken.
- The two SQLite race tests gained started-events proving the racing
  task ran before the sample, so a loaded box cannot pass them
  vacuously by never scheduling it; the remaining grace periods are
  annotated (SQLite exposes no lock-wait state to observe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: background purge tick in the resource manager

Nothing but the inline drain in drop_session_partition ever called
purge_deleted_partitions, so a drain interrupted by a crash or a
dropped connection left its queue entry (and the partition's rows)
waiting for the next session deletion anywhere in the deployment.

The resource manager -- the component that owns each segment store --
now runs one background task per store: one bounded purge call per
fixed tick, exceptions logged and retried next tick, cancelled in
close() before the stores shut down. One call per tick keeps the
background work bounded by construction (a backlog drains over
successive ticks), and no purger coordination is needed at any
instance count because the store's claiming already makes racing
purgers safe. The store itself still never schedules reclamation;
this loop is the caller-side scheduler the ABC contract calls for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: purge loop reads the backlog signal

purge_deleted_partitions() returning True is the API's statement that
more work remains; discarding it drained a backlog at one bounded
call per tick (~167 rows/s at the defaults). The loop now runs
bounded calls back-to-back while the store reports more and sleeps
one tick only when it reports done or a call fails -- full-rate
recovery, still bounded per call, still one idle call per tick.
Pinned by a test that drains a three-call backlog under a
deliberately huge tick interval.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: empty-input calls MAY skip the handle check

The contract permits the shortcut rather than mandating it; an
implementation that checks anyway still conforms.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: pin the config-mismatch guard directly

The guard, its error type, and the ABC declaration predate this
branch, but no test staged a mismatch -- only the lifecycle-churn
test tolerated it as a domain outcome. Plaintext is the only concrete
codec config, so the test stands in a subclass for a future variant
(pydantic instances survive validation unrevalidated and compare
unequal by class). Verified to fail with the guard ablated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* purge: bound the integrity-escape link delete; warn on it and on collisions

The retire-path guard delete was the one unbounded statement in the
purge, unbounded precisely when it was not a no-op. It is now batched
under the same per-call budget as the segment rows: a full batch
leaves the queue entry for the next call (the existing call-until-
False contract absorbs it, callers unchanged), the normal case still
costs one zero-row statement, and reclaiming rows there logs a
warning naming the incarnation, since it means referential integrity
failed somewhere. Pinned by a test that stages orphan link rows
through a second SQLite engine without the foreign-key pragma and
drains them in warned batches; verified to fail against the unbounded
form.

The module's logger also gains the only other events worth an
operator's attention: a minted incarnation colliding (with garbage or
in the registry) is warning-logged at the detection site -- a genuine
collision is astronomically unlikely, so the log marks either broken
randomness or a misclassified persistent database error, visible even
when retries eventually succeed. Everything else either raises to the
caller or is normal operation, and stays unlogged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: shared purge budget is measured, not a guessed ratio

Measured with the purge's batched-delete shape on 100k rows each
(3 interleaved rounds): a segment row deletes at ~3.3 us and a
derivative-link row at ~1.0 us, so link rows are about 3x cheaper --
they are narrower, carry fewer indexes, and fire no cascade. That is
why integrity-escaped links draw count-for-count on the segment
budget instead of getting their own limit: one budget calibrated on
the most expensive row type upper-bounds the call, whereas a separate
link limit would be safe only under an assumed cost ratio.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: the row-cost direction is the shared budget's precondition

The shared purge budget stays conservative only while a link row
deletes cheaper than a segment row; widening the link table or adding
indexes to it revisits the choice.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: pace the purger, state the drain's real guarantee, close cleanly

Round-3 findings 1, 2, 5 and 6 -- the first three introduced by the
background purger itself.

- The purge loop now pauses briefly after every productive call
  instead of running delete transactions back-to-back: the pause
  yields the database (and SQLite's single write lock) to request
  serving, while a backlog still drains at one bounded call per pause
  and an idle store costs one call per tick. This also removes the
  in-process busy-timeout window between the background task and the
  inline drain on SQLite.
- The inline drain's comment claimed "the server schedules no other
  purger", which the purger commit falsified, and "reclaimed before
  returning", which SKIP LOCKED claiming never strictly guaranteed
  under any concurrent purger. Comment, design doc, and PR body now
  state the actual promise: rows are reclaimed promptly -- normally
  before the drain returns, and otherwise within the bounded call of
  whichever purger claimed the entry, moments later.
- close() clears the purge-task list and the store registry, so a
  second close is a no-op and a post-close get_segment_store can no
  longer hand back a shut-down store that silently never purges.
- The design doc no longer implies deployments can already tune the
  purge bounds through server configuration: the server constructs
  its stores with the defaults, and config plumbing is future work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: one datetime-normalization rule per filter path

Round-3 findings 3 and 11.

- The properties_json In branch bound raw values while its Comparison
  sibling normalized through _cast_properties_json_value; a datetime
  In list would bind datetime objects against the stored ISO-string
  form (an InterfaceError on Python 3.14's sqlite3, a never-matching
  comparison on PostgreSQL). Both leaf shapes now cast and normalize
  through the one function, which also aligns the float and bool
  casts the old branch fell through to as_string/as_integer. Same
  defensive-reachability status as the column-leaf In normalization
  kept deliberately: unreachable by In's declared value types,
  reachable at runtime.
- The episode store's start_time/end_time bounds re-implemented the
  UTC normalization inline; sql_filter_util's normalize_column_value
  is now public and both bounds use it, so the storage convention has
  one definition across compiled filters and dedicated bounds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review: StaticPool guard raises; empty add_segments short-circuits

Round-3 findings 9 and 10.

- The params validator's StaticPool guard was a bare assert, stripped
  under python -O -- and the design depends on multiple connections
  (the registry fence, deletion waiting out writers, SKIP LOCKED
  claiming all degrade on one shared connection). It now raises
  ValueError like the ephemeral-SQLite check beside it; pinned by a
  test, and the check stays a ValueError because pydantic converts
  only ValueError/AssertionError into a ValidationError.
- add_segments returns early on empty input, matching delete_segments
  and the ABC's empty-input permission; previously it opened a
  transaction and, on SQLite, took the write lock to insert nothing.
  The stale-handle test pins the shortcut.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* filter: datetime values normalize to UTC at node construction

The filter language now owns datetime semantics: a value denotes an
instant, and a naive value means UTC. Comparison.__post_init__
normalizes datetime values to UTC-aware instants (In gets the same
defensively -- its declared types exclude datetimes, but runtime lists
are unchecked), so every consumer -- parsed trees and programmatically
built ones, SQL compilers and vector stores alike -- receives
normalized instants by construction, and compilers only choose a
representation.

This is where the rule the recent fixes kept restating per leaf
actually belongs: the same aware-to-UTC-or-naive-means-UTC conversion
appeared in the SQL column leaf, the properties_json leaf, the episode
bounds, and twice in the Milvus store, and two of the drifted copies
were bugs fixed this round. With the invariant at the node, the SQL
column leaf's re-normalization became redundant and is reverted (it
binds tree values as-is); the properties_json leaf keeps its routing
because datetime-to-ISO-string is representation, not normalization;
the episode start/end bounds keep the shared helper because they are
raw API values outside any tree; other backends' now-idempotent
defenses are left for separate cleanup.

Pinned by tests that a programmatically built Comparison and a parsed
date() literal with a non-UTC offset both carry the UTC instant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* filter: drop normalize_column_value; contract stated on the protocol

With datetime normalization at node construction, the compiler-side
helper had no filter role left, and its one remaining consumer -- the
episode store's start/end bounds, which arrive outside any filter
tree -- now spells the convention inline as the two explicit steps,
ensure_tz_aware(...).astimezone(UTC). A composed to_utc() helper was
considered and rejected: the name does not pin the naive-means-UTC
tagging decision (an alternative design under the same name could
reject naive datetimes entirely), so the explicit steps are clearer
at each site.

The FilterExpr protocol docstring now states the construction-time
contract where the next value-carrying node's author will read it:
such a node normalizes datetime values to UTC-aware instants, and
compilers bind instants without re-normalizing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: the two-step datetime spelling is deliberate

Record the datetime convention in the design doc so a future cleanup
does not consolidate the repeated ensure_tz_aware(...).astimezone(UTC)
sequences back into the composed helper b636d62a deliberately removed:
a name that pins only the conversion, not the naive-means-UTC tagging,
hides a real design decision, so the repetition is load-bearing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: second ground for rejecting the composed datetime helper

A shared helper earns its place only when the name honestly pins the
unit AND the composition structurally prevents half-applied
normalization. The second condition fails here regardless of naming:
read paths legitimately need the tagging step alone (segment reads
reapply the stored original offset; cluster and episode reads only tag
naive database values), so ensure_tz_aware stays independently
available and the helper could not have removed the partial-use error
class.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: SQLite foreign keys enforced from engine creation

Round-4 findings 1, 4 and 5. The store registered its foreign_keys
pragma as a per-store connect listener, which has two structural
faults: connections the caller's shared engine pooled before the store
existed never receive the pragma, so cascade deletes silently leave
orphaned link rows for LIV…
The decorator was stacked twice in the speedkick merge (MemMachine#1548); one
application is the whole effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…emMachine#1588)

* Atomically swap vector search engine index files on save

SQLiteVectorStore persists each collection's index by calling the search
engine's save(), which wrote directly to the final path. A crash mid-write
left a truncated/corrupt file. Because index_saved=True makes the on-disk
index a durable contract (missing/corrupt is a hard IndexLoadError, not a
silent empty rebuild), an interrupted save could render a collection
unrecoverable.

Write the index to a sibling temp file and swap it into place with
os.replace (atomic on POSIX and Windows on the same filesystem), so a reader
sees either the old or new index, never a partial write; a failed save leaves
the previous index intact. Leftover temp files are cleared on load so a crash
does not leak them across restarts.

Implemented in the engines (shared index_persistence helper) rather than in
SQLiteVectorStore/SQLiteVectorStoreCollection, since the index save location
and number of files written differ across engine implementations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Make the index swap durable, not only atomic

The swap protects a reader from a torn index, but the vector store also
trims its pending-operation log once `save` returns -- and that log is the
only other copy of those vectors, since the records table stores no vector
column. So the swap reaching disk is load-bearing rather than a bonus:

- fsync the parent directory after the replace, since POSIX `rename(2)`
  leaves the new directory entry in the page cache. Best-effort and ignored
  on failure, matching SQLite's `unixSync`; a no-op on Windows, which has no
  equivalent operation.
- stop swallowing a failed fsync of the temp file. SQLite draws the same
  line -- a file fsync failure raises SQLITE_IOERR_FSYNC while a directory
  fsync failure is ignored -- and `EIO` means the writeback already failed
  and the dirty pages were dropped, which is exactly when the save must not
  be reported as committed. The existing cleanup then leaves the previous
  index in place with the log untrimmed, so the next save retries.
- use F_FULLFSYNC on macOS, where plain `fsync` leaves the data in the
  drive's volatile write cache, falling back when a filesystem refuses it.

State the resulting obligation on `VectorSearchEngine.save` itself, since
that is what the store now relies on: replace atomically, then make the
replacement as durable as the platform allows. An engine whose backend
already implements the whole protocol can delegate to it and skip these
helpers; the rest use `atomic_index_write`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Let the engine own index durability, not the vector store

The pending log holds the only durable copy of a vector between
checkpoints -- the records table has no vector column -- so trimming it is
safe only at an instant when the index provably holds those vectors. The
temp-write + rename protocol this PR shipped could not provide that
instant. A rename changes a directory entry, and Windows exposes no way
to flush one: os.fsync is _commit, which is FlushFileBuffers, which is
for file data, and you cannot open a directory to fsync it. The decisive
evidence is SQLite's own -- it threads a directory-sync flag through
every commit-relevant directory operation, honors it in unixDelete, and
declares it /* Not used on win32 */ in winDelete. So os.replace could
return, _save_collection_index could commit its trim durably behind it,
and a power cut could still roll the rename back: records forward, index
back, no copy of the difference left. MOVEFILE_WRITE_THROUGH is not a
fix; its documented guarantee covers copy-and-delete (cross-volume)
moves, not same-volume renames.

Take SQLite's answer, which was not to harden the directory operation
but to stop using one as a commit point (PERSIST commits by zeroing a
header, TRUNCATE by truncating, WAL by appending frames).

A base path now expands into two index slots plus a generation record
each, created once and thereafter only overwritten. A checkpoint writes
the index over the inactive slot and flushes it, then writes that slot's
generation record and flushes that. The record is the commit, and it is
a write into a file that already exists. It holds the generation and its
bitwise complement, so a torn write reads as absent rather than as some
other generation -- all or nothing without needing single-sector
atomicity from the hardware. load takes the highest believable
generation, and deliberately does not fall back to the older slot when
the published index will not parse: the log was trimmed against the
newer one, so the older is stale by exactly the ops that can no longer
be replayed.

Both backends already write straight to the path they are given, which
is what this protocol wants -- verified that repeated saves preserve the
inode and leave no stray files -- so no engine gains a temp file, a
buffer, or a rename.

Durability is entirely the engine's, including which artifact is live.
The store keeps no slot pointer, manifest, or generation, so no schema
change and no migration: what remains is one rule, never trim past what
save says is durable, and _save_collection_index already had that order.
index_path becomes index_base_path since it no longer names a file, and
discarding a collection asks the engine layer which files that covers.

BREAKING CHANGE: an index written by the previous protocol is not
published under the new one, so a collection with index_saved=True
raises IndexLoadError until its index directory is cleared and the
records re-ingested.

Anomaly tests walk every crash point in the publish sequence by
constructing the on-disk state each would leave, plus one that pins the
ordering itself (a failed index write must publish nothing) since
state-based tests cannot observe it. Verified against three deliberate
breaks -- dropping the complement check, writing the record first, and
reusing one slot instead of alternating -- each caught.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Publish the index atomically, and stop promising durability

The two-slot generation-record protocol bought a guarantee we have
decided not to make: that a save survives a power failure. Every engine
would have to implement and maintain that protocol, and the failure it
buys out is bounded -- search recall for the records applied since the
last checkpoint, repaired by re-ingesting them. The direction that
actually costs, a published index that will not parse, is closed by the
atomic swap on its own.

So this returns to the temp-file-plus-rename publication and spends the
difference on stating the contract instead of strengthening it: `save`
publishes atomically, never durably; the store trims the pending log
behind a publication a power failure can revert; a record whose vector
is lost that way still resolves by uuid, is absent from search until it
is upserted again, and nothing here detects the gap for the caller.

Reverts the durability and engine-owned-publication commits, keeps the
atomic swap, and adds a store-level test that reconstructs a reverted
publication deterministically -- restore the previous index bytes after
the trim -- to pin the direction it fails in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Report a lost embedding as lost, not as a missing feature

`update_feature` reads the stored embedding back when a caller updates a
feature without supplying one, and that is the only place in the server
that depends on the index still holding a vector. With publication now
atomic rather than durable, a power failure can leave a feature whose
row is intact and whose vector is not -- a state this path reported as
"Vector record not found", which points the caller at the wrong thing
and hides the repair.

Split the two cases. A record that is genuinely absent keeps the old
message; a record whose embedding the index no longer holds says so and
names the fix, which is to pass a fresh embedding.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Let a failed fsync fail the save, and cut the essay around it

`_flush_to_disk` wrapped its fsync in `contextlib.suppress(OSError)` and
called itself best-effort. A failed fsync is exactly the evidence that the
bytes are not safe to publish -- on Linux an EIO from fsync means writeback
failed, reported once and then cleared -- so swallowing it and renaming
anyway published a file we had positive evidence was bad. The safeguard
cost something and, in the one case it existed for, guaranteed nothing.
Nothing tested it either.

Let it propagate. `atomic_index_write` already unlinks the temp and
re-raises, so a failed flush now leaves the previously published index
standing, which is the correct outcome. A test pins that.

The fsync is not best-effort, and the docstring should not have said so: it
rules out a class rather than narrowing a window. Because the flush
completes before the rename is issued, and a durable write does not
un-happen, the new name can never appear over incomplete bytes. What the
missing directory fsync costs is the other direction -- the rename may not
survive, so the publish reverts -- and that is the benign one this store
already accepts.

The module docstring was 76 lines against 49 of everything else, most of it
argument rather than documentation: a walk through SQLite's `unixDelete` /
`winDelete` sync-flag handling, and a rejected two-slot commit protocol.
That is the PR's case for the design, not something to re-read every time
someone opens a 20-line module, and the PR body carries it. What a reader
here needs is the guarantee, the non-guarantee, and the cost.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Say why the swap is a rename, and what the reopened fd cannot see

Two things the module was silent on.

Why a rename at all. The stronger answer is to put the commit point inside
the file, where an fsync reaches it portably -- SQLite never renames, and
commits by truncating or zeroing its rollback journal, or in WAL mode by
appending frames whose checksums make a torn tail self-identifying. Both
need the writer to own the file format. A search engine owns its own and
exposes `save(path)`, so above that call a rename is the only atomicity
primitive left, and an engine whose format already commits that way needs
none of this. Worth saying, because "why not do the better thing" is the
first question the module invites.

What the reopened descriptor cannot see. Flushing is fine on a fresh fd --
dirty pages belong to the file, not to the descriptor that dirtied them --
but error reporting is not: Linux hands a writeback error to descriptors
open when it was recorded, so one recorded between the engine's close and
this open is never reported and the save proceeds on bytes already known
bad. Same shape as the 2018 PostgreSQL fsync report. It cannot be closed
from here: the engine writes through its own descriptor and closes it
before returning, and closing the window needs an engine that writes
through a handle the caller supplies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Fsync the index on a descriptor that predates the write

The fsync was on a descriptor opened after the engine had written and
closed its own, which flushes correctly -- dirty pages belong to the file,
not to the descriptor that dirtied them -- but reports nothing useful.
Linux samples the writeback error sequence when a file is opened, so a
descriptor opened after an error was recorded never learns of it: the fsync
returns success and the save publishes bytes already known bad. Same shape
as the 2018 PostgreSQL fsync report.

Open the temp before yielding it and hold it across the caller's write, so
the descriptor predates the bytes and any error from writing them is
reported here, where it fails the save.

That assumes the caller writes in place. Both engines do -- verified: the
inode is unchanged across `save_index` and `save`, and the held descriptor
sees the written size -- but it is their behaviour, not their contract. An
engine that built a file of its own and renamed it over the temp would
leave this descriptor on an orphaned inode, and the fsync would report on a
file nobody is about to publish. So it is checked before the fsync, and a
mismatch fails the save rather than passing it quietly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* State the in-place rule where the caller reads it

The descriptor held across the body only flushes what the body wrote if the
body writes the yielded path in place, and that requirement was recorded in
`_flush_to_disk` -- a private function nobody writing an engine opens. It
belongs on `atomic_index_write`, which is the API they use, alongside what
happens when it is broken: an `OSError` and no publication, so the mistake
surfaces at the first save rather than at a power cut.

`_flush_to_disk` keeps the mechanism -- why the descriptor has to predate
the write -- and now points at the rule instead of restating it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Ask the drive to flush on macOS, where fsync does not

`os.fsync` is not the same guarantee on all three platforms this ships to.
On Linux it flushes to the device, and on Windows `FlushFileBuffers` does
the same. Darwin's `fsync` explicitly does not: it returns once the data
reaches the drive, which may hold it in a volatile write cache. So on macOS
the ordering this module is built on -- data durable before the rename is
issued -- did not hold at the device, which is exactly the case it claims
to rule out.

`F_FULLFSYNC` asks the drive to flush that cache. Filesystems that cannot
refuse it, and there `fsync` is the most that can be asked, so a refusal
falls back; any other error is a write failure and propagates, as before.

The flush-failure test patched `os.fsync`, which Darwin no longer reaches.
It patches the module's own `_fsync` instead, which every platform does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Stop guessing which errnos mean "this drive cannot do that"

The fallback from `F_FULLFSYNC` to `fsync` was gated on an errno allowlist
-- ENOTSUP, EOPNOTSUPP, EINVAL -- so that a genuine write failure would
propagate rather than be quietly downgraded. Probing the actual returns on
Darwin shows the list is both incomplete and partly invented:

  ENOTSUP=45  EOPNOTSUPP=102   distinct here, so both are needed
  /dev/null   F_FULLFSYNC -> ENODEV(19), while fsync succeeds
  pipe/socket F_FULLFSYNC -> EBADF(9)
  EINVAL      never came from F_FULLFSYNC at all; it came from fsync

So ENODEV -- a real refusal, on a path anyone can reproduce -- would have
raised instead of falling back, and EINVAL was in the list by analogy
rather than evidence. What a network mount answers is not knowable from
here, which makes the whole list a guess that fails closed on whatever it
missed. This codebase does not classify driver errors by guessing, and
this was that.

Fall through on any failure instead. It is not a suppression: `fsync` runs
on the same descriptor and raises in its turn, so a flush that cannot
happen still fails the save. What the fallback gives up is the drive-cache
flush -- the guarantee this had before `F_FULLFSYNC` was asked for at all.
That is also what SQLite does with this same call, for the same reason.

The test drives it through `/dev/null`, which refuses with ENODEV and
accepts `fsync`; it fails against the allowlist and passes without it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Fail the save when the drive cannot be told to flush

`F_FULLFSYNC` failing fell through to `fsync`, on the reasoning that a
refusal is a statement about the filesystem rather than a write failure.
That reasoning does not survive asking what `fsync` alone actually buys on
Darwin.

Against a process or kernel crash it is enough: the data has left the OS
for the drive before the rename is issued, so the rename dies in the page
cache and the old index stands. Against power loss it is not. The data
sits in the drive's volatile cache, the rename's metadata joins it moments
later, and nothing orders them -- and the rename is a few bytes against an
index of megabytes, so a drive flushing as it pleases can easily put the
new name on media while the bytes behind it are still queued. That is the
torn publication this module exists to prevent, in precisely the scenario
its docstring is about.

So the fallback answered a request for ordering with a flush that does not
provide it, and said nothing. A filesystem that cannot order data ahead of
a rename is not one to publish an index onto; raise, and let the operator
point `index_directory` at storage that can.

This is also the simpler code. Refusal and failure now take the same path,
so no errno is inspected -- there is no line to draw and no list to get
wrong, which is what the previous two revisions kept getting wrong in
opposite directions. The `/dev/null` test went with the fallback it pinned.

The earlier defence of falling back rested on network and FUSE mounts
being a realistic home for an index directory. They are not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 21105d6)
…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 779362c)
The Qdrant and Milvus clients were built without a timeout, so a remote
write could hang a request indefinitely. `request_timeout` on QdrantConf
and MilvusConf, in seconds, is passed to the client; it is required, with
no default, so a deployment states how long it is willing to wait, and the
configuration wizard supplies 30 seconds as the starting point. The sample
configurations and the configuration docs show the option.

A breaking configuration change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A required field bounds only the deployments that added it and fails the
rest at load; a default bounds every deployment, including one whose
cfg.yml predates the option, and matches every other field on QdrantConf
and MilvusConf. The wizard no longer carries the value: it constructs the
confs and the field supplies it. Zero and negative values are rejected at
load rather than handed to httpx as the request timeout and to pymilvus
as the gRPC deadline, where zero expires every request on arrival.

Without the option, qdrant-client already bounded a request at 5 seconds
(httpx's default for REST, DEFAULT_GRPC_TIMEOUT for gRPC); pymilvus passed
no deadline, so a Milvus request could wait forever. The default applies
30 seconds to both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Neither client's keyword carries the unit, and the field mirrors neither
(both take `timeout`), so it follows max_retry_interval_seconds on the
embedder and language model configurations instead. The sample
configurations lose their "seconds" comments, which the name now carries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
MilvusClient's constructor timeout is the time it waits for the channel
to become ready, at construction and on reconnect; a request is bounded
only by the timeout passed to that request, and pymilvus keeps no default
for it, so every request the store made had no deadline. The store now
takes request_timeout_seconds and passes it on every request; the client
keeps it as its connection bound. A test wraps every request method and
checks the timeout reaches each call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
qdrant-client takes its timeout as an int and rounds a fraction up, so a
fractional value was honored by pymilvus and silently changed for Qdrant.
An int is honored exactly by both, and matches max_retry_interval_seconds
on the embedder and language model configurations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…e the stores are configured

MilvusVectorStoreCollection.get() was the one client request without
`timeout=`; the spy test now exercises it, so a request without the
timeout fails the test. The configuration parameter table gains
`request_timeout_seconds`, the databases page's Milvus example carries it,
and the configuration page gains a Qdrant example beside the Milvus one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
QdrantVectorStore was the fourth component built without a metrics factory.
OperationTracker accepts metrics_factory=None and then discards every timing
without an error, so the store looked instrumented and emitted nothing - the
same defect as the Neo4j store, the episode store and the session store, which
is why no Qdrant latency was observable.

QdrantConf gains MetricsFactoryIdMixin so it can resolve one, and
database_manager passes it through.

test_qdrant_creates_vector_store pinned the exact params and had to change.
It now asserts metrics_factory is not None rather than pinning it: passing the
keyword is not the property worth guarding, since None is accepted and silently
discards everything. Removing the wiring fails it.

Ported to main without MemMachine#1532's Dockerfile change (the EXTRAS build arg),
which is unrelated to the wiring; the `metrics_factory_id` key is added to
the database configuration table in the docs.

(cherry picked from commit b6c90ab)

Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The simple chatbot example, the TypeScript REST demo and the Dify plugin's
add-memory tool wrote to a project without creating it, relying on the
write to create it. Each now creates its project before its first memory
request and accepts 409 as the project already existing. No behavior
changes for them; they stop depending on a write creating a project.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
(cherry picked from commit 4cae58a)
Adding memories to, or searching, a project that did not exist created it,
with the server's default configuration, without the caller's knowledge.
Now only the create-project request creates a project: a write or a
search opens the session and answers 404 for an unknown project, as the
search endpoint already promised; the manager's open-or-create goes.

Two callers depended on the implicit creation. `org_id` and `project_id`
default to `universal`, so the API promises the project
`universal/universal`; the server creates it, once, at startup, and leaves
one that already exists as it is. The MCP add tool names its own project
and has no create-project counterpart, so it creates the project it writes
to, once, and says so. The API doc strings and the OpenAPI document say
which requests create a project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
(cherry picked from commit d21a5d0)
…anything, on every entry point

A write persisted its episodes before opening episodic memory, and a
write or search that targeted only semantic memory never opened it, so a
request naming a project nobody created could still insert episode rows
under that key and answer 200. Every write now checks the registry first,
and a search that does not open episodic memory checks it too; the check
is one registry read, and the episodic open, which refuses an unknown
project itself, is unchanged.

`memmachine-server --stdio` built its resources without setting or
starting the module-level MemMachine the tools read, and so never created
the default project either; it now starts and stops through the same
calls as the HTTP servers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
… (speedkick) (MemMachine#1598)

* Answer with cosine scores, not with stored vectors

Two changes to one contract, taken together so its signatures are not
churned twice.

Cosine is now the only similarity. Every embedder MemMachine ships already
produced vectors meant to be compared that way -- OpenAI hard-coded it,
Bedrock defaulted to it, SentenceTransformer only reported what the model
declared -- while every layer that touched a score paid for the other
three metrics in direction flags, threshold directions, and per-backend
tables mapping the enum onto native metric names. `SimilarityMetric` is
gone; scores are cosine similarities in [-1, 1] and the names say so:
`QueryMatch.score` and `SearchMatch.score` become `cosine_similarity`,
`query(score_threshold=)` becomes `query(min_cosine_similarity=)`, which
no longer needs a direction to be meaningful. The Bedrock embedder's
`similarity_metric` config key goes with it, and the install and
configuration docs drop it.

And `VectorStoreCollection.get` is removed, with nothing offered in its
place. It had one production caller: the semantic storage read a feature's
stored embedding back so it could write the same embedding again with
fresh properties, because `upsert` demands a whole record.
`return_vector=True` was passed at that one call site and nowhere else,
and `VectorSearchEngine.get_vectors` existed to serve it. `set_properties`
serves that caller directly -- correcting a record's properties no longer
requires holding its vector -- and on SQLite and sqlite-vec it is an
UPDATE that never touches the index.

No scoring-by-id entry point takes `get`'s place. One would be needed if a
caller assembled a candidate set outside the store and asked for those ids
to be scored, which is what a selective-filter plan running above the
store would do. Property filtering stays inside the store instead, so the
candidate set stays there too, and a store-side regime can reach engine
keys directly without a public method addressed by record UUID.

Two consequences beyond the vector store. The vector graph stores carried
a metric per stored embedding, as a companion property beside every
vector; that is gone and `Node.embeddings` holds plain vectors.
NebulaGraph indexes only L2 and IP and its `cosine()` cannot be
APPROXIMATE, so with cosine alone no index it can build serves a query --
its ANN branch and vector index creation could no longer run and are
removed; search there is always exact KNN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Let each store own its own mapping, and stop copying it

The vector store is not the authority for anything but vectors, so a
consumer that read a record's properties back out of it was reading a copy.
Both consumers did that, and for the same reason: to get from a search hit
back to a domain id. What the copy cost them differed. Semantic memory's
was of mutable columns, correct only as far as the last write to it and
stale from then on. Event memory's was an immutable uuid pairing, so it
stayed correct -- it simply duplicated a mapping the segment store already
served from an index, and spent a reserved property name doing it.

`QueryMatch` therefore answers `record_uuid` and a score. Properties stay
stored and filterable, because a filter is evaluated against the copy
rather than trusted as the record, and are no longer returned. That is the
end of `return_properties`, of `Record` on the read path, and of
`set_properties`, which existed only to keep a copy fresh that nobody
reads now.

Event memory used a `_segment_uuid` property to reach a derivative's
segment. The segment store already holds that mapping on the derivative's
own row, non-null, under a primary key that leads on exactly the columns
the lookup filters -- so the copy bought nothing an indexed read does not,
and `get_segment_uuids_by_derivative_uuids` mirrors the forward lookup that
was already there. The property is gone, and with it a reserved field name.

Semantic memory had no mapping to reach for. Its record uuid was
`uuid5(namespace, feature_id)`, which is one-way, so the feature id had to
ride along in the properties. It now owns a `vector_uuid` column, unique
and minted per feature, and resolves hits through it -- the vector id is
the vector's own, and the caller keeps the correspondence. Every property
that collection carried was a copy of a column on the feature row, never
filtered on, so the payload goes entirely.

The delete paths read that column before deleting the rows, since the row
is what says which vector record a feature owns.

Data on speedkick does not survive this: existing semantic features have no
`vector_uuid`, and existing collections carry properties nothing reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Restore the Nebula tests a bad edit took with it

Cutting `test_similarity_metric_mappings` searched for the next `\ndef ` or
`\nclass ` to find where the function ended. Every test after it is `async
def`, so both searches missed, the end fell through to end-of-file, and the
edit deleted 429 lines: ten test functions, of which four had anything to
do with similarity metrics.

Gone with it were the empty-input cases for adding nodes and edges, the
none-property cases for both, the wrong-collection delete, the
nonexistent-uid read, the multi-property directional search -- and
`test_search_similar_nodes_cosine_metric`, the test for the one metric that
survives this change.

Nothing caught it because these tests skip everywhere, CI included: the
vector support they exercise needs NebulaGraph Enterprise >= 5.0, which the
fixture reaches at NEBULA_HOST and skips without.

Rebuilt from the file on speedkick with the metric stripping applied, then
removing only what has no subject left: the mapping helpers' test, the dot
and manhattan searches, and the ANN search -- Nebula indexes only L2 and
IP, so with cosine alone no index it can build serves a query.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Keep NebulaGraph's ANN by spelling cosine as an inner product

Nebula's `cosine()` is KNN-only -- it cannot take APPROXIMATE -- and its
vector indexes offer only L2 and IP. From that I concluded cosine could
never be approximate here and deleted the ANN search branch and vector
index creation as unreachable. That was wrong: cosine similarity between
unit vectors *is* their inner product, so an IP index over normalized
vectors gives cosine ranking with ANN.

Vectors are normalized in `_vector_to_gql_literal`, which is the single
place any vector becomes a literal -- node writes, edge writes, ANN queries
and exact queries all pass through it, so the stored side and the query
side cannot disagree about it. `inner_product()` DESC against an IP index
replaces the metric lookups.

IP rather than L2, though both rank identically over unit vectors
(‖a-b‖² = 2(1-cos), monotone in cos, and verified equal by argsort). The
reason is not numerical: measured over near-duplicate float32 unit vectors,
recovering cosine as 1 - d²/2 is 1.1x worse than reading it off IP, which
is nothing. It is that the contract answers a cosine similarity, and with
IP over unit vectors the index score already is one -- no conversion, no
assumption about whether the engine hands back the distance or its square,
and no result landing outside [-1, 1] needing a clamp.

`test_search_similar_nodes_ann` comes back with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

* Drop the constant aliases left over from configurable metrics

Collapsing the metric enum left an instance attribute that only ever
copied a class constant (self._space = self._SPACE) and, in the Nebula
store, a local that copied a constant into a second local before use.
Read the constants where they are used.

The Nebula metric names move to module scope alongside the package's
other fixed identifier constants; they are neither per-instance nor
overridden.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit abf92a3)
…emMachine#1603)

Require a vector on the record type nothing reads back

`Record` is written and never returned, so `vector` being optional
described nothing a caller could do. Every store rejected `None` at its
own `upsert` with the same message, four copies of one rule that the type
could state once. `properties` was optional in the same way, so Qdrant
carried `record.properties if record.properties is not None else {}`.

The vector is required and the properties default to `{}`. The four
checks go, and rejection moves to the model, where a caller finds it at
construction rather than at a write.

Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit cdeef88)
…ck) (MemMachine#1606)

* Regenerate the OpenAPI document under the locked FastAPI

`docs/openapi.json` predates the FastAPI release in `uv.lock`
(0.141.1), whose `ValidationError` component carries `input` and `ctx`;
regenerating the document with `docs/tools/generate_openapi.py` adds the
two fields and changes nothing else. Separate from the API changes above
it so their diffs of this file show only what they change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

* Remove per-project filterable properties

A project could declare `properties_schema`, a set of caller property keys
with types, on its long-term memory configuration; the event backend merged
it into the vector store collection's indexed schema and rejected filters on
any other `m.<key>`. That let a tenant create database resources (indexes,
columns) by naming them in a request, which is what forced per-collection
native resources named by a hash of their schema on the backends that limit
them.

The option is removed from the server configuration, the project API and
the memory-configuration API, the Python SDK, the sample configurations,
the configuration docs and the OpenAPI document. A filter may name any
`m.<key>`; the stores evaluate it on the properties they hold. What a store
indexes is decided by the deployment, not per project.

A breaking API change on `speedkick`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…quest

The event backend created a session's vector store collection and segment
store partition on the first request that opened the session, so a search
or a write for an unknown session created storage as a side effect, and
the service locator was the only place that knew both stores' create paths.

The owner is the session. Every path that creates a session row runs
through EpisodicMemoryManager._create_session, which inserts the row and,
when the row is new, creates the session's partitions in its segment store
and its vector store (create_episodic_memory_storage); an equivalent
re-create accepts the row and leaves the storage as it is. The request
path binds handles with the stores' lookups and raises
SessionPartitionMissingError when a partition is absent: a session without
its storage is broken, not new. Deleting a session with no open instance
deletes its partitions by key, so a session whose storage was never fully
created can still be deleted. MemMachine.create_session goes through the
manager for the same reason.

The semantic manager owns its one collection and creates it, once, at the
storage's first use. With that, nothing calls the stores' open-or-create.

The API is unchanged: the manager's open-or-create still creates a session
a memory request names, now through the same path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
Nothing calls them since a session's storage is created with the session:
`open_or_create_collection` and `close_collection` leave the vector store
interface and its four backends, `open_or_create_partition` and
`close_partition` leave the segment store interface and its
implementation, and the two config-mismatch errors that only open-or-create
raised go with them. A store creates on `create_*`, strictly, and looks up
on `open_*`, answering None; create-if-absent is the owner's, where the
key's provenance is known.

Source changes are deletions only. The tests that exercised
open-or-create as a fixture use a test-side create-if-absent instead, and
the tests of its own semantics go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…res' lookup to get_partition

A vector store's logical collection becomes a partition, the segment store's word for the same thing, and both stores' lookup is get_partition, answering None like a Python get. Identifiers only, produced by the script below; the (namespace, name) identity, the per-partition config and every docstring are as they were, and the next change gives them their meaning. The native clients' create_collection and delete_collection keep their names.

```sh
set -e
cd "$(git rev-parse --show-toplevel)"
git mv packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_collection.py \
       packages/server/server_tests/memmachine_server/common/vector_store/in_memory_vector_store_partition.py
git ls-files -z 'packages/server/*.py' | xargs -0 perl -0pi -e '
  s/VectorStoreCollection(?!Config)/VectorStorePartition/g;
  s/in_memory_vector_store_collection/in_memory_vector_store_partition/g;
  s/vector_store_collection(?!_schema|_namespace)/vector_store_partition/g;
  s/open_collection/get_partition/g;
  s/def create_collection\(/def create_partition(/g;
  s/def delete_collection\(/def delete_partition(/g;
  s/\.create_collection\((\s*namespace=)/.create_partition($1/g;
  s/\.delete_collection\((\s*namespace=)/.delete_partition($1/g;
  s/\.create_collection(?=\s*=\s*AsyncMock|\.assert_)/.create_partition/g;
  s/\.delete_collection(?=\s*=\s*AsyncMock|\.assert_)/.delete_partition/g;
  s/"create_collection"/"create_partition"/g;
  s/"delete_collection"/"delete_partition"/g;
  s/only delete_collection is invoked/only delete_partition is invoked/g;
  s/test_delete_collection_/test_delete_partition_/g;
  s/open_partition/get_partition/g;
'
uv run ruff check --fix --quiet packages/server
uv run ruff format --quiet packages/server
```

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…uilt by the composition root

A vector store was a factory of logical collections, each identified by a
(namespace, name) pair and created with its own dimensions and schema; a
backend that limits native collections shared one among logical
collections of equal configuration, under a name derived from a hash of
that configuration, and a registry per namespace mapped names to it.

A store is now one collection: `VectorStore(collection, vector_dimensions,
indexed_properties)` names its one native collection (or its tables and
index files) at construction, every partition of it shares the
collection's dimensions and schema, and `provision()` creates the
collection's durable resources idempotently, before `startup`.
`create_partition(key)`, `get_partition(key)` and `delete_partition(key)`
take a string key; a partition is a payload value (Qdrant), a partition-key
value (Milvus) or a pair of tables (the SQLite stores) inside the
collection, and the registry beside it records what each partition was
created under, so a store built with other dimensions or another schema
raises VectorStorePartitionSchemaMismatchError instead of reading columns
and vectors that are not there. Collection names may be 64 bytes; the
hash-derived native names go, and with them `VectorStoreCollectionConfig`
and the per-partition config.

`DatabaseManager.get_vector_store(backend, collection=, vector_dimensions=,
indexed_properties=)` builds and caches one store per (backend,
collection), keyed by the service's system keys; asking for a collection
again with other dimensions or keys is a configuration error. The event
backend's collection is `long_term_memory__<embedder>` and the semantic
memory's `semantic_memory__<embedder>`, one cell of the purpose-by-embedder
matrix each; the two SQLite stores of one backend share its engine, and
MemMachine warms the event backend's store through the locator, since
building it needs the embedder's dimensions.

The data path is as it was: a partition stores every property of a record
and filters on any key, with the declared keys indexed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…bitrates; delete logically, reclaim by purge

A partition is identified to callers by its key and inside every store by
an incarnation minted when the partition is created. Records, points and
index files are keyed by the incarnation, never by the key, so a partition
deleted and re-created under the same key starts empty and its
predecessor's storage is never adopted by, or reclaimed out from under,
the successor. Handles are bound to one incarnation: once it is deleted,
every operation of the handle raises VectorStorePartitionHandleStaleError.

The registry that maps a key to its incarnation lives in SQL on every
backend, so any server process may create, delete and purge, with nothing
coordinated outside the database: creation is an insert the primary key
arbitrates (a racing creator on any process gets AlreadyExists; a minted
incarnation colliding with a live or queued one is re-minted, up to
_MAX_MINT_ATTEMPTS, then VectorStoreAttemptsExhaustedError), deletion is
one transaction that queues the incarnation for purge and deletes the
row, and a purge claim is a row lock the database hands to one purger at
a time. delete_partition therefore makes the partition unreachable at
once, and the new purge_deleted_partitions reclaims the storage
afterward, oldest deletion first, a bounded amount per call, safe to
repeat and to run from several processes.

Both SQLite stores keep the registry beside their data in the one file:
every partition of a collection in shared tables (records, vec0 with the
incarnation as its partition key, pending-operation log) with a purge
queue beside the registry, writes fenced with a self-checking UPDATE or a
registry SELECT under BEGIN IMMEDIATE. Qdrant and Milvus have no
transactions, unique constraints or conditional writes, so their registry
is `SqlPartitionRegistry` in the deployment's relational database: one
table pair per backend kind (`vector_store_qdrant_pt`/`_gc`,
`vector_store_milvus_pt`/`_gc`), shared by every store on that kind of
backend, a purge claim under `FOR UPDATE SKIP LOCKED` on PostgreSQL so
concurrent purgers split a backlog, retired when the backend's
filter-delete returns and kept when it raises, and a per-operation fence
that is a registry lookup by incarnation. The registry collections the
two stores kept inside their backends go, and with them the per-process
lock tables that serialized creation within one process only, and
`registry_replication_factor`. `QdrantConf` and `MilvusConf` gain a
required `registry_database`, the name of a relational database under
`resources.databases`; DatabaseManager hands its engine to the store, and
`provision()` creates the two tables on it, surviving a racing
provisioner.

The `VectorStore` contract no longer restricts a partition to one
process: every operation is safe from any process sharing the backend,
and a store that cannot give that says so itself (the engine-backed
SQLite store holds a partition's index in the process that opened it;
the sqlite-vec store is shared by the processes of one node).

The SQLite stores' on-disk layout changes (shared tables per collection
in place of tables per partition); existing SQLite vector store files are
not migrated.

partition_lifecycle_contract.py holds the contract tests every backend
mixes in: stale handles, empty re-creation, idempotent deletion, purge
reclaiming what deletion deferred and leaving live partitions alone; run
against Qdrant in local, REST and gRPC modes. test_sql_partition_registry.py
runs the registry on SQLite and PostgreSQL, concurrent creators included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
The first time a vector store is handed out, the resource manager starts
the same purge loop it runs for segment stores, one per (backend,
collection); close() cancels both sets. Mechanical churn in the same
change: the loop's interval and pause constants lose their SEGMENT_STORE_
prefix and the loop takes a label for its failure log line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A Qdrant or Milvus handle read its incarnation as live and then made the
remote call; a deletion committing between the two left the write under
a dead incarnation and the operation reporting success. The handle now
reads the row again after the call and raises the stale error if the
incarnation died meanwhile, so a completed operation never reports
success on a dead partition. No lock spans the remote call and the
logical delete waits for nothing; a write that landed under a dead
incarnation is the purge's to reclaim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
One purge could not be the last: a Qdrant or Milvus write the registry
read as live can land after the purge that followed the deletion, and
the queue entry was retired at that first purge, so nothing would ever
find those points. The queue entry is now the incarnation's tombstone.
A purge round looks for one point under the incarnation and deletes by
filter if there is one; a round that found points keeps the entry due;
a round that found none stamps it clean; and the entry is removed only
by a round that finds nothing again once `tombstone_retention_seconds`
(a day unless configured, per store, on the database clock) has passed
since the clean stamp. A round that finds a late write clears the stamp
and the rounds start over. Until removal the incarnation is never
re-minted. `purge_deleted_partitions` returns True when the round found
points, so the sweeper comes back for another round, and False when it
found none or nothing was due.

The retention is the one clock in the design and it decides nothing
about validity: a stale write is refused by the liveness check after
the operation, and the retention only has to exceed, by orders of
magnitude, the longest a request can be in flight, which
request_timeout_seconds bounds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
A retention is set in days and compared on the database clock; a
fraction of a second has no use, and the field then matches
request_timeout_seconds and max_retry_interval_seconds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…example and the parameter table

The configuration page's Qdrant example lacked the registry database the
store now requires, and neither new key had a row in the parameter table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…tization

`QdrantConf` gains `hnsw_config`, `optimizers_config` and
`quantization_config`, plain mappings mirroring qdrant-client's
`HnswConfigDiff`, `OptimizersConfigDiff` and `QuantizationConfig`, so
qdrant-client stays optional for configuration parsing; the store's
params validate them against qdrant's own models. They apply to the
store's data collection, never to its registry collection.

`m` must be 0 or unset: the collection is multi-tenant and disables the
global graph in favor of per-partition payload indexing, so a deployment
tunes `payload_m`, which defaults to 16 as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
…w them in the samples and the parameter table

The option docstrings and descriptions still spoke of "native collections"
and "registry collections": since MemMachine#1631 the store is one collection and
its registry is relational tables. The sample configurations gain a
commented block with the three keys, checked against qdrant-client's
models, and the configuration page's parameter table gains their rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn
@edwinyyyu
edwinyyyu force-pushed the feat/qdrant-collection-options-speedkick branch from c9b1f5e to c533194 Compare September 18, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants