Connor-Matthew · Oct 16, 2026 · Oct 16, 2026 · Oct 17, 2026 · Oct 18, 2026 · Oct 18, 2026
diff --git a/README.md b/README.md
diff --git a/docs/source/_static/custom.css b/docs/source/_static/custom.css
@@ -6,3 +6,31 @@
        margin: 0;
     }
 }
+
+.latest-blogs .sd-card {
+    height: 100%;
+    border-radius: 0.75rem;
+    transition: border-color 0.15s ease, box-shadow 0.15s ease;
+}
+
+.latest-blogs .sd-card:hover {
+    border-color: var(--pst-color-primary);
+    box-shadow: 0 4px 14px color-mix(in srgb, var(--pst-color-text-base) 8%, transparent);
+}
+
+.latest-blogs .sd-card-img-top {
+    width: 100%;
+    height: 148px;
+    object-fit: cover;
+}
+
+.latest-blogs .sd-card-title {
+    font-size: 1rem;
+    line-height: 1.4;
+}
+
+.latest-blogs .blog-card-date p {
+    margin-bottom: 0;
+    color: var(--pst-color-text-muted);
+    font-size: 0.875rem;
+}
diff --git a/docs/source/api-reference/cpp/mooncake-store.md b/docs/source/api-reference/cpp/mooncake-store.md
@@ -64,7 +64,7 @@ struct ReplicateConfig {
 };
 ```
 
-`dfs_replica_num` may currently be `0` or `1`. When it is `1`, `replica_num >= 1` is required, so DFS-only placement is not supported. DFS replicas currently support only the `default` tenant and require the master and client DFS backends to be configured with the same shared root and shard layout. See the {ref}`DFS deployment documentation <dfs-storage>` for setup and lifecycle limitations. Native C++ clients must initialize a `DistributedStorageBackend` and attach it with `SetDfsStorageBackend()` before issuing DFS reads or writes. DFS descriptors are carried by each `PutStart`, `UpsertStart`, or query response; there is no client-side descriptor cache. The Python/RealClient setup path attaches the backend through `FileStorage`.
+`dfs_replica_num` may currently be `0` or `1`. DFS-only placement (`replica_num = 0`, `nof_replica_num = 0`, `dfs_replica_num = 1`) is supported; the DFS replica is then the object's only copy, so DFS eviction or removal deletes the object. DFS replicas currently support only the `default` tenant and require the master and client DFS backends to be configured with the same shared root and shard layout. See the {ref}`DFS deployment documentation <dfs-storage>` for setup and lifecycle limitations. Native C++ clients must initialize a `DistributedStorageBackend` and attach it with `SetDfsStorageBackend()` before issuing DFS reads or writes. DFS descriptors are carried by each `PutStart`, `UpsertStart`, or query response; there is no client-side descriptor cache. The Python/RealClient setup path attaches the backend through `FileStorage`.
 
 Soft pinning starts when the first replica becomes readable and has a fixed
 lifetime: reads do not extend it. `PRESERVE` keeps the committed deadline on an

diff --git a/docs/source/api-reference/python/mooncake-store.md b/docs/source/api-reference/python/mooncake-store.md
@@ -30,7 +30,7 @@ imported and accepts cpp or rs.
 This page describes the shared facade and the default C++ backend. See the
 [Store-RS Python API](store-rs.md) for backend-specific runtime behavior, API
 coverage, and compatibility notes. The
-[source-checkout quickstart](../../getting_started/store-rs.md) shows how to
+{ref}`Build with Store-RS <build-with-store-rs>` section shows how to
 build the root wheel with Store-RS enabled.
 
 ## Quick Start
@@ -667,9 +667,10 @@ config.nof_replica_num = 1
 **Default:** `0`
 **Status:** **Work in progress; development and evaluation only.**
 **Description:** Requests an additional replica in the configured shared
-distributed filesystem. The supported values are currently `0` and `1`. When
-set to `1`, `replica_num` must be at least `1`, so DFS-only placement is not
-supported. DFS replicas currently support only the `default` tenant.
+distributed filesystem. The supported values are currently `0` and `1`.
+DFS-only placement (`replica_num = 0` with `dfs_replica_num = 1`) is supported;
+the DFS replica is then the object's only copy, so DFS eviction or removal
+deletes the object. DFS replicas currently support only the `default` tenant.
 
 ```python
 config = ReplicateConfig()
@@ -744,6 +745,40 @@ config = ReplicateConfig()
 config.prefer_alloc_in_same_node = True
 ```
 
+#### session_ids
+**Type:** `Optional[List[str]]`
+**Default:** `None`
+
+Optional per-key write attribution. Supply one entry for each input key (one
+entry for a single-key write). Each ID must be valid UTF-8 and at most 256 bytes.
+An empty string means no new attribution. The field does not change object identity,
+replication, group membership, leases, or eviction policy.
+
+```python
+config = ReplicateConfig()
+config.session_ids = ["session-a", "session-a", "session-b"]
+store.batch_put_from(keys, buffer_ptrs, buffer_sizes, config)
+```
+
+The tag becomes committed when the first replica of the write becomes readable.
+A skipped duplicate Put leaves the original tag unchanged. Upsert with a
+non-empty ID replaces attribution at commit; an omitted or empty ID preserves
+the previous tag, including when storage is reallocated. Failed/preempted
+writes do not commit their pending tags. Committed tags survive metadata
+OpLog/snapshot recovery and standby promotion; old metadata defaults to no tag.
+
+With KV events enabled, each `stored` / `BlockStored` map carries a single
+`session_id` string, or MessagePack `nil` when untagged. Replica availability
+changes reuse the committed object tag. `removed` and `cleared` keep their
+existing schemas: they retract availability, not a particular session's
+association. This matches SGLang's single-session stored-event attribution;
+it does not enumerate every session that has reused a shared object.
+
+Old RPC readers can ignore the compatible extension, but do not persist or
+publish session tags. Upgrade the Store client and Master to enable attribution.
+A storage node's disk scan alone cannot reconstruct metadata tags after all
+Master metadata is lost; preserve the metadata recovery artifacts as well.
+
 #### group_ids
 **Type:** `List[str] | None`
 **Default:** `None`
@@ -1619,19 +1654,51 @@ else:
 ---
 
 #### batch_probe_key()
-Point-in-time existence check for multiple objects in a single batch
-operation, granting no read leases.
+Probe multiple objects, optionally selecting and leasing the last complete
+candidate. The default `"none"` policy remains a lease-free existence check.
 
 ```python
-def batch_probe_key(self, keys: List[str]) -> List[int]
+def batch_probe_key(
+    self, keys: List[str], policy: str = "none", candidate_size: int = 1
+) -> List[int]
 ```
 
 **Parameters:**
 - `keys` (List[str]): List of object identifiers to check
-
-**Returns:**
-- `List[int]`: List of existence results (1=existed at probe time,
-0=not exists, -1=error)
+- `policy` (str): `"none"` or `"LastHitOnly"`.
+- `candidate_size` (int): Number of consecutive keys forming each candidate.
+  Ignored for `"none"`. For a nonempty `"LastHitOnly"` request, it must be
+  positive and divide the number of keys exactly. Empty input returns `[]`.
+
+**Returns:**
+- Both policies return per-key existence results in the same length and order
+  as `keys`: 1=readable when checked, 0=missing or unreadable. Negative values
+  indicate errors, not cache misses.
+- With `"LastHitOnly"`, the last group of `candidate_size` all-1 results is
+  the selected, leased candidate. Earlier complete groups and incomplete
+  groups retain their existence results. If no group is all-1, no complete
+  candidate was selected.
+
+Candidates are ordered by the caller. A candidate is complete when every key
+has a readable replica. Earlier candidates may be missing or incomplete.
+The caller must include all required components,
+cache groups, and rank shards; Store does not infer prefix or model semantics.
+Duplicate key positions are allowed and must not alter the grouping.
+
+The Master first probes without leases, then rechecks and leases complete
+candidates in reverse order using the existing per-shard lookup. Recheck
+results replace that candidate's initial results; a candidate that loses a
+member cannot remain all-1. A failed candidate may retain partial leases
+while selection falls back to an earlier candidate. This is not an atomic
+snapshot of all keys, and unselected existence results are not lease guarantees.
+
+Selected objects receive the Master's default read-lease TTL, protecting them
+from normal eviction while the leases remain valid; soft-pin deadlines are
+unchanged. The policy does not protect against lease expiry, forced removal,
+or replica loss. Existing shared group leases (or a physical key repeated
+outside the selected candidate) may also protect unselected positions. No
+persistent group registration is added. Invalid candidate sizes return
+per-key parameter errors; unknown policy strings raise `ValueError` in Python.
 
 **Example:**
 ```python
@@ -1641,6 +1708,29 @@ candidates = [key for key, exists in zip(keys, results) if exists == 1]
 print("Probed candidates (unprotected from eviction):", candidates)
 ```
 
+For checkpoint selection, first use `batch_is_exist()` to protect the required
+KV data and determine its usable boundary. Pass only valid checkpoint
+candidates within that boundary, in increasing resume-position order:
+
+```python
+keys = ["checkpoint128.rank0", "checkpoint128.rank1",
+        "checkpoint256.rank0", "checkpoint256.rank1"]
+results = store.batch_probe_key(keys, policy="LastHitOnly", candidate_size=2)
+if any(result < 0 for result in results):
+    raise RuntimeError("Checkpoint lookup failed")
+selected = next((i for i in reversed(range(len(keys) // 2))
+                 if all(result == 1 for result in results[i * 2:(i + 1) * 2])), None)
+# Both complete: [1, 1, 1, 1] selects 1.
+# Last incomplete: [1, 1, 1, 0] selects 0.
+# Load only the selected checkpoint, with the KV prefix it needs.
+```
+
+**Upgrade note:** the policy changes the `BatchProbeKey` Master RPC and the
+DummyClient-to-RealClient batch-probe RPC signatures. Upgrade the Master,
+client daemon, and clients together; mixed-version batch-probe calls are not
+supported. Default Python arguments preserve the old call syntax, not wire
+compatibility. `is_exist()`, `batch_is_exist()`, and `probe_key()` are unchanged.
+
 ---
 
 #### get_size()

diff --git a/docs/source/api-reference/python/store-rs.md b/docs/source/api-reference/python/store-rs.md
@@ -4,7 +4,7 @@ The Python Store facade selects its implementation when mooncake.store is
 first imported. The shared API and default C++ backend are documented in
 [Mooncake Store Python API](mooncake-store.md). This page describes Store-RS
 runtime and API behavior. See the
-[source-checkout quickstart](../../getting_started/store-rs.md) to build and
+{ref}`Build with Store-RS <build-with-store-rs>` section to build and
 install the unified root wheel.
 
 ## What the Python Layer Provides

diff --git a/docs/source/deployment/mooncake-store-deployment-guide.md b/docs/source/deployment/mooncake-store-deployment-guide.md
@@ -361,7 +361,7 @@ With `ha_backend_type=etcd`, Mooncake Store supports a Primary-Standby HA model
 HA leadership and metadata replication are configured separately:
 
 - The HA coordinator elects the active master. Configure it with `--enable_ha`, `--ha_backend_type`, `--ha_backend_connstring`, and `--cluster_id`. For `ha_backend_type=etcd`, legacy `--etcd_endpoints` is used only when `--ha_backend_connstring` is empty.
-- The optional batch-record OpLog records metadata mutations so standby masters can catch up and later be promoted. It is the etcd Primary-Standby path. Enable it explicitly with `--enable_oplog=true`; it is disabled by default. `ha_backend_type=etcd` requires a build with `STORE_USE_ETCD`. On etcd, acknowledged batches form the durable prefix used for promotion. `ha_backend_type=redis` requires a build with `STORE_USE_REDIS` and is an experimental HA log storage method. A successful Redis write only means Redis accepted the record. Do not treat Redis persistence, including AOF `appendfsync everysec`, as a strict durability guarantee. 
+- The optional batch-record OpLog records metadata mutations so standby masters can catch up and later be promoted. It is the etcd Primary-Standby path. Enable it explicitly with `--enable_oplog=true`; it is disabled by default. `ha_backend_type=etcd` requires a build with `STORE_USE_ETCD`. On etcd, acknowledged batches form the durable prefix used for promotion. `ha_backend_type=redis` requires a build with `STORE_USE_REDIS` and is an experimental HA log storage method. A successful Redis write only means Redis accepted the record. Do not treat Redis persistence, including AOF `appendfsync everysec`, as a strict durability guarantee.
 - The optional standby-generated batch OpLog snapshot path is enabled with `--enable_oplog_snapshot=true` together with `--enable_oplog=true`. It uses the batch snapshot provider/coordinator and does not use the legacy catalog snapshot manager. Startup fails when the required etcd, cluster ID, object-store, or chunk configuration is invalid; a temporary upload failure leaves OpLog apply running for a later attempt.
 
 
@@ -745,6 +745,7 @@ mooncake_master \
 | `--ha_backend_connstring` | empty | HA backend connection string |
 | `--etcd_endpoints` | empty | Backward-compatible etcd HA endpoints, used only for `ha_backend_type=etcd` when `--ha_backend_connstring` is empty |
 | `--cluster_id` | `mooncake_cluster` | Cluster ID for HA persistence |
+| `--ha_leader_lease_ttl_sec` | `5` | Leadership lease TTL in seconds for `ha_backend_type=etcd` (range `2`-`60`; the JSON config key `ha_leader_lease_ttl_sec` is also accepted). Raise it to ride out etcd apply stalls longer than the default. A longer TTL delays failover: a new leader waits one full TTL before serving, and a standby waits for a crashed leader's lease to expire. Ignored by the Redis and k8s backends |
 | `--enable_oplog` | `false` | Enable the batch OpLog writer and reader; requires `enable_ha=true` and `ha_backend_type=etcd` or `redis`. etcd uses it for Primary-Standby promotion. Redis uses it as an experimental HA log store: a successful write only means Redis accepted the record |
 | `--enable_oplog_snapshot` | `false` | Enable standby-generated batch OpLog snapshots; requires batch OpLog, HA/etcd, valid object-store configuration, and persistent local snapshot storage when applicable |
 | `--snapshot_chunk_object_count` | `1000000` | Maximum objects per batch OpLog snapshot chunk; must be positive when the new snapshot path is enabled |
@@ -998,9 +999,9 @@ config.dfs_replica_num = 1
 store.put("key", b"value", config)
 ```
 
-`dfs_replica_num` may currently be `0` or `1`. A DFS replica must be requested
-with at least one memory replica (`replica_num >= 1`), so DFS-only placement is
-not supported.
+`dfs_replica_num` may currently be `0` or `1`. DFS-only placement is supported
+by setting `replica_num = 0` and `nof_replica_num = 0`; the object then has no
+memory replica, and DFS eviction or removal deletes it.
 
 Allocation first tries the key's hash-selected DFS shard, then tries other
 ready shards if that shard has no suitable extent. This lets new shards accept
@@ -1022,7 +1023,9 @@ failed key is revoked without downgrading successful keys.
 For a same-size `Upsert`, if either the existing object or the new request has
 a DFS replica, the requested memory, NoF, and DFS replica counts must match the
 existing topology. A different-size update releases the old placement and
-allocates a new topology.
+allocates a new topology. For example, after the memory replica of a
+memory-plus-DFS object is evicted, a same-size `Upsert` must request the
+DFS-only topology.
 
 On reads, the master returns the readable replica list through the normal query
 path, and the client selects the first complete replica. If it selects DFS, any
@@ -1032,8 +1035,7 @@ reads for that descriptor.
 #### Current limitations
 
 - Only the `default` tenant is supported.
-- `dfs_replica_num` must be `0` or `1`, and `replica_num >= 1` is required when
-  it is enabled.
+- `dfs_replica_num` must be `0` or `1`.
 - C and Rust clients cannot currently request or access descriptor-based DFS:
   their replication configuration does not expose `dfs_replica_num`, and their
   setup API cannot initialize the distributed `FileStorage` backend. Use the

diff --git a/docs/source/deployment/store-rs/index.md b/docs/source/deployment/store-rs/index.md
@@ -2,7 +2,7 @@
 
 Use these guides to build the optional Store-RS component, configure a runtime,
 operate tenant policy and routes, and validate deployment scenarios. The
-[source-checkout quickstart](../../getting_started/store-rs.md) starts with a
+{ref}`Build with Store-RS <build-with-store-rs>` section starts with a
 root CMake build and the unified Python wheel.
 
 | Guide | Use it to |

diff --git a/docs/source/deployment/store-rs/python.md b/docs/source/deployment/store-rs/python.md
@@ -1,11 +1,11 @@
 # Store-RS Python Operations
 
 The root wheel installs the standalone Store-RS commands. Build and install it
-using the [source-checkout quickstart](../../getting_started/store-rs.md).
+using {ref}`Build with Store-RS <build-with-store-rs>`.
 
 ## Standalone Store-RS Commands
 
-The root CMake target build_store_rs_cli builds the client, admin, and benchmark binaries. The unified root wheel installs the standalone commands under their mooncake-store-rs-* names. Use the [source-checkout quickstart](../../getting_started/store-rs.md) to configure the root build and install the wheel.
+The root CMake target build_store_rs_cli builds the client, admin, and benchmark binaries. The unified root wheel installs the standalone commands under their mooncake-store-rs-* names. Use {ref}`Build with Store-RS <build-with-store-rs>` to configure the root build and install the wheel.
 
 Start a storage client:
 

diff --git a/docs/source/deployment/store-rs/rolling-upgrade.md b/docs/source/deployment/store-rs/rolling-upgrade.md
@@ -8,7 +8,7 @@ This document describes how to perform a rolling upgrade on a Mooncake Store clu
 
 - The new version only increments `store_api_minor_version` (minor version) and does **not** change `store_api_version` (major version)
 - The changes in the new version are backward-compatible (e.g., adding fields, adding RPCs) and do not alter existing API semantics
-- Both versions were built from their own root CMake build trees using the [Store-RS source-checkout quickstart](../../getting_started/store-rs.md)
+- Both versions were built from their own root CMake build trees using {ref}`Build with Store-RS <build-with-store-rs>`
 
 ### Compatibility Guarantees
 

diff --git a/docs/source/deployment/store-rs/route-migration.md b/docs/source/deployment/store-rs/route-migration.md
@@ -202,6 +202,6 @@ What this feature is not:
 ## 9. Related Docs
 
 - [Python Guide](../../api-reference/python/store-rs.md)
-- [Features](../../getting_started/store-rs-features.md)
+- [Features](../../design/store/store-rs/features.md)
 - [Configuration Reference](configuration.md)
 - [Architecture](../../design/store/store-rs/architecture.md)
diff --git a/docs/source/design/store/ssd-offload.md b/docs/source/design/store/ssd-offload.md
@@ -144,6 +144,10 @@ Bucket IDs are monotonically increasing timestamps with a sequence suffix, so `b
 
 **Grouping strategy** (`GroupOffloadingKeysByBucket`): objects are accumulated into a bucket until either `bucket_size_limit` (default 256 MB) or `bucket_keys_limit` (default 500) is reached. Objects that do not fill a complete bucket are held in `ungrouped_offloading_objects_` and retried on the next heartbeat.
 
+Set `MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS=1` before starting a client to write its final partial bucket in the same offload cycle. The default is `0`, preserving the existing grouping policy. The setting is read when the bucket backend configuration is created; an existing backend does not change when the environment changes.
+
+This opt-in policy avoids the deferred-task loss described in [issue #3006](https://github.com/kvcache-ai/Mooncake/issues/3006): carrying only keys and sizes across heartbeats does not preserve their offload tasks. Flushing the tail keeps eligible objects in the cycle that owns those tasks. It does not change completion notification, write-failure handling, or the on-disk format. Smaller buckets can increase file/metadata overhead and reduce write efficiency; this policy does not guarantee a disk replica if an offload fails. An alternative is to retain the tasks across cycles while waiting for full buckets, as proposed in [PR #3009](https://github.com/kvcache-ai/Mooncake/pull/3009).
+
 **In-flight read tracking**: A `BucketReadGuard` RAII object increments `BucketMetadata::inflight_reads_` on construction and decrements it on destruction. This allows safe deletion of bucket files even when concurrent reads are in progress.
 
 **Ownership contract (`storage_path`)**: LOCAL_DISK allows **one live client per `storage_path`**. `BucketStorageBackend::Init()` opens `.mooncake_local_disk.lock` under that directory and takes a non-blocking exclusive `flock`; a second live client fails Init immediately with an actionable error (`storage_path already held by another live client … use standalone store or per-client directories`). Do not delete the lock file while a client is running — that would release the flock for other processes. `flock` is only meaningful on local filesystems (NFS and similar may not honor exclusive locks the same way).