You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f279d9e
Browse filesBrowse the repository at this point in the historyBrowse files
fix(file-search): fill dispatch capacity in selection order and keep reaped claims ahead of the rotation (#9671)
* fix(file-search): fill dispatch capacity in selection order and reap before the backfill
The claim cut ranked every selected owner's candidates by revision age, so with less capacity
than candidates a woken owner selected first could lose every slot to an older backlog selected
after it, and the claim's stamp then filed it behind the rotation. Claims now fill capacity owner
by owner in the pass's selection order (each still up to its free slots).
The backfill page ran before the reaper in the same transaction, so an owner it queued as a
reconciliation wake-up absorbed the reaper's wake-up (both writes at the transaction time) and
the released claim waited behind the rotation. The reaper now runs first; the backfill's wake-ups
leave queued owners untouched.
Tests: a two-slot pass with a woken owner against an older backlog, and a reaped claim whose
owner the backfill queues in the same pass; the capacity filler's files are deleted so the
backfill never re-queues it.
* fix(file-search): keep the dispatch pass's file-then-queue lock order
Reaping before the backfill held the owner's queue row while the backfill took FOR SHARE on its
unindexed files; a concurrent save holds its file while its trigger waits on that queue row, so the
two could deadlock. The pass runs backfill then reap again, and a wake that conflicts with a row this
transaction's backfill just stamped as a reconciliation clears the stamp, so a reaped claim stays
ahead of the rotation.
Migration `0426_file_search_owner_scope.sql` adds canonical workspace/Project ownership to the existing build, revision, and chunk pipeline. Legacy workspace inserts are translated by triggers, and the existing workspace trigram index remains usable. Project rows use a separate owner-scoped trigram index on the same chunk table. Organizations and personal users are not enabled search owners.
73
73
74
-
The new `file_search_dispatch_queue` replaces the workspace-keyed scheduler because older workers cannot decode a Project without a workspace. During expansion, workspace mutations enqueue both queues, while the new dispatcher reads the owner queue. Both deployments share the dispatcher lock, revision claim token, build lease, and publication fence. Drain old app dispatchers and Trigger workers/retries before enabling Project dispatch. Retire the legacy queue and its dual enqueue in a later contract deploy; they are not a second indexing pipeline. The `file-search-owners-v3` cursor performs bounded owner/file keyset backfill and hourly reconciliation without rewriting extracted text in the migration. Claims take owners woken since their last dispatch first (`wokenSinceDispatch` in `dispatcher.ts`); backfill wake-ups enter as if just dispatched and never touch an owner already queued, so a page of them cannot delay a file write by several passes.
74
+
The new `file_search_dispatch_queue` replaces the workspace-keyed scheduler because older workers cannot decode a Project without a workspace. During expansion, workspace mutations enqueue both queues, while the new dispatcher reads the owner queue. Both deployments share the dispatcher lock, revision claim token, build lease, and publication fence. Drain old app dispatchers and Trigger workers/retries before enabling Project dispatch. Retire the legacy queue and its dual enqueue in a later contract deploy; they are not a second indexing pipeline. The `file-search-owners-v3` cursor performs bounded owner/file keyset backfill and hourly reconciliation without rewriting extracted text in the migration. Each pass selects owners woken since their last dispatch that still have a free slot first (`wokenSinceDispatch` in `dispatcher.ts`), then the rest by dispatch age, and fills its capacity owner by owner in that order. Backfill wake-ups enter as if just dispatched and never touch an owner already queued, so a page of them rotates in behind owners woken for work.
75
75
76
76
Generated document indexing never executes source. A cached artifact must match its owner and current input manifest. Each published build stores at most 500 input identities. Queries exclude a build when any input is changed, archived, deleted, or outside the owner, and report it as pending coverage. Input mutations wake their owner queue; the dispatcher retires stale manifests in bounded batches before rebuilding. Expired build cleanup also removes its dependency records. A cache miss indexes the stored generation source as text until an authorized rendering path creates a current artifact.
* Whether an owner was woken after its last dispatch: never dispatched, or touched since. A file
183
-
* write, a released claim, and a reaped claim advance `updated_at`; a claim stamps both columns
184
-
* with one time, and a reconciliation wake-up enters with them equal. A write whose transaction
185
-
* began before the claim that stamped its owner reads as earlier and waits one rotation.
187
+
* write, a released or reaped claim, and an artifact-ready requeue advance `updated_at`; a claim
188
+
* stamps both columns with one time, and a reconciliation wake-up enters with them equal. A write
189
+
* whose transaction began before the claim that stamped its owner reads as earlier, so the owner
190
+
* keeps the dispatch-age rotation instead.
186
191
*/
187
192
constwokenSinceDispatch=sql`(${fileSearchDispatchQueue.lastDispatchedAt} IS NULL OR ${fileSearchDispatchQueue.updatedAt} > ${fileSearchDispatchQueue.lastDispatchedAt})`
0 commit comments