status: fix FSMonitor history and clean-proof gaps - #74
Closed
ttaylorr-oai wants to merge 465 commits into
Closed
Conversation
Untracked status used one helper both to traverse the worktree and to copy untracked and ignored entries into status output. Checking whether a traversal actually validated the repository's UNTR cache requires that directory walk without copying results or recording user-facing timing. Factor the walk into wt_status_collect_untracked_1() with an explicit collection flag. Return whether the traversal used the index's own untracked cache, and populate the result lists and advice timing only when collection is requested. Retain wt_status_collect_untracked() as the collecting wrapper. Every existing production caller still requests collection, so status output and ordinary traversal behavior remain unchanged. Token adoption and validation-only production use are not added by this preparatory patch. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An external attributes file can be missing, reached through a symbolic link, or redirected when an ancestor is replaced. Hashing its contents alone cannot distinguish stable absence from a changed containing namespace, or identical bytes reached through a different path. Extend the filesystem-identity primitives from S06/P03 to capture the lstat identity or absence of every component of an absolute path. Compare snapshots component by component and hash their explicit states and canonical identity fields with length-delimited framing. Reject capture with EAGAIN when the platform cannot report reliable object identity instead of hashing fabricated identity fields. Expose whether the final component exists and release all snapshot storage explicitly. Tests verify equal snapshots and hashes, a missing target that subsequently appears, and an ancestor replacement that changes the namespace even when the replacement has identical content. The snapshot is independently testable. It does not read an external attribute source or establish a status speedup. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S11/P01 retains scan results for later processing, but treats every changed stat observation as a pending content check. A nonzero cached size that differs from the observed size already proves that a tracked file changed. Racy timestamps, zero cached sizes, type changes, and the Windows symlink-size sentinel cannot establish that conclusion. Classify an entry as definitively modified only when its mode and type remain comparable, its cached size is nonzero, and match_stat_data() reports an actual data-size difference. Pass that terminal state through the existing preload result, skip its speculative restat, and emit a separate definitive-modification Trace2 count. Preserve ordinary content verification for every ambiguous observation. Update the APFS dirty-file test to require the new terminal classification, no speculative lstat, and the still-required authoritative refresh. This verifies the new state at its first consumer without claiming that status already consumes it directly. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
read_one() allocates a subtree array even for leaf cache-tree nodes. It also inserts each serialized child through cache_tree_sub(), which searches children that the writer already emits in increasing order. Allocate a child array only for non-leaf nodes and append increasing child names directly. Retain subtree_nr + 2 pointer slots for each non-leaf, but allocate them without zeroing because only populated slots are inspected. Keep cache_tree_sub() as the compatibility fallback for older, unsorted input. Existing t/t0090-cache-tree.sh tests exercise ordinary cache-tree decoding. This change adds no dedicated unsorted-input regression or isolated benchmark. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Once refresh_cache_ent() verifies that an entry's content still matches, it allocates and copies a replacement, fills its stat data, and preserves a caller-cleared CE_VALID bit under assume_unchanged. Keeping that sequence in one caller would require another verified refresh path to duplicate the allocation and validity handling. Extract make_refreshed_cache_entry() as a private helper and keep refresh_cache_ent() as its first consumer. Pass !ignore_valid through the existing condition so the entry name, observed stat data, and CE_VALID behavior remain unchanged. This is a behavior-preserving refactor. It introduces no new index write, configuration, test claim, or independent performance claim. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider token obtained before a tracked or untracked scan does not cover worktree changes racing with that scan. Publishing it as an FSMN or FSUC proof can make a later status trust an index or untracked-cache snapshot that was never valid at that boundary. Keep bootstrap tokens pending while tracked entries are refreshed and any rooted untracked cache is traversed. For builtin providers, query again after the scans, apply intervening paths, and repeat the affected scans until a clean boundary is found or three closing queries are exhausted. Treat a trivial closing reply as complete invalidation followed by another scan; accept its replacement token only after a later clean reply. Reject provider errors, incomplete cache proofs, and exhausted retries with strong invalidation and complete fallback scans. Hook providers cannot perform a closing IPC query, so accept their token only after a complete tracked and applicable untracked collection; reject failed or trivial hook replies. A matching on-disk FSUC token can now authorize replay of recursive UNTR validity established by S01. Reconstruct that validity only after the entire extension has decoded, and only for directories without a cached per-directory exclude digest. This lets a warm status prune known-empty subtrees while still rechecking a changed .gitignore, including changes made through an unwatched hardlink alias. Trust an indexed exclude's metadata alone only when its identity is reliable and it has exactly one link; otherwise retain the complete content-hash check. Route both status collection and commit index refresh through the shared closure. Preserve ordinary behavior for existing paired state, path-limited requests, and ignored-mode collection. Cover clean and changed closures, trivial replies, retry exhaustion, provider errors, on-disk FSMN/FSUC publication, warm empty-subtree pruning, descendant events, and cached exclude changes. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An external attributes file can change conversion without changing a worktree attribute manifest. Content alone is also insufficient: an ancestor or linked target can be replaced, and a missing source is safe to reuse only while its containing namespace remains stable. Capture the normalized absolute-path namespace with S07/P09 before and after observing each enabled source. For a present source, require nonblocking-open support, a regular singly linked file below the attribute-file limit, and matching descriptor, pathname, and target identities. Read the entire file into one allocation. Record source configuration and contents in one framed digest, and component and target identities in a separate namespace digest. Recheck the complete namespace for stable missing sources. Enabled sources inherit the namespace capture's fail-closed identity check; disabled sources remain unobserved and safely digestible. Reject instability rather than publishing an incomplete fingerprint. Register the fingerprint library and Clar suite in both Make and Meson. Tests separate content from metadata changes, detect an altered ancestor of a missing source, preserve disabled-source digests, and exercise both object formats. This does not select repository attribute sources or integrate fingerprints into status. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
add_patterns() rejects pattern files larger than 100 MiB only after allocating and reading their complete contents. An oversized filesystem input can therefore exhaust the memory the limit is meant to protect, or terminate Git when GIT_ALLOC_LIMIT rejects the allocation. Check the size obtained from fstat() before allocating a filesystem pattern buffer. Preserve the existing warning, close the descriptor, and return the existing failure result. Keep the later size check for index-backed fallback data, whose size is unavailable before it is read. Strengthen the existing EXPENSIVE regression by reading its 101 MiB .gitignore under GIT_ALLOC_LIMIT=1m. The old ordering dies in xmallocz(); the early rejection preserves the expected warning without attempting the oversized allocation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A complete APFS preload can already prove that tracked entries are deleted or have definitive size changes. Ordinary status nevertheless refreshes those entries and later asks worktree diff to rediscover them. Skipping refresh without preserving ambiguous content checks would either duplicate work or misreport metadata-only changes. Request terminal-result deferral explicitly from porcelain status and retain a complete per-entry result on its index. Let refresh defer proven modifications and deletions while marking ambiguous entries for the ordinary content check. Insert terminal changes into the normal status change list before running worktree diff, temporarily mark only those entries up to date, and restore their flags afterward. Clear retained results before another preload and release them with the index. Other refresh callers keep their existing behavior. Extend the APFS tests to assert direct modified and deleted results and no redundant refresh stats. Add a metadata-only mismatch that must still reach worktree diff and produce clean porcelain output. Retained terminal state trades additional temporary memory for removing the second classification of proven changes. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The index extension worker decodes TREE and UNTR serially even though their parsers read the same immutable mapping and publish to different index_state fields. An unconditional additional worker would consume cache-entry workers and interfere with split-index assembly. Use the bounded framing from S02/P01 to select exactly one TREE and one UNTR extension. Require extension-offset metadata and at least four index workers; start an additional TREE worker only when both payloads reach 1 MiB. Leave at least two cache-entry workers available and join the TREE worker before unmapping the index. Keep LINK, duplicate or missing extensions, insufficient workers, small payloads, and auxiliary-worker creation failures on the existing serial path. Malformed framing still reports index file corruption. Allow GIT_TEST_PARALLEL_INDEX_EXTENSIONS to bypass only the payload threshold. Add a PTHREADS, UNTRACKED_CACHE, and SHA1 regression that compares parallel and serial status, cache-tree, and untracked-cache results and checks the extension/parallel/tree-untracked Trace2 marker. The regression unsets GIT_TEST_SPLIT_INDEX because split indexes intentionally remain on the serial path. The eligible path adds one auxiliary worker and its stack. The benchmark covers the complete series, not this patch in isolation. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Bulk preload defers metadata-mismatched entries to run_diff_files() for a content check. When writable status confirms that such an entry is clean, it still leaves old stat data in the index. The next status must therefore repeat a content check already known to match. Request DIFF_UPDATE_INDEX_STAT only when status holds the index lock and bulk preload covers every indexed entry. After a real stat and a successful content and mode check, refresh only entries marked CE_CONTENT_CHECK_REQUIRED. Build the replacement with the helper from S15/P01 and install it with replace_index_entry(), preserving existing CE_VALID and index-change handling. Read-only status, incomplete bulk scans, dirty entries, and ordinary diff callers keep their existing behavior. The APFS regression compares both status output and the written index with ordinary status, and requires one bulk content check with no refresh-time lstat. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S08 depends on S07 and S05. S05 now contains the former S01/P06 prerequisite, so join only the two independent histories before applying S08. Keep every fsmonitor test registration in sorted order in Makefile and t/meson.build. In builtin/commit.c, wt-status.c, and wt-status.h, start the attribute snapshot and UNTR preload before the token-closing refresh, and preserve each state through cleanup.
A clean-status configuration digest cannot establish which index it describes while it remains detached from the repository and index that will consume it. External attribute content and namespace must also be recorded before an index can reuse conversion-dependent history. Attach a finalized, repository-bound digest at the beginning of do_read_index(), fingerprint the system, global, and info attribute sources, and store the resulting state on the index. Ignore an unfinalized digest, another repository's digest, and a second attachment. Release the state with release_index(). Extend the existing clean-status configuration unit suite to exercise repository binding, one-shot attachment, semantic and attribute hashes, unsafe-filter state, and index-lifetime cleanup. Register the new production object with both Make and Meson. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Index attachment cannot recover the configuration seen by git status or git commit if their callbacks finish without recording it. A separate configuration pass could also bind a different stream from the one that established the commands' existing behavior. Wrap each existing status or commit callback so the original callback and clean-status digest consume the same key, value, and context. Finalize and stage the digest after the existing configuration pass and before either command reads its index. Preserve determine_whence(), advice_enabled(), the original callback, configuration order, and option handling. The index-owned attachment and its existing configuration unit coverage are supplied by S08/P01; this patch adds no command-specific regression. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An index with a null trailing checksum cannot be bound to the file that was actually read unless the platform supplies a durable file identity. Treating a directory, multiply linked file, or unsupported platform as equivalent would turn identity comparison into an unwarranted correctness guarantee. Add clean_status_identity_from_stat() for single-link regular files and make clean_status_identity_is_durable() return true only on Apple platforms. Keep unsupported platforms explicitly ineligible instead of inferring durability from stat fields alone. Register the identity object and its unit suite with Make and Meson. The tests reject directories and multiply linked files, accept a single-link regular file, and check the appropriate platform result. Actual null-checksum index verification remains a separate change. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
With index.skipHash enabled, a null trailing checksum cannot prove that verify_index_from() reopened the index that do_read_index() parsed. Replacing the pathname between those operations can otherwise make an unread index appear valid. Record the identity from the index reader's existing fstat() result. When verifying a null-checksum index on an Apple platform, compare it with the identity from the verifier's existing file observation. Reject an absent, nonregular, multiply linked, or replaced identity. Leave checksummed indexes and platforms without durable identities on their existing paths. Reuse the identity classification from S08/P03 without adding an index-read system call. Register the new object and unit suite with Make and Meson; the unit test replaces the index pathname and checks the unsupported fallback. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A filesystem-monitor token does not establish that saved configuration, conversion rules, attribute inputs, or their complete manifest still describe the current index. Accepting duplicate, stale, or partially bound history could let status trust cached worktree state under different semantics. Recognize the FSCF index extension and delegate malformed-record rejection to the bounded clean-proof parser from S07/P07. Publish its token, configuration and semantic hashes, attribute hash, and manifest only after the complete record validates. Reject duplicate records, and adopt a manifest only when the current token, hashes, complete proof flags, and filter policy all agree. Record stronger semantic mismatches and withhold incoherent history. Integrate validation into post_read_index_from(), release all owned record and manifest storage with the index, and document the extension layout. Register the history object and unit suite with Make and Meson. A SHA-1 fixture rejects duplicate records; a SHA-256 fixture accepts coherent history and detects a changed semantic hash. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Reading a validated FSCF record is not enough to preserve it during a generic index rewrite. Writing fresh token or stat bindings before the current provider token is revalidated would claim a semantic proof that the index has not established. Write a newly bound FSCF extension only when configuration, attributes, the complete manifest, the valid provider token, and its revalidated token all agree. Otherwise preserve an existing validated record with its token and stat bindings cleared; never serialize malformed or missing history. Add the extension to the existing index writer. Extend the history unit tests to distinguish closed proofs from preserved unbound manifests. Add a test-tool round trip and t7519 coverage that read, write, and reread a coherent FSCF record through a real index. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
move_index_extensions() transfers extensions to a replacement index, but index-owned FSCF history would otherwise remain on the old state. A generic rewrite could silently discard a validated manifest, while sharing its storage would create a lifetime hazard. Copy only a parsed, valid serialized record into independently owned destination storage. Reload the saved manifest through its validated parser, copy the existing token and hashes, and leave an absent or invalid source untouched. Invoke the transfer from move_index_extensions() so ordinary index release owns each copy. Extend the existing history unit suite with a real extension transfer. Verify the copied record and manifest, invalidate the source, reject a second transfer from that source, and confirm that the independent first destination remains valid. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
Fingerprinting an external attribute file while reading the index does not prevent the attribute parser from reopening a replaced file during preload or status collection. Cached stat data could then be evaluated with conversion rules that the original fingerprint did not cover. Capture the system, global, and info attribute bytes and namespace once and keep the immutable snapshot active from untracked-cache preload through collection. Parse snapshot lines with the ordinary attribute rules, including byte-order marks, embedded NULs, and line endings. End the snapshot and release its bounded source buffers with status. Make a failed capture or changed attribute content sticky and invalidate fsmonitor validity and the untracked cache before ordinary refresh. Preserve hook-provider behavior when semantic history is absent or only the namespace changes: hooks have no closing query and retain their reported-path contract. An observed content change still invalidates hook-derived state. Add t7531 integration coverage for file-parser parity, missing attribute history, an observed hook-time attribute change, and the hook missing-history exception. Update the existing history unit test to exercise the public strong-mismatch predicate. The namespace-only hook branch has no dedicated regression in this patch. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An fsmonitor token can mark an entry valid even when the index has no coherent history for the configuration and attributes that determine its content. With minimal stat checks, a same-size rewrite can then be reported as clean. Rebuild the attribute manifest for expanded indexes during IPC bootstrap. Compare it with the current in-process or retained on-disk manifest, invalidate only the tracked and untracked scopes whose attribute sources changed, and preserve the last complete manifest when a rebuild fails. A legacy index with no FSCF extension is different from a mismatched proof: it contains no claim about semantic history to disprove. When it also has a valid nontrivial FSMN token with core.trustctime enabled and full core.checkStat, clear FSMN validity and seed a forward baseline through ordinary configured stat checks. This avoids hashing every tracked file solely because the index predates FSCF. The baseline still needs to finish in the bootstrap command. Preserve the freshly-proven FSMN-valid bit on entries replaced by that refresh, so that the accepted token does not defer the same migration work into the next status. Keep strong global invalidation for semantic or attribute mismatches, weak stat settings, a present FSCF without complete manifest history, provider reset or failure, manifest rebuild failure, and fresh indexes without a prior nontrivial FSMN token. Retain ordinary provider handling when reliable file identity is unavailable. The migration exception has ordinary Git stat semantics rather than a content-proof guarantee; same-size changes hidden by the platform's configured stat identity can remain hidden at that boundary. Add coverage for the forward-baseline lane, the weak-stat same-size rewrite, and the refreshed baseline FSMN bits, along with unit coverage for coherent, manifest-only, missing, and present-without-manifest history. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An exact clean-result sidecar must remain bound to one physical index, but resumable fsmonitor history needs to survive a format-only rewrite by another Git implementation. It cannot use the index checksum or file identity as its cross-implementation key. Promote the checksummed path snapshot operations needed by an external store. They open the named index without following its final symlink and retain the descriptor, then require the descriptor and current pathname to identify the same valid index. Null checksums remain ineligible for durable snapshot pins. When fstat identity is reliable, retain the validated reader descriptor for process-local proof epochs only; generic certification and persisted CSHS still require a non-null checksum. Define a canonical digest of the ordered logical entries. Include the entry count and each path, stage, object ID, mode, CE_VALID, skip-worktree, and intent-to-add state, while excluding index encoding, cached stat data, and acceleration-only flags. Unsupported transient state rejects the digest rather than disappearing with the process. Add the checksummed CSHS codec and a local-APFS-only, nofollow, atomically-replaced store bounded to eight 16-MiB namespace slots. This commit has no status caller; the following history patch restores and saves complete checkpoints through this persistence layer. Cover both object formats, malformed and null-checksum snapshots, pathname replacement, logical-entry bindings, malformed and independent checkpoint namespaces, bounded retention, and idempotent writes. A checkpoint may contain only the required FSMN and FSCF payloads. Skip absent optional payloads rather than handing a NULL source and zero length to memcpy(). Extend the malformed-checkpoint unit test to round-trip that minimal valid form before its rejection cases. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A collapsed sparse index cannot enumerate every tracked path needed for a complete attribute manifest. Expanding the live index would discard the sparse representation that status is supposed to preserve. Pin the named index with S09/P02, reread the verified index into a scratch index, and expand only that scratch copy. Build the complete manifest from the expanded scratch index. Check that both the parsed scratch state and original live state still match the held descriptor and stored trailer checksum; discard the manifest if either check fails. Add a sparse-checkout regression that checks the collapsed outside entry before and after status while detecting a same-size tracked rewrite. Extend the existing index unit case with a parsed A-to-B-to-A mismatch. Failed snapshot validation retains ordinary full-invalidation fallback. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A clean provider response closes only the filesystem interval after its starting token. It cannot certify a refresh that started before the named index, configuration, attributes, and manifest were captured, or one whose semantic inputs subsequently changed. Capture the proof epoch before each refresh whose provider token may be accepted. Pin the named index, starting token, repository configuration, external attribute fingerprint, and complete full-index manifest. Recheck those inputs after the closing query. Record semantic history only for the accepted token; reject missing or changed inputs and fall back to a complete refresh. For a null-checksum index, let only the proof-epoch pin use the process-local reader descriptor retained by the preceding patch. The proof-only exception rechecks both the retained source descriptor's original stat identity and the current named path when pinning and closing the epoch. Generic certification and persisted CSHS continue to reject the null trailer. Always rebuild the manifest when physical history is unavailable, even if the stored semantic configuration already matches. Without that manifest, a trivial response invalidates the old binding and leaves the closing query with no complete epoch to bind, so each later status repeats the fallback. Teach this lifecycle to restore and save complete external history checkpoints through the preceding CSHS store. A restore validates the logical index and all FSMN, UNTR/FSUC, and FSCF sections in scratch state, then rechecks the pinned index before installing them together. A save requires the same logical entries before and after status and a closed, writable proof. Keep both paths dormant until a later patch enables them only for a normal top-level status. A retry inside a captured epoch can also lose a freshly acquired CE_FSMONITOR_VALID bit when replace_index_entry() applies its generic conservative invalidation. Mark proof-epoch refreshes explicitly and restore only a validity bit acquired by the replacement itself. Changed or rejected closures still invalidate those provisional bits before falling back. Register clean-status-epoch.c in Make and Meson alongside its first production consumer in wt-status.c. Add scripted regressions for capture-before-refresh ordering and recovery from unbound physical history. Add unit coverage for the complete full-index manifest, the restricted post-status logical-digest exception, retained-descriptor lifetime, a stat-visible same-inode size change, and atomic path replacement. Later activation patches cover external checkpoint recovery and the immediate warm run. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
A provider event can change a tracked .gitattributes file after status captures its conversion inputs. Accepting the resulting token against the previous manifest can incorrectly reuse tracked validity when a complete status would report a content change. Record when provider invalidation expires the current manifest and semantic proof. Before retrying token closure, rebuild that manifest and recapture external attribute sources when their content changes. Keep the response token pending until the new scan and current attribute epoch are both closed. Preserve ordinary provider handling when file identity is unreliable. Preserve reusable manifest history across ordinary index rewrites without retaining expired bindings. Extend the history and manifest unit cases and the index round-trip helper. Add a scripted regression for tracked attribute changes. Manifest or snapshot failure still forces a complete scan. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An IPC provider cannot safely adopt missing semantic history merely because tracked entries are marked fsmonitor-valid. Minimal stat checks can conceal a content rewrite, and a clean token cannot retroactively certify workers started under different attributes. Capture the complete proof epoch before preparing semantic workers. Prime each worker's attribute frames and verify the starting token and complete epoch before hashing. After a clean closing query, apply the proof only if the pinned index, configuration, attribute content, manifest, worktree identity, and token remain consistent. Permit attribute-namespace bookkeeping to change only after its source bytes and initial namespace were verified. Accept tracked validity independently of untracked validity. Keep a query pending when the untracked scan has not run. Leave collapsed sparse indexes, pathspecs, ignored-file requests, unreliable file identity, non-IPC providers, and failed proofs on ordinary closure or complete refresh. Add scripted regressions for adopting missing tracked history without hiding a same-size rewrite and for preserving a collapsed sparse index on the ordinary closure path. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An untracked-cache preload can inspect cached excludes and directory state before tracked semantic adoption restores verified stat data. One provider response also cannot certify an untracked traversal performed after the tracked scan that response closes. Defer provider-backed untracked validation until the tracked proof has been applied and its first query has closed. Prime the untracked cache afterward, issue a second closing query, and recheck the full tracked proof before accepting either result. If the later query reports a change, invalidate both results, reprime during ordinary closure, and retry within the existing query bound. Factor the existing proof-current checks into the predicate used by proof application and deferred closure. Preserve automatic untracked preload when no provider is enabled or file identity is unreliable. Fall back to a complete scan if untracked validation or token closure fails. Add prerequisite-guarded scripted cases for successful deferred scans, failed untracked closure, and a change reported by the second closing query. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
An as-is commit refreshes its index before running the pre-commit hook. If the hook rewrites a tracked path without changing its size or mtime, the later in-process status must not certify the earlier refresh as though it covered the hook. For a nonsplit index using an IPC provider, perform the initial refresh through status token closure. After an invoked hook, release the saved attribute snapshot and reopen the last accepted provider token before status runs again. Reject unavailable token state and invalidate the manifest, tracked semantics, and untracked cache before falling back to a complete refresh. Pin the post-hook named index before persisting strong invalidation. Write refreshed state only while its held descriptor, pathname, stored trailer checksum, and in-memory index still match. Preserve a hook-replaced index and the existing reread. Split indexes, platforms without reliable file identity, and non-IPC providers retain their original initial refresh. Add prerequisite-guarded scripted cases for successful post-hook closure without an untracked cache, failed closure with complete worktree refresh, and a hook that updates the index itself. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
S13 depends on S09, S11, and S12. Join those three histories once before applying S13. S09 and S11 add adjacent cleanup and test-list entries. Release both clean-status and bulk-preload state in read-cache.c, and keep t7529 before t7531 in t/meson.build. S12 requires no further resolution.
A bulk untracked scan cannot reuse its result merely because an ignore file has familiar stat data. A file or its parent may be replaced while the scan runs, an absent source may appear, and repeated reads of the same source may observe different patterns. Record each source beneath its nearest available anchored parent, along with its path, symlink policy, presence, size, and blob identity. Check descriptor and parent identities while capturing an observation, then resolve the current parent again and compare the actual source bytes at validation. Coalesce equivalent observations and invalidate the proof immediately when observations conflict. Validation uses nonblocking opens, so replacing a source with a FIFO cannot hang. Equal contents remain acceptable even if the source or its parent has a different identity. This also preserves an empty /dev/null and an equivalent empty FIFO; changed or missing contents, unavailable anchored primitives, and failed parent callbacks invalidate the proof. Register the implementation and focused unit suite in both Make and Meson. The tests cover source and parent replacement, stable absence, repeated and conflicting observations, missing buffers, no-follow policy, /dev/null, FIFO replacement, and parent-opener failure. Signed-off-by: Taylor Blau <ttaylorr@openai.com>
The Darwin daemon treats delivery of its cookie-file event as proof that all earlier worktree changes have been published. That assumes the callback containing the cookie cannot overtake logically older work. A retained FSEvents trace disproves that assumption. The cookie callback completed before a later callback published removals that had happened before the cookie was created. A status query could therefore answer from incomplete event history and report a dirty worktree as clean. After the ordinary cookie wait, ask a long-lived worker to flush the FSEvents stream and then drain its serial callback queue. The flush schedules provider events; the queue drain waits for those callbacks to finish publishing. Accept the boundary only when the cookie was seen in the same token generation, and coalesce overlapping requests onto a single fence. If the bounded fence times out or intersects shutdown, return a conservative result and retire the daemon before unsafe stream teardown. Advertise the stronger boundary as a capability and token suffix so new clients replace unfenced daemons while older clients retain prefix compatibility. Exercise split and blocked callbacks, timeout replacement, generation reset, listener shutdown, concurrent coalescing, second-wave requests, rename and cache scopes, and protocol compatibility. Keep status proof tests outside the split-index matrix where that proof is deliberately disabled, and materialize externally restored tokens before raw-index helpers consume them. The provider fence adds work to each Darwin query, while overlapping queries share a fence when their cookies are already registered.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 27, 2026 22:40
08f9074 to
7bed8a3
Compare
ae161e8 (fsmonitor: validate builtin daemon responses before applying them, 2026-07-10) validates each worktree path with verify_path(). That helper enforces index-entry rules and rejects a .git component anywhere in a path. Filesystem providers can legitimately report such a component for an untracked nested repository. The client therefore rejects the entire response after an event such as scratch/.git/file, forcing a full worktree scan. Commands that need a current provider boundary cannot persist a clean proof from that query. Validate the narrower daemon-response contract instead: require a relative path with nonempty, non-dot components and at most one trailing separator. Keep rejecting absolute and traversal paths, but allow .git components that already exist in the worktree. Cover the parser directly and exercise a real Linux daemon event against an optional-lock-free status oracle.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
2 times, most recently
from
August 28, 2026 05:07
ad39dc8 to
2ebd9f8
Compare
fbd7a23 (rebase: introduce and use pseudo-ref REBASE_HEAD, 2018-02-11) records the commit currently being replayed. The sequencer normally deletes that ref before executing each todo item. When the final item stops for a conflict, rebase --continue commits the resolved result. pick_commits() then reaches the end of the list without entering another iteration and removes the rebase state directly. The merge backend skips finish_rebase() because the sequencer owns cleanup, so REBASE_HEAD survives a successful rebase. The same path also strands the fsmonitor proof when index.skipHash is enabled. After committing the resolution, the sequencer reloads the canonical index and repairs its proof through a close-only index.lock witness. A skip-hash witness has a null trailer and a fresh file identity, so proof-epoch validation cannot bind it to the in-memory index. Rebase succeeds without FSUC, and read-only status cannot repair it. Delete REBASE_HEAD whenever interactive-rebase state is removed. This matches finish_rebase() cleanup and also avoids retaining a ref for an explicitly quit operation. Propagate a failed deletion so rebase does not report success after leaving the stale ref behind. Give only PROVISIONAL_LOCK witnesses a real checksum. The final index rewrite continues to honor index.skipHash, preserving the normal index write fast path while giving proof repair an authenticated epoch. Extend the final-conflict test to require REBASE_HEAD during resolution, its removal after completion, and a reported failure when the ref cannot be deleted. Exercise skip-hash proof repair after a clean-prefix, conflicted replay in primary and linked worktrees, with plain and configured-filter repositories.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 28, 2026 05:34
2ebd9f8 to
16d98d8
Compare
A proof repair can close one provider token, collect untracked results,
reopen the token, and close it again with the same struct wt_status. If
the first closure published untracked output, the second closure tries
to publish another snapshot over it and hits:
BUG: publishing untracked results over collected status
This is reachable from stash pop when an index writer repairs a complete
FSMonitor proof while an untracked path is present.
Before closing a required new token, discard output explicitly marked as
coming from an earlier authenticated token closure or bulk preload.
Keep the BUG for ordinary caller-collected results, which must not be
silently overwritten. Extend refresh invalidation to discard both
authenticated forms as well.
Allow the scripted provider to opt into proof repair, and add a
regression covering the two-token stash path with visible untracked
output.
An fsmonitor provider can reset while merge is reading an index with an authenticated clean-status proof. The reset leaves that proof available for revalidation, but merge updates the worktree before repairing it. The resulting index can lose FSUC after a clean merge. Read-only status cannot persist the missing proof, so every later status falls back. A multi-strategy merge can lose the same history after preparation. An external strategy may replace the index before it declines or reports a conflict. restore_state() then reloads that index while rewinding the worktree. A later built-in strategy sees the original repair decision, but no longer has the paired proof from which to repair. Resolved conflicts expose a separate instance of the same failure. merge clears the resolve-undo extension before updating the worktree. That removal sets RESOLVE_UNDO_CHANGED, which prevented checkout from transferring an otherwise current proof. The result had neither a live provider token nor a pending token from which the writer could repair. Revalidate an authenticated proof before a non-fast-forward merge updates the worktree. Repair it before built-in results are published, after successful external strategies, and after each restore_state() rewind. Permit transfer after the resolve-undo map has been cleared, since removing that optional extension changes neither tracked entries nor worktree contents. Continue rejecting a live resolve-undo map. A repaired writer stats only entries that lack provider validation or stat data before certifying the new index. Fast-forward merges retain their existing path, while conflicts continue to fail closed. Cover built-in ort with and without retained resolve-undo history, trivial and content-level resolve merges, an external strategy that declines, and one that leaves a three-stage conflict before a clean ort retry. Verify that clean results remove resolve-undo data, keep a paired proof, and leave repeated read-only status unable to rewrite the index.
378744b (status: reuse closed proofs for scoped queries, 2026-08-11) taught the untracked cache to reconcile a provider-reported direct child without reopening its directory. A valid cached directory can still have a null exclude_oid when its existing contents are all tracked, since traversal never needed to load its tracked .gitignore. prep_exclude() interprets that null OID as proof that no per-directory exclude exists. The targeted refresh can therefore report a newly created ignored file as untracked. Before refreshing a provider-dirty cached directory with a null exclude OID, use its tracked exclude as the expected identity and load the worktree source. Prefer the exact stage-zero entry, then look for a case-folded alias on case-insensitive worktrees. add_patterns() still opens and hashes the actual source when only an alias exists, so a case collision can only force invalidation. Use the empty-blob ID for an unmerged, non-regular, removed, or intent-to-add match so that it forces a source read and conservative invalidation unless the source is truly empty. Cover both exact and case-folded tracked excludes with read-only status calls. They must match a cold oracle without opening the directory or writing the index, and a changed source must invalidate the cache. Also pin the sparse-index boundaries: an in-cone event retains targeted refresh without expansion, while a vivified outside-cone source takes the existing conservative expansion path.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 30, 2026 00:24
fd910c1 to
b4d251b
Compare
The provider fence added in 7bed8a3 (fsmonitor: fence Darwin callbacks before answering queries, 2026-08-27) calls FSEventStreamFlushSync() from a long-lived worker. The client gives that worker one second before it retires the daemon. Under sustained status traffic, the synchronous provider call can cross that deadline and return immediately afterward. The timeout still forces a daemon restart, and the next status conservatively scans the worktree. A retained trace showed this turning a clean status into a 17-second outlier. On local APFS and HFS volumes, register sticky vnode watches on each watched root and its canonical ancestors before starting the FSEvents stream. Use FSEventStreamFlushAsync() and wait until the callback has published through its returned event ID and a serial queue barrier. The worker waits on its existing condition variable, so the bounded timeout can interrupt it without racing an uncancellable provider call. WatchRoot notifications have event ID zero and cannot be represented by that monotonic token. Treat the kqueue poll as the fence's linearization point, and reject the fence if any watched namespace edge occurred or a watched root changed identity. This also covers a root or ancestor moving away and back before the fence completes. Fall back to the synchronous provider fence when the vnode proof cannot be installed, preserving the existing conservative behavior on unsupported filesystems and resource failures. Exercise the positive event-ID wait, the zero-ID rename ABA, the synchronous fallback, and 512 consecutive read-only status calls. Require every stress-test request to complete without restarting the daemon.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 30, 2026 00:48
b4d251b to
837f89d
Compare
An authenticated clean-status sidecar is bound to the identity of the index file it certifies. A stash push can restore a complete FSMonitor and untracked-cache proof after its child processes rewrite the index, but the existing sidecar still names the old inode. The next read-only status rejects it and takes the slower history path even though stash left the worktree clean. Remember whether stash started with a regular, singly linked sidecar. When optional locks are available and no post-index-change hook is configured, retain the status data gathered by proof repair, commit and reread the final index, then issue a replacement sidecar from that same certifying scan. This avoids a second worktree traversal while binding the proof to the final index identity. Other stash paths keep the existing repair behavior. Cover a scoped push with a subsequent read-only status that must take the clean-proof fast path. Let the existing writer-proof test accept both authenticated sidecar hits and coherent-history reuse, since both are valid read-only fast paths.
An authenticated clean-status sidecar is bound to the identity of the index file it certifies. During an interactive rebase, the child commit run by "rebase --continue" can replace that index. The worktree and index are clean when the replay finishes, but the remaining sidecar still names the old inode. The next read-only status rejects it with a fast-index-mismatch and falls back to the slower history path. Remember whether the rebase started with a regular, singly linked sidecar and persistent FSMonitor proof history. After the replay finishes successfully, reread the final index and use the existing writer-proof repair to publish a replacement sidecar. Only do so when optional locks are available and no post-index-change hook is configured, matching the existing stash guardrails. Move the sidecar-presence check and sidecar-capable repair helper into wt-status so stash and sequencer can share them. Cover a conflicted interactive rebase whose continuation must publish a replacement sidecar and whose next read-only status must hit it.
The clean-status sidecar path accepts only the main worktree. A newly created linked worktree therefore cannot publish a proof after a full clean scan. Later read-only status commands rescan the worktree even though each linked worktree has its own index and sidecar path. Accept a linked worktree only when its per-worktree gitdir remains registered in the common directory and the registered path names the current worktree. Reject an alternate worktree paired with a linked- worktree gitdir; the repository fingerprint continues to bind the resolved worktree, gitdir, common directory, index, and filesystem identities. New worktrees are commonly probed with "git status --short". Let the exact top-level --short and -s forms certify empty output. A fresh worktree index can still be racy, so write and re-read it before saving resumable history and issuing the sidecar. This binds both proofs to the new on-disk index epoch. Cover issuance and optional-lock-free reuse in a registered linked worktree, and verify that an impostor worktree sharing its gitdir falls back.
Stash and rebase may repair and reissue a clean-status sidecar after their primary operation has completed. A lock, read, or proof-repair failure in that optional work currently replaces the successful command result. The command then reports failure even though it has already updated the repository and worktree. The repair path also commits its updated index before rereading it and issuing the sidecar. Another writer can replace the index in that gap. The old clean scan could then be bound to the replacement index. A later status could hide a newly staged change. Treat sidecar repair as best-effort after stash and rebase complete. Retain a descriptor-backed snapshot of the index produced by each postwrite clean scan. Require it to match both the reread index state and the canonical index path before installing the sidecar. A failed repair or intervening write therefore omits the cache and falls back to ordinary status without changing the primary command result. The normal sidecar-hit path remains unchanged. Cover lock contention after successful stash and rebase operations, and replace the index at deterministic postwrite barriers in both repair and status issuance paths.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 30, 2026 11:11
23f8d90 to
d5c2374
Compare
Clean-status sidecars require a durable index identity on local APFS. The history behavior is still valid on other filesystems, but two tests required index.csts after their substantive assertions passed and failed during cleanup on Linux. Hardlink metadata events can also arrive before status refreshes the index. If that refresh creates a racy index, status may conservatively withhold a sidecar until a later scan restores its process-local proof. Requiring immediate reissuance made the test depend on provider timing. Gate sidecar removal on local APFS. For the racy-index case, accept either immediate reissuance or the conservative fallback, but require clean output and recovery to a new proof within three status calls.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
5 times, most recently
from
August 31, 2026 07:30
a43fd03 to
02e1d29
Compare
b65fc91 (status: retain the identity of an index it rewrites, 2026-08-17) lets status keep a race-proof receipt for an index it rewrites. Receipt preparation duplicates the writer descriptor so it can hash the final bytes, and requires the in-memory checksum to match the configured null trailer. Three owned write paths can leave an otherwise valid index without a usable clean proof. A worktree-update repair first writes a checksummed provisional index, then reopens the lockfile write-only before the final skipHash write while the index still records the provisional checksum. Receipt preparation rejects both states, so scoped stash cannot publish a sidecar for the index it installs. Worktree add creates its linked index before the new worktree has a closed FSMonitor provider epoch. The index has FSMonitor and untracked-cache extensions, but lacks the authenticated clean-config proof. Later read-only status processes remain correct, but cannot persist that proof and repeat the full fallback on every invocation. A post-checkout hook can also change worktree-specific configuration. Checking the invoking worktree's settings after the hook can therefore skip priming when the hook enables FSMonitor and the untracked cache only in the new worktree. A clean non-fast-forward merge repairs its authenticated index proof before committing, but leaves the existing sidecar bound to the old index and HEAD tree. The next read-only status rejects it with a fast-index-mismatch and scans the semantic manifest. Only a later writable status can replace the stale sidecar. On Apple, add a read-write reopen operation only for the provisional index lock so receipt preparation can read the final index. Fall back to the original write-only reopen when read access is unavailable, allowing the write to succeed without a receipt. Other platforms retain the write-only reopen. Finish provisional writes through a cold helper that restores the null object ID before the receipt-aware write. After worktree add successfully runs the post-checkout hook, read the linked worktree's effective FSMonitor and untracked-cache settings. If both features are enabled, run one silent status to establish the provider epoch and persist the complete proof. Do so only when the caller permits optional locks; never override an explicit --no-optional-locks request. Factor the best-effort sidecar reissue used by rebase into wt-status. After a successful merge commit has installed its final HEAD and index, use that helper to authenticate the settled state. Sidecar failure still falls back to ordinary status and never changes the merge result. Cover receipt publication and adoption after scoped stash, preserve the generic write-only tempfile contract, and require worktree add to honor post-checkout index writes, linked-worktree configuration, and disabled optional locks. Also require a clean non-fast-forward merge to publish a sidecar that its next read-only status can consume.
ttaylorr-oai
force-pushed
the
tb/codex/fsmonitor-hardlink-inodes-unstable
branch
from
August 31, 2026 08:59
02e1d29 to
8055b6c
Compare
A command can disable worktree-to-Git filters with empty clean and process commands and required=false. It does not need to disable smudge, which converts in the other direction. The status fingerprint only normalizes the four-setting form, so an otherwise equivalent three-part override discards scoped FSMonitor history and forces a tracked-file scan. Recognize the complete read-side override as well. Continue to fingerprint partial or mixed-driver overrides, and retain the normalized-filter bit so that temporarily disabling filters cannot publish a clean sidecar. The scoped proof must still establish that no tracked path uses a filter. Exercise all subsets with both hash algorithms, the three-setting diff invocation, and active-filter write and priming attempts with either form.
The reset used by worktree add has no index proof to repair, and its explicit GIT_WORK_TREE prevents proof authentication. A later optional status can establish history, but GIT_OPTIONAL_LOCKS=0 suppresses that priming step. Subsequent read-only status calls cannot persist the missing proof and keep repeating the tracked-file scan. When FSMonitor and writable untracked caching are enabled, let the checkout discover the already registered worktree. Preserve the explicit repository environment for unsupported contexts, including relative configuration-file overrides whose meaning would change with cwd. For a hard reset that creates the index, certify the checkout under the mandatory index lock before committing it. Attach the current config to the new index state, query the provider after checkout, and reuse the writer repair machinery to bind the tracked and untracked proofs. Allow certification under this owned lock even when optional locks are disabled; ordinary read-only status still cannot write or repair its index. Require a complete untracked scan and a closing provider query. Active filters, provider errors, and provider resets must leave the checkout usable without certifying it. Cover those failures, post-checkout edits, and repeated read-only status calls that reuse the proof without writing the index or scanning tracked files.
ttaylorr-oai
force-pushed
the
codex-unstable
branch
from
September 8, 2026 02:28
fded12c to
3e7318a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current candidate
Head:
5f1339f46d. This pull request remains review-only; do not merge orenable auto-merge. Fresh approval is required for this head before
controller admission.
Two additional patches address command patterns that prevented proof reuse:
filter.<driver>.cleanandprocesswithrequired=falseas complete read-side disabling, without requiring asmudge override. Scoped FSMonitor reuse still requires a filter-free scope.
including when
GIT_OPTIONAL_LOCKS=0. Later read-only status calls reusethe paired proof without writing the index or scanning tracked files.
Regression controls cover active filters, provider errors and resets,
post-checkout edits, partial overrides, and both hash algorithms.
Focused native validation passed:
t7519-status-fsmonitor.sh: 123 testst7527-builtin-fsmonitor.sh: 200 testst7530-status-clean-sidecar.sh: 67 testsHosted push and pull-request CI each completed with 45 successful jobs and
two expected skips. Style and whitespace checks passed.
A fresh clean developer-mode full suite passed: 33,507 successes, zero
unexpected failures, and 378 known breakages. Full status and FSMonitor
suites also passed at both commit boundaries, as did a separate incremental
build through both patches.
Two executable Git workflows passed on each of macOS and Linux, covering
staging, stash, conflicting rebase, linked worktrees, merge, and worktree
removal. Optimized results matched conservative oracles, and the required
read-only checks left the index unchanged. The macOS workflows used the
same sealed artifact as the latency campaign.
Qualification remains open. An earlier incremental full-suite run failed
the linked-worktree stash-history test in
t7519. Its failing assertionwas not captured. Subsequent clean, incremental, and paired stress runs
pass; branch traces do not support the proposed cause in these patches.
Three further full
t7519repeats per commit also passed. The simulatedreview identified no supported source fix; the original failure remains
recorded and unexplained.
Same-base macOS latency qualification is still open. The corrected
four-repository run passed its oracle, provider-fence, identity, and
read-only-index checks, but did not meet the mean-latency thresholds for
clean writable and dirty read-only status. Median and p95 limits passed.
No samples were discarded and no threshold was relaxed.
A bounded diagnostic captured multi-second stalls in both builds before
Git initialized its process clock; the exact startup cause is unresolved.
These observations are not a latency pass. No release will be admitted
before the remaining latency qualification and fresh approval.
Earlier implementation and validation notes
Summary
open(2)shutdown wake reliable across
EINTR, and containSIGPIPEduring gentleclient writes
accepting a query boundary, with bounded timeout, daemon retirement, token
generation checks, and request coalescing
Controller scope
This review-only pull request presents one prerequisite and two topic patches
on top of
1293167e46. Approve it for controller admission, but do not mergeit into
codex-unstable.Performance
Overlapping Darwin queries whose cookies are already registered share one
provider fence. A timeout or shutdown intersection falls back conservatively
and retires the daemon. Paired same-base latency qualification is being run on
this exact source and binary; this description does not make a latency claim
before that evidence completes.
Validation
At commit
7bed8a334f:t0052-simple-ipc.sh(11/11)unique paths)
EINTRshutdown wake stress (50/50, with 50 verified injections)The preceding candidate,
08f907411e, completed unit tests (406/406),t7519-status-fsmonitor.sh(110/110),t7527-builtin-fsmonitor.sh(194/194),t7530-status-clean-sidecar.sh(59/59), and theASan+UBSan
clean_status_manifestunit tests (4/4). The only tree change fromthat candidate is replacing an open-coded allocation in
t/helper/test-simple-ipc.cwithCALLOC_ARRAY()as required by staticanalysis. Production sources, FSMonitor tests, and the FSMonitor topic
patch-id are unchanged.
Exact-head hosted CI, same-base latency qualification, and executable workflow
qualification are in progress.