Skip to content

ngclient: Update root.json link atomically - #3010

Open
Sachith77 wants to merge 1 commit into
theupdateframework:developfrom
Sachith77:fix/atomic-root-symlink
Open

Sachith77 wants to merge 1 commit into
theupdateframework:developfrom
Sachith77:fix/atomic-root-symlink

Conversation

@Sachith77

Copy link
Copy Markdown
Contributor

_update_root_symlink() deletes root.json before creating the new link, so if the process dies in that window or the link can't be created, root.json just stays gone:

after failed update, root.json exists: False
bootstrap=None now fails: FileNotFoundError

Since that's the trust anchor for Updater(bootstrap=None), the client can't start after that. It also runs from a finally block in _load_root(), so this can happen on failure paths too.

Fixed this by creating the link under a temp name and using os.replace() to put it in place, same idea as _persist_file(). I used a fixed .tmp name instead of tempfile since os.symlink() needs a path that doesn't exist yet, and multiple Updater instances aren't supported anyway.

Added a test that fails on develop.

Removing root.json before creating the new link means a failure
leaves no root.json at all, and bootstrap=None can no longer be
used. Use a temp name + os.replace(), like _persist_file() does.

Signed-off-by: Sachith Reddy <sachith.24bcs10403@sst.scaler.com>
@Sachith77
Sachith77 requested a review from a team as a code owner September 3, 2026 11:17
@Sachith77

Copy link
Copy Markdown
Contributor Author

@jku could you please have a look when you get a chance .

@albertvillanova

Copy link
Copy Markdown

This also matters for concurrent processes sharing the metadata directory (#2836), but the fixed root.json.tmp name moves the race to the temp file: processes remove or replace each other's temp link. With 8 or 16 processes creating an Updater at the same time on an empty cache (via sigstore Verifier.production()), every trial still had failed processes, now also with FileNotFoundError on os.replace(temp_name, linkname).

A temp name unique to the process makes this step concurrency-safe on POSIX:

temp_name = f"{linkname}.{os.getpid()}.tmp"

With that change, the same test had no failures (0 out of 15 trials, vs 15 out of 15 with this PR and 18 out of 18 without it). The other metadata writes already go through _persist_file, which uses a unique NamedTemporaryFile + os.replace.

Context: since kernels 0.18, every Hugging Face kernel load verifies a sigstore signature, so the ranks of a multi-GPU launch hit this on a fresh machine: huggingface/kernels#970

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants