Concurrent get_kernel calls on a fresh machine fail with TUFError: Failed to refresh TUF metadata #970

Description

Since kernels 0.18.0, get_kernel verifies the sigstore signature of Hub kernels by default (SignatureValidator, added in #828). When no verification receipt is cached yet, verify_signature calls sigstore.verify.Verifier.production(), which refreshes the TUF trust root in ~/.local/share/sigstore-python/tuf/. python-tuf replaces the root.json symlink there with a non-atomic os.remove + os.symlink, so when several processes load a kernel at the same time some of them fail with FileExistsError, raised as sigstore.errors.TUFError: Failed to refresh TUF metadata.

This is the normal situation of a multi-GPU launch (torchrun, accelerate launch) on a fresh machine, container or CI runner: every rank loads the same kernel at the same time. It is not a rare race: with 8 processes, every trial I ran had at least one failed load.

The error is not caught by SignatureValidator, although its docstring says verification issues are currently reported as warnings, so the kernel load itself fails.

python-tuf documents that concurrent instances sharing a cache are not supported, see:

• ngclient: Be better with concurrent instances theupdateframework/python-tuf#2836
• Concurrency errors from multiple TUF Updaters sigstore/sigstore-python#1403

We hit this in the TRL CI, where pytest-xdist workers load kernels-community/flash-attn3 at the same time: huggingface/trl#8130

Steps to reproduce

# Load the same Hub kernel from 8 processes at the same time on a fresh machine:
# empty sigstore TUF cache (in HOME) and empty kernels cache (no signature verification receipts).
import multiprocessing as mp
import os
import tempfile


def load(barrier):
    from kernels import get_kernel

    barrier.wait()
    get_kernel("kernels-community/activation", version=1)


if __name__ == "__main__":
    home = tempfile.mkdtemp()
    os.environ["HOME"] = home
    os.environ["KERNELS_CACHE"] = os.path.join(home, "kernels")
    ctx = mp.get_context("spawn")
    barrier = ctx.Barrier(8)
    procs = [ctx.Process(target=load, args=(barrier,)) for _ in range(8)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    print(f"{sum(p.exitcode != 0 for p in procs)}/8 loads failed")

Output (3 out of 3 runs): 2/8 loads failed, each with:

Traceback (most recent call last):
  File ".../sigstore/_internal/tuf.py", line 117, in __init__
    self._updater = Updater(
  File ".../tuf/ngclient/updater.py", line 148, in __init__
    self._update_root_symlink()
  File ".../tuf/ngclient/updater.py", line 373, in _update_root_symlink
    os.symlink(current, linkname)
FileExistsError: [Errno 17] File exists: 'root_history/15.root.json' -> '.../tmpdsposr6f/.local/share/sigstore-python/tuf/https%3A%2F%2Ftuf-repo-cdn.sigstore.dev/root.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "repro_kernels.py", line 12, in load
    get_kernel("kernels-community/activation", version=1)
  File ".../kernels/load.py", line 175, in get_kernel
    return get_kernel_with_resolver(
  File ".../kernels/load.py", line 99, in get_kernel_with_resolver
    tree_only_local.validate_kernel(kernel_validator)
  File ".../kernels/deps.py", line 72, in validate_kernel
    validator.validate_kernel(kernel=self.location)
  File ".../kernels/validate.py", line 249, in validate_kernel
    validator.validate_kernel(kernel=kernel)
  File ".../kernels/validate.py", line 186, in validate_kernel
    result = verify_signature(kernel.variant_path, policy=self.policy, location=location)
  File ".../kernels/verify.py", line 243, in verify_signature
    verifier = Verifier.production()
  File ".../sigstore/verify/verifier.py", line 112, in production
    config = ClientTrustConfig.production(offline=offline)
  File ".../sigstore/models.py", line 937, in production
    return cls.from_tuf(DEFAULT_TUF_URL, offline)
  File ".../sigstore/models.py", line 963, in from_tuf
    updater = TrustUpdater(url, offline, bootstrap_root)
  File ".../sigstore/_internal/tuf.py", line 129, in __init__
    raise TUFError("Failed to refresh TUF metadata") from e
sigstore.errors.TUFError: Failed to refresh TUF metadata

Loading the kernel once in a single process first (so the verification receipt is cached) avoids it: the 8 concurrent loads then succeeded in 5 out of 5 runs.

Expected behavior

Concurrent get_kernel calls from several processes succeed, as with kernels 0.17. For example:

• serialize the signature verification across processes, e.g. with a file lock around Verifier.production() / the verification, or
• report a failure to refresh the TUF metadata as the warning that the SignatureValidator docstring describes, instead of failing the load.

Environment

• kernels version: 0.18.1 (sigstore 4.5.1, tuf 7.0.1)
• Python version: 3.13.9
• PyTorch version: 2.15.0+cu132
• CUDA version: 13.2
• GPU model: NVIDIA H100 80GB HBM3
• OS: Linux-6.8.0-1057-aws-x86_64-with-glibc2.35

Additional context

The race also happens with kernels-community/flash-attn3 (3 out of 3 trials with 8 processes).