discard the sticky disk's resolv-host.conf before buildkitd starts

[shreyas-blacksmith]

What broke

A repo's --network=host RUN steps can fail with getaddrinfo EAI_AGAIN <hostname> for any external name during the image build, while the VM's own DNS works fine. It only hits VMs that had been up for a few minutes before the job started, and only Dockerfiles whose first container is not a host-network RUN (for example ones that start with # syntax=docker/dockerfile:…, which runs the frontend as a sandboxed container first).

Why

BuildKit generates the /etc/resolv.conf it bind-mounts into RUN containers once and keeps it on disk under the worker root: executor/resolv.conf for sandboxed RUNs, executor/resolv-host.conf for --network=host RUNs. We configure buildkitd with dns.nameservers = [<this VM's eth0 IP>], so the file names the VM's own address.

The worker root is /var/lib/buildkit, which is the repo's sticky disk. So the saved resolv-host.conf is inherited from whichever VM last committed the disk, and it names that VM's eth0 address. From this VM that address is dead.

BuildKit guards against stale files two ways, and both miss this case:

• runcexecutor.New deletes executor/hosts and executor/resolv.conf on daemon start. It does not delete resolv-host.conf (added later for host networking).
• GetResolvConf regenerates on the daemon's first container of either mode. After that it reuses the saved file unless its mtime is older than /etc/resolv.conf. On our VMs /etc/resolv.conf is written at boot, so a file committed by another VM after this VM booted looks fresh and is kept.

With a # syntax= frontend, the frontend container (sandboxed mode) is the daemon's first container and uses up the first-run regeneration. The first host-mode RUN then falls through to the mtime check and inherits the other VM's nameserver.

flowchart LR
  subgraph before [Before]
    direction TB
    a1[mount sticky disk] --> a2[buildkitd starts: rm hosts, resolv.conf] --> a3[frontend container: first run, regenerate resolv.conf] --> a4[host-mode RUN: saved resolv-host.conf newer than /etc/resolv.conf, keep it] --> a5[nameserver = other VM's eth0: EAI_AGAIN]
  end
  subgraph after [After]
    direction TB
    b1[mount sticky disk] --> b2[new: rm executor/resolv-host.conf] --> b3[buildkitd starts] --> b4[frontend container: first run] --> b5[host-mode RUN: no saved file, regenerate with this VM's eth0] --> b6[lookups work]
  end
  before ~~~ after

Loading

Reproduced on devbox with the v0.29.5-blacksmith binary: plant a bogus nameserver in resolv-host.conf, run a sandboxed step first, and the host-mode step inherits it; run the host-mode step first and it regenerates.

The change

Right after setupStickyDisk mounts the disk, remove /var/lib/buildkit/runc-*/executor/resolv-host.conf (as root, so the glob expands). Failure to remove only warns. buildkitd then writes a fresh one on the first host-mode RUN.

A per-VM file should never ride along on the sticky disk. The matching fork/upstream fix is to add resolv-host.conf to the startup cleanup in runcexecutor.New; this PR is the no-roll version.

Verification

pnpm eslint ., pnpm tsc --noEmit, pnpm vitest run (31 tests) and pnpm run build pass locally; dist/ is rebuilt from this source. The live action-test jobs in CI exercise the new removal on real sticky disks.

[blacksmith-staging]

✅ Host-network RUN after sticky disk carries another VM's resolv-host.conf

Before, the host-network RUN inherited the other VM's nameserver and the lookup timed out. After, the post-mount cleanup removed the file and the RUN got this VM's eth0 and resolved.

✅ 2 of 2 claims verified with evidence

	Case	What Proof saw
✅ As described	Host-network RUN after sticky disk carries another VM's resolv-host.conf	Before, the host-network RUN inherited the other VM's nameserver and the lookup timed out. After, the post-mount cleanup removed the file and the RUN got this VM's eth0 and resolved.
✅ As described	Cleanup touches only resolv-host.conf	Only the executor's resolv-host.conf disappeared. The rest of the disk was unchanged, the decoy survived, and a fresh disk ran clean with no warning.

_{Proof on 45d9b56c}

View full Proof report

[ajwerner]

LGTM