Summary
Wire the freestanding execve restore-stub (already landed, tested, but #[allow(dead_code)]) into the live restore path, and replace the ptrace injection engine with it. The stub restores a checkpoint into a fresh execve'd address space that holds only the checkpoint image plus a fresh kernel vDSO/vvar, eliminating the leftover-launcher-mapping contamination that limits the current injection engine.
Background
Live-process restore today (Sandbox::restore_interactive -> checkpoint::resume::restore_into) works by forking a child that parks before execve, then ptrace-injecting the checkpoint image over that parked launcher. This works for an idle libc program (file-backed maps, anon data via process_vm_writev, fd reopen, and [vdso]/[vvar] relocation via plan_vdso_moves are all handled), but a busy workload segfaults on resume: the parked launcher's own mappings (libc, stack, unswept trampolines) coexist with the injected image and get hit once the restored program runs freely.
The execve-stub apparatus was landed incrementally to fix this at the root by giving restore a clean address space:
checkpoint/restore_blob.rs - control-blob wire format + serializer (anon-only today; n_fds hardcoded 0; file-backed regions marked NO_DATA).
checkpoint/restore-stub.c - freestanding x86_64 stub: maps anon regions MAP_FIXED, registers userfaultfd (missing mode), READY/GO handshake with the supervisor, rt_sigreturns into the checkpoint registers. No file-backed regions, no fd table, no vDSO relocation, no FP restore yet.
checkpoint/pager.rs - supervisor-side userfaultfd page server for the anon working set.
tests/integration/test_restore_stub.rs - end-to-end proof with a hand-assembled single-region program.
All of the above is exercised only by its own unit/integration tests and is not called from restore_interactive/restore_into. See the checkpoint/mod.rs comment ("not yet wired into the live restore code ... drop [the allow] then").
Goal (first milestone)
Restore a real, idle libc program via the stub (concrete target: the musl sleep-loop service process we already restore via injection). Definition of done: checkpoint the idle service, restore it through the stub path, and observe it resume and keep running with correct clock_gettime/vDSO behavior. The busy-workload case is explicitly out of scope for this milestone (it is the next milestone the clean address space unlocks).
Approach: pure execve stub
The stub reconstructs the whole address space itself; the supervisor only serves anon pages lazily over userfaultfd.
Stub (restore-stub.c) gains:
- File-backed regions -
openat the region's in-chroot path and mmap MAP_PRIVATE|MAP_FIXED at file_off with the recorded prot. Writable-private file-backed pages that were dirtied are captured as anon data and overlaid (uffd-served) on top.
- fd table - reopen each restorable fd:
openat the in-chroot path with saved flags, lseek to the saved offset, dup2 to the target fd number.
- vDSO/vvar relocation - find the fresh vDSO via
AT_SYSINFO_EHDR in auxv and mremap [vdso] (and [vvar]/[vvar_vclock]) onto the checkpoint-recorded base, so glibc/musl's cached vDSO pointers resolve. (vvar has no auxv entry; locate it relative to vDSO / from a supervisor-passed base if needed.)
- FP register restore - populate the
rt_sigframe fpstate from the captured fpregs instead of the current m->fpstate = 0.
Blob format (restore_blob.rs) gains (bump BLOB_VERSION):
- Real fd table entries (
n_fds > 0): fd number, in-chroot path, flags, offset.
- Per file-backed region: the in-chroot path string + file offset.
- memfd /
(deleted)-backed private regions (e.g. /memfd:sandlock-exec (deleted)) marked anon-with-data so their captured bytes are paged in (this pairs with the capture-side dump already added in the chroot-restore fix).
Supervisor (restore_interactive / a new resume entrypoint): fork child, apply confinement (chroot + Landlock + seccomp), install the fixed inherited fds (control-blob memfd at CTRL_FD, READY_FD/GO_FD eventfds, UFFD_SLOT), execve the stub, pidfd_getfd the uffd, run the pager, GO, wait.
Remove the duplicated logic
Once the stub reaches parity on the idle case, delete the injection engine and de-duplicate:
- Remove
checkpoint/inject.rs and the ptrace-injection path in checkpoint/resume.rs (restore_into, RemapFromFile/WriteBytes application, injected fd reopen).
- Drop the
#[allow(dead_code)] markers on restore_blob/pager/the CTRL_FD/READY_FD/GO_FD/UFFD_SLOT constants in checkpoint/mod.rs.
- Single home for shared planning. The region/fd/vDSO planning logic (
build_memory_plan, build_fd_plan, plan_vdso_moves, and the host->in-chroot path translation chroot::resolve::host_to_virtual) should live in exactly one place and feed the blob serializer, rather than being re-implemented in the stub. Keep the decisions (what to map, which fds are restorable, which paths translate) in Rust at serialize time; the stub stays a dumb executor of the blob. Anything in resume.rs that survives only to serve injection goes away with it.
Confinement / seccomp consideration
The stub runs inside the sandbox (confinement must be applied before execve because rt_sigreturn hands control to the restored program with no later setup hook). The seccomp policy must therefore permit the stub's syscalls during restore: openat, mmap, mremap, lseek, fstat, ioctl, userfaultfd, dup2, read, write, rt_sigreturn, exit. Decide whether these are always-allowed by policy or gated behind a restore-time allowance (note: seccomp is one-way, so anything the stub needs, the restored program keeps). Landlock must grant read/exec on the image files at their in-chroot paths; chroot must already be in place so those paths resolve.
Out of scope (future milestones)
- Busy-workload restore (the clean address space is the prerequisite; fidelity of an actively-running register/stack/vdso state is its own milestone).
- Non-x86_64 architectures (stub is x86_64-only; keep the existing arch guard).
- Sockets/pipes/eventfd fds (remain surfaced via
restore_skipped / the app_state hatch).
Acceptance
Summary
Wire the freestanding execve restore-stub (already landed, tested, but
#[allow(dead_code)]) into the live restore path, and replace the ptrace injection engine with it. The stub restores a checkpoint into a freshexecve'd address space that holds only the checkpoint image plus a fresh kernel vDSO/vvar, eliminating the leftover-launcher-mapping contamination that limits the current injection engine.Background
Live-process restore today (
Sandbox::restore_interactive->checkpoint::resume::restore_into) works by forking a child that parks beforeexecve, then ptrace-injecting the checkpoint image over that parked launcher. This works for an idle libc program (file-backed maps, anon data viaprocess_vm_writev, fd reopen, and[vdso]/[vvar]relocation viaplan_vdso_movesare all handled), but a busy workload segfaults on resume: the parked launcher's own mappings (libc, stack, unswept trampolines) coexist with the injected image and get hit once the restored program runs freely.The execve-stub apparatus was landed incrementally to fix this at the root by giving restore a clean address space:
checkpoint/restore_blob.rs- control-blob wire format + serializer (anon-only today;n_fdshardcoded 0; file-backed regions markedNO_DATA).checkpoint/restore-stub.c- freestanding x86_64 stub: maps anon regionsMAP_FIXED, registers userfaultfd (missing mode), READY/GO handshake with the supervisor,rt_sigreturns into the checkpoint registers. No file-backed regions, no fd table, no vDSO relocation, no FP restore yet.checkpoint/pager.rs- supervisor-side userfaultfd page server for the anon working set.tests/integration/test_restore_stub.rs- end-to-end proof with a hand-assembled single-region program.All of the above is exercised only by its own unit/integration tests and is not called from
restore_interactive/restore_into. See thecheckpoint/mod.rscomment ("not yet wired into the live restore code ... drop [the allow] then").Goal (first milestone)
Restore a real, idle libc program via the stub (concrete target: the musl
sleep-loop service process we already restore via injection). Definition of done: checkpoint the idle service, restore it through the stub path, and observe it resume and keep running with correctclock_gettime/vDSO behavior. The busy-workload case is explicitly out of scope for this milestone (it is the next milestone the clean address space unlocks).Approach: pure execve stub
The stub reconstructs the whole address space itself; the supervisor only serves anon pages lazily over userfaultfd.
Stub (
restore-stub.c) gains:openatthe region's in-chroot path andmmapMAP_PRIVATE|MAP_FIXEDatfile_offwith the recorded prot. Writable-private file-backed pages that were dirtied are captured as anon data and overlaid (uffd-served) on top.openatthe in-chroot path with saved flags,lseekto the saved offset,dup2to the target fd number.AT_SYSINFO_EHDRin auxv andmremap[vdso](and[vvar]/[vvar_vclock]) onto the checkpoint-recorded base, so glibc/musl's cached vDSO pointers resolve. (vvar has no auxv entry; locate it relative to vDSO / from a supervisor-passed base if needed.)rt_sigframefpstate from the capturedfpregsinstead of the currentm->fpstate = 0.Blob format (
restore_blob.rs) gains (bumpBLOB_VERSION):n_fds> 0): fd number, in-chroot path, flags, offset.(deleted)-backed private regions (e.g./memfd:sandlock-exec (deleted)) marked anon-with-data so their captured bytes are paged in (this pairs with the capture-side dump already added in the chroot-restore fix).Supervisor (
restore_interactive/ a newresumeentrypoint): fork child, apply confinement (chroot + Landlock + seccomp), install the fixed inherited fds (control-blob memfd atCTRL_FD,READY_FD/GO_FDeventfds,UFFD_SLOT),execvethe stub,pidfd_getfdthe uffd, run the pager, GO, wait.Remove the duplicated logic
Once the stub reaches parity on the idle case, delete the injection engine and de-duplicate:
checkpoint/inject.rsand the ptrace-injection path incheckpoint/resume.rs(restore_into,RemapFromFile/WriteBytesapplication, injected fd reopen).#[allow(dead_code)]markers onrestore_blob/pager/theCTRL_FD/READY_FD/GO_FD/UFFD_SLOTconstants incheckpoint/mod.rs.build_memory_plan,build_fd_plan,plan_vdso_moves, and the host->in-chroot path translationchroot::resolve::host_to_virtual) should live in exactly one place and feed the blob serializer, rather than being re-implemented in the stub. Keep the decisions (what to map, which fds are restorable, which paths translate) in Rust at serialize time; the stub stays a dumb executor of the blob. Anything inresume.rsthat survives only to serve injection goes away with it.Confinement / seccomp consideration
The stub runs inside the sandbox (confinement must be applied before
execvebecausert_sigreturnhands control to the restored program with no later setup hook). The seccomp policy must therefore permit the stub's syscalls during restore:openat,mmap,mremap,lseek,fstat,ioctl,userfaultfd,dup2,read,write,rt_sigreturn,exit. Decide whether these are always-allowed by policy or gated behind a restore-time allowance (note: seccomp is one-way, so anything the stub needs, the restored program keeps). Landlock must grant read/exec on the image files at their in-chroot paths; chroot must already be in place so those paths resolve.Out of scope (future milestones)
restore_skipped/ theapp_statehatch).Acceptance
running == true), vDSO calls work.inject.rs+ inject path inresume.rs) removed; no#[allow(dead_code)]left on the stub machinery.