Skip to content

fix(sandbox): make mediated signal, scheduling, and limit calls behave natively - #5212

Open
EmilienM wants to merge 3 commits into
NVIDIA:mainfrom
EmilienM:fix/5185-sandbox-signal-sched/EmilienM
Open

EmilienM wants to merge 3 commits into
NVIDIA:mainfrom
EmilienM:fix/5185-sandbox-signal-sched/EmilienM

Conversation

@EmilienM

@EmilienM EmilienM commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three compatibility fixes for the mediated signal and scheduling syscalls, so common tooling (pthread affinity, taskset -p, nice, prlimit(getpid()), self-signals, sigqueue) behaves inside a sandbox as it does in a plain container.

Related Issue

Closes #5185

Changes

  • rt_sigqueueinfo mediation now sets si_signo to the requested signal before pidfd_send_signal. The kernel rejects a mismatched siginfo with EINVAL there, while native rt_sigqueueinfo overwrites the field.
  • Self-directed kill/rt_sigqueueinfo notifications (the caller's PID or a sibling TID) are continued instead of emulated through a pidfd. The kernel executes the syscall after the notification wait, so errno (for example EAGAIN under an exhausted RLIMIT_SIGPENDING), siginfo identity, and handler timing are the workload's own, and the delivered signal can no longer interrupt the call being emulated (the EINTR and restart-and-renotify loops from the issue).
  • sched_setaffinity, sched_setattr, sched_setparam, sched_setscheduler, prlimit64, and setpriority move from static child-filter denies to broker mediation, following the existing tkill resolve-then-continue pattern: the caller's own TGID and sibling TIDs in its thread group are allowed; every other target stays EPERM. setpriority accepts only PRIO_PROCESS (PRIO_PGRP and PRIO_USER stay denied because they can reach the shared workload UID or a rejoined process group), and invalid arguments keep their native EINVAL/ESRCH. A static equal-deny on the trusted sandbox leader remains as defense in depth.
  • Calls with pid 0 (the caller itself) are allowed in the listener's cBPF filter and never notified, so they stay non-interruptible and off the broker, as before this PR. Only named targets round-trip.
  • The six syscalls, each one's PID argument, and the pid 0 rule are defined once in MEDIATED_PROCESS_SCHED_SYSCALLS; the notified set, the cBPF pid 0 path, broker dispatch, static deny, and argument decode all derive from it.
  • A child without the workload listener (the socket probe) takes child_seccomp::prepare_without_broker, which keeps the previous static rules: pid 0 only, setpriority denied.
  • New e2e test sandbox_compat_signals_sched runs inside a real sandbox via python3: thread affinity, self and sibling-TID signals, os.nice, raw prlimit64, a queued signal with a mismatched si_signo, and denial probes against a same-UID child and the sandbox parent. Docker and Kubernetes lanes pick it up by default; it is added to the podman CI allowlist.

Accepted risk: PID reuse on a continued target

Linux has no pidfd form of sched_*, prlimit64, or setpriority, so the broker resolves a named target through procfs and continues the syscall with the numeric TID. A PID reused between that check and kernel execution could redirect the call to another same-UID task. This is the same class as the existing tkill resolve-then-continue, the window is one broker round trip, and the trusted sandbox leader stays covered by the static equal-deny. Documented in the process_sched module doc. Reviewers, please sign off on this explicitly.

Merge coordination: #5161 also changes child_seccomp::prepare and mediate_process_signal; whichever lands second needs a rebase.

Release Notes

Sandboxed workloads can now set their own CPU affinity, nice value, and resource limits, signal themselves, and queue signals, the same as in a plain container. Calls targeting processes outside the workload's own thread group are still denied.

Testing

  • cargo test -p openshell-sandbox (345 lib tests, including allow/deny, native-errno, EAGAIN, sibling-TID, and pid 0 native-path tests)
  • cargo clippy -p openshell-sandbox --all-targets --features perf-harness -- -D warnings; cargo fmt --all --check
  • cargo zigbuild check --target x86_64-unknown-linux-musl -p openshell-sandbox (the previous head did not build on musl)
  • e2e:podman lane with sandbox_compat_signals_sched
  • Note: mise run rust:lint/pre-commit fail on this workstation before reaching any code check (rpm-shipped /etc/uv/uv.toml the pinned uv cannot parse); the cargo-level equivalents above all pass.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (crate README module list)

@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mrunalp

mrunalp commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

/ok to test bdeb980

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. api-behavior — No explicit risk acceptance found

PID-zero prlimit64 and sched_set* calls now enter interruptible seccomp mediation instead of executing natively.

  • Impact: Existing C/C++ workloads using signal handlers without SA_RESTART can receive unexpected EINTR failures from previously non-interruptible resource-limit and scheduling operations.
  • Action: Keep existing PID-zero prlimit64 and sched_set* calls on a native seccomp fast path while mediating explicit targets, since this train does not allow breaking existing callers.
  • PR: fix(sandbox): make mediated signal, scheduling, and limit calls behave natively #5212
Review details
  • Execution: complete
  • Candidate: PR #5212 (a4adf573eb5ea98191373d613c638f4d3c8c0243)
  • Baseline: 53cfadcdd22dfc3b2bb70c7db2de2d23eb245cac (53cfadcdd22dfc3b2bb70c7db2de2d23eb245cac)
  • Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

@mrunalp mrunalp added retired:area:sandbox Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. test:e2e Requires end-to-end coverage labels Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied for bdeb980. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@mrunalp

mrunalp commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

/ok to test bdeb980

@EmilienM
EmilienM force-pushed the fix/5185-sandbox-signal-sched/EmilienM branch from bdeb980 to 8e8a4f6 Compare November 9, 2026 21:33
@purp purp added area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. topic:security Vulnerability management, security analysis/auditing, bypass detection, and abuse prevention. and removed retired:area:sandbox Retired; do not apply. See #5151 for item-specific migration to the current taxonomy. labels Nov 15, 2026
The rt_sigqueueinfo emulation passed the workload's siginfo to
pidfd_send_signal unchanged, but the kernel rejects a siginfo whose
si_signo differs from the signal argument with EINVAL, while native
rt_sigqueueinfo overwrites si_signo with the requested signal. A
workload that leaves si_signo unset therefore failed only under
OpenShell.

Mirror the kernel by overwriting si_signo after the existing si_code
check, and cover it with a forked-child test that queues a signal with
a deliberately mismatched si_signo and asserts the delivered signo and
value.

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
@EmilienM
EmilienM force-pushed the fix/5185-sandbox-signal-sched/EmilienM branch from 8e8a4f6 to 97fbbf6 Compare November 16, 2026 17:00
@FrostGod

Copy link
Copy Markdown
Collaborator

/ok to test 97fbbf6

@FrostGod FrostGod left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough fix and tests. A few general notes; inline comments follow.

  • Logging: every Err from the new mediation is logged as "sandbox network notification denied" (network_broker.rs:328). That includes plain usage errors like ESRCH/EINVAL, and none of these are network events. Cross-group EPERM denials are policy decisions and would warrant an OCSF event per AGENTS.md. The signal path already works this way, so a follow-up issue is fine.
  • PID reuse (process_sched.rs:31-39): accepting this risk seems reasonable and is well documented. Please call it out in the PR description so reviewers sign off explicitly.
  • Merge order: agree on ordering with #5161 before either lands, since both change the signal mediation paths.

Comment thread e2e/rust/e2e-podman.sh Resolved
A self-directed kill or rt_sigqueueinfo was emulated by sending through
the retained pidfd while the caller was still blocked in its seccomp
notification, so the delivered signal interrupted the very syscall
being emulated: without SA_RESTART the call returned EINTR, and with
SA_RESTART it restarted and was notified again.

Continue the notification instead when the target group is the
notifying thread's own group. The kernel then executes the syscall
after the notification wait, so its result and siginfo are the
workload's own: an errno such as EAGAIN under an exhausted
RLIMIT_SIGPENDING stays visible, si_pid and si_uid name the workload
rather than the broker, and a handler runs exactly when it would
without mediation. The frozen-workload SIGCONT gate and the
queued-signal si_code check stay ahead of the continuation; other
targets keep the retained-pidfd delivery. Continuing an inspected
numeric target has the same PID-reuse window as the tkill precedent.

Cover it with forked-child tests: a self-directed kill, a kill that
names a sibling thread's TID, and a self-directed queued signal
complete natively with the handler observed, and a queued signal under
a zero RLIMIT_SIGPENDING reports the native EAGAIN instead of success.

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
…ss group

The child hardening filter denied sched_setaffinity, sched_setattr,
sched_setparam, sched_setscheduler, and prlimit64 for any nonzero PID
argument and setpriority outright, so thread pinning
(pthread_setaffinity_np and GOMP_CPU_AFFINITY pass a kernel TID),
taskset -p, psutil cpu_affinity(pid), prlimit(getpid()), and nice all
failed with EPERM inside sandboxes while working in a plain container.

A cBPF filter cannot compare an argument against the caller's own
thread group, and SECCOMP_RET_ERRNO outranks the user-notification
action, so the denies cannot be narrowed in place. Move the six
syscalls to broker mediation instead: the workload listener now
notifies them when they name a target, and the broker continues the
call when the target is in the caller's own thread group (own TGID or
a sibling TID resolved through /proc), preserves ESRCH for vanished
targets, and fails closed with EPERM for everything else, including a
caller it cannot identify. setpriority is limited to
PRIO_PROCESS, since PRIO_USER names every process of the shared
workload UID and PRIO_PGRP names a group an untrusted child can rejoin.
The static filter keeps an equal-deny on the trusted sandbox leader as
defense in depth, on argument 1 for setpriority whose PID is the
second argument.

A call with pid 0 names only the caller, which cBPF can check, so the
listener filter allows it without a notification (for setpriority only
under PRIO_PROCESS). Those calls keep their native, non-interruptible
behavior, and the prlimit64(0, ...) the pre-exec hardening hook issues
never waits on the broker. A child that does not inherit the workload
listener, the socket probe, takes prepare_without_broker, which keeps
the previous static rules. Shared /proc thread-group resolution moves
to linux/proc_status and the new mediation lives in
linux/process_sched.

Cover the filter and broker with allow, deny, ESRCH, and native pid 0
path tests, and add an e2e that runs inside a real sandbox: thread
affinity, self and sibling-TID signals, nice, a raw prlimit64, a
queued signal with a mismatched si_signo, and denied calls against a
same-UID child and the sandbox parent.

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
@EmilienM
EmilienM force-pushed the fix/5185-sandbox-signal-sched/EmilienM branch from 97fbbf6 to d4fe133 Compare November 18, 2026 13:53
@EmilienM

Copy link
Copy Markdown
Contributor Author

Pushed d4fe133 with everything below folded into the three commits. Inline replies are on each thread.

  • Heads-up: the head you reviewed didn't build for musl (PRIO_* is c_int there, c_uint on glibc), which is why Core E2E failed and no e2e lane ran. Fixed, and the podman lane passes locally. It needs a fresh /ok to test.
  • Logging: agreed, I'll open a follow-up. Usage errors shouldn't be logged as "network notification denied", and cross-group EPERM should be an OCSF event, on the signal path too.
  • PID reuse: it has its own section in the PR description now, asking for explicit sign-off.
  • Merge order: I'd land this one first. feat(sandbox): opt-in job-control signals to the caller's own process group #5161 conflicts with main today and needs a rebase either way, and the overlap is small (child_seccomp::prepare and mediate_process_signal). If you'd rather have feat(sandbox): opt-in job-control signals to the caller's own process group #5161 first, say so and I'll rebase on it.

@EmilienM

Copy link
Copy Markdown
Contributor Author

Follow-ups are filed: #5705 for the logging and OCSF events, #5706 for ioprio_set.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:runtime Sandbox execution, isolation, supervision, policy enforcement, networking, and provider lifecycle. test:e2e Requires end-to-end coverage topic:security Vulnerability management, security analysis/auditing, bypass detection, and abuse prevention.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: some calls fail or are interrupted inside the sandbox but work natively (rt_sigqueueinfo, self-directed kill, sched_setaffinity, prlimit64, nice)

4 participants