Skip to content

aarch64 strict gate: fcntl contention oracle reds with FAIL:hold_safety_release because the driver's pin starts after the holder is spawned #836

Description

@ryanbreen

Observed on fix/819-fcntl-oracle-arming-rendezvous at the branch's shipping
kernel bytes (7ed348f plus doc comments only), 1 of 5 boots of
./docker/qemu/run-aarch64-boot-test-strict.sh:

[FCNTL_PM_CONTENTION_ORACLE:aarch64:arm_wait_us=17:armed=0:acquired=1:holder_cpu=1:pm_busy_probe=0:calls=0:eagain=0:first_errno=18446744073709551615:first_wait_us=0:hold_safety=1:hold_done=1:joined=1:FAIL:hold_safety_release]
[TEST:syscall:fcntl_pm_contention_oracle:FAIL:fcntl contention oracle's peer released the process-manager lock on its safety deadline; the arming rendezvous did not complete]

This is the residual of #828 after that issue's rendezvous repair: a different
arm (hold_safety_release, not arming_timeout), correctly named by the repair's
own per-arm verdict text, with a mechanism the marker's own fields decode.

Mechanism, from the marker's fields

hold_safety=1 means the holder waited the full FCNTL_PM_HOLD_SAFETY_US
(250 ms) from its acquire without seeing FCNTL_PM_RELEASE_REQ. armed=0 with
arm_wait_us=17 means the driver's rendezvous loop exited after 17 us on
FCNTL_PM_HOLD_DONE, not on FCNTL_PM_HOLD_ACTIVE: by the time the driver
reached the loop, the holder had already acquired, held out its 250 ms and
finished. The driver's first RELEASE_REQ store on this path happens after that
loop. So at least 250 ms elapsed between the holder acquiring and the driver
reaching arm_start.

The driver spawns the holder and only then takes its pin:

match kthread_run_on_cpu_for_test(fcntl_pm_holder_body, "fcntl-pm-holder", peer) {
    Ok(handle) => {
        crate::per_cpu::preempt_disable();          // <-- pin starts here
        let arm_start = crate::tracing::trace_timestamp();

kernel/src/test_framework/registry.rs, run_fcntl_pm_contention_oracle(). The
holder can start holding as soon as it is spawned, and the driver is preemptible
from the spawn until preempt_disable(). The pin closes the window from that
call onward and leaves the window before it open.

Corroboration in the same serial

process_list_populated starts and passes between the oracle's START and its
marker:

[TEST:syscall:fcntl_pm_contention_oracle:START]
...
[TEST:process:process_list_populated:START]
[TEST:process:process_list_populated:PASS]
[FCNTL_PM_CONTENTION_ORACLE:...:FAIL:hold_safety_release]

That test takes the blocking crate::process::manager(), which on aarch64 masks
DAIF before it spins for the lock. It can only have passed after the holder
released, i.e. it was masked and spinning for the hold's duration -- and while it
was, whichever CPU it was on could not dispatch. That is the same starvation path
the round doc's root cause (a)(2) names, arriving in the interval the pin does
not cover.

Rate and host conditions

25 strict boots at these bytes on the shared Mac: 24 PASS oracle lines, this 1
FAIL:hold_safety_release. The same 25 boots produced 4 Exec smoke did not complete reds, the #826 signature, all 4 with a PASS oracle line in the same
serial. Host load average was 5.39 / 7.45 / 6.84 immediately after the 20-boot
run, with other lanes' QEMUs on the machine; #826 and #828 are both load-sensitive,
and this signature is expected to be as well, since it is a question of how long
the driver waits for a dispatch.

Candidate repair, not attempted here

Take the pin before the spawn so the holder cannot start holding while the driver
is preemptible. That moves preempt_disable() across Thread::new_kernel() and
spawn_on_cpu_for_test() -- kernel-stack allocation and scheduler publication --
which is a path with its own history (#653, P4 kstack custody), so it wants its
own round and its own soak rather than a tail-end patch. A ~1-in-25 liveness
signature is not acceptable on a 5- or 20-boot run either way.

Serial: docs/planning/green-program/syscalls/serials/819-oracle-arming/11-branch-strict-hold-safety-release.txt

Related: #828 (the arming failure this repair addresses), #826 (the
exec-smoke reds in the same battery), #819.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions