Skip to content

Gateway's interceptor credential expires after a forward clock jump or suspend; interceptor-bound RPCs fail closed until the next refresh #5044

Open

Description

@thenashef

User Story

I use OpenShell v0.1.5 as the sandbox runtime of a self-hosted agent platform I am building (Fort): one gateway as a systemd service on Ubuntu 24.04, Docker driver, a supervisor middleware and gateway interceptors on Unix sockets. I hit this while testing that setup on two VMs.

Problem Statement

With an authenticated gateway interceptor (Unix socket, extension auth on), every interceptor-bound RPC fails closed for tens of minutes with PERMISSION_DENIED "gateway interceptor '' failed closed: interceptor transport error: … 'The request does not have valid authentication credentials', message: "extension bearer token has expired"". The interceptor never receives these calls: the gateway itself refuses to send a credential it considers expired (BearerTokenSlot::authorization_at, crates/openshell-extension-core/src/auth.rs:86-100).

Observed: the gateway mints its interceptor credential with a 3600 s lifetime and refreshes it 2880 s (4/5) later, measured on the monotonic clock, while expiry is checked against the wall clock. When the wall clock moves ahead of the monotonic clock by more than the remaining 720 s, the credential expires before the next refresh. We reproduced this with a forward clock step: right after a refresh, with NTP off, date -s "+45 min"; 14 minutes later every interceptor-bound RPC failed, and it kept failing for 32 minutes, until the refresh fired exactly 2880 s of monotonic time after the previous one. A guest suspend (ACPI S3) opens the same gap: after a 6 min suspend CLOCK_MONOTONIC was 353 s behind CLOCK_BOOTTIME (we did not run a suspend long enough to expire the credential). We first saw the failure without a deliberate step, twice in one day on a laptop-hosted VM (26 min until a gateway restart, and 21–43 min recovering on its own).

Impact / Why This Matters

During the outage CreateSandbox, DeleteSandbox, UpdateConfig and provider writes all fail; the gateway's own SnapshotProviderProfiles calls fail too, so StartSandbox fails ("Docker supervisor exited before becoming ready") and leaves the sandbox in Error until a Stop + Start after recovery. Running sandboxes, their model calls through credential placeholders and StopSandbox keep working. Policy tightening and credential rotation cannot reach running sandboxes until it clears. Today the only immediate fix is restarting the gateway, which restarts every sandbox.

Acceptance Criteria

  • A forward wall-clock step or a VM suspend/resume does not make interceptor-bound RPCs fail with "extension bearer token has expired".
  • If the credential is found expired at send time, the gateway obtains a fresh one instead of failing the call.

Reproduction Steps

  1. Run the gateway with an authenticated interceptor (Unix socket) and wait until the gateway has refreshed its interceptor credential once.
  2. timedatectl set-ntp false; date -s "+45 min".
  3. Every minute, call an interceptor-bound RPC (e.g. DeleteSandbox of a missing name with allowMissing: false).
  4. About 12 minutes after the last refresh, every call returns PERMISSION_DENIED "… extension bearer token has expired", until roughly 48 minutes after that refresh.

Environment

  • OpenShell: v0.1.5 (bdb956f), gateway release tarball, run as a systemd service (non-root openshell user)
  • OS: Ubuntu 24.04 (arm64 on a Lima VM, kernel 6.8; x86_64 on a GCP VM, kernel 7.0.0-1014-gcp), Docker 29.9.2
  • Deployment: single gateway, Docker compute driver, supervisor middleware on HTTPS loopback, two gateway interceptors on Unix sockets (extension auth on)

Logs

gateway interceptor 'arcology' failed closed: interceptor transport error: code: 'The request does not have valid authentication credentials', message: "extension bearer token has expired"
provider profile source 'interceptor/arcology' snapshot failed: ... SnapshotProviderProfiles failed for 'arcology': code: 'The request does not have valid authentication credentials'
start sandbox failed: Docker supervisor exited before becoming ready

Activity

  1. johntmyers commented on Oct 30, 2026

    @johntmyers
    Collaborator

    Why is step 2 part of your workload?

  2. thenashef commented on Oct 30, 2026

    @thenashef
    Author

    Changing the clock isn't something we do; it's just a fast way to reproduce the bug. In practice the wall clock jumps ahead of the monotonic clock when a VM is suspended and resumed (after a 6-minute suspend we measured CLOCK_MONOTONIC 353 s behind CLOCK_BOOTTIME) or when NTP steps a drifted clock. We first hit this with no manual step, twice in one day on a laptop-hosted VM. The date -s step reproduces the same wall-vs-monotonic gap on demand.

  3. yjjnls commented on Nov 18, 2026

    @yjjnls

    Corroborating the dual-clock-domain root cause here with an independent production incident from a different system — a gateway node in our LLM inference platform ran its wall clock ~8.8 s fast, and the blast radius looked exactly like an auth bug until we reconciled clocks.

    How we found it (the method may be the useful part). Cross-component timestamp reconciliation: the same request is logged by two components, and one component's log showed responses completing before the upstream component reported sending them. That impossible-ordering signature is what a constant clock offset produces, and it is searchable. A one-line comparison of the node's clock against an authoritative time source confirmed ~8.8 s of forward skew; it self-healed once NTP resynced. The operating rule we adopted afterwards: before debugging any expiry/auth logic across components, reconcile clocks first — and emit clock-sync state (or both wall and monotonic readings) in diagnostics, because a skew of this size is invisible in any single component's own logs while corrupting every cross-component comparison and every wall-clock-based validity judgment.

    Why the 4/5 refresh margin makes this issue's math exact. The refresh-at-2880 s-of-3600 s design leaves a tolerance window of exactly lifetime − refresh_interval (720 s here): that is the maximum forward wall-clock step the credential survives. The failure this issue reports is not an edge case of a bad environment — it is the deterministic consequence of checking validity against wall clock while scheduling renewal on a clock that NTP steps and VM suspends do not advance. Any positive step larger than the remaining margin expires the credential early, and the system then waits out the full monotonic interval regardless.

    Strong endorsement of acceptance criterion 2 (refresh at send time when found expired). This is the fix that converts the whole family into bounded latency instead of an outage. We built the same shape for route-level credential expiry in our gateway: operations that hit an expired credential are parked in a pending worker and a periodic sweep refreshes and replays them, rather than failing the request path for the remainder of the TTL window. With lazy refresh-at-use, the scheduled 4/5 refresh becomes an optimization rather than a correctness requirement, and the clock-domain mismatch stops being able to produce a tens-of-minutes fail-closed window at all.

    Suspend/resume TTL semantics. Your 353 s measurement (CLOCK_MONOTONIC behind CLOCK_BOOTTIME after a 6-minute S3 suspend) is exactly the suspend duration, which is the expected behavior — monotonic does not count suspended time. The general rule we apply: a credential that must remain valid across suspend/resume should have its lifetime accounted in a suspend-aware domain (wall or BOOTTIME), or expiry should be evaluated lazily at use time. A cheap hardening addition for the resume case specifically: on resume (or first use after an observed wall-vs-monotonic divergence), treat "time since last refresh" as unknown and force a refresh if wall-clock elapsed since the last refresh exceeds the lifetime.

    If a candidate fix lands, we can exercise it with a suspend/resume + NTP-step harness (forward steps sized just under and just over the remaining margin, suspend durations straddling the margin) and report behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions