bug(kubernetes): reconciliation suspends creation before fence identity publication

[matthewgrossman]

User Story

The operator is using OpenShell's K3s conformance lane to validate sandbox creation and file operations, and requested investigation and a separate fix after an intermittent creation failure in draft PR #4072. Reliable sandbox creation is necessary to assess the actual conformance scenarios.

Problem Statement

The Kubernetes compute driver's periodic reconciler can inspect a fresh Sandbox CR in the preparing phase before the creator publishes the namespace fence UID and generation annotations. It interprets those absent annotations as a changed fence and attempts to suspend an otherwise valid creation.

The actual namespace fence policy has already been validated. The missing binding is expected at this stage: creation publishes it after observing the workload Pod and before releasing the gated generation. A deterministic fixture exercising the real reconciler confirms the incorrect suspension attempt. A live K3s investigation observed two such attempts; both were rejected by resource-version conflicts, allowing creation to succeed.

Impact / Why This Matters

The background reconciler can interfere with active provisioning. A successful suspension would request workload teardown while creation is still in progress. This is a demonstrated lifecycle race; the precise Kubernetes object behind the original CI create sandbox failed: sandbox not found response remains unidentified, so this issue does not claim that failure's root cause is confirmed.

Acceptance Criteria

• Fresh preparing generations with verified fence policies are not suspended solely because their binding annotations have not yet been published.
• Released generations with missing or mismatched fence bindings still fail closed.
• Altered fence policies still cause suspension during preparation.
• Abandoned preparation still expires under the existing bootstrap deadline.
• Relevant Kubernetes E2E passes with artifacts built from the fix's own commit.

Reproduction Steps

1. Deploy OpenShell on K3s and create sandboxes while the gateway's periodic reconciliation loop is active.
2. Inspect gateway logs for namespace workload fence generation changed; suspending stale boundary during active creation and Kubernetes audit metadata for the corresponding Sandbox PATCH.
3. For deterministic regression coverage, invoke the real driver reconciler with a fresh preparing Sandbox CR, the expected namespace fence policies, and no published fence identity annotations. The current implementation patches the Sandbox to Suspended.

Environment

• OpenShell candidate: 508035eefd57a1d13a6e126a34597db3ba735b2e, artifacts from Branch E2E run 36938749555.
• Live investigation: Ubuntu 24.04/systemd, K3s v1.36.3+k3s1, Agent Sandbox v0.5.0, dedicated Docker container with host-backed K3s storage.
• Existing-candidate baseline: 8/8 conformance scenarios and 20/20 additional sandbox creations passed. Two erroneous suspension attempts returned HTTP 409. These results demonstrate the race but do not validate the proposed fix or reproduce the CI failure.

Proposed Change

Defer the persisted fence identity comparison until the generation leaves preparing. Continue verifying the actual fence policy before that guard and retain bootstrap expiration afterward. No CLI, API, configuration, or deployment changes are needed.