Skip to content

feat: draft native Kubernetes execution provider - #1614

Draft
JAORMX wants to merge 35 commits into
plan/native-kubernetes-executionfrom
impl/native-kubernetes-execution
Draft

JAORMX wants to merge 35 commits into
plan/native-kubernetes-executionfrom
impl/native-kubernetes-execution

Conversation

@JAORMX

@JAORMX JAORMX commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Stage: production-completion repairs (draft, existing stack)

Plan: #1579 (plan/native-kubernetes-execution), docs/acceptance/native-kubernetes-execution.md, draft ADR 0350. Existing PRs only; no additional PRs and no merge. Draft development does not imply human contract approval or production readiness.

Current implementation head: 19097d3674dc1dc6c1b8d0ac82169705da9b9deb.

Implemented repairs

  • Distinct administratorFor operations identities with creator-scoped authorization on all admin paths; no implicit data-plane/owner-attestation privilege.
  • Continuing rate-limited PVC/Pod provisioning recovery, meaningful autonomous quota recovery tests, and bounded quota-accounting fixture readiness.
  • Retained Helm network policies, configuration, security high-water and capacity state; safe same-release/namespace adoption and quiesced upgrade checks.
  • Exact migration receipts including source-schema presence, receipt expiry on replacement, and fail-closed changed-input replay.
  • Multi-controller retained-deletion finalizer race repaired with UID/resourceVersion guards. Holder-loss qualification now initiates exact Pod termination independently of executor descendant cleanup.
  • Immutable rotation material staging and narrowly classified read-only convergence; deterministic tests reproduce replica lag and mixed-material poisoning without claiming either caused a historical timeout.
  • Real remote filesystem/tool conformance, grant authorization, physical confinement, CAS behavior, bounded UTF-8 output and honest truncation.
  • Host-project skill/command/Git exclusion while preserving operator-global instructions and, importantly, configured global/explicit-file Deny and Ask permissions under remote auto posture.
  • Existing live workflow extended with a manual-only, exact-SHA native qualification job; deterministic prerequisite, trusted runtime credential loader, bounded stages, recorded ownership, UID-safe cleanup and allowlisted artifacts. Docker/retained-image alias and Helm tag/digest handoff fixes included.
  • User guidance, lifecycle inventory, readiness tracker, and 32-AC source-backed proof map updated. Current main and fix(ci): repair live compaction slot and module metadata #1728 are integrated.

Verification completed

Normal latest-head PR workflows passed: 31 checks succeeded, 8 skipped (excluding the separate failed explicit manual qualification described below). Analysis/actionlint is green. Draft full-race/macOS lanes remain skipped.

Native production Kind+Calico job passed on the PR merge candidate. Production also passed in the exact-head manual job. The unfiltered production suite exercises all 11 scenarios, including scoped administration, rotation/revocation, replicas, migration, quotas, holder loss, Clear/Fork, retention/Helm lifecycle and bounded failure artifacts. General remote tool qualification passed too.

Full local lint/test (race and standalone modules), build, API, docs/site, offline demo and strict trace passed during the repair iterations; subsequent script-only fixes passed targeted race/fake-CLI/chart/actionlint checks. Native tracing resolves all 32 AC proof names; this is not a substitute for live evidence or human acceptance.

Independent Spec, Standards, Test Adequacy, Security, Kubernetes, Architecture, Reuse, UX and DevEx reviews and scoped follow-ups drove the repairs. No final human approval is claimed.

Remaining native-live blocker — not shippable yet

Manual run 35669238058 checked out exact head 19097d3674dc1dc6c1b8d0ac82169705da9b9deb. Production, mock preflight, credential staging and real-provider rollout succeeded. TestKindExecutionLiveQualification then timed out after 480 seconds at POST /v1/sessions, before its model prompt. No model Write, exact successful go test ./..., independent live artifact verification, or positive native usage was observed. No live summary was produced.

The underlying creation stall is unresolved. Source tracing identifies storage probing and remote Ensure/Attach readiness as possible waiting stages, but does not establish the cause. Context-window admission is a run-entry gate, not this create path. Next step is bounded, sanitized create-stage/placement diagnostics; do not increase the timeout or retry unchanged and call that a repair.

Cleanup succeeded: mock harness restored, run-owned Secret deleted with UID precondition, private credential file removed, and the exact owned cluster deleted. The allowlisted status artifact records production=success, credential=success, live=failure, cleanup=success. No credential bytes were read into model/tool context or artifacts.

Both existing PRs remain draft. Human API/schema/security/release decisions and required native live qualification remain outstanding. No production-ready or merge-ready claim.

JAORMX and others added 9 commits September 21, 2026 10:20
Implement a separately deployed provider/controller, confined executor, optional mecak8s mTLS client, charts, and mock kind qualification. Keep lifecycle, distributed ownership, and acceptance gaps explicit for draft review.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Hold the actual model stream until both cleanup guards are checked instead of racing a fast finite mock stream.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Replace the private REST protocol with typed protobuf RPCs over mTLS. Add a real-key kind coding smoke with runtime-only credential loading, UID-pinned Secret cleanup, mock restoration, and independent gRPC artifact and command verification.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Add run claims, transactional reference reconciliation, UID-fenced executor replacement and retirement, schema migration, durable grant revocation, reloadable TLS/keyring authority, deployment limits and release definitions. Qualification and independent review remain in progress.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Close ownership, schema pruning, rotation and termination-proof review findings. Add Calico-backed lifecycle, quota, rotation and failure scenarios, strict release provenance, and operational runbooks. Local production qualification remains blocked by host inotify capacity; CI proof is pending.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Fix CRD admission, termination and replay edges, bounded renewal, exact intent lookup and release provenance. Add real API-server legacy migration and replica/network proof corrections. Final local production qualification is blocked by kernel keyring quota, not reported as passed.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Restore forward-generation trust for new connections, reacquire expired test claims safely, use the post-release epoch, and surface holder start failures without weakening lifecycle checks.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Restore owner attestation in intent responses, verify presented intermediate chains against current trust, and tie readiness to authoritative state. Qualify phase-scoped claims and recreate endpoint connections after rollouts.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
@JAORMX
JAORMX force-pushed the impl/native-kubernetes-execution branch from 6aa84d1 to 7c9c523 Compare September 21, 2026 07:35
JAORMX and others added 2 commits September 21, 2026 12:04
Reconcile operation-holder expiry without waiting for another Kubernetes event and improve bounded qualification diagnostics.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
@JAORMX JAORMX added the e2e-live Run the live e2e suite on this PR label Sep 21, 2026
JAORMX and others added 6 commits September 21, 2026 19:43
Preserve the canonical amended draft contract from 622a57c; no plan approval or runtime change.

Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Integrate origin/main including the generic live compaction fix, reliable acceptance checker validation, and the renumbered draft ADR while preserving integrated runtime fixes.

Co-Authored-By: OpenAI <noreply@openai.com>
Keep the implementation-only tracker link aligned with the integrated plan; preserve draft status and runtime behavior.

Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
JAORMX and others added 9 commits September 21, 2026 23:39
…e proofs

Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
… skew

Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
…thority

Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Stage the live credential only after same-run production qualification, validate owned cleanup, and allowlist bounded evidence. Prove durable revocation after restart with a current signed claim and exact generation receipt.

Co-Authored-By: OpenAI <noreply@openai.com>
Verify image alias targets before reuse, bound qualification stages with cleanup headroom, and bind downstream ownership to production step outputs. Cover retained loads, conflicts, partial setup, and signals with offline fake commands.

Co-Authored-By: OpenAI <noreply@openai.com>
JAORMX and others added 5 commits September 22, 2026 01:32
Co-Authored-By: OpenAI <noreply@openai.com>
Keep admitted retained deletion driveable across peer reconciliation and deletion timestamps. Replace the holder-loss descendant timer with exact-owned kubelet termination, respect quota resync in the fixture, and collect bounded sanitized failure evidence before cleanup.

Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Co-Authored-By: OpenAI <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

e2e-live Run the live e2e suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant