Skip to content

fix(api): time out every apiserver call; retry read-modify-write on conflicts - #82

Merged
jbouder merged 1 commit into
mainfrom
fix/k8s-timeouts-conflicts
Oct 19, 2026
Merged

jbouder merged 1 commit into
mainfrom
fix/k8s-timeouts-conflicts

Conversation

@jbouder

@jbouder jbouder commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Closes #47

What changes

  • Timeouts. Every KubernetesAppStore call passes _request_timeout=(5, 30) (connect, read), so a slow apiserver can't pin threadpool workers indefinitely. The readiness check keeps its own 3s.

  • Paged lists. List calls go through _list_custom / _list_typed, which pass limit=500 and follow continue tokens. A bare limit would silently truncate. pod_logs asks for limit=1, since it only ever used the first pod.

  • Field-selected events. A field selector can't match a name prefix, so instead of listing every event in the namespace and filtering in Python, app_events queries by exact name:

    • involvedObject.kind=App,name=<app>
    • app-<app>, which covers the Deployment, Service, ConfigMap and NebariApp
    • each current pod, plus the ReplicaSet named in its ownerReferences

    Events of pods or ReplicaSets that no longer exist are no longer returned.

  • Conflict retry. PATCH, stop and start share update_app. It re-reads the App, which also re-checks access, re-applies the change (including PATCH's subdomain check) and writes once more. A second conflict returns 409 "was changed by someone else; reload and try again" instead of a 500.

Upgrade notes

None.

Testing

  • Local tests:
    • test_every_call_has_a_timeout drives all 13 store methods against mocked clients and asserts _request_timeout on every call.
    • Paging follows continue.
    • Events are queried only by the expected field selectors and merged newest first.
    • A FakeStore that raises ConflictError N times shows PATCH, stop and start succeed after one conflict and return 409 after two.
    • Totals: 134 API tests and 25 MCP tests pass, plus ruff and mypy strict.
  • nebari-local, with the API image built from this branch:
    • A new app's events endpoint returned Deployment, NebariApp, Pod and ReplicaSet events (11). Logs, stop, start and delete all worked.
    • 16 concurrent PATCHes to one App: 8 × 200, 8 × 409, no 500s. The same burst against the image built from main gave 6 × 200, 10 × 500.

🤖 Generated with Claude Code

https://claude.ai/code/session_016pvMUQJoLkBDk6B94TTm45

…onflicts

- Every Kubernetes client call passes a (5s connect, 30s read)
  _request_timeout. List calls page with limit=500 and follow continue
  tokens rather than truncating.
- Events are fetched by field selector (the App, app-<name>, each
  current pod and its ReplicaSet) instead of listing the namespace.
- PATCH, stop and start re-read and re-apply once when the write
  conflicts on resourceVersion; a second conflict returns 409, not 500.

Closes #47

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pvMUQJoLkBDk6B94TTm45
@jbouder
jbouder merged commit 109356e into main Oct 19, 2026
12 checks passed
@jbouder
jbouder deleted the fix/k8s-timeouts-conflicts branch October 19, 2026 13:18
jbouder added a commit that referenced this pull request Oct 19, 2026
…, restart 409, safe owner label (#96)

- Models carry what the CRD allows: env[].valueFrom, runtime.keepAlive and
  any resource name (ephemeral-storage, nvidia.com/gpu, claims).
- PATCH merges `runtime` and `access` field by field: fields the request
  leaves out keep their stored value (null/empty removes one), so a UI or
  MCP edit no longer drops what kubectl or GitOps set. The UI form carries
  resource names it does not edit, sends null to clear, and shows a
  valueFrom variable read-only.
- POST and PATCH (and so launch_app) return 413 when inline files exceed
  MAX_UPLOAD_BYTES, the cap uploads already had.
- restart returns 409 for a stopped app (replicas 0), as documented.
- The owner label is written only when the username is a valid label
  value; spec.owner stays the owner, so `a@b.c` can create and edit apps.

Event matching by exact child names already landed in #82.

Closes #57


Claude-Session: https://claude.ai/code/session_016pvMUQJoLkBDk6B94TTm45

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add timeouts to every Kubernetes client call and retry PATCH/stop/start on resourceVersion conflicts

1 participant