feat: structured JSON logs with request ids; readiness that checks dependencies

[jbouder]

Closes #46

What changes

• JSON logs (api, mcp). logs.py sets up structlog, with stdlib integration so uvicorn's own lines are JSON too: one object per line with timestamp, level, logger, event and fields. The level comes from LOG_LEVEL.
• Request ids. RequestIdMiddleware (pure ASGI):
  • It keeps an incoming X-Request-ID if it is well formed (≤128 chars of A-Za-z0-9._:-), otherwise it makes one.
  • It binds the id to the log context, writes it back into the request headers and echoes it on the response.
  • It writes one access line per request: method, path, status, duration_ms, and on the API the authenticated user. Probe paths log at DEBUG.
• Propagation.
  • The UI's nginx forwards the incoming id on /api and /mcp, or generates one when none arrives.
  • MCP tools run in the session task, not the request's. So ToolLogMiddleware re-binds the id from the HTTP request, logs tool call with tool, outcome and duration, and ApiClient sends the id to apps-api.
• GET /api/v1/readyz:
  • It checks that the Kubernetes API answers list namespaces (limit=1, 3s timeout, already in the API's cluster role) and, with auth on, that the realm JWKS loads (3s; a success is reused for 60s so probes don't hammer Keycloak).
  • It returns 503 when a check fails. The body names the failing check but not the reason, since the endpoint is public; the reason is logged.
  • The chart's API readiness probe uses it (timeoutSeconds: 8). Liveness stays on the static /healthz, so an outage takes pods out of rotation without restarting them.
• MCP GET /healthz replaces the tcpSocket probes.
• Docs: request ids and /readyz in the API reference, plus a "Logs and probes" section and LOG_LEVEL in configuration.

Not done here: the operator's readyz is still healthz.Ping. The issue mentions it, but its acceptance criteria don't require it.

Upgrade notes

• Log output changes from uvicorn's text format to JSON, so update any log parsing.
• The API pod now goes unready while Keycloak's JWKS or the Kubernetes API is unreachable.

Testing

• Locally:
  • 11 new API tests: id generated, kept, or replaced when malformed; the access line carries the id; readyz ok, failing store → 503 without leaking the reason, JWKS down → 503 then up → 200; liveness unaffected.
  • 3 new MCP tests: /healthz over HTTP with the id echoed; ApiClient forwards the id or sends none.
  • Totals: 126 API tests and 25 MCP tests pass, plus ruff and mypy strict. nginx -t passes on the rendered config.
• nebari-local, with the api, mcp and ui images built from this branch:
  • Every API and MCP line is JSON, including uvicorn's startup lines.
  • Request ids, browser to API: Envoy assigns the id at the gateway. The id on the response matched the API's access line, which also named the user (authz-alice).
  • Request ids, MCP: a tool call logged tool call with the same id as its /mcp access line. ApiClient run inside the MCP pod sent exec-trace-1, and the API logged it.
  • /api/v1/readyz returned 200 with kubernetes and jwks ok. With OIDC_JWKS_URL pointed at an unresolvable host it returned 503 {"jwks":"failed"} and logged the DNS error. After the override was removed it went back to 200.
  • The probe definitions were checked in helm template. The cluster runs its vendored chart, so the new probes weren't exercised by kubelet.

🤖 Generated with Claude Code

https://claude.ai/code/session_016pvMUQJoLkBDk6B94TTm45

[github-actions]

📄 Docs preview for feat/logging-readiness:
https://feat-logging-readiness.apps-pack.pages.dev