Skip to content

FR-VAL-03: driver observability parity — truthful /status, visible worker panes, periodic watchdog rows, driver heartbeat #291

Description

@linhdmn

Part of the §23 Driver Validation family (PRD). Closes the two live defects found 2026-09-10 and adds the missing liveness surface. Prior art: OpenTelemetry's GenAI semantic conventions standardize agent telemetry (gen_ai.* attributes, provider.name discriminator) — the lesson is that telemetry must be structured, uniform across spawn paths, and always-on, not best-effort.

(a) Truthful runs.active — closes #287. internal/daemon/endpoints.go:290 countActiveRuns counts ~/.devagent/locks/*.lock, but the task path never acquires one (pipeline.TryAcquireRun has no non-test caller in the task flow). Wire the run lock into the task entrypoint (the primitive + its test already exist: TestTaskTryAcquireRun) or derive active from the run registry. Verified live: /status said active:0 while a worker was executing.

(b) Visible worker panes — closes #288. internal/workers/herdrruntime.go:26 declares var HerdrPaneRunner with a TODO and no production assignment — every worker CLI launch takes the direct-spawn fallback regardless of spawn.visibility: "visible". Wire it to the internal/herdr pane runtime at init (keep the nil seam as test hook). Live evidence: omp worker was a direct child of devagent task while herdr agent list returned [].

(c) Periodic watchdog-health on ALL spawn paths. Watchdog rows are currently teardown-only on the direct path: a worker wedged at first-token burned 20 minutes (600s budget = 3× over) with no kill and no row until teardown (watchdogFired:false, recorded only after a manual kill). Emit watchdog-health periodically (e.g. every 30s) on every spawn path and enforce the no-progress kill there, matching the herdr-pane path.

(d) Loopdriver heartbeat on /status. The driver writes its phase to .selfbuild/logs/loop-N.log only; the daemon never sees it. Write .selfbuild/heartbeat.json ({iteration, phase, pid, updatedAt}) at every phase boundary and include it in GET /status (e.g. loop:{...}), so the TUI header can show 'iteration N · phase X' from the daemon instead of scraping the ledger.

Acceptance. Live protocol (the one used 2026-09-10): dispatch a task via the loop → /status shows active>=1; herdr agent list shows the worker pane; a deliberately hung fake worker (emits nothing) is killed by the watchdog within the budget and its periodic rows appear; TUI header shows the loop phase from /status.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:P2Backlog — later iterationsselfbuildSelf-build loop work item (issue-first selection)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions