Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/control-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The operator credential is an owner-only regular file. Every dispatch creates a

## Updates and operator inbox

Updates are immutable and ordered; read up to 200 after a cursor. Comments leave workflow state alone. A question atomically holds a task in `needs-human`: its own, or the asking task (the update's `task`) when a task's session asks on its plan. A task's session asks only on its task or plan. Acknowledgements and replies leave it unresolved. Resolution records the decision and atomically restores each held task's earlier state, unless another question holds it, its state changed or its execution ended. Unresolved questions block completion and merge checks; a plan's also block its tasks' state changes, planner handoff, child dispatch and automatic quota resume.
Updates are immutable and ordered; read up to 200 after a cursor. Comments leave workflow state alone. A question atomically holds a task in `needs-human`: its own, or the asking task (the update's `task`) when a task's session asks on its plan. A task's session asks only on its task or plan. Acknowledgements and replies leave it unresolved. Resolution records the decision and atomically restores each held task's earlier state, unless another question holds it, its state changed or its execution ended. A task's own questions, including one it asked on its plan, block its state changes, completion, merge and quota resume. A plan's question blocks only planner handoff and the first dispatch of its tasks.

Messages addressed to the operator appear in the cross-repository inbox. Informational messages leave the inbox on acknowledgement; questions leave on resolution. Idle, unrestricted sessions get wake-up pointers for others' messages on their item; tasks also for others' plan messages and resolutions, except coordinator notices and those to the operator. Messages persist when wake-up fails.

Expand All @@ -54,9 +54,9 @@ Messages addressed to the operator appear in the cross-repository inbox. Informa

The process forwards only read operations (`list`, `plan-view`, `attempts`, `get`, `updates`, `session-view`) and answers GET and HEAD for a loopback host name on any port, so an SSH tunnel works. It sends `Cache-Control: no-store`. When the control server fails, the page keeps its last data, marked stale. Descriptions render as CommonMark plus tables: raw HTML stays text, images show alt text, and only absolute http, https and mailto links act. Messages are text; the CSP allows only the page's scripts.

The per-plan view lists unfinished prerequisites, including other plans' tasks. A queued task's feedback is the plan's question. On both tabs, "Needs feedback" opens the question's thread at its update over the tab, on the plan's thread when it asked (`internal/ui/static/sessions.js`).
The per-plan view lists unfinished prerequisites, including other plans' tasks. A todo task waiting to start has the plan's question as feedback. On both tabs, "Needs feedback" opens the question's thread at its update over the tab, on the plan's thread when it asked (`internal/ui/static/sessions.js`).

Each task has one summary: done is done; failed and needs-human are blocked; an active attempt, or a running or in-review state, is in progress; an unfinished prerequisite or the plan's open question (as sessions report it) blocks it; todo under an approved plan is ready; anything else is blocked. Coordination does not change it.
Each task has one summary: done is done; failed and needs-human are blocked; an active attempt, or a running or in-review state, is in progress; an unfinished prerequisite or a todo task under the plan's open question (its read's `question`) is blocked; todo under an approved plan is ready; anything else is blocked. Coordination does not change it.

## Session view

Expand All @@ -65,7 +65,7 @@ Each task has one summary: done is done; failed and needs-human are blocked; an
- `role` is `planner` for a plan and `executor` for a task.
- `last_seen` is when a host last reported the window live; it survives `unknown` and `exited` observations.
- `observed` is the harness's `working`, `waiting` or `limited`; `live` for an open window with no harness state; `exited` once a reachable host shows the window gone; otherwise `unknown`. An exited attempt stays listed until engine ends it.
- `feedback` is the item's unresolved question, its own or its parent plan's. Waiting without one is not a feedback request.
- `feedback` is the item's unresolved question, including one its session asked on its plan. Waiting without one is not a feedback request.
- `latest` is the item's newest `say`, `ask`, `reply` or `resolved` update with a body, excluding coordinator notices. A task's preview never shows its plan's messages.

Observations live in an in-memory cache keyed by attempt, so a request never contacts a host. Coordinator passes store their host reads, and a background loop refreshes stale ones, one read in flight per host. The cache drops ended and replaced attempts and starts empty with the server. An unreachable host, unreadable run records or an old observation give `unknown`; the timings are in `internal/controlrun/observe.go`.
2 changes: 1 addition & 1 deletion docs/coordinator.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ A window gone from a host that answered ends its attempt `vanished` if the task

Each repository has one merge queue across its plans. `engine task merge-ready`, for the in-review PR at its reviewed, pushed head, queues the task; a later request keeps its place.

Only the queue's head is called: the first eligible entry in request order (in-review, coordinated, no unresolved question). It stays head until it merges or leaves. An entry that stops being eligible waits, keeping its place; waiting entries are never called. The head is called through its task host's `merge` op, since the server has no GitHub credential: at once when it becomes head or requests again, then once per `CheckEvery` (`internal/reconcile/queue.go`). A restart adds no call. A merge marks the task done with its PR. A head that is behind, conflicting or moved gets `merge-turn`, waking the executor to refresh and request again. A closed or branch-ruled head ends `merge-refused` and tells the operator; a red head ends `merge-refused` without a notice. A PR merged elsewhere is done, and the operator told.
Only the queue's head is called: the first eligible entry in request order (in-review, coordinated, no unresolved question of its own). It stays head until it merges or leaves. An entry that stops being eligible waits, keeping its place, and is not called. The head is called through its task host's `merge` op, since the server has no GitHub credential: at once when it becomes head or requests again, then once per `CheckEvery` (`internal/reconcile/queue.go`). A restart adds no call. A merge marks the task done with its PR. A head that is behind, conflicting or moved gets `merge-turn`, waking the executor to refresh and request again. A closed or branch-ruled head ends `merge-refused` and tells the operator; a red head ends `merge-refused` without a notice. A PR merged elsewhere is done, and the operator told.

**After the merge**, the `merged` update gives the session a cleanup turn, ending in `engine task cleaned`. Idle after that, it is released; `--left` tells the operator. Without that, it is released once idle `CleanupWait` after the merge, or at once if its window is gone, telling the operator where it ran. A cleanup turn still busy at `CleanupCap` is noticed (`internal/reconcile/session.go`). Other done tasks' sessions go once idle 10 minutes. Releasing a running background job is a notice.

Expand Down
18 changes: 11 additions & 7 deletions internal/cli/ui_acceptance_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,13 @@ func acceptPlans(t *testing.T, d *dashboard, w *world) {
var hardening ui.PlanDetail
d.get(t, "/api/plans/"+w.hardening.ID, &hardening)
for _, row := range hardening.Tasks {
// The plan's open question holds up both: the coordinator dispatches
// no task of a plan whose question is unresolved.
check(t, row.Feedback != nil && row.Feedback.Item == w.hardening.ID && row.Feedback.Update == w.hardeningQuestion, "%s must link its plan's question: %+v", row.ID, row.Feedback)
// The plan's open question holds up only the task waiting to start:
// the running one carries on.
if row.ID == w.keys.ID {
check(t, row.Feedback != nil && row.Feedback.Item == w.hardening.ID && row.Feedback.Update == w.hardeningQuestion, "%s must link its plan's question: %+v", row.ID, row.Feedback)
} else {
check(t, row.Feedback == nil, "%s is running and must not link its plan's question: %+v", row.ID, row.Feedback)
}
}
check(t, hardening.Counts == ui.Counts{InProgress: 1, Blocked: 1}, "a queued task under an open plan question is blocked, not ready: %+v", hardening.Tasks)
status, env := d.get(t, "/api/plans/p99999", nil)
Expand Down Expand Up @@ -162,8 +166,8 @@ func acceptSessions(t *testing.T, s *controlStack, d *dashboard, w *world) {
for _, r := range list.Sessions {
order = append(order, r.Item)
}
want := []string{w.links.ID, w.hardening.ID, w.rotate.ID, w.rollout.ID, w.sessionsUI.ID, w.plansView.ID}
check(t, status == 200 && list.Feedback == 3 && slices.Equal(order, want), "sessions %v (feedback %d), want %v", order, list.Feedback, want)
want := []string{w.links.ID, w.hardening.ID, w.rollout.ID, w.sessionsUI.ID, w.plansView.ID, w.rotate.ID}
check(t, status == 200 && list.Feedback == 2 && slices.Equal(order, want), "sessions %v (feedback %d), want %v", order, list.Feedback, want)
rows := sessionMap(list)
for id, want := range map[string]struct{ role, host, observed string }{
w.rollout.ID: {"planner", "devbox", "working"}, w.plansView.ID: {"executor", "devbox", "waiting"}, w.links.ID: {"executor", "devbox", "waiting"},
Expand All @@ -179,14 +183,14 @@ func acceptSessions(t *testing.T, s *controlStack, d *dashboard, w *world) {
check(t, links.State == "needs-human" && links.Feedback.Update == w.linksQuestion && links.Latest.Update == w.linksReply && links.Latest.Author == "operator",
"links keeps its question open after an acknowledgement and a reply, and previews the reply: %+v %+v", links.Feedback, links.Latest)
rotate := rows[w.rotate.ID]
check(t, rotate.Feedback != nil && rotate.Feedback.Item == w.hardening.ID && rotate.Latest == nil, "a task links its plan's question but previews only its own messages: %+v", rotate)
check(t, rotate.Feedback == nil && rotate.Latest == nil, "a running task is not held by its plan's question: %+v", rotate)
build := rows[w.sessionsUI.ID]
check(t, build.Latest.Update == w.lastProgress && build.Latest.Author == w.attempts[w.sessionsUI.ID].ID && build.Latest.Summary == fmt.Sprintf("Progress note %d", progressSays), "latest: %+v", build.Latest)

resolution := s.post(t, control.Request{Op: "resolve", ReplyTo: w.linksQuestion, Body: "Yes, link the same feedback"})
list = readSessions(t, d)
links = sessionMap(list)[w.links.ID]
check(t, list.Feedback == 2 && links.Feedback == nil && links.Latest.Update == resolution && links.Latest.Kind == "resolved",
check(t, list.Feedback == 1 && links.Feedback == nil && links.Latest.Update == resolution && links.Latest.Kind == "resolved",
"a recorded resolution clears the request: %+v", links)
check(t, links.State == "running" && links.Observed == "waiting", "the resolution restores the task's state, and host state stays: %+v", links)
check(t, list.Sessions[0].Item == w.hardening.ID, "the remaining requests lead: %+v", list.Sessions[0])
Expand Down
8 changes: 4 additions & 4 deletions internal/cli/ui_browser_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,8 @@ func sessionsAndOutage(t *testing.T, b *browser, s *controlStack, w *world, shot
b.waitFor("six sessions", "document.querySelectorAll('#sessions tbody tr').length === 6")
var order []string
b.value("[...document.querySelectorAll('#sessions tbody tr')].map((r) => r.id)", &order)
want := []string{"session-" + w.links.ID, "session-" + w.hardening.ID, "session-" + w.rotate.ID}
if strings.Join(order[:3], ",") != strings.Join(want, ",") {
want := []string{"session-" + w.links.ID, "session-" + w.hardening.ID}
if strings.Join(order[:2], ",") != strings.Join(want, ",") {
t.Errorf("feedback requests must lead: %v", order)
}
if got := text(b, "#tab-sessions"); got != "Sessions · 6" {
Expand All @@ -168,8 +168,8 @@ func sessionsAndOutage(t *testing.T, b *browser, s *controlStack, w *world, shot
if text(b, pv+" .observed") != "Waiting" || text(b, pv+" .feedback") != "" || !strings.Contains(text(b, pv+` td[data-label="Latest attached update"]`), "Waiting for the harness") {
t.Errorf("an ordinary wait must show as waiting, without feedback: %q", text(b, pv))
}
if got := text(b, "#session-"+w.rotate.ID+" .feedback"); got != "Needs feedback · plan "+w.hardening.ID {
t.Errorf("the parent plan's question: %q", got)
if got := text(b, "#session-"+w.rotate.ID+" .feedback"); got != "" {
t.Errorf("a running task is not held by its plan's question: %q", got)
}
b.tabTo(fmt.Sprintf(`a[data-key="s-fb-%s"]`, w.links.ID), 40)
b.press("Enter")
Expand Down
2 changes: 1 addition & 1 deletion internal/control/external.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ func (s Service) completeExternal(st *State, a Actor, r Request) (Response, erro
if active(st, item) || !slices.Contains([]string{"todo", "failed"}, item.State) {
return Response{}, problem(409, "external completion requires an inactive, unfinished task without a hard stop")
}
if PendingQuestion(st, item.ID) > 0 {
if OwnQuestion(st, item.ID) > 0 {
return Response{}, problem(409, "operator question is unresolved")
}
if !completionPR(r.Item.PR, st.Items[item.Repo].Slug) {
Expand Down
55 changes: 28 additions & 27 deletions internal/control/external_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,33 +104,23 @@ func TestExternalCompletionInactiveStatesAndQuestions(t *testing.T) {
}
})
}
for _, parentQuestion := range []bool{false, true} {
t.Run(fmt.Sprint("question-", parentQuestion), func(t *testing.T) {
s := service(t)
repo, plan := repoPlan(t, s)
task := newTask(t, s, repo.ID, plan.ID)
target := task.ID
if parentQuestion {
target = plan.ID
}
if !parentQuestion {
if err := s.Store.Transaction(func(st *State) error {
item := st.Items[task.ID]
item.State = "running"
st.Items[item.ID] = item
return nil
}); err != nil {
t.Fatal(err)
}
}
call(t, s, op, Request{Op: "ask", ID: target, Body: "Need a decision"})
if !parentQuestion {
task.Version++
task = *call(t, s, op, Request{Op: "state", ID: task.ID, Version: task.Version, State: "failed"}).Item
}
reject(t, s, op, Request{Op: "complete-external", ID: task.ID, Version: task.Version, Item: Item{PR: externalPR(repo.Slug)}}, 409)
})
}
t.Run("question", func(t *testing.T) {
s := service(t)
repo, plan := repoPlan(t, s)
task := newTask(t, s, repo.ID, plan.ID)
if err := s.Store.Transaction(func(st *State) error {
item := st.Items[task.ID]
item.State = "running"
st.Items[item.ID] = item
return nil
}); err != nil {
t.Fatal(err)
}
call(t, s, op, Request{Op: "ask", ID: task.ID, Body: "Need a decision"})
task.Version++
task = *call(t, s, op, Request{Op: "state", ID: task.ID, Version: task.Version, State: "failed"}).Item
reject(t, s, op, Request{Op: "complete-external", ID: task.ID, Version: task.Version, Item: Item{PR: externalPR(repo.Slug)}}, 409)
})
s := service(t)
repo, plan := repoPlan(t, s)
for _, item := range []Item{repo, plan} {
Expand All @@ -139,6 +129,17 @@ func TestExternalCompletionInactiveStatesAndQuestions(t *testing.T) {
reject(t, s, op, Request{Op: "complete-external", ID: "t999", Version: 1}, 404)
}

func TestExternalCompletionIgnoresItsPlansQuestion(t *testing.T) {
s := service(t)
repo, plan := repoPlan(t, s)
task := newTask(t, s, repo.ID, plan.ID)
call(t, s, op, Request{Op: "ask", ID: plan.ID, Body: "Need a decision"})
done := call(t, s, op, Request{Op: "complete-external", ID: task.ID, Version: task.Version, Item: Item{PR: externalPR(repo.Slug)}}).Item
if done.State != "done" {
t.Fatalf("task is %s, want done", done.State)
}
}

func TestCompletionPR(t *testing.T) {
good := externalPR("owner/demo")
for _, raw := range []string{"", "https://%", "http" + good[5:], good + "/", good + "?q=x", good + "#fragment", good + "/files", externalPR("other/repo"), fmt.Sprintf("https://github.com/owner/demo/pull/%d", 0), fmt.Sprintf("https://github.com/owner/demo/pull/+%d", 42), fmt.Sprintf("https://user@github.com/owner/demo/pull/%d", 42), fmt.Sprintf("https://github.com:443/owner/demo/pull/%d", 42)} {
Expand Down
2 changes: 1 addition & 1 deletion internal/control/merge.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ func (s Service) mergeRequest(st *State, a Actor, r Request) (Response, error) {
if item.PR != r.Merge.PR {
return Response{}, problem(409, "the merge request's PR is not the task's; record it with the in-review state")
}
if PendingQuestion(st, item.ID) > 0 {
if OwnQuestion(st, item.ID) > 0 {
return Response{}, problem(409, "operator question is unresolved")
}
m := *r.Merge
Expand Down
24 changes: 20 additions & 4 deletions internal/control/messages.go
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ func (s Service) release(st *State, a Actor, question Update) []Update {
var out []Update
for _, id := range slices.Compact(ids) {
item, ok := st.Items[id]
if !ok || item.Kind != "task" || item.State != "needs-human" || !active(st, item) || PendingQuestion(st, id) > 0 {
if !ok || item.Kind != "task" || item.State != "needs-human" || !active(st, item) || OwnQuestion(st, id) > 0 {
continue
}
prior := heldFrom(st, id)
Expand Down Expand Up @@ -247,11 +247,27 @@ func (s Service) Dispatch(item Item, host, harness, key string) (Response, error
return s.Execute(Actor{Name: "coordinator", Role: "coordinator"}, Request{Op: "claim", ID: item.ID, Version: item.Version, Body: host, State: harness, Key: key})
}

// PendingQuestion includes a task's own questions and its parent plan's questions.
func PendingQuestion(st *State, id string) int {
// OwnQuestion is the first open question that holds item id: one asked on
// it, or asked by its session on its plan. It is 0 when none is open. A
// plan's question holds the task that asked it, not the plan's other tasks.
func OwnQuestion(st *State, id string) int {
for _, u := range st.Updates {
if u.RequiresReply && (u.Item == id || u.Task == id) && !resolved(st, u.ID) {
return u.ID
}
}
return 0
}

// PlanQuestion is an open question on the plan of task id, or 0. It blocks
// only the first dispatch of a task in that plan.
func PlanQuestion(st *State, id string) int {
parent := st.Items[id].Plan
if parent == "" {
return 0
}
for _, u := range st.Updates {
if u.RequiresReply && (u.Item == id || parent != "" && u.Item == parent) && !resolved(st, u.ID) {
if u.RequiresReply && u.Item == parent && !resolved(st, u.ID) {
return u.ID
}
}
Expand Down
Loading