Skip to content

feat(pr-pipeline): trust reconciler re-runs of lost review, refine and feedback-dispatch runs #8524

Closed

Description

@kyle-sexton

Refs: #8468

Problem

The lost-run reconciler (.github/workflows/maintenance-reconcile-lost-runs.yml,
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs) re-runs lost CI runs and, through
the reconciler-workflow input of check-trusted-trigger, lost pr-verify runs. Every other lane
run an outage loses is only listed in the run summary for a person, because a re-run that
github-actions[bot] starts is refused further down:

  • Review lanes (.github/workflows/pr-review-hosted.yml,
    .github/workflows/pr-review-security-hosted.yml) call the reusable workflows
    melodic-software/ci-workflows/.github/workflows/pr-review.yml and pr-review-security.yml
    (pinned v0.48.0). Their trigger gate lives in ci-workflows, which has no reconciler-workflow
    input to pass through.
  • Refine (.github/workflows/pr-refine.yml -> .github/workflows/pr-run-activity-write.yml)
    gets its lane App token from the lanes token broker through
    .github/actions/request-lane-token. The broker (Azure Function in melodic-software/azure-iac,
    ADR 0058, github-iac ADR 0015) refuses a mint for a run github-actions[bot] re-ran; it trusts
    a github-actions[bot] run only as a recorded pr-unblock-dispatch.yml dispatch
    (dispatcher_run_id).
  • Feedback dispatch (.github/workflows/pr-feedback-dispatch.yml) writes the
    pr-feedback-dispatched-* records that check-trusted-trigger's dispatcher check reads; that
    check accepts a dispatcher run only at attempt 1 or re-run by a listed human, so a
    github-actions[bot] re-run of it dispatches nothing the lanes trust.

Proposed change

Extend the pr-verify rule (reconciler record artifact reconcile-rerun-<run id>-2-<head sha>,
judged by the run GitHub attributes it to, never its contents) to:

  1. ci-workflows: add a reconciler-workflow input to pr-review.yml and pr-review-security.yml
    and pass it to their trigger check; bump the pin in both caller workflows here.
  2. Broker (melodic-software/azure-iac): accept a github-actions[bot] re-run at attempt 2 when
    the same reconciler record exists, with the same run, branch, event, actor, tag and ancestry
    checks as verifyReconcilerRerun in
    .github/actions/check-trusted-trigger/check-trusted-trigger.mjs; send the needed field from
    .github/actions/request-lane-token/action.yml.
  3. check-trusted-trigger dispatcher check: accept a dispatcher run at attempt 2 started by
    github-actions[bot] with a reconciler record, or keep feedback dispatch person-only and say
    so.
  4. Add each workflow to RECORDED_LANES in reconcile.mjs, and give each a "work already done"
    test like pr-verify's verdict-upload-step check, so a cancelled run that already acted (posted
    a review, pushed a refine commit, dispatched) is never re-run.

Acceptance

  • A lost review, refine or feedback-dispatch run is re-run once by the reconciler and its gate
    passes; a github-actions[bot] re-run without a record still stops.
  • A run that already reached its side effect is left to a person.
  • Broker and ci-workflows changes land and are pinned before this repo turns them on.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 20, 2026
  2. github-actions commented on Oct 20, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: structural
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    [quote truncated; the full decision is at the link]

    #8524 (comment)

    The decision settles the open questions. Review runs are recovered through the babysit merge gate. Refine and feedback-dispatch runs get one fresh dispatch on their trusted path. The ci-workflows, azure-iac and check-trusted-trigger code stay unchanged. The remaining work is in this repository. It leans on #9073's single re-run trigger definition and #10142's rerun_binds. I could not confirm from the open-issue list whether either is still open, so I listed no blocker.

    Automated triage, applied by the lanes App.

  3. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 20, 2026
  4. removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 25, 2026
  5. kyle-sexton commented on Oct 25, 2026

    @kyle-sexton
    MemberAuthor

    Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

    To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

  6. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: needs-decisionAwaiting a human or maintainer judgment call.
    on Oct 25, 2026
  7. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: the issue is at status: ready.

  8. added
    work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 25, 2026
  9. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Extend the pr-verify reconciler rule so lost review and refine runs are re-run once by the reconciler and pass their gates, with feedback dispatch likely kept person-only, landing ci-workflows, azure-iac and this repo as separate changes.

    Dependencies

    • melodic-software/ci-workflows: reconciler-workflow input on pr-review.yml and pr-review-security.yml must be released before the pin here is bumped
    • melodic-software/azure-iac: broker must accept a recorded github-actions[bot] attempt-2 re-run before refine is added to RECORDED_LANES

    Related work

    Open questions

    1. Should the check-trusted-trigger dispatcher check accept an attempt-2 feedback-dispatch run started by github-actions[bot] with a reconciler record, or keep feedback dispatch person-only?
      RECOMMENDED: Keep feedback dispatch person-only, say so in the check-trusted-trigger README, and leave it out of RECORDED_LANES. Basis: judgment: intake triage recommendation; the records it writes are what other lanes trust, so this is the largest trust widening
    2. Does the owner approve the broker trust change letting a github-actions[bot] attempt-2 re-run mint a lane token when a reconciler record exists?
      RECOMMENDED: Yes, with the same run, branch, event, actor, tag and ancestry checks as verifyReconcilerRerun. Basis: judgment: security call about token minting that lives in melodic-software/azure-iac, outside this repo
    3. Should this issue be split into one issue per repo (ci-workflows, azure-iac, this repo) since a lane run here can only change this repo?
      RECOMMENDED: Yes: a ci-workflows issue, an azure-iac issue, and a this-repo issue that waits on both and then edits RECORDED_LANES (.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94), the pins and the work-already-done tests. Basis: .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94 lists only pr-verify.yml

    Sources read

    • .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94: RECORDED_LANES contains only .github/workflows/pr-verify.yml
    • .github/workflows/pr-verify.yml:117: passes reconciler-workflow to check-trusted-trigger
    • .github/actions/check-trusted-trigger/README.md:141: documents the reconciler re-run rule at attempt 2

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  10. 1 remaining item

  11. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    No owner ledger (.work/*/interview-checklist.md, .work/gh-monitor-owner-decisions/) names this issue.

    Decided with no trust widening: no change to the ci-workflows trigger gate, the azure-iac broker, or check-trusted-trigger's dispatcher check. The facts show each kind of lost run can be recovered without one, so the body's items 1 to 3 are not built.

    Questions from intake's refinement (#8524 (comment)), plus the review-lane premise:

    1. Do the review lanes need a ci-workflows reconciler-workflow input, or a reconciler re-run?
      Decided: Neither. Both callers now use pr-review-cli.yml (.github/workflows/pr-review-hosted.yml:172, .github/workflows/pr-review-security-hosted.yml:177), so the body's v0.48.0 premise is out of date. The reconciler stays out of review lanes: its own record says they are "listed in the summary for a person: their gates or the lanes token broker refuse a re-run github-actions[bot] starts" (.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:43-45), and making github-actions[bot] a re-run actor for them would widen a trust gate. A lost review run is recovered through the one path that already re-runs AI review checks: the babysit merge gate, which re-runs under the owner's token on the pinned live head, attempt 1 only, at most once per check per head (plugins/source-control/skills/babysit-prs/reference/safety.md:847-858). The lost conclusions (cancelled, timed_out, startup_failure, the reconciler's LOST set at reconcile.mjs:89) join that gate's re-run triggers, beside rate-limit, publish rate-limit and pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073's install-failed, read from the one definition pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073 creates. This changes no actor, token or gate.
    2. Feedback dispatch: accept a github-actions[bot] attempt-2 dispatcher?
      Decided: No. It stays person-only: check-trusted-trigger accepts a dispatcher at attempt 1 or re-run by a listed human (.github/actions/check-trusted-trigger/check-trusted-trigger.mjs:732), unchanged. A lost pr-feedback-dispatch.yml run is recovered by the half-hourly sweep, a fresh attempt-1 run the gate already trusts (.github/scripts/pr-unblock/feedback-dispatch.mjs:52-62). The sweep looks for recovery only when its run listing shows a lost pr-feedback-dispatch.yml run, using the reconciler's definitions: a LOST conclusion with at least one job started (reconcile.mjs:89, :123-126), within its lookback window (LOOKBACK_HOURS = 24, .github/scripts/maintenance-reconcile-lost-runs/infra-failure.mjs:36), so a run replaced in its concurrency group (.github/workflows/pr-feedback-dispatch.yml:63-66) does not count and one failed tick does not lose the recovery. In the common case that is one call per tick, which keeps within the owner's rate-limit constraint ("within our environmental, github, rate limit, etc. constraints", owner-quotes.md, Issue Triage line). It then dispatches a head that dispatchFeedback would dispatch. dispatchFeedback already skips a head with an address-feedback check run or a queued or running address-feedback run (feedback-dispatch.mjs:29-34, :301-318), so a lost run that already dispatched is not dispatched twice. pr-pipeline: react to pull request comments without waiting for the next review run #8440 extends the same sweep for fix requests; both use one selection function.
    3. Broker: let a github-actions[bot] attempt-2 refine re-run mint a token?
      Decided: No. Refine is a workflow_dispatch from the dispatcher (.github/workflows/pr-refine.yml, input dispatcher-run-id), and fix-docs-dispatch.mjs runs in that dispatcher, so "the trigger gate and the broker check it as they check a resolve-conflicts dispatch" (.github/scripts/pr-unblock/fix-docs-dispatch.mjs:6-8). A lost refine run is recovered by one fresh dispatch on that path, which the broker already trusts. The rule is defined once, in pr-unblock: derive fix-ci-subscription's start and the restart cap from trusted values #8890's record: a fix-docs run lost to an outage with no check posted gets one more dispatch, and a completed run without a check otherwise makes the PR handled.
    4. Split per repository?
      Decided: Not needed. With no ci-workflows or azure-iac change, every edit is in this repository (question 0, reconcile.mjs:89-98).

    Converted from the body's "Acceptance" list, wording unchanged except: "re-run once by the reconciler" becomes a re-run by the babysit merge gate for review runs, and refine and feedback-dispatch runs get one fresh dispatch on their trusted path (questions 1 and 2), and the "without a record" clause applies to those two only, since a review re-run passes its unchanged gate (question 0). The third bullet ("Broker and ci-workflows changes land and are pinned ...") is dropped, since neither repository changes.

    Acceptance criteria:

    • A lost review run (cancelled, timed_out or startup_failure) is re-run once by the babysit merge gate under its existing head, attempt and job-binding rules, read from the single re-run trigger definition pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073 creates; a lost refine or feedback-dispatch run gets one fresh dispatch on its trusted path; a github-actions[bot] re-run of a refine or feedback-dispatch run still stops.
    • A run that already reached its side effect is left to a person: a review run whose lane posted its review or reviewed-head marker on that head, a refine run after which a lanes App commit landed on the PR, a feedback-dispatch run that already dispatched. A test covers each.
    • The babysit merge gate re-runs a review run on a lost conclusion and never on a failure outside its listed classes; a test covers a plain failure that is not re-run.
    • The reconciler still lists lost review runs for a person and re-runs none (reconcile.mjs review-lane exclusion unchanged).
    • The sweep's recovery path makes one run-listing call on a tick with no lost pr-feedback-dispatch.yml run in the lookback window, and a run with no started job does not count as lost. A test pins each.
    • RECORDED_LANES and check-trusted-trigger.mjs are unchanged, and nothing changes in melodic-software/ci-workflows or melodic-software/azure-iac.
    • .github/actions/check-trusted-trigger/README.md states that feedback dispatch stays person-only and that the sweep recovers its lost runs.

    Depends on #9073's single re-run trigger definition, and on #10142's rerun_binds so attempt 2 binds.

  12. melodic-automation-lanes commented on Oct 26, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: blocker #9073 is not closed by a listed human or a merge.

  13. melodic-automation-lanes commented on Oct 28, 2026

    @melodic-automation-lanes
    Contributor

    The lanes App's pull request #10512 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage, which decides what work is left and admits it again.

  14. removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 28, 2026
  15. github-actions commented on Oct 28, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (attempt 1 of 2).

  16. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 28, 2026
  17. melodic-automation-lanes commented on Oct 28, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped partway through this issue and needs a person.

    Why, in the run's own words:

    Nothing left to build: every acceptance criterion of the recorded decision (2026-10-26) is already implemented and tested on main at 2633f07, so this run made no change. A person should check the evidence below and close #8524 as done (or name a remaining gap).

    What the run says it did not do:

    • Did not run the node and pytest suites named above (this lane runs no interpreter); I only read them to confirm the tests exist. Main's CI runs them.
    • Shallow clone: git history could not show which PR landed this work, so the issue is not linked to it here.
    • No diff, so no review, build or lint step ran.

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40621391499

    To retry, remove needs-human and add agent-ready again.

  18. removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 28, 2026
  19. kyle-sexton commented on Oct 28, 2026

    @kyle-sexton
    MemberAuthor

    Closing as completed. Every criterion of the 2026-10-26 decision is on main: #10512 (merged 2026-10-27T21:15Z) and its dependencies #9073 (closed) and #10142 (merged). Feedback dispatch stays person-only, as .github/actions/check-trusted-trigger/README.md:91-94 on main states, and the half-hourly sweep in .github/scripts/pr-unblock/feedback-dispatch.mjs recovers a lost run. Lane run 40621391499 found nothing left to build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions