MR-level release gating: four specialised agents inspect the diff in parallel; an arbitration layer collapses their signals into a single HOLD / REVIEW / PROCEED recommendation that lands at the top of the MR comment.
Traditional CI runs the full test suite plus rule checks and reports a single red/green bit. Real release risk isn't that flat — a single diff simultaneously touches "which tests need to run", "how risky is the rollout", "who knows this code", and "any code-level anti-patterns?". Squashing those four orthogonal views into one pass/fail throws away the signal that reviewers actually need.
ReleaseGuard splits these views into four specialised agents that run in parallel and emit structured signals, then a Decision Arbitration Layer collapses them into one actionable recommendation. The reviewer opens the MR and sees the verdict (HOLD / REVIEW / PROCEED) plus the specific triggered_signals that drove it.
Three MR comments produced by docker compose up against the local harness — the same analyzer code path, three different diffs, three different recommendations:
 |
 |
 |
| ✅ PROCEED docs-only change |
🟡 REVIEW handler + config drift (4 medium zones) |
🔴 HOLD critical drift in secrets config |
Side-by-side breakdown plus the Topology 0 integration validation (real LCOV + reverse BFS over 17K symbols) live in docs/case_study.md.
flowchart LR
A[GitLab MR] --> B[Analyzer]
B --> C{4 agents in parallel}
C --> D1[Selective Test]
C --> D2[Rollout Risk]
C --> D3[Ownership]
C --> D4[AI Reviewer]
D1 --> E[Arbitration]
D2 --> E
D3 --> E
D4 --> E
E --> F[HOLD / REVIEW / PROCEED]
F --> G[MR comment]
Full pipeline, T0/T1 topology comparison, and the arbitration decision matrix (three mermaid diagrams) are in docs/architecture.md.
- Many insights → one decision. The arbitration layer reconciles "four views speak at once" with "the reviewer wants one clear call". The
triggered_signalsarray keeps every contributing signal traceable. - Ownership stays neutral. No reviewer scores, no auto-written approval rules, and the suggestion order is shuffled (a lint test enforces
score/kindare not present in the JSON schema). The system surfaces "who has touched this area" without making personnel judgements. - L1 / L2 / L3 confidence ladder. Selective Test degrades gracefully: path heuristics (confidence 0.5) →
coverage_mapintersect (0.75) → callgraph reverse BFS with a dynamic-ratio confidence penalty (0.9). Real coverage is generated bycmd/cover2lcov, which runsgo test -coverprofileperTest*across the whole repo and converts the output to LCOV before feeding the indexer — not a hand-crafted seed. - Topology 1 invites the zero-infra adopter. The full topology needs Postgres + a nightly indexer; Topology 1 runs the analyzer container alone (degraded but usable), so an infrastructure prerequisite never blocks the first taste.
Plans A / B / C are complete (see CHANGELOG.md). Plan D (RAG) has a written spec only and is not implemented. The RAG-related pieces of the AI Reviewer remain design-only:
| Capability | Status |
|---|---|
| AI Reviewer Channel A (prompt stacking) | ✅ implemented |
| AI Reviewer Channel B (RAG: vector + BM25 retrieval) | ❌ spec-only |
| AI Reviewer self-reflection | ❌ config flag only (RG_REVIEWER_SELF_REFLECTION), no logic |
| Plan D RAG pipeline (embedding backfill + retrieval) | ❌ spec-only |
See docs/spec/task_ai_reviewer.md and docs/spec/superpower/plan_d_rag.md for the deferred designs.
make build
./bin/analyzerPrebuilt images: docker pull ghcr.io/twjohnwu/releaseguard-analyzer:0.1.0 (and releaseguard-indexer), published on every v* tag.
cd deploy/compose
mkdir -p artifacts
docker compose up --build --abort-on-container-exit
for f in artifacts/note-proj*.md; do echo "=== $f ==="; cat "$f"; echo; doneThree analyzer instances run in parallel against a mock GitLab and post the three arbitration outcomes (PROCEED / REVIEW / HOLD) as separate MR comments. See deploy/compose/README.md for details.
include:
- project: 'platform/releaseguard'
ref: main
file: 'deploy/ci/releaseguard.yaml'
releaseguard-review:
extends: .releaseguard-full
variables:
RG_SERVICE_NAME: my-service
RG_SERVICE_TYPE: backendSet AGENT_TIMEOUT_SEC to cap each agent independently; it defaults to ANALYZE_TIMEOUT_SEC and must not exceed it. Agents are cancelled cooperatively through context, so a blocking syscall inside an agent (for example a stalled network mount under PROJECTS_DIR) is not interrupted.
The AI Reviewer is the only component that calls a paid LLM API; the other three agents are pure Go. Per MR, the reviewer sends roughly:
| Component | Tokens |
|---|---|
System prompt (stacked .md context) |
~3k–5k input |
| MR diff + summary | ~2k–10k input |
| Structured findings (tool-use JSON) | ~0.5k–2k output |
At Sonnet-class pricing (claude-sonnet-4-6: $3 / 1M input, $15 / 1M output) that is roughly $0.05–0.30 per MR. Larger diffs or a pricier model raise it proportionally.
The model is configurable via RG_AI_MODEL (default claude-sonnet-4-6). Set it to any current Claude model id (e.g. claude-opus-4-8, claude-haiku-4-5) to trade cost for capability. Disable the reviewer entirely with RG_AGENT_AI_REVIEWER_ENABLED=false for a zero-API run.
HOLD and REVIEW gates can misfire. To measure that, every HOLD/REVIEW MR comment ends with a one-line invitation: if a reviewer believes the gate was wrong, they add the label releaseguard:false-positive to the MR.
The feedback subcommand then scans recent merged MRs, matches ReleaseGuard's own decision (parsed from its MR comment) against that label, and prints an MVP precision report:
# scans CI_PROJECT_ID by default; --project overrides
./bin/analyzer feedback --project 42 --since 2026-01-01T00:00:00Z
# machine-readable:
./bin/analyzer feedback --project 42 --jsonReviewers may also add releaseguard:confirmed to a HOLD/REVIEW MR to record that the verdict was right. Output tallies HOLD count, confirmed/false-positive-labeled HOLDs, and the precision metrics below. This is a Postgres-free MVP — the collected data is what the selective-test confidence constants (internal/agents/testselect/confidence.go) await for calibration.
analyzer feedback emits four metrics, not one:
confirmed_hold_precision_pct— precision computed only over MRs labelledreleaseguard:confirmedorreleaseguard:false-positive; every other HOLD is excluded from the denominator.weak_signal_hold_precision_pct— the old formula: every unlabeled HOLD is counted as correct. This is what earlier versions of this report reported asprecision_pct.confirmed_label_coverage_pct— the share of HOLDs that carry an explicit label at all.unlabeled_rate_pct— the share of HOLDs with no label.
An unlabeled HOLD is not evidence that the verdict was correct — it just means no one has looked yet. Always read confirmed precision together with coverage: a high confirmed precision on low coverage is selection bias (only the reviewer's favorite cases got labelled), not a reliable estimate. An MR carrying both releaseguard:confirmed and releaseguard:false-positive is counted as hold_conflict and excluded from every percentage. Percentages are null (not 0) when their denominator is zero.
analyzer replay --dataset <dir> runs the deterministic selective-test and rollout-risk agents (no AI or database) over recorded MR diffs stored as <dir>/<case>/{diff.json,expected.json}. It reports exact match, confirmed_hold_precision_pct, weak_signal_hold_precision_pct, confirmed_label_coverage_pct, unlabeled_rate_pct, and missed_risk_count.
go run ./cmd/analyzer replay --dataset testdata/replay --jsonThe seed dataset under testdata/replay/ is copied from the mock-gitlab fixtures and is not real MR data. Import real merged MRs with replay-import (below).
Case 04-t0demo needs Postgres for its intended L3 result; with deterministic agents only it yields PROCEED, so its expected.json records that observed baseline, not ground truth.
expected.json may carry an optional label object recording a human judgment, distinct from the recommendation string:
{
"recommendation": "HOLD",
"label": {
"human_outcome": "correct",
"evidence_source": "reviewer_comment",
"derivation": "explicit",
"confidence": "high",
"evidence_ref": "https://gitlab.example.com/g/p/-/merge_requests/42#note_1",
"reviewed_by": "jdoe",
"reviewed_at": "2026-09-01T00:00:00Z"
}
}Enum values (internal/report/label.go):
human_outcome:correct,overcautious,missed_risk,unlabeledevidence_source:reviewer_comment,release_decision,post_merge_change,deployment,incidentderivation:explicit,inferredconfidence:high,medium,low
derivation: explicit means a human stated the outcome directly and is the only kind of label that feeds confirmed_hold_precision_pct and confirmed_label_coverage_pct. derivation: inferred is ReleaseGuard's own reconstruction (for example, from the releaseguard:false-positive label) and only feeds weak_signal_hold_precision_pct — the same explicit-vs-inferred distinction as the feedback loop's limitation note above: an inferred or missing label is not proof the verdict was right. Cases with an empty recommendation are counted as unlabeled and skipped from every metric.
replay-import never writes derivation: explicit; imported labels are either inferred (a false-positive label on the source MR becomes human_outcome: overcautious) or left unlabeled. To hand-label a case, edit expected.json directly and set derivation: explicit plus reviewed_by/reviewed_at. Re-running replay-import preserves any label already marked explicit; pass --force to overwrite it anyway.
# GitLab: expected verdict is derived from ReleaseGuard's own MR comment
GITLAB_API_BASE=https://gitlab.example.com/api/v4 GITLAB_TOKEN=... \
go run ./cmd/analyzer replay-import --source gitlab --project 42 --since 2026-01-01T00:00:00Z --out .replay
# GitHub: PRs carry no ReleaseGuard comment, so every case is written unlabeled (empty recommendation)
GITHUB_TOKEN=... go run ./cmd/analyzer replay-import --source github --repo owner/name --since 2026-01-01T00:00:00Z --out .replay
# Stratified random sample instead of importing everything (recommend 30-50 cases for a first hand-labelled set):
go run ./cmd/analyzer replay-import --source gitlab --project 42 --sample 40 --seed 1 --out .replay
go run ./cmd/analyzer replay --dataset .replayHow expected.json is derived (GitLab): the newest ReleaseGuard recommendation: note gives the verdict; if the MR also carries the releaseguard:false-positive label and the verdict was HOLD or REVIEW, expected becomes PROCEED. MRs with no ReleaseGuard note are skipped unless --allow-unlabeled, which writes an empty recommendation and "human_outcome": "unlabeled"; replay excludes such cases from metrics and reports them as unlabeled until you fill in the recommendation by hand. Re-running replay-import into the same --out never overwrites an expected.json whose label has derivation: explicit; pass --force to overwrite anyway.
--sample N --seed S: instead of importing every merged MR, take a stratified random sample of N cases across the HOLD/REVIEW/PROCEED verdict strata (GitHub imports use a single stratum, since PRs carry no verdict). Each stratum contributes ceil(N / number of strata) cases; the selection is deterministic for a given seed, so re-running with the same --seed reproduces the same sample. The chosen sample size and seed are recorded under the sample key in .manifest.json. A first hand-labelled set of 30-50 cases is a reasonable starting point. With --allow-unlabeled, MRs without a ReleaseGuard note form a fourth stratum, so each stratum then gets ceil(N/4).
What is stripped and what is kept: no MR title, author, description, URL or note text is ever written. diff.json keeps file paths and patch bodies verbatim — they are the signal the agents read — so the dataset is identity-stripped, not anonymized code. Case directories are named by a hash; the only file that maps a hash back to a project/MR is <out>/.manifest.json. Both .replay/ and .manifest.json are gitignored; move cases into testdata/replay/ only if that code may be public.
docs/case_study.md— three MR-comment outcomes side by side, with analysis.docs/architecture.md— pipeline / topology / decision-matrix mermaid diagrams.docs/learnings.md— design reflections ("what I learned from making these decisions").docs/decisions_log.md— twenty design pivots in "initial idea → why wrong → current → lesson" form.docs/one_pager.md— single-page summary for sharing.docs/spec/— full spec (ten design docs plus four implementation plans undersuperpower/).deploy/compose/README.md— local-harness instructions.