Skip to content
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,12 @@ repository still gets a decision, never by following the link; no release carrie
- :memo: docs(openspec): amend REQ-EX-S07-03/04 to match the measured implementation
- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane follow-ups
- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S07 lane
- :memo: docs(walkthrough): pin the walkthrough to measured multi-format CLI output (EX-S09)
- :memo: docs(walkthrough): close review gaps in the EX-S09 pins and repo snapshot
- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane
- :memo: docs(walkthrough): fix Step 3 lede self-contradiction on .tf format count
- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 lane follow-ups
- :memo: docs(changelog): regenerate CHANGELOG.md for the EX-S09 F1 fix

### Features
- :sparkles: feat(docs): gate example pack and format claims against dogfood
Expand All @@ -140,6 +146,7 @@ repository still gets a decision, never by following the link; no release carrie
- :bug: specs(p5-ex): make S10 schema freeze and S07 fence non-vacuous
- :bug: fix(docs-gates): fail on unmapped format tokens and de-confound the tests-dir mutation (EX-S01 review)
- :bug: fix(examples): make privilege-tier's proving fixture load-bearing
- :bug: fix(docs): pin case+decision pairs, not just case names, in the walkthrough (F1)

### Refactoring
- :recycle: refactor(examples): discover dogfood packs instead of a hardcoded 3-name loop
Expand Down
129 changes: 117 additions & 12 deletions docs/usage/walkthrough.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,34 @@

## The repo

`topic-registry` — one YAML file per Kafka topic:
`topic-registry` — one YAML file per Kafka topic, map-at-root keyed by topic name.
Trimmed from a real shipped fixture
(`examples/packs/topic-registry/.assent/tests/topics/wildcard-grant/head/topics/prod/billing.invoices.v1.yaml`):

```yaml
# topics/prod/orders.yaml
name: orders
owner: team-orders
partitions: 12
retentionMs: 604800000
# topics/prod/billing.invoices.v1.yaml
billing.invoices.v1:
owner: billing-team
partitions: 6
replication_factor: 3
retention_hours: 168
schema:
format: avro
subject: billing.invoices.v1-value
compatibility: BACKWARD
acl:
grants:
billing-team: read
finance-team: read
```

Today every MR waits for a platform engineer. Goal: routine changes merge themselves.

`partitions`, `replication_factor` and `retention_hours` are scalar fields; `acl.grants`
and (on other topics) `consumers` are nested maps. The starter packs also govern nested
JSON objects, nested tfvars maps, and — opaquely, falling back to REVIEW — HCL `.tf`
blocks. Step 3 below is real `assent test` output across all four.

## Step 1 — scaffold a policy tree

> **Planned — `assent init` does not exist.** There is no scaffolding subcommand in the
Expand All @@ -35,10 +51,12 @@ assent lint .
```

That gives you `.assent/config.yaml` (environments prod/dev by path, class
`kafka-topic`), `.assent/bindings.yaml`, `.assent/packs/topics/` (ownership,
bounded-change, non-destructive, schema-valid) and `.assent/tests/topics/` with a passing
fixture for every rule — `assent lint` rejects a rule with no test case, so the two ship
together by construction.
`kafka-topic`), `.assent/bindings.yaml`, `.assent/packs/topics/rules/` — nine rule files
today (`ownership`, `bounded-change`, `non-destructive`, `schema-valid`,
`schema-compatibility`, `list-no-shrink`, `wildcard-grant`, `soft-delete`,
`referenced-resource-ownership`) — and `.assent/tests/topics/` with a passing fixture for
every rule — `assent lint` rejects a rule with no test case, so the two ship together by
construction.

Committed starter packs: [`examples/packs/`](https://github.com/PlatformRelay/assent/tree/main/examples/packs)
(`topic-registry`, `service-catalog`, `infra-vars`).
Expand Down Expand Up @@ -69,22 +87,109 @@ hard errors, not warnings.
$ assent test .
PASS topics/bounded-change (APPROVE)
PASS topics/bounded-change/negative (REVIEW)
PASS topics/list-no-shrink (APPROVE)
PASS topics/list-no-shrink/negative (REVIEW)
PASS topics/ownership (APPROVE)
PASS topics/ownership/negative (REVIEW)
PASS topics/quota-ceiling (APPROVE)
PASS topics/quota-ceiling/facts-omitted (REVIEW)
PASS topics/quota-ceiling/negative (REVIEW)
PASS topics/resource-ownership (APPROVE)
PASS topics/resource-ownership/negative (REVIEW)
PASS topics/schema-compatibility (APPROVE)
PASS topics/schema-compatibility/negative (REVIEW)
PASS topics/schema-valid (APPROVE)
PASS topics/schema-valid/negative (BLOCK)
PASS topics/soft-delete (APPROVE)
PASS topics/soft-delete/negative (REVIEW)
PASS topics/soft-delete/unrelated-modify (APPROVE)
PASS topics/wildcard-grant (APPROVE)
PASS topics/wildcard-grant/negative (BLOCK)
PASS topics/non-destructive (APPROVE)
PASS topics/non-destructive-delete (REVIEW)
```

That is real output from the shipped starter pack. Each case is a directory with
`base/`, `head/`, `facts.yaml` and `expect.yaml` (or an inline `cases.yaml` entry); a
That is real output from the shipped `topic-registry` starter pack — **YAML** with nested
`acl.grants` and keyed `consumers` maps, not a single flat field. Each case is a directory
with `base/`, `head/`, `facts.yaml` and `expect.yaml` (or an inline `cases.yaml` entry); a
failing case prints the expected and actual decision plus the findings that differ.
`--update` rewrites expectations from the produced actuals (refused when `CI` is set),
`--coverage` is the read-only both-polarity completeness gate. Exit `0` every case
matched; `1` a mismatch, write or load error; `2` usage, discovery, or the CI guard
refusing `--update`.

`topic-registry` is one of **four** governed formats — three of them structurally
diffed, with `.tf` governed only as an opaque whole (below):

```console
$ assent test examples/packs/service-catalog
PASS catalog/allowed-fields (APPROVE)
PASS catalog/allowed-fields/negative (REVIEW)
PASS catalog/context-fresh (APPROVE)
PASS catalog/context-fresh/negative (REVIEW)
PASS catalog/nested-fields (APPROVE)
PASS catalog/nested-fields/negative (REVIEW)
PASS catalog/non-destructive (APPROVE)
PASS catalog/non-destructive/negative (REVIEW)
PASS catalog/ownership (APPROVE)
PASS catalog/ownership/negative (REVIEW)
PASS catalog/privilege-tier (REVIEW)
PASS catalog/privilege-tier/negative (REVIEW)
PASS catalog/schema-valid (APPROVE)
PASS catalog/schema-valid/negative (BLOCK)
PASS catalog/unkeyed-list-opaque (REVIEW)
PASS catalog/file-non-destructive (APPROVE)
PASS catalog/file-non-destructive-delete (BLOCK)
```

`service-catalog` is **JSON** with nested objects (`catalog/nested-fields`) and a tier
allow-list (`catalog/privilege-tier`). `catalog/unkeyed-list-opaque` is a measured, not
wished-for, limit: an *unkeyed* list is opaque to the differ by design (D-061), so a
change inside one falls back to REVIEW rather than a false-precision partial diff.

```console
$ assent test examples/packs/infra-vars
PASS vars/bounded-change (APPROVE)
PASS vars/bounded-change/negative (REVIEW)
PASS vars/max-replicas-change (APPROVE)
PASS vars/max-replicas-change/negative (REVIEW)
PASS vars/min-replicas-change (APPROVE)
PASS vars/min-replicas-change/negative (REVIEW)
PASS vars/nested-map-change (APPROVE)
PASS vars/nested-map-change/negative (REVIEW)
PASS vars/ownership (APPROVE)
PASS vars/ownership/negative (REVIEW)
PASS vars/placement (APPROVE)
PASS vars/placement/negative (REVIEW)
PASS vars/tf-opaque (REVIEW)
PASS vars/companion-delete (REVIEW)
```

`infra-vars` is **tfvars** — keyed `workloads.*` maps, including the deeper
`vars/nested-map-change` case. `vars/tf-opaque` and `vars/companion-delete` are the two
honest edges of the current differ, both pinned as expected **REVIEW**, never a silent
APPROVE:

- **`.tf` is governed but not structurally diffed.** The differ only routes the
`.tfvars` extension to the HCL parser; a `.tf` file's content — blocks or bare literals
alike — is opaque and falls back to REVIEW, never a partial parse. Assent does **not**
understand Terraform expressions or resource blocks; it treats a whole changed `.tf`
file as one un-provable unit. This is a permanent v1 limitation, not a bug — see
`examples/README.md`.
- **A companion file outside the pack's class match** (e.g. a `NOTES.md` next to the
`*.tfvars` files) deleted alongside a real change is caught only by the class-agnostic
unmatched-whole-file-delete fail-safe — REVIEW, no obligation attached. v1 does not
correlate "delete A and append B" across files; that is out of engine scope today.

These packs also carry the **REF-EX C1–C8** governance patterns: keyed-map entry removal
(C1, `topics/list-no-shrink`), a tier allow-list (C2, `catalog/privilege-tier`),
wildcard-grant blocking (C3, `topics/wildcard-grant`), soft-delete-as-field-add (C4,
`topics/soft-delete`), a fact-derived quota ceiling (C5, `topics/quota-ceiling`), a
placement allow-list (C6, `vars/placement`), referenced-resource ownership (C7,
`topics/resource-ownership`), and the companion-file-delete REVIEW above (C8,
`vars/companion-delete`) — all runnable today from
[`examples/packs/`](https://github.com/PlatformRelay/assent/tree/main/examples/packs).

## Step 4 — backtest before trusting it

> **Planned — `assent scan` and `assent stats` do not exist.** There is no historical
Expand Down
130 changes: 130 additions & 0 deletions hack/docs/truthlag_pins_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,136 @@ else
fail "EX-S01: docs-gates does not run hack/docs/example_format_inventory_test.sh"
fi

# --- REQ-EX-S09-01 — every (case name, decision) pair the walkthrough shows is real -------
#
# The walkthrough's console blocks are supposed to be COPIED from a real `assent test`
# run, not invented. This builds the binary (same pattern as hack/dogfood-examples.sh /
# readme_smoke_test.sh) and runs it against every example pack, then diffs the
# `name (DECISION)` pairs against every `PASS <case> (<decision>)` line the walkthrough
# shows. NAME ALONE IS NOT ENOUGH: an earlier version of this pin extracted only the
# case name (awk '{print $2}'), so a doc that renamed `PASS vars/tf-opaque (REVIEW)` to
# `(APPROVE)` stayed green — the walkthrough's own prose claims tf-opaque and
# companion-delete are "pinned as expected REVIEW, never a silent APPROVE", and the pin
# did not actually check that. Comparing the full pair closes it, and generalizes past
# just those two cases: ANY decision flip on ANY shown case now reddens. The comparison
# is bidirectional (comm -23 AND comm -13): a real case silently missing from a console
# block that claims to be a complete pack run is drift too, not just an invented one.
WT=docs/usage/walkthrough.md
BIN="${ASSENT_BIN:-bin/assent}"
if [[ ! -x "$BIN" ]]; then
echo "== building $BIN for EX-S09 console pins =="
CGO_ENABLED=0 go build -o "$BIN" ./cmd/assent
fi

REAL_CASES="$(mktemp)"
WT_CASES="$(mktemp)"
trap 'rm -f "$MIRROR_EXPECTED" "$REAL_CASES" "$WT_CASES"' EXIT

for pack in topic-registry service-catalog infra-vars; do
"$BIN" test "examples/packs/$pack" 2>/dev/null
done | awk '{print $2, $3}' | LC_ALL=C sort -u > "$REAL_CASES"

grep -oE '^PASS [A-Za-z0-9/_-]+ \([A-Z]+\)' "$WT" | awk '{print $2, $3}' | LC_ALL=C sort -u > "$WT_CASES"

wt_case_count="$(wc -l < "$WT_CASES" | tr -d ' ')"
if [[ "$wt_case_count" -eq 0 ]]; then
fail "REQ-EX-S09-01: $WT has no 'PASS <case> (<decision>)' console lines to pin — vacuous"
else
missing_cases="$(comm -23 "$WT_CASES" "$REAL_CASES")"
extra_cases="$(comm -13 "$WT_CASES" "$REAL_CASES")"
if [[ -n "$missing_cases" ]]; then
fail "REQ-EX-S09-01: $WT shows case/decision pair(s) absent from real 'assent test' output (invented case, or a decision that doesn't match the real run):"
printf ' %s\n' "$missing_cases" >&2
elif [[ -n "$extra_cases" ]]; then
fail "REQ-EX-S09-01: real 'assent test' output has case(s) the walkthrough's console blocks omit — a block claiming to be a complete run is now incomplete:"
printf ' %s\n' "$extra_cases" >&2
else
pass "REQ-EX-S09-01: all $wt_case_count walkthrough console (case, decision) pairs are real, and no real case is missing"
fi
fi

# --- REQ-EX-S09-02 — the walkthrough names all four governed formats, honestly ------------
#
# Before EX-S09 the walkthrough's only console block was the topic-registry pack's
# original 8-line output; there was no evidence any other format was governed at all.
# Pin each pack's namespace AND the two hardest cases (the `.tf` opaque fallback and the
# C8 companion-file-delete REVIEW) by name, plus the four format words, so a revert to
# the old one-format sketch reddens on more than one axis.
declare -A ex_s09_prefix_hint=(
[topics/]="YAML (topic-registry)"
[catalog/]="JSON (service-catalog)"
[vars/]="tfvars (infra-vars)"
)
for prefix in "${!ex_s09_prefix_hint[@]}"; do
if grep -q "PASS ${prefix}" "$WT"; then
pass "REQ-EX-S09-02: $WT shows a real ${ex_s09_prefix_hint[$prefix]} case"
else
fail "REQ-EX-S09-02: $WT has no '${prefix}' case — missing ${ex_s09_prefix_hint[$prefix]} coverage"
fi
done

for must_case in "vars/tf-opaque" "vars/companion-delete"; do
if grep -q "PASS $must_case" "$WT"; then
pass "REQ-EX-S09-02/03: $WT shows the $must_case case"
else
fail "REQ-EX-S09-02/03: $WT is missing the $must_case case (the .tf opaque / C8 REVIEW demonstrator)"
fi
done

for word in "YAML" "JSON" "tfvars" "HCL"; do
if grep -q "$word" "$WT"; then
pass "REQ-EX-S09-02: $WT names format $word"
else
fail "REQ-EX-S09-02: $WT never names format $word"
fi
done

# --- REQ-EX-S09-03 — HCL/.tf claims are honest: REVIEW/known-limitation, never "understands" -
for f in "$WT" examples/README.md; do
if grep -qiE 'assent (fully |now )?(understands|parses|supports) terraform|full terraform support|evaluates terraform expressions' "$f"; then
fail "REQ-EX-S09-03: $f overclaims Terraform/.tf support"
else
pass "REQ-EX-S09-03: no Terraform-support overclaim in $f"
fi
done

if grep -qE '\.tf\b' "$WT"; then
# A bare "REVIEW|opaque" grep over the whole page is vacuous — those words already
# appear ~30 times in the console blocks above (every REVIEW-decision case line).
# Pin the STABLE SUBSTRING of the actual honest sentence instead, so softening or
# deleting that specific claim (not just any REVIEW/opaque word anywhere) reddens.
if grep -q 'opaque and falls back to REVIEW, never a partial parse' "$WT"; then
pass "REQ-EX-S09-03: $WT carries the measured .tf opaque/REVIEW sentence"
else
fail "REQ-EX-S09-03: $WT mentions .tf but lost the opaque/REVIEW-never-partial-parse sentence"
fi
else
fail "REQ-EX-S09-03: $WT no longer mentions .tf at all — the honesty pin would be vacuous"
fi

# --- Step 1's rule-file inventory must match the shipped tree (same truth-lag class as
# REQ-EX-S09-01/02: a stale parenthetical rule list is exactly the kind of drift this
# story exists to kill, even though it is prose rather than a console block) -----------
RULE_DIR="examples/packs/topic-registry/.assent/packs/topics/rules"
real_rule_count=0
missing_rule_files=0
for rf in "$RULE_DIR"/*.yaml; do
[[ -f "$rf" ]] || continue
real_rule_count=$((real_rule_count + 1))
base="$(basename "$rf" .yaml)"
if ! grep -q "\`$base\`" "$WT"; then
echo " $WT Step 1 does not name rule file \`$base\`" >&2
missing_rule_files=$((missing_rule_files + 1))
fi
done
if [[ "$real_rule_count" -eq 0 ]]; then
fail "EX-S09: $RULE_DIR has no rule files — the Step 1 inventory pin would be vacuous"
elif [[ "$missing_rule_files" -ne 0 ]]; then
fail "EX-S09: $WT's Step 1 rule-file list is missing $missing_rule_files real rule file name(s)"
else
pass "EX-S09: $WT Step 1 names all $real_rule_count real topic-registry rule files"
fi

if [[ "$fails" -ne 0 ]]; then
echo "FAILED: $fails truth-lag pin(s) reopened" >&2
exit 1
Expand Down