From 38a2e2c4aea26d083656cf40a8cf1a2dfacce490 Mon Sep 17 00:00:00 2001 From: Lakhan Samani Date: Fri, 24 Jul 2026 13:46:48 +0530 Subject: [PATCH] docs: add release runbook for server/SDK version sync Captures the multi-repo dependency graph (server -> BSR -> proto packages -> Go/Python SDKs; GraphQL schema as the one hand-maintained, undetected drift source), release ordering, pre-release checklist, and the verification bar that would have caught every cross-repo bug found in this project's actual history (pagination schema drift, a removed field SDKs kept sending, stale vendored proto stubs, an unresolvable Go module path, a missing PyPI publish secret). --- docs/release-runbook.md | 213 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 docs/release-runbook.md diff --git a/docs/release-runbook.md b/docs/release-runbook.md new file mode 100644 index 00000000..e2e6cfe3 --- /dev/null +++ b/docs/release-runbook.md @@ -0,0 +1,213 @@ +# Release runbook: keeping the server and SDKs in sync + +Authorizer ships as one server repo plus five satellite repos that must stay +in lockstep with it: three hand-maintained client SDKs (`authorizer-go`, +`authorizer-py`, `authorizer-js`) and two generated-only proto packages +(`authorizer-proto-go`, `authorizer-proto-python`) that the Go/Python SDKs +depend on. Nearly every cross-repo bug found in this project's history has +been the same shape: **the server's actual wire contract changed, and some +downstream artifact — a vendored stub, a hand-written type, a doc page — +didn't**. This runbook exists to make that class of bug structurally harder, +not just something a reviewer has to remember to check for. + +## The dependency graph + +``` +proto/*.proto (source of truth) + │ + ├─▶ gen/go, gen/openapi (server-side; make proto-gen / proto-check) + │ + ├─▶ buf.build/authorizerdev/authorizer (BSR module, pushed on every merge to main) + │ │ + │ ├─▶ authorizer-proto-go (own repo, regenerates from BSR on its own schedule) + │ │ └─▶ authorizer-go depends on it as a real go.mod dependency + │ │ + │ └─▶ authorizer-proto-python (own repo, regenerates from BSR on its own schedule) + │ └─▶ authorizer-py depends on it as a real PyPI dependency + │ + └─▶ internal/graph/schema.graphqls (GraphQL — independently maintained, NOT + generated from proto; can drift from it silently, see "Known drift + sources" below) + └─▶ authorizer-js hand-types its GraphQL/REST surface directly + (cannot use gen/ts's message types — see "Why authorizer-js + stays hand-typed" below) +``` + +Two facts fall out of this that drive the whole runbook: + +1. **`authorizer-go`/`authorizer-py` are no longer built by copying generated + code.** They depend on `authorizer-proto-go`/`authorizer-proto` as real, + versioned dependencies — the same shape as `prometheus/client_golang` + depending on `prometheus/client_model`. There is no manual "regenerate, + then copy into 3 repos" step left for those two. +2. **`authorizer-js`'s GraphQL/REST surface and the GraphQL schema itself are + both hand-maintained and can drift independently of proto.** This is the + one place in the whole graph without a generated source of truth, so it + needs the most manual vigilance. + +## Release order + +Always release in this order — each step's version needs to exist before the +next step can depend on it correctly: + +1. **`authorizer` (server)** — merge to `main`, cut a release (`gh release + create --prerelease` for an RC). This pushes the updated schema to + BSR automatically (`.github/workflows/buf.yml`'s `push to main -> publish + (buf push)` job — verify it actually ran and succeeded, don't assume). +2. **Wait for the published Docker image to actually exist** before running + any SDK CI that pulls it. `release.yaml` triggers on `release: created`, + not on merge — the image build+push takes real time (observed: ~10-12 + minutes end to end). Check `docker manifest inspect + quay.io/authorizer/authorizer:` succeeds before re-running SDK CI + against it — CI that runs too early gets a 404/timeout, not a useful + signal that something's actually broken. +3. **`authorizer-proto-go` / `authorizer-proto-python`** — if the proto + change affects generated messages, trigger `regenerate.yml` + (`workflow_dispatch`, don't wait for the weekly cron if you need it now), + review the auto-opened PR, merge, tag a new version, and for + `authorizer-proto-python` confirm the PyPI publish actually succeeded + (`release.yml` is tag-triggered — the workflow that runs is the one + **at that tag's commit**, not whatever's on `main` right now; if you + retag after fixing the release workflow itself, you must move the tag to + the new commit or the old workflow content still runs). +4. **`authorizer-go` / `authorizer-py`** — bump their dependency on the + proto package(s), update any code touching removed/changed fields, run + the FULL live-integration suite against a locally built server image + (not just unit tests — see "Verification bar" below), then release. +5. **`authorizer-js`** — only if the GraphQL/REST surface actually changed + (see "Known drift sources"); it never touches the proto packages. +6. **Examples repo, docs repo** — update pinned SDK versions and any + documented method signatures last, once the SDKs they reference are + actually published and installable. + +Skipping the order (e.g. releasing an SDK before its proto dependency is +published) produces exactly the failure this session hit: CI that resolves +a version that doesn't exist yet, or worse, silently resolves an *older* +cached version and passes for the wrong reason. + +## Pre-release checklist + +Run all of these before cutting **any** release, server or SDK: + +- [ ] **`make proto-check`** (server-side gen/go, gen/openapi) is clean. +- [ ] **`make proto-check-clients`** (gen/ts) is clean. Note this does + **not** cover `authorizer-proto-go`/`authorizer-proto-python` — those + regenerate from BSR independently on their own repos, not from this + repo's `gen/` at all anymore. +- [ ] **GraphQL schema vs. proto schema**: if you touched a field/message + that exists conceptually in both (pagination, request envelopes, + anything with a GraphQL input type mirroring a proto message), grep + the other one and confirm they still agree. There is **no automated + check for this** — it's the one drift class this whole graph doesn't + structurally prevent. When in doubt, diff the GraphQL input type + against the proto message field-by-field by hand. +- [ ] **Full test suite green** — `go build ./...`, `go vet ./...`, `make + test` (SQLite), `make lint`. For storage-layer changes, at least one + non-SQL backend too. +- [ ] **`make smoke`** before a server release specifically (build tag + `smoke`, boots the real binary, exercises GraphQL/REST/gRPC/MCP). +- [ ] **Docs**: if a public method's signature, a request/response shape, or + a config flag changed, grep `docs/sdks/*/functions.md` and + `docs/sdks/*/admin.md` in the `authorizerdev/docs` repo for the old + shape. These are hand-written and have gone stale before (46 of 81 Go + admin methods were once undocumented — caught by an explicit audit, + not any automated process). + +## Verification bar for SDK releases + +"The unit tests pass" is not sufficient — the bugs this project has actually +shipped were wire-shape mismatches that only a real round trip catches. +Before tagging an SDK release: + +1. Build a local server image from the exact commit you're releasing + against: `docker build -t authorizer:localmain .` from the server repo. +2. Run it with the flags the SDK's own test suite expects (check + `test/authorizer_test.go` / `tests/integration/test_live.py` for the + exact `client-id`/`admin-secret`/ports/`--disable-mfa` defaults — they're + documented at the top of those files specifically so this doesn't need + re-discovering every time). +3. Run the **full** live-integration suite (not `-m "not live"`, not just + unit tests) across every transport the SDK supports (graphql, rest, + grpc) — a bug can be transport-specific (the `_users` admin query's stale + `PaginatedRequest` type broke GraphQL only; REST/gRPC use proto messages + directly and were unaffected, which is exactly the kind of thing a + single-transport check would miss). +4. For a **published-package** change (not just a git branch), do an actual + clean install from the registry (`go get module@version` in a throwaway + module, `pip install package==version` in a fresh venv) and confirm it + resolves and imports — `go.sum`/PyPI propagation lag and packaging + mistakes (a missing `/v2` module suffix, a broken sdist) don't show up + from testing a local checkout. + +## Known drift sources (from this project's actual incident history) + +Keep this section updated — every entry here is a real bug that shipped, +not a hypothetical: + +- **GraphQL `PaginatedRequest` double-wrapper vs. proto's flat + `PaginationRequest`.** The two schemas were never generated from the same + source for this type; they silently diverged. Fixed by standardizing on + the proto shape everywhere and removing the wrapper — but nothing + prevents a *new* divergence like this from being introduced again. Grep + both schemas by hand for anything that looks like a shared concept. +- **A field removed from the server (`is_multi_factor_auth_enabled` on + `SignupRequest`, a security fix) kept getting sent by SDKs that hadn't + been updated**, in some cases for weeks. The removal was correct and + reviewed; the *propagation* to dependents wasn't tracked anywhere. +- **`gen/go-client`/`gen/python` (before they were replaced by dedicated + proto packages) went stale for the same reason** — the regen step was a + manual "clone the server repo, run buf generate, copy the output in" + dance that nobody was reminded to do. This is why those directories don't + exist anymore; the dedicated proto-package repos regenerate themselves + from BSR instead of depending on a human remembering. +- **`authorizer-go`'s `go.mod` never got the `/v2` module-path suffix** Go + requires once a module is tagged v2+. Every v2.x release before this was + found was **completely unresolvable** via `go get` — not a subtle bug, a + release that silently could never be consumed, for an unknown amount of + time before anyone tried a clean install and noticed. This is exactly why + step 4 of the verification bar above (a real clean install, not a local + checkout test) exists. +- **A CI-config secret dependency wasn't set until the first real release + attempt.** A repo's `release.yml` referencing `secrets.PYPI_API_TOKEN` + looks complete in code review; it doesn't fail until someone actually + tries to cut a release and the token doesn't exist. Prefer PyPI's Trusted + Publishing (OIDC via `pypa/gh-action-pypi-publish`) for new packages — + there's no secret to forget to set in the first place. + +## Why `authorizer-js` stays hand-typed + +Documented here because the temptation to "just use the generated `gen/ts` +types" comes up naturally once Go/Python are on generated packages, and the +answer is a real architectural mismatch, not an oversight: + +- `gen/ts` (protobuf-es) generates **camelCase** message shapes + (`phoneNumber`, `confirmPassword`) — the standard JS/TS convention for + generated protobuf code. +- This server's REST transport is **deliberately configured to snake_case** + (`internal/gateway/mount.go`, `UseProtoNames: true`) specifically to keep + REST payloads aligned with the GraphQL surface, which is also snake_case. +- `authorizer-js` only speaks GraphQL and REST (browsers can't speak raw + gRPC, so it never needed the generated message types for that reason + either) — both of its real transports are snake_case, so generated + camelCase types would require a rename/mapping layer to actually match + the wire format, at which point the "just use the generated types" + shortcut isn't actually a shortcut. + +If this server's REST config ever changes to emit camelCase by default, +this tradeoff should be re-evaluated — but as long as REST intentionally +mirrors GraphQL's snake_case, generated TS types would be actively wrong +for both of `authorizer-js`'s transports, not just unnecessary. + +## SDK-level e2e coverage + +`e2e-playground/` (this repo's live-playground Playwright suite) exercises +the full feature surface — OIDC, SAML, SCIM, WebAuthn, TOTP/SMS-OTP/WebOTP, +10 social OAuth providers, MFA enforcement, OTP lockout — but does so via +raw GraphQL/REST calls and browser automation, never through the actual +published SDKs. `e2e-playground/sdk-tests/{go,python}/` exercises the same +live stack through the real `authorizer-go`/`authorizer-py` packages +specifically to catch the wire-shape-mismatch class of bug described above +before it reaches a real consumer. When adding a new feature to the server, +consider whether it needs coverage in both places — the Playwright suite +proves the feature works; the SDK-driven suite proves the SDKs correctly +expose it.