feat(scripts/spectro): automate the Spectro Cloud Palette pack

[ausbru87]

Summary

Adds a generator and a release-triggered workflow that keep the Coder community pack in spectrocloud/pack-central in step with the stable Coder release.

Companion pull request adding the regenerated pack: spectrocloud/pack-central (linked once opened).

Problem

The Coder pack in pack-central is coder-2.23.3, contributed by hand in July 2025. Stable is now 2.35.6. It is not only stale, it is wrong in two ways, and both came from hand edits:

1. It ships the wrong chart. The directory and Chart.yaml say 2.23.3, but every template and value byte-matches upstream 2.21.3. Chart.yaml was version-bumped without re-vendoring the chart body, and the archive charts/coder-2.23.3.tgz internally reports version: 2.21.3.
2. Every Palette override is inert. The pack nests chart values at charts.coder.env, but the chart's root key is coder:, so the correct path is charts.coder.coder.env. Rendering the chart with the pack's own values leaves the defaults in place. Every knob, including image, ingress, service, tls and resources, is silently discarded today.

The second defect also means the pack fails pack-central's current CI. validator/check-values-structure.py landed in pack-central PR #308 a month after the Coder pack merged, and the existing pack produces 27 errors against it. Any future PR touching packs/coder-* trips this.

Palette community packs are maintained by the contributing vendor, not by Spectro Cloud, and Palette deprecates a pack minor once two newer minors exist, then disables it after three months and deletes it three months later. Without automation this pack ages out of Palette on its own.

Fix

scripts/spectro/build-pack.sh <version> builds packs/coder-<version>/ from the published Helm chart. Nothing is hand-edited. It refuses to emit a pack unless:

• the chart tarball sha256 matches the digest in https://helm.coder.com/v2/index.yaml;
• the chart body's version and appVersion equal the requested version, and the repackaged archive agrees with its own filename (the check the 2.23.3 pack would have failed);
• pack.json still carries name: coder-chart and displayName: Coder, since changing either forks a new pack in Palette;
• charts.coder mirrors the chart's root keys and pack-central's own check-values-structure.py exits 0;
• rendering with the pack's charts.coder subtree produces ghcr.io/coder/coder:v<version>, and an override probe injected into that subtree reaches the rendered manifests. A pack with inert values renders identically either way, which is precisely why the current breakage went unnoticed;
• crane manifest resolves every entry in pack.content.images;
• every chart value path documented in the pack README still exists in the chart.

.github/workflows/publish-spectro-pack.yaml subscribes to the coder-release repository_dispatch that coder/coder already sends to this repository, so no change to coder/coder is required. It is gated on release_channel == 'stable', matching the conditional aws/submit_ce.sh already uses. It syncs a bot fork, guards against an already-open PR, and opens Upgrade coder pack to <version> against pack-central. A second job runs a weekly drift check and files a tracking issue if the pack falls behind, because silent rot is the failure mode that produced this situation.

The three Spectro-specific Chart.yaml edits are applied line-wise rather than with yq, because yq reflows sequence indentation and would make every future pack diff noisy against the previous version directory.

Verification

Generated 2.35.6 and ran pack-central's real validator against the output:

SUCCESS: JSON syntax is valid
SUCCESS: Schema validation passed
SUCCESS: pack.json version has no leading 'v'
SUCCESS: Logo file is present
SUCCESS: README.md file is present
SUCCESS: All chart tarballs present
SUCCESS: Chart values structure check passed
SUCCESS: Successfully verified Image content section
SUCCESS: Packs validation successful...

The same validator against the in-tree coder-2.23.3 reports Structural + duplicate check failed for 'charts.coder' with 27 path errors.

pack.json diff against 2.23.3 is two lines, the version and the chart path. logo.png is byte-identical. Chart.yaml differs from the previous pack by exactly version and appVersion.

Follow-ups and open questions

• CDRCI_GITHUB_TOKEN in this repository. Referenced by coder/coder for the same bot identity, but repo-level secrets here are not readable with my access. Needs confirming before the first real run.
• Fork ownership. It is unconfirmed that cdrci holds a fork of spectrocloud/pack-central. The owner is read from vars.SPECTRO_FORK_OWNER, defaulting to cdrci; the first run fails loudly at gh repo sync if the fork is missing.
• vars.SPECTRO_PACK_REVIEWERS is unset, so the cc step no-ops. Set it to route pack PRs to a human.
• CODEOWNERS is new and deliberately scoped to the Spectro paths only. Adjust or drop the handles if that is not the right routing.
• Action pinning. Actions are pinned by tag to match the existing workflows in this repository. coder/coder pins by SHA. Happy to switch if the org wants SHA pinning on new files here.
• Palette validation. pack-central asks contributors to test in a Palette environment before opening a PR. That needs a tenant. Coder is working with Spectro Cloud on a partner tenant; the seam for it is documented in scripts/spectro/README.md and is designed as a separate secret-gated job so this workflow keeps working if the tenant is delayed.

Implementation plan

Keeping the Coder pack in spectrocloud/pack-central up to date

Direct answer

Yes — build the pipeline in coder/packages. The trigger already exists and already points there: coder/coder's release workflow fires repository_dispatch: coder-release at coder/packages with {coder_version, release_channel}. Three workflows there already consume it. A Spectro job is a fourth subscriber to a working event bus, requiring zero change to coder/coder.

But a pipeline alone does not fix this. The pack is not merely outdated — it is mislabeled and functionally broken, and it fails pack-central's current CI. A remediation PR must land first, by hand, before any automation is worth wiring up.

---

What the research found

The pack is worse than "outdated"

Claim	Evidence
Directory says coder-2.23.3; content is upstream 2.21.3	diff -ru of packs/coder-2.23.3/charts/coder against coder_helm_2.21.3.tgz is empty except a 3-line Chart.yaml patch. Against 2.23.3 it differs by exactly the upstream 2.21.3→2.23.3 delta. The shipped charts/coder-2.23.3.tgz reports version: 2.21.3, appVersion: 2.21.3. Someone hand-edited Chart.yaml without re-vendoring.
Every user-facing value is inert	Pack values.yaml nests chart keys at charts.coder.env, but the chart's root key is coder:, so the correct path is charts.coder.coder.env. Proven: helm template with charts.coder.image.repo overridden still renders ghcr.io/coder/coder. Correct nesting works. Every knob — env, image, ingress, service, tls, resources — is silently discarded in Palette today.
It fails pack-central CI as it stands	python3 validator/check-values-structure.py → 27 errors, exit 1. Five recently-merged packs → exit 0. Coder is the outlier. This check landed in PR #308 (2026-08-25), a month after the Coder pack merged (#136, 2025-07-27). Any PR touching packs/coder-* triggers it.
Version drift	Pack content 2.21.3. Current stable 2.35.6, mainline 2.36.3. 14 minor versions behind.

The nesting bug was flagged during the original review — vishwanaths on PR #134: "coder should be within parent charts: ... This is the format to be followed" — and merged anyway.

How pack-central contribution actually works

Fork-and-pull. Add a new packs/<name>-x.y.z/ directory; nothing is ever edited in place, nothing is ever deleted. Required per version: pack.json, values.yaml, logo.png, README.md, charts/<chart>-<ver>.tgz (+ unpacked tree, the 174/237 convention).

CI (pack-validation.yml, no secrets, fork-safe) enforces: pack.json JSON-schema validity; version with no leading v; logo.png and README.md present; every charts[]/kubeManifests[] path exists; charts.<chartName> in pack values is a structural subset of the chart's own values.yaml; pack.content.images non-empty and each image resolvable via crane manifest. Bulwark gitleaks + CVE scans also run; CVE findings are informational (PR #303 merged with 235).

Identity rule: pack.json name (coder-chart) and displayName (Coder) must stay byte-stable across versions or Palette treats it as a brand-new pack.

Human gates that no bot can shorten:

• Merge is manual. vishwanaths merged 58 of the last 60 PRs. Median time-to-merge 4.2 days (p90 15d). 40 PRs open now, oldest 2.5 months.
• Publication is manual. push-packs.yml is workflow_dispatch-only with Spectro-held registry secrets. Stated cadence: biweekly, Tuesdays and Thursdays. Merge ≠ published.

Deprecation clock: Palette deprecates a pack minor once two newer minors exist → 3 months Deprecated → 3 months Disabled → Deleted. A vendor that stops publishing ages out of Palette automatically. Community packs are explicitly "not actively maintained by Spectro Cloud; updates depend on community contributors."

Re-vendoring is trivial

Upstream publishes a fully self-contained chart four redundant ways, all byte-identical (verified sha256 match between the HTTP tarball and the OCI artifact):

• https://helm.coder.com/v2/coder_helm_X.Y.Z.tgz
• helm pull coder/coder --version X.Y.Z
• helm pull oci://ghcr.io/coder/chart/coder --version X.Y.Z
• GitHub release asset coder_helm_X.Y.Z.tgz

libcoder is pre-vendored inside — no helm dependency update, no submodules. index.yaml carries a per-version digest for integrity checking.

The only genuine Spectro delta is a 3-line Chart.yaml patch (repository: file://libcoder, icon: file://assets/icons/coder.png, kubeVersion: '>=1.19-0') plus dropping the upstream README.md.

2.21.3 → 2.35.6 is additive only

Nothing removed, nothing renamed, no new required values. Two new opt-in templates (httproute.yaml, listenerset.yaml, both default-off, Gateway API). New defaulted keys: priorityClassName, podSecurityContext, serviceAccount.workspaceNamespaces, serviceAccount.labels, hostAliases, readinessProbe.*, livenessProbe.*, listenerset.*, httproute.*. libcoder/_rbac.yaml was refactored for multi-namespace RBAC — behaviorally equivalent in the default single-namespace case.

One behavior change to document: coder.livenessProbe.enabled now defaults to false (was unconditionally on in 2.21/2.23).

Exactly one container image, so pack.content.images stays a single entry: ghcr.io/coder/coder:vX.Y.Z (note the v on the image tag; no v on the pack/chart version).

2.35.6 (stable) and 2.36.3 (mainline) are template-identical — only Chart.yaml differs. Targeting stable costs nothing.

Precedent for the automation shape

coder/coder's release workflow already opens PRs against third-party repos:

• publish-winget → microsoft/winget-pkgs — the true analogue. gh repo sync cdrci/winget-pkgs, submit from a bot-owned fork with CDRCI_GITHUB_TOKEN, then locate the PR and gh pr comment a cc for human tracking.
• publish-homebrew → coder/homebrew-coder — raw git + gh, branch auto-release/$version, with an idempotency guard: gh pr list --search "head:$branch", bail if one is already open.

coder/packages is definitionally the marketplace repo (AWS Marketplace AMI + Container/EKS, GCP Marketplace, ECR, Snap Store) and already carries aws/helm_values.yaml. It already implements stable-gating in the exact shape needed (submit_ce.sh). greg-the-coder — who contributed the Spectro pack — is an active coder/packages committer.

Decisions taken

• Channel: stable only. Gate on client_payload.release_channel == 'stable'. ~1 PR/month, comfortably outruns the deprecation clock, and does not pile onto a 40-PR backlog.
• Validation: offline now, with a pre-built seam for Palette API validation once the partner tenant lands.

---

Plan

Phase 0 — Remediation PR to pack-central (blocking; do this first)

Nothing downstream matters until the pack is correct. Hand-drive one PR that adds packs/coder-2.35.6/, built from upstream 2.35.6 with the nesting bug fixed.

1. Fetch and verify:

   helm pull oci://ghcr.io/coder/chart/coder --version 2.35.6 -d /tmp/pull
   # cross-check sha256 against .entries.coder[] | select(.version=="2.35.6").digest
   # in https://helm.coder.com/v2/index.yaml

2. Scaffold packs/coder-2.35.6/; copy logo.png verbatim from coder-2.23.3.
3. Unpack into charts/, apply the 3-line Chart.yaml patch, delete upstream README.md, sync Chart.lock's repository field for consistency. Do not hand-edit version/appVersion — they already read 2.35.6. Repackage with helm package and assert the produced tgz reports 2.35.6.
4. pack.json: change only version and charts[0]. name: "coder-chart" and displayName: "Coder" stay byte-identical.
5. Regenerate values.yaml, do not hand-merge. Emit the pack: header, then indent the chart's own values.yaml four spaces under charts.coder:. This produces charts.coder.coder.*, charts.coder.provisionerDaemon.*, charts.coder.extraTemplates — the layout the validator and Palette expect — and fixes the inert-values bug for free.
6. Rewrite README.md against templates/README-template.md (Title / Prerequisites / Parameters table / Upgrade / Usage / References). The current README uses ## Resources and has no Parameters table. The Upgrade section must call out the livenessProbe.enabled: false default change and that overrides now actually take effect.
7. Verify locally before pushing — these two checks catch both historical failure modes:

   python3 validator/check-values-structure.py \
     packs/coder-2.35.6/values.yaml packs/coder-2.35.6/charts/coder/values.yaml coder   # exit 0
   helm template coder packs/coder-2.35.6/charts/coder \
     -f <(yq '.charts.coder' packs/coder-2.35.6/values.yaml) | grep 'image:'            # v2.35.6
   crane manifest ghcr.io/coder/coder:v2.35.6 >/dev/null
   MODIFIED_FILES_FILE=/tmp/modified_files GITHUB_WORKSPACE=$PWD ./validator/validate-packs.sh

8. PR title Upgrade coder pack to 2.35.6, matching repo convention. Body names the validated scenarios and states plainly that 2.23.3 shipped 2.21.3 content with non-functional overrides. PR carries the Coder Agents disclosure.

Open decision inside Phase 0: the published 2.23.3 pack is live in the prod registry deploying 2.21.3 with all config ignored. Options: (a) supersede silently, (b) an in-place +2 -2 correction to 2.23.3 (precedent: PRs #294, #251), or (c) request removal of the broken version (precedent: PR #304, vendor-initiated). Recommend (a) plus a note in the PR body, escalating to (c) only if Spectro asks.

Phase 1 — Generator script in coder/packages

Land scripts/spectro/build-pack.sh <version> behind its own PR, tested via workflow_dispatch before any dispatch trigger is wired. Pure function: version in, packs/coder-<version>/ out. Steps 1-7 above, scripted. Vendor validator/check-values-structure.py's contract as the local oracle, or fetch it from pack-central at runtime and pin by SHA.

Guardrails the script must assert, because they are the easy ways for a bot to break Palette:

• name == "coder-chart" and displayName == "Coder" unchanged.
• version has no leading v; the image tag does.
• Four coupled edits move together: dir name, pack.json.version, pack.json.charts[0] filename, Chart.yaml version/appVersion, pack.content.images[0], pack.content.charts[0].version.
• charts.coder node exists and is a mapping; structural-subset check exits 0.
• values.yaml uses spaces only, 2-space multiples, no trailing whitespace. (validate-values.sh is currently dead in CI due to a stale ^stable/ matcher, but conform pre-emptively.)

Phase 2 — Workflow coder/packages/.github/workflows/publish-spectro-pack.yaml

on:
  repository_dispatch:
    types: [coder-release]        # already dispatched by coder/coder release.yaml
  workflow_dispatch:
    inputs: { coder_version: {...} }   # manual backfill, mirrors sibling workflows

Gated if: github.event.client_payload.release_channel == 'stable' || github.event_name == 'workflow_dispatch' — the same conditional aws/submit_ce.sh already uses.

Steps: run the Phase 1 generator → gh repo sync <fork>/pack-central -b main (winget pattern) → idempotency guard gh pr list --repo spectrocloud/pack-central --head "<fork-owner>:$branch", bail if open (homebrew pattern; essential here, since Spectro's 4-day median merge means PRs will routinely still be open when the next Coder release lands) → push branch auto-release/coder-pack-$VERSION → gh pr create --repo spectrocloud/pack-central.

PR body: version, upstream chart digest, the offline checks that passed, and an explicit note that it is automated from coder/packages.

On success, gh pr comment a cc to the marketplace owners, mirroring winget — the dispatch is fire-and-forget, so a failure here is otherwise invisible from the coder/coder release run.

Phase 3 — Palette validation (deferred until the partner tenant exists)

Add as a separate job gated on secret presence, so Phase 2 keeps working unchanged if the tenant is delayed or revoked. Design the seam now:

1. oras push the pack to a GHCR repo under <org>/spectro-packs/archive/coder-chart:<version>. Palette requires that exact namespace path. ORAS must be pinned to v1.0.0 — Spectro documents this as a hard requirement.
2. POST /v1/registries/oci/basic (once, or reuse a persistent registry UID) → POST /v1/registries/oci/{uid}/basic/sync → poll /sync/status.
3. POST /v1/clusterprofiles/validate/packs — schema-validates a profile using the pack without deploying a cluster, so no cloud spend and no 15-30 minute wait.
4. GET /v1/packs/{packName}/registries/{registryUid}, /readme, /logo to confirm it rendered.

Auth is an apiKey: <key> header, tenant-scoped. Rate limits are 10 rps on /v1/packs and /v1/registries. Registry must be reachable inbound from Palette SaaS — GHCR satisfies this; a self-hosted throwaway registry would not without tunneling.

When this lands, extend the PR body to name the validated scenarios, which is what pack-central reviewers explicitly ask for.

Phase 4 — Housekeeping in coder/packages

• Add a Spectro Cloud Palette row to the README platform matrix. That table is the org's de facto marketplace inventory and Spectro is currently invisible in it.
• Add a CODEOWNERS naming the marketplace maintainers. coder/packages has none, so there is no review routing today.

---

Prerequisites to confirm with an admin

These could not be resolved with the available token (gh api repos/coder/packages/actions/secrets → 403):

1. Is CDRCI_GITHUB_TOKEN already available to coder/packages? It is wired into coder/coder, but coder/packages currently uses cloud credentials, not GitHub PATs. This is the one genuinely new capability the plan needs.
2. Can the cdrci machine account hold a spectrocloud/pack-central fork? pack-central mandates fork-and-pull.
3. Do coder/openshift-deployment or coder/aws-deployment (both INTERNAL, contents inaccessible) contain competing marketplace automation?

---

Risks

Risk	Mitigation
Spectro's 40-PR backlog means a bump sits open when the next release fires	Idempotency guard skips; the open PR simply lands one version behind. Consider amending the existing PR to the newest stable instead of skipping — decide after observing the first two cycles.
The pipeline silently rots (the exact failure mode that produced 2.21.3-in-a-2.23.3-box)	Add a scheduled drift check: compare gh release view --repo coder/coder --json tagName (which authoritatively returns stable, not newest) against the highest packs/coder-* directory, and open an issue on mismatch.
check-values-structure.py changes upstream and breaks generation	Pin by SHA and surface a clear diff on failure rather than generating a broken pack.
A bot breaks pack identity and Palette forks a new pack	Hard assertions in Phase 1; name/displayName are never templated.
Reviewers push back on packs not tested in Palette	Phase 0 and early Phase 2 PRs state exactly which offline checks ran. Phase 3 closes the gap properly.
CVE scans post CHANGES_REQUESTED	Informational at pack-central; a maintainer dismisses them. Non-blocking, but budget for a round trip.
Repo growth — no version is ever pruned	Accepted by Spectro; fluentbit has 12 live directories. No action.

---

Success criteria

1. packs/coder-2.35.6/ merged, with check-values-structure.py exit 0 and helm template rendering ghcr.io/coder/coder:v2.35.6.
2. Overrides set in Palette actually reach Helm — verifiable by rendering the pack values.
3. A stable coder-release dispatch produces a correct pack PR with no human in the loop up to the point of Spectro's review.
4. The pack never falls more than two minors behind stable, keeping it clear of Palette's deprecation clock.

---

Raised by Coder Agents on behalf of @ausbru87.