Skip to content

ci: fail release if any expected artifacts are missing#14

Merged
kenvandine merged 9 commits into
lemonadefrom
fix/validate-release-artifacts
Jul 2, 2026
Merged

ci: fail release if any expected artifacts are missing#14
kenvandine merged 9 commits into
lemonadefrom
fix/validate-release-artifacts

Conversation

@kenvandine

Copy link
Copy Markdown
Member

Summary

Ports the artifact-completeness gate from lemonade-sdk/llama.cpp#16 to this repo's release job, adapted to its build matrix. Previously the release job had no check at all — it created a release and uploaded whatever showed up in ./artifact, so a build failure anywhere could silently ship a release missing entire backends.

Changes

Each backend family is checked as an all-or-nothing unit; if any artifact in a family is missing, every artifact for that family is dropped from the release (other families are unaffected), and the job still fails so the gap is visible:

  • cuda: 8 sm_* variants on ubuntu x64 + windows x64, plus sm_121-only on ubuntu arm64 (GB10) — 17 files total.
  • rocm-therock: TheRock nightly ROCm 7.13.0 on windows + ubuntu.
  • rocm-hipsdk: official/legacy HIP SDK ROCm on windows (7.1.1) + ubuntu (7.2.1) — kept separate from rocm-therock since they're different SDKs, not just different platforms of the same backend.
  • cpu: ubuntu + windows avx2 builds.
  • metal: macos-14 + macos-latest builds.

A release with zero artifacts overall is never created.

Also brings the release/upload mechanism up to parity with llama.cpp's: an idempotent create-or-get-release step (safe to re-run after a partial failure) and a retrying, resumable asset upload, replacing the old anzz1/action-create-release + github-script@v3 flow.

Test plan

  • Validated YAML parses.
  • Simulated the per-family check/drop logic locally (a missing artifact in one family drops only that family; other families and the zero-artifact hard guard behave as expected).
  • Watch the next scheduled/manual release run to confirm end-to-end behavior.

kenvandine and others added 2 commits July 2, 2026 09:23
Port the artifact-completeness gate from lemonade-sdk/llama.cpp#16 to
this repo's release job, adapted to its build matrix:

- cuda: 8 sm_* variants on ubuntu x64 + windows x64, plus sm_121-only
  on ubuntu arm64 (GB10) — 17 files total.
- rocm-therock: TheRock nightly ROCm 7.13.0 on windows + ubuntu.
- rocm-hipsdk: official/legacy HIP SDK ROCm on windows (7.1.1) +
  ubuntu (7.2.1) — kept separate from rocm-therock since they're
  different SDKs, not just different platforms of the same backend.
- cpu: ubuntu + windows avx2 builds.
- metal: macos-14 + macos-latest builds.

Each family is all-or-nothing: if any artifact in a family is
missing, every artifact for that family is dropped from the release
so a release never contains a partial/inconsistent set (e.g. CUDA
split across sm_* or platforms). Other families are unaffected by a
gap in one. A release with zero artifacts overall is never created.

Also brings the release/upload mechanism up to parity with
llama.cpp's: an idempotent create-or-get-release step (safe to
re-run after a partial failure) and a retrying, resumable asset
upload, replacing the old anzz1/action-create-release +
github-script@v3 flow.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without always(), GitHub Actions' default skip-if-a-needed-job-failed
behavior meant a single failed build job (e.g. one CUDA sm_* matrix leg,
several of which use fail-fast: false) could skip the release job
outright, before the per-family completeness check ever ran — silently
producing no release instead of a correct partial one.

The other issue found in the equivalent llama.cpp fix (a hard-fail step
that could block unrelated families) doesn't apply here: this workflow
has no cross-job merge step comparable to llama.cpp's Windows CPU-into-
other-zips step, so there's nothing else that needs the same treatment.

Follow-up to #14.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kenvandine

Copy link
Copy Markdown
Member Author

Pushed 03e249d: applied one of the two fixes from the follow-up review on lemonade-sdk/llama.cpp#16.

  • Needed: added always() to the release job's if:. Without it, GitHub Actions' default "skip if a needed job failed" behavior meant a single failed matrix leg (several build jobs here use fail-fast: false, e.g. ubuntu-latest-cuda, windows-latest-cuda, ubuntu-arm64-cuda, macos-arm64-cmake) could skip the whole release job before the per-family completeness check ran.
  • Not applicable: llama.cpp also had a hard-fail step where a missing Windows CPU artifact aborted the entire job (blocking unrelated Ubuntu/CUDA artifacts too), since CPU binaries get merged into other Windows zips there. This repo's release job has no equivalent merge/rename step — every build job already produces its final, self-contained artifact filename directly — so there's nothing else that needed the same treatment.

stable-diffusion.cpp's Windows CUDA build fails with:

  fatal error C1128: number of sections exceeded object file format
  limit: compile with /bigobj

src/stable-diffusion.cpp combined with the CUDA backend's generated
code exceeds MSVC's default COFF section limit per object file. The
windows-latest-cmake job already works around the same class of issue
with -DCMAKE_CXX_FLAGS='/bigobj'; apply the same flag to the CUDA
build.

@fl0rianr fl0rianr left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is moving in the right direction and I like the strict all-or-nothing checks per backend family, especially for CUDA.

I would still not merge this yet because the release job still has normal needs: on all build jobs and no always() as stated on the other PR. So again suggesting:

if: ${{ always() && (github.event_name == 'schedule' || github.event.inputs.create_release == 'true') }}

Then the artifact check can decide which complete families to publish.

One more subtle edge case: because the release is now reused and existing uploaded assets are skipped, a previous partial release could keep stale/partial assets from a family that is dropped in the current run. If a family is dropped locally, matching existing release assets for that family should probably be deleted too, or at least this limitation should be documented.

The release is reused across runs against the same tag (idempotent
create-or-get). If a family (e.g. rocm-therock) was complete and
published in an earlier run but the current run finds it incomplete and
drops it locally, its assets from that earlier run were never removed —
the release would keep looking complete for a family this run couldn't
actually reproduce.

The upload step now reconciles existing release assets against the
current ./artifact file set: anything not fully uploaded (a broken
partial upload, as before) or not present in this run's file set (stale
leftovers from a dropped family) gets deleted before uploading.

Addresses review feedback on #14. The
always() fix requested in the same review was already applied in
03e249d.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kenvandine

Copy link
Copy Markdown
Member Author

Both addressed:

  1. `always()` was already applied in 03e249d (pushed as a follow-up before this review came in) — confirmed the `release` job's `if:` is `${{ always() && (github.event_name == 'schedule' || github.event.inputs.create_release == 'true') }}`.

  2. Stale asset pruning, pushed in bd5e37f. Good catch — since the release is reused across runs against the same tag, a family that was complete and published in an earlier run but gets dropped as incomplete in a later run would otherwise leave its old assets sitting on the release, making it look complete when this run couldn't actually reproduce it.

    Rather than threading per-family knowledge into the upload step, I had it reconcile the release against the current `./artifact` file set directly: any existing asset that's either not fully uploaded (broken partial upload, existing behavior) or not present in this run's file set (stale leftover from a dropped family) now gets deleted before uploading. Verified the reconciliation logic standalone: a stale rocm-therock pair from a prior run gets deleted when the current run only has `cpu`, while an intact `cpu` asset is correctly skipped and a broken partial upload gets deleted and re-uploaded.

kenvandine added a commit to lemonade-sdk/llama.cpp that referenced this pull request Jul 2, 2026
Same fix as lemonade-sdk/stable-diffusion.cpp#14: the release is reused
across runs against the same tag (idempotent create-or-get). If a
family (e.g. rocm) was complete and published in an earlier run but the
current run finds it incomplete and drops it locally, its assets from
that earlier run were never removed — the release would keep looking
complete for a family this run couldn't actually reproduce.

The upload step now reconciles existing release assets against the
current ./release file set: anything not fully uploaded (a broken
partial upload, as before) or not present in this run's file set (stale
leftovers from a dropped family) gets deleted before uploading.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@fl0rianr fl0rianr left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now matches the intended release behavior for me.
The all-or-nothing family logic is exactly what I like. Thanks!

kenvandine and others added 5 commits July 2, 2026 16:30
ci(windows-cuda): pass /bigobj to fix C1128 build failure
Port the artifact-completeness gate from lemonade-sdk/llama.cpp#16 to
this repo's release job, adapted to its build matrix:

- cuda: 8 sm_* variants on ubuntu x64 + windows x64, plus sm_121-only
  on ubuntu arm64 (GB10) — 17 files total.
- rocm-therock: TheRock nightly ROCm 7.13.0 on windows + ubuntu.
- rocm-hipsdk: official/legacy HIP SDK ROCm on windows (7.1.1) +
  ubuntu (7.2.1) — kept separate from rocm-therock since they're
  different SDKs, not just different platforms of the same backend.
- cpu: ubuntu + windows avx2 builds.
- metal: macos-14 + macos-latest builds.

Each family is all-or-nothing: if any artifact in a family is
missing, every artifact for that family is dropped from the release
so a release never contains a partial/inconsistent set (e.g. CUDA
split across sm_* or platforms). Other families are unaffected by a
gap in one. A release with zero artifacts overall is never created.

Also brings the release/upload mechanism up to parity with
llama.cpp's: an idempotent create-or-get-release step (safe to
re-run after a partial failure) and a retrying, resumable asset
upload, replacing the old anzz1/action-create-release +
github-script@v3 flow.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without always(), GitHub Actions' default skip-if-a-needed-job-failed
behavior meant a single failed build job (e.g. one CUDA sm_* matrix leg,
several of which use fail-fast: false) could skip the release job
outright, before the per-family completeness check ever ran — silently
producing no release instead of a correct partial one.

The other issue found in the equivalent llama.cpp fix (a hard-fail step
that could block unrelated families) doesn't apply here: this workflow
has no cross-job merge step comparable to llama.cpp's Windows CPU-into-
other-zips step, so there's nothing else that needs the same treatment.

Follow-up to #14.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The release is reused across runs against the same tag (idempotent
create-or-get). If a family (e.g. rocm-therock) was complete and
published in an earlier run but the current run finds it incomplete and
drops it locally, its assets from that earlier run were never removed —
the release would keep looking complete for a family this run couldn't
actually reproduce.

The upload step now reconciles existing release assets against the
current ./artifact file set: anything not fully uploaded (a broken
partial upload, as before) or not present in this run's file set (stale
leftovers from a dropped family) gets deleted before uploading.

Addresses review feedback on #14. The
always() fix requested in the same review was already applied in
03e249d.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e-artifacts' into fix/validate-release-artifacts
@kenvandine
kenvandine merged commit 9dcaab1 into lemonade Jul 2, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants