ci: fail release if any expected artifacts are missing

[kenvandine]

Summary

Part of lemonade-sdk/lemonade#2163

The previous "Check release artifacts" step only verified that at least one artifact existed. This meant a release could be published with missing CUDA builds — as happened with b9549, which was missing all 7 Windows CUDA artifacts and ubuntu-cuda-sm_90-x64.

Changes

The check now explicitly enumerates every required artifact and fails with actionable error messages before a release is created:

• All 7 CUDA SM variants (sm_75, sm_80, sm_86, sm_89, sm_90, sm_100, sm_120) for each platform:
  • Ubuntu x64 (.tar.xz)
  • Ubuntu arm64 (.tar.xz)
  • Windows x64 (.7z)
• Ubuntu ROCm (.tar.gz)
• Ubuntu OpenVINO (.tar.gz)
• Windows ROCm (.zip)
• Windows CPU (.zip)

If any artifact is absent, the step fails and lists exactly which files are missing — so a partially-populated release can never be tagged.

[fl0rianr]

I don't think this should be the final fix. The artifact manifest check is useful, but this makes releases all-or-nothing: a transient Windows CUDA failure now blocks the entire release.

I'd prefer this as a first step only if we also add a recovery path: publish available artifacts with a clear missing-artifacts list, then allow a manual backfill workflow to rebuild/upload a missing sm_* target for the same tag.

[fl0rianr]

Do you want to have a slight adaption? I'm just thinking if we have a major issue with either rocm or cuda not able to get the artifact out why hold of the other? Hence why I did this one: lemonade-sdk/lemonade#2518

[kenvandine]

Good call — pushed 73fe29e which decouples the checks per backend family (cuda, rocm, openvino, cpu). A gap in one family is still reported with ::error:: and still fails the job (so it doesn't get silently missed), but it no longer blocks creating the release or uploading the artifacts from families that built fine. So a transient Windows CUDA sm_* failure will publish ROCm/OpenVINO/CPU as before, just with the run ending red and a clear list of what's missing from CUDA.

That should cover your second comment directly (rocm failure no longer holds back cuda, and vice versa).

Didn't add the manual backfill workflow yet — happy to follow up with that in a separate PR if you think it's still needed on top of this, since re-running the whole nightly job is idempotent already (skips already-uploaded assets) but doesn't let you target just one missing sm_*.

[fl0rianr]

Thanks, this is much closer to the behavior I was hoping for.

One thing I would still keep as a blocker: please preserve a hard guard for the files == 0 case before creating or updating the release. If no artifacts made it into ./release, we should fail before touching the GitHub release, otherwise we could create an empty release.

Optional/nit: I’m okay with not publishing a given OS/backend family when one of its required pieces failed — for example, if the Windows side is incomplete, it’s fine for that part of the release to be absent. The main thing is that this should not accidentally create an empty release, and the final error should clearly say which family/artifacts were missing.

[kenvandine]

Pushed 34e6acd:

• Added the hard guard you asked for: if ./release ends up with zero artifacts, the workflow now fails immediately with ::error::No release artifacts found in ./release — aborting before creating a release. before the "Create or get release" step runs, so we can never touch/create a GitHub release with nothing to publish. This check runs independently of the per-family logic, so even if the per-family bookkeeping were ever wrong, an empty ./release is always caught.
• The final failure message is now self-contained — it includes the per-family summary directly (e.g. 2 artifact(s) missing (cuda: 1 missing, rocm: 1 missing)) instead of just pointing back at the earlier step's log.

On the optional nit: I left individual working artifacts within a partially-broken family published (e.g. if one Windows CUDA sm_* fails, the other Windows CUDA sm_* builds that succeeded still get uploaded) rather than dropping the whole family. That seemed like a strict improvement over dropping everything in the family, and you'd said you're fine either way as long as it's never an empty release and the error is clear — let me know if you'd rather I drop the whole family on any partial failure instead.

[kenvandine]

Pushed 33312dd: switched from "publish whatever built within a family" to strict all-or-nothing per family.

If any required artifact in a family (cuda, rocm, openvino, cpu) is missing, every artifact belonging to that family is deleted from ./release before upload — so a release can never contain a partial/inconsistent set for a family (e.g. CUDA sm_90 failing on Windows now drops all 21 CUDA files, not just the one that failed). Other families are unaffected. The zero-artifact hard guard from the last round still applies on top of this, and the final error message lists which families were dropped and why, e.g.:

2 artifact(s) missing (cuda: 1 missing, family dropped) — the incomplete families were dropped from this release; other families were still published.

So: if CUDA shows up in a release, it's guaranteed to be the complete set for ubuntu x64/arm64 and windows, no mixed versions.

[fl0rianr]

This is much closer and I like the strict all-or-nothing behavior per family: if CUDA is present, it must be complete across Ubuntu x64, Ubuntu arm64, and Windows x64; if not, CUDA should be dropped while ROCm/OpenVINO/CPU can still publish. Great!

One blocker remains: the release job still has normal needs: on all build jobs and no always(). If a CUDA matrix job actually fails, the release job may be skipped before this new family-drop logic can run. To make the intended behavior work, the release job should use something like:

if: ${{ always() && (github.event_name == 'schedule' || github.event.inputs.create_release == 'true') }}

Then the artifact check can decide what complete families to publish.

Also worth checking: Move artifacts still hard-fails if the Windows CPU artifact is missing before the family check runs. That may be intentional, but if the goal is true per-family independence, this should either be handled by the family logic or documented as a required global prerequisite.

[kenvandine]

Both addressed in e8991e5:

1. always() added. The release job's if: is now ${{ always() && (github.event_name == 'schedule' || github.event.inputs.create_release == 'true') }}. You were right that the previous condition still fell under the default "skip if a needed job failed" behavior — a single failed CUDA matrix leg (or windows-cuda getting skipped because windows-cpu failed) could skip the whole release job before the family-drop logic ever ran, silently producing no release at all instead of a partial-but-correct one.

2. Windows CPU dependency handled by the family logic, not just documented. Windows CPU turned out to be a genuine structural prerequisite — its binaries get merged directly into e.g. the ROCm zip — so a Windows zip published without it would be a broken artifact, not just an "optionally missing" one. Rather than hard-exiting the whole job (which was blocking unrelated Ubuntu ROCm/OpenVINO/CUDA artifacts too), the merge step now just skips moving any Windows zip-based artifact into release/ when CPU is unavailable — Ubuntu/CUDA .7z artifacts proceed untouched. The existing completeness check then naturally sees the missing cpu/rocm files in release/ and drops those two families, same as any other gap.

Tested both changes locally (bash logic for the CPU-merge skip path, confirming Ubuntu artifacts still land in release/ while the un-mergeable Windows zip is correctly left out and caught by the family check).

[fl0rianr]

This is basically there now.

One small remaining issue: the artifact/*.tar.gz move loop should also guard against an unmatched glob, like the CUDA .tar.xz and .7z loops already do.

If both .tar.gz producers are missing, the loop may try to move the literal artifact/*.tar.gz and fail before the family completeness check can drop ROCm/OpenVINO and still publish CUDA/Windows CPU.

Suggested fix:

for tar_file in artifact/*.tar.gz; do
  [ -f "$tar_file" ] || continue
  ...
done

After that I’m good with approving.

[kenvandine]

Fixed in 87f8d0f — added the same `[ -f "$tar_file" ] || continue` guard the CUDA loops already had. Verified locally that the loop now exits cleanly with zero iterations when no `.tar.gz` files are present, instead of attempting to `mv` the literal unmatched glob string.

[fl0rianr]

Go for it!

[kenvandine]

Proactively applied the same stale-asset fix that came up in review on lemonade-sdk/stable-diffusion.cpp#14, since this repo has the identical idempotent create-or-reuse-release pattern and is exposed to the same bug: if a family (e.g. rocm) was complete and published in an earlier run against a tag, but a later run against that same tag finds it incomplete and drops it locally, the earlier run's assets for that family were never cleaned up — the release would keep looking complete for something the current run couldn't reproduce.

Pushed in f0883c9: the upload step now reconciles existing release assets against the current `./release` file set — anything not fully uploaded (broken partial upload, existing behavior) or not present in this run's file set (stale leftover from a dropped family) gets deleted before uploading, instead of just skipping re-upload of whatever's already there.

[fl0rianr]

Nice, all for go from my side