Skip to content

[precompiled]: rebuild/overwrite the image even if the tag already exists#860

Open
shivakunv wants to merge 1 commit into
mainfrom
overwrite-nvstaging-image
Open

[precompiled]: rebuild/overwrite the image even if the tag already exists#860
shivakunv wants to merge 1 commit into
mainfrom
overwrite-nvstaging-image

Conversation

@shivakunv

Copy link
Copy Markdown
Contributor

Add FORCE_PUSH override to overwrite already-published precompiled images

Issue: When a CVE is found during a release and the precompiled image is already published, the pipelines skip re-pushing because the image tag already exists. If no new kernel has been released, there is no way to rebuild and overwrite the fixed image

Solution: Write the existing FORCE_PUSH into both scheduled pipelines(github and gitlab) so it can be toggled via a CI/CD variable.
Setting FORCE_PUSH=true forces a rebuild, e2e test, and overwrite of the existing tag.
Unset keeps the normal skip.

…ists

Signed-off-by: Shiva Kumar (SW-CLOUD) <shivaku@nvidia.com>
@shivakunv
shivakunv marked this pull request as ready for review July 8, 2026 14:06
@shivakunv

Copy link
Copy Markdown
Contributor Author

@rajathagasthya

Copy link
Copy Markdown
Contributor

Let's not do this. We never want to overwrite tags. If there are CVE fixes, they will be carried in the next precompiled driver image. This is how it works for non-precompiled images as well.

@shivakunv

shivakunv commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Let's not do this. We never want to overwrite tags. If there are CVE fixes, they will be carried in the next precompiled driver image. This is how it works for non-precompiled images as well.

Why force write is necessary

  1. Non-precompiled images: these go through testing on every release, we fix the critical bug and then release. So, a CVE fix naturally lands in the next release.

  2. Precompiled images: a precompiled release happens only with a new kernel release, and precompiled testing is done only for a major release:

    a) A new kernel version is detected automatically (proceed to (b) and (c) only if the tag does not exist in both ghcr and nvcr).
    b) github pipeline runs build -> e2e -> publish to ghcr.
    c) gitlab pipeline pushes it to nvstaging (only if the tag does not exist).
    d) nspect manual MR creation and publishing of images.
    
  3. Problem:
    because the tag already exists, this image tag and this specific kernel version of the GPU driver container will never be published in github pipeline. The buggy image (CVE or scan failure) will stay in nvstaging permanently, and we will miss that kernel release.

  4. Waiting does not reliably fix it:
    if we wait for the next kernel (normally 10–15 days) and it again has a scan failure, we have to wait again and we may end up skipping 3–4 kernel versions this way.

  5. Scope is limited to staging:
    this force write only writes to nvstaging. It is never pushed to the public nvcr registry and does not update the actual release on nvcr.

  6. The gitlab pipeline already has this CI/CD variable to overwrite the image.

  7. With this change, we no longer miss a new kernel release.

  8. FORCE_PUSH runs the normal github pipeline job (build -> e2e -> publish), the only difference is that it overwrites the buggy image already sitting in nvstaging (in this case ubuntu26.04)

If you agree, I will add an vuln exception for this CVE and proceed with releasing the precompiled driver container. We can address the CVE in the next release. Please let me know your suggestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[precompiled]: rebuild/overwrite the image even if the tag already exists

2 participants