Mirror history: skip re-synchronizing already-mirrored (and promoted) digests - #164
Merged
Conversation
…gests Propose a durable mirror-history OCI artifact stored at quarantine/<image>:mirror-history recording every synchronized source digest (keyed by source tag), so a digest that was mirrored once is not re-synchronized after promotion deletes it from quarantine. Design only; proposed and awaiting review. Refs #157.
- O2: a recorded digest suppresses copy_referrers re-copy too (force refreshes) - O3: no history seeding; a one-time re-mirror of pre-existing images is accepted - O4: reserved tag name mirror-history confirmed - O1 (open): detail how the CSSC Dashboard Acquisition view would keep showing a history-only quarantine package; likely a small packages-service follow-up Refs #157.
…shboard Instead of hiding the persisted mirror-history package, extend the CSSC Dashboard Acquisition view to read the :mirror-history artifact and show a per-repo synchronized history, and exclude the reserved tag from the in-quarantine signal. Adds a Dashboard integration section and deliverable. Refs #157.
New .github/actions/mirror-history/ composite action maintaining a durable append-only history of synchronized source digests under the reserved tag <dest-image>:mirror-history. - check: outputs already-synchronized=true|false for (source-tag, source-digest) - record: appends an entry and pushes the updated OCI artifact (created on first use); force always appends (audit trail); reads a missing tag as empty but fails loudly on other registry errors - pushed via oras push --artifact-type (empty config); JSON log is the layer Align design doc artifact-format wording with the implementation. Refs #157. Closes #158.
- New record_history input (default true). - Resolve source digest up front, then mirror-history 'check'; skip the copy when the (tag, digest) is already recorded, unless force. copy_referrers is suppressed too once recorded. - mirror-history 'record' after a run that actually copied or found the dest up to date; skipped-by-history runs record nothing. force appends. - oras setup + GHCR login now also enabled by record_history. - Job summary reports a distinct 'skipped, already synchronized' outcome. Caller workflow_call signature stays backward compatible. Refs #157. Closes #159.
…oint)
- cssc_common: new OciRegistryClient (minimal registry manifest+blob JSON read,
GHCR token exchange, injectable httpx client) + MIRROR_HISTORY_TAG constant +
MirrorHistoryEntry model (camelCase aliases).
- packages-service: PackagesClient.get_history(name) reads <name>:mirror-history
from the registry; new GET /packages/{name}/history endpoint (empty when the
tag is absent or no registry configured).
- Tests: entries parse, empty-on-404, empty-without-registry.
Refs #157. Closes #160.
- dashboard-web PackagesServiceClient gains get_tags + get_history. - AcquisitionProvider computes in_quarantine from real (non-history) tags so a history-only, already-promoted package is no longer shown as awaiting promotion, and attaches the synchronized history to each card. - Acquisition template renders a collapsible Synchronized section (source tag, short digest, synced time, run link, forced badge) + small CSS. - Tests: synchronized surfaced, history-only not-in-quarantine, fragment render. Refs #157. Closes #161.
- New docs/reference/mirror-history.md (artifact location, format, behaviour, retrieval) + linked from reference README. - workflow-actions.md: mirror-history terminology row + action catalogue entry. - image-mirror-workflows.md: record_history input row + Mirror history bullet in the Implemented section. - Design doc status -> implemented on branch (pending validation). Refs #157. Closes #162.
Contributor
There was a problem hiding this comment.
Pull request overview
Implements the “mirror-history” mechanism to make mirroring idempotent across quarantine tag deletion by persisting synchronized (sourceTag, sourceDigest) pairs in a per-repository :mirror-history OCI artifact, and surfaces that history in the Dashboard Acquisition view.
Changes:
- Adds a new
mirror-historycomposite action and wires it into_mirror-image.ymlto short-circuit already-recorded digests (unlessforce) and record newly-synchronized digests. - Extends packages-service with registry reads for the
:mirror-historyartifact and aGET /packages/{name}/historyendpoint, plus Dashboard UI rendering/tests for “Synchronized” history and correctedin_quarantine. - Adds architecture/reference documentation for the mirror-history artifact, schema, and behavior.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/reference/workflow-actions.md | Documents the new mirror-history action in the action catalogue. |
| docs/reference/README.md | Adds the mirror-history reference page to the reference index. |
| docs/reference/mirror-history.md | New reference doc describing artifact location, schema, and retrieval. |
| docs/architecture/acquire/README.md | Adds mirror-history to the Acquire architecture index. |
| docs/architecture/acquire/mirror-history.md | New design/architecture doc for mirror-history behavior and integration. |
| docs/architecture/acquire/image-mirror-workflows.md | Updates mirror workflow docs with the new record_history behavior. |
| apps/python-app/services/packages-service/tests/test_history.py | New unit tests for history retrieval and empty-history cases. |
| apps/python-app/services/packages-service/src/packages_service/client.py | Adds registry-backed get_history() to PackagesClient. |
| apps/python-app/services/packages-service/src/packages_service/app.py | Wires registry client and adds /packages/{name}/history endpoint. |
| apps/python-app/services/dashboard-web/tests/test_app.py | Extends stage rendering test to expect “Synchronized” output. |
| apps/python-app/services/dashboard-web/tests/test_acquisition.py | Adds provider tests for synchronized history and history-only packages. |
| apps/python-app/services/dashboard-web/src/dashboard_web/templates/stages/acquisition.html | Renders “Synchronized” history table in the Acquisition stage UI. |
| apps/python-app/services/dashboard-web/src/dashboard_web/static/css/styles.css | Styles the new synchronized history <details> block. |
| apps/python-app/services/dashboard-web/src/dashboard_web/stages/acquisition.py | Fetches tags/history per package; excludes reserved tag for in_quarantine. |
| apps/python-app/services/dashboard-web/src/dashboard_web/clients.py | Adds packages-service client calls for tags and history endpoints. |
| apps/python-app/libs/cssc_common/cssc_common/registry.py | New minimal OCI registry read client for JSON artifacts. |
| apps/python-app/libs/cssc_common/cssc_common/models.py | Adds MirrorHistoryEntry model to parse camelCase artifact entries. |
| apps/python-app/libs/cssc_common/cssc_common/init.py | Exposes registry client, reserved tag constant, and MirrorHistoryEntry. |
| .github/workflows/_mirror-image.yml | Adds record_history input; resolves digest; checks/records history; new summary path. |
| .github/actions/mirror-history/action.yml | New composite action implementing check and record operations via oras/jq. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- action: fail fast when 'record' is missing source-image (avoids empty source). - OciRegistryClient: decouple the token-exchange Basic-auth username from the package owner (owner may be an org); new username param + GITHUB_USERNAME setting, defaulting to owner. - acquisition: note the deliberate per-package tags+history reads (demo scale). - docs: design doc opening now says 'describes' (implemented); acquire README marks mirror-history implemented (pending validation). Refs #157 #164.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the mirror-history feature designed in #157.
Problem
The mirror is stateless: it copies whenever the source digest differs from the quarantine digest. After promotion deletes the tag from quarantine (and GHCR deletes the whole package when its last tag goes), the next scheduled mirror sees an absent destination and re-synchronizes the already-promoted digest, looping it back through the pipeline.
Solution
A durable, deletion-surviving mirror-history OCI artifact per synchronized repo (
quarantine/<image>:mirror-history) — an append-only JSON log of every synchronized source digest, keyed by(sourceTag, sourceDigest). Being a separate tag, it survives image-tag deletion (and keeps the package alive, sidestepping the GHCR "cannot delete the last tagged version" case). The mirror checks it before copying and skips a digest already recorded (unlessforce).What's included
check+record) —oras-based,--artifact-type application/vnd.cssc.mirror-history.v1+json._mirror-image.ymlwiring — newrecord_historyinput (default on); resolve digest → check → gated copy → record; distinct skipped, already synchronized summary.copy_referrersre-copy is suppressed once recorded;forcebypasses and still records.packages-servicereads the artifact (GET /packages/{name}/history); the Acquisition view surfaces a per-repo Synchronized list and fixes thein_quarantinesignal (excludes the reserved tag) so a history-only package isn't shown as awaiting promotion.docs/reference/mirror-history.md, action catalogue + terminology, architecture updates.Decisions (from #157)
Location colocated in
quarantine/<image>:mirror-history; key(sourceTag, sourceDigest);forcebypasses + records; append-only unbounded; nodestDigestfield;forceappends; dashboard surfaces history (resolves O1).Validation
get_errors,actionlint, andshellcheck -S warningclean on new/changed run blocks; jq check/record logic unit-tested locally.Closes #158, #159, #160, #161, #162. Part of #157.