[schmitch]

Fixes #185

Retention policies can now run on a cron schedule in a time zone with a time limit per run, package rules get three new triggers and three new filters, image rules get tag patterns and glob protections, and the format contract gets the two retention hooks (version class, keep-N partition) that apt and Maven need. The admin API validates every new field, the Retention screen edits all of it, and docs/retention.md now lists every rule with examples.

1. Cron schedules (package, raw file and image policies)

• New envelope fields, the same in all three policy shapes (model.rs, files.rs, images.rs):
  • schedule: a 5-field cron pattern (0 3 * * *; @daily and the other nicknames work). Seconds and years are refused.
  • schedule_timezone: an IANA zone (Europe/Berlin). UTC when absent. Refused without a schedule.
  • max_run_seconds: from 60 s to a week.
  • min_run_interval_seconds stays the default way to schedule, and a policy with a schedule ignores it.
• New module crates/core/src/retention/schedule.rs: parsing, validation, next_after/next_runs, and reading these fields from a stored policy of any shape.
• Worker (worker.rs):
  • next_run_at is the schedule's first time at least the one-minute floor after the run started. The three per-shape interval computations are now one next_run_at, which a failed run uses too.
  • The deadline is the claim time + max_run_seconds.
• Time limit: apply, apply_files and apply_images check a RunClock before each batch or eviction.
  • Once the deadline has passed, the run stops and counts what it left as RunSummary.deferred. The next run re-evaluates and goes on with the rest.
  • The first step of a run always goes, so a run whose evaluation alone outlasts its window still makes progress.
• Saving a policy (PUT …/retention):
  • A policy with a schedule sets the ledger's next run to the schedule's next time (write_policy_at), so it does not run the moment it is saved.
  • A schedule replaced by an interval, or a shorter interval, brings the next run forward to last start + interval (never before now).
• Run now still runs at once. The run after it is at the next scheduled time.
• Policy changed while a run works: finish_run/fail_run take the stored policy the run evaluated. When the stored policy is a different one by then, they keep the next run that the new policy's write set. Without this, an old interval's now + 6h could land a run in working hours.

2. New triggers and filters

Package rules (evaluate.rs):

• prerelease_superseded_for_days: selects a prerelease (or snapshot) once its package has had a listed release that orders after it for N days. For SemVer, that means a release with the same or a higher base version.
  • The clock starts when the first such release was published.
  • It is computed in one pass per package.
  • The reason records superseded_by.
• unlisted_for_days: selects a version that has been unlisted for N days, read from the new component.unlisted_at.
  • It needs action: hard_delete; unlist is refused, because unlisting an unlisted version does nothing.
• max_versions_per_package: a pure count. It selects what is beyond the newest N versions, listed or not, of the rule's pool in its partition.
  • Also allowed on proxy_cached rules (cached versions).
  • The reason records rank.
• Filters:
  • version_pattern / exclude_version_pattern: globset globs on the whole normalized version, ignoring case, with {a,b} alternation.
  • selector: an existing content selector, by name, matched like a grant scoped by it.
  • release_type: snapshots.
• selector handling:
  • The worker and the preview resolve it (retention::resolve_selectors).
  • PUT/preview refuse a policy that names a missing selector (400).
  • A run fails without mutating anything if the selector was deleted since (fail closed).
  • DELETE /admin/content-selectors/{id} answers 409 and names the repositories while a retention rule uses the selector (retention_dao::repositories_using_selector, a jsonb_path_exists query).
• Filters are compiled once per evaluation (RuleFilter). A glob that does not compile, or a selector that is not resolved, makes the rule admit nothing.

Image rules (images.rs):

• tag_pattern / exclude_tag_pattern choose which tags a tag rule looks at, and keep_last_n_tags counts only those.
• protected_tags takes globs (release-*) as well as names, in the rules and in the size budget's protections.
• They are validated: no empty glob, it must compile, and it needs a tag criterion.

Protections still win over every new trigger. A version that any rule's keep_last_n / keep_n_most_downloaded keeps is never selected, in the same rule or another one.

Validation (deny_unknown_fields, as before):

• 400 for a zero count or zero days, an invalid glob, an empty selector, unlisted_for_days on an unlist rule, or prerelease_superseded_for_days on a releases rule.
• 400 for hosted-only triggers on proxy_cached rules, and for an invalid cron pattern, time zone or max_run_seconds.
• 422 for a misspelled field.

3. Format-specific rules: plugin contract hooks

• FormatPlugin::version_class(version) -> VersionClass (Release / Prerelease / Snapshot, in forge-plugin-api).
  • The default is SemVer's rule (a - makes a prerelease).
  • release_type and the supersession trigger use it.
• FormatPlugin::retention_partition(id, version) -> Option<String>: the key that keep-N counts within (keep_last_n, keep_n_most_downloaded, max_versions_per_package).
  • The default is None: the whole package is one partition.
• Both are mirrored on forge_core::Format and forwarded by PluginFormat.
• NuGet and npm keep the defaults, so the generic rules apply. Helm charts live in docker repositories and follow the image rules.
• apt and Maven rules are not in this PR: those formats (apt (Debian/Ubuntu) repositories: hosted, proxy and group, Nexus-style #176, Maven repositories: hosted (releases and snapshots), proxy and group, Nexus-style #177) are not on main, and the issue's criteria say their rules land with them.
  • The hooks are what those formats need. The evaluator test format_hooks_partition_keep_n_and_class_snapshots drives the generic rules with a Maven-like test format:
    • newest N builds per -SNAPSHOT version;
    • snapshot builds older than N days;
    • a version's snapshots deleted once its release exists;
    • release rules per groupId:artifactId.
  • apt would return the architecture as its partition. docs/format-plugins.md has a new "Retention" section spelling this out.
• Raw (Raw (generic file) repositories: hosted, proxy and group, Nexus-style #42): checked against main, which already has path-prefix rules (path_prefix), keep-N files per directory (keep_newest_per_directory) and the age/download triggers, as Raw (generic file) repositories: hosted, proxy and group, Nexus-style #42 specifies. Raw policies only gain the schedule fields here.

4. Schema

• Migration 0011_retention_unlisted_at.sql: adds component.unlisted_at, maintained by a BEFORE INSERT OR UPDATE OF listed, unlisted_at trigger:
  • set when a version is unlisted or arrives unlisted;
  • kept while it stays unlisted (unlisting again changes nothing);
  • cleared on relist;
  • an explicit value on an unlisted row is kept, which allows backdating.
  • A trigger rather than DAO changes, so every unlist path is covered: NuGet's DELETE, npm, retention's own unlist, pushes and imports of unlisted versions, a redeploy that unlists, and future paths.
• Migration default (my decision): versions already unlisted get their updated_at. Every unlist path sets updated_at = now(), so the default is never earlier than the real unlisting. A version changed after it was unlisted counts from that later change, which only defers its deletion. 185.upgrade.md says so.
• .sqlx:
  • generated by compiling forge-db against Postgres 18 migrated through 0011 (SQLX_OFFLINE_DIR);
  • 6 new entries: finish_run, fail_run, the two candidate loaders (now with unlisted_at), write_policy_at's update and repositories_using_selector;
  • the 4 stale versions of the changed queries are removed.

API

• PUT /admin/repositories/{name}/retention accepts and validates every field above, and sets the next run for a schedule.
• POST …/retention/preview:
  • decisions carry the new reasons (superseded_by, prerelease_superseded_for_days, unlisted_for_days, max_versions_per_package, rank);
  • a policy with a schedule also returns next_runs (its next 5 times, UTC).
• The run summary (last_run.last_run_summary) gains deferred.
• DELETE /admin/content-selectors/{id}: 409 while a retention rule names the selector.

Web UI (Repositories → repository → Retention)

• Envelope:
  • "When it runs" switches between the interval and a cron schedule (pattern and time zone fields).
  • "Stop a run after" sets max_run_seconds with a unit.
• Package and proxy rules:
  • a content selector picker listing the existing selectors (a text field until they load);
  • version pattern and except-pattern;
  • Snapshots in "Versions";
  • triggers "Superseded by a release for (days)", "Unlisted for (days)" and "More than N versions".
• Image rules: tag pattern and except-pattern; protected tags take globs.
• Runs panel: shows the schedule (0 3 * * * (Europe/Berlin); a run stops after 1 hour) and, after a run that hit its limit, "Left for the next run".
• Preview: lists a scheduled draft's next runs. "Why" explains the new reasons ("superseded by 2.3.0 for over 14 days", "number 5 of its package, beyond the newest 3").
• Fields the screen does not know still survive a save, as before.

Docs

• docs/retention.md: now the full retention reference (title "Retention" in SUMMARY/README):
  • the envelope;
  • when a policy runs: interval, cron with an examples table, DST behaviour, Run now, max_run_seconds, missed times;
  • package rules: filters, protections, every trigger, and examples (CI prereleases, PR builds, unlist-then-delete, a selector-scoped night run);
  • proxy rules;
  • formats and their hooks;
  • raw and image rules with examples;
  • preview, runs and audit;
  • the existing size-budget text as a section.
• docs/docker.md: tag patterns, glob protected tags, schedule fields.
• docs/raw.md: schedule fields, and links updated.
• docs/npm.md: links updated.
• docs/format-plugins.md: a new "Retention" section with the hooks and what Maven and apt need. The "Prerelease is contains a -" limitation now applies only to the search filter.
• Changelog: docs/changes/2.2.0/185.feature.md and 185.upgrade.md (the unlisted_at default, and the 409 on selector deletion).

Tests

Unit tests (forge-core):

• schedule.rs:
  • DST, deterministic: 0 3 * * * in Europe/Berlin across the 2026-03-29 spring change and the 2026-10-25 autumn change stays at 03:00 local (02:00 → 01:00 → 02:00 UTC);
  • 30 2 * * * in the skipped hour runs once, at 03:00 CEST;
  • 30 2 * * * in the repeated hour runs once, at its first occurrence;
  • UTC default and strictly-later search;
  • validation;
  • reading a stored policy;
  • the RunClock first-step rule.
• evaluate.rs, one or more tests per new trigger and filter:
  • supersession: days threshold, the first-published newer release, an unlisted release never supersedes, superseded_by;
  • unlisted_for_days, including an unknown unlisting time, and the two-step unlist-then-delete policy;
  • max_versions_per_package: pure count with ranks, unlisted versions counted, counting within the rule's pool and ANDed with age;
  • version include/exclude globs with alternation, ignoring case;
  • selectors, and an unresolved one admitting nothing;
  • protections win over every new trigger, in the same rule and from another rule;
  • the Maven-like hooks test above;
  • proxy count and filters.
• model.rs: the new fields round-trip, absent fields are not serialized, unknown fields are refused, every new validation error, hosted-only triggers refused on proxies, schedule validation.
• images.rs: tag patterns and keep-N over the pool, glob protected tags in rules and the budget, validation.
• files.rs: schedule validation.

Plugin contract (forge-plugin, retention_hooks_reach_the_host): the defaults, and a plugin's own hooks forwarded through PluginFormat.

DB (crates/db/tests):

• unlisting_records_when_on_every_path: insert unlisted, client unlist/relist, re-unlist keeps the time, backdating, retention's update, candidates carry unlisted_at.
• a_saved_schedule_sets_the_next_run_and_a_stale_run_keeps_it: the guard in finish_run/fail_run.
• The summary's deferred serialization.
• repositories_using_selector.
• migrations.rs versions_unlisted_before_the_upgrade_count_from_their_last_change: migrates to 0007, seeds rows, migrates on, checks the default and then the trigger.

Core integration (crates/core/tests):

• retention_apply a_run_past_its_deadline_stops_and_the_next_one_resumes: three packages with the deadline already passed take three runs, each deleting one and deferring the rest; with a far deadline, one run applies everything.
• retention_worker:
  • a scheduled policy's next_run_at is the next 03:00 Berlin;
  • unlisted_for_days and supersession end to end, with the audit reasons;
  • a missing selector fails the run without mutating, and after the selector is created the rule deletes only what it admits.

Server (crates/server/tests/admin_retention.rs):

• schedules_triggers_and_selectors_are_validated_and_stored:
  • missing selector 400;
  • fields stored;
  • next run = next 03:00 Berlin;
  • Run now makes the policy due;
  • schedule → interval pulls the next run in;
  • selector deletion 409;
  • 400/422 for each invalid value or typo.
• preview_shows_the_new_triggers_decisions_and_the_next_runs: supersession and unlisted decisions with their reasons, 5 next_runs at 02:30/03:00 Berlin, the count trigger with a protection from another rule.
• image_rules_take_tag_patterns_and_glob_protected_tags.

Web:

• RetentionPolicyForm.test.ts: schedule, time limit and new rule fields round-trip; interval switch; cron and time-limit checks; new reasons; describeSchedule.
• retention.test.tsx: a scheduled policy loads into the form; the Runs panel shows the schedule and deferred count; the selector picker lists the selectors; editing the new fields sends them in the preview; the preview shows reasons and next runs; switching back to the interval.

Checks run

• Environment: shared machine, CARGO_PROFILE_{DEV,TEST}_DEBUG=line-tables-only, SQLX_OFFLINE=true, CARGO_INCREMENTAL=0. The DB-backed tests ran against a Postgres 18 container (mirror.gcr.io/library/postgres:18) through FORGE_TEST_DATABASE_URL, on a fresh database per run.
• cargo test -p forge-core --lib: 349 passed (114 of them retention).
• cargo test -p forge-core --test retention_apply --test retention_worker: 9 + 22 passed.
• cargo test -p forge-db --test retention: 19 passed. --test migrations: 4 passed.
• cargo test -p forge-plugin --lib: 10 passed.
• cargo test -p forge-server --test admin_retention: 17 passed.
• cargo test -p forge-server --test raw retention: 1 passed. --test registry retention: 5 passed. These are the image/file apply callers whose signature changed.
• cargo test -p forge-server --test admin_rbac: 15 passed (content selectors).
• cargo clippy -p forge-plugin-api -p forge-plugin -p forge-core -p forge-db -p forge-server --all-targets -- -D warnings: clean.
• cargo fmt --all -- --check: clean.
• cargo xtask changelog --check: "33 changelog fragment(s) under docs/changes, all for milestones after 2.1.0."
• cargo xtask docs --check: "16 page(s) of the site, every link resolves."
• Web: pnpm install --frozen-lockfile && pnpm run tsc && pnpm run test && pnpm run build: tsc clean, 40 files / 538 tests passed, build ok.
• Not run: cargo sqlx prepare --check (no sqlx-cli here; the .sqlx entries were generated as described under Schema).

Notes

• Crates:
  • croner 4.0.1 (MIT, hexagon/croner-rust, ~12M downloads, actively released). Pinned in Cargo.lock to 4.0.1, which is about a month old; 4.0.2 is 10 days old and only adds exports.
    • It parses 5-field patterns, and seconds and years can be disallowed.
    • It works on chrono DateTime<Tz>, the workspace's time type.
    • It documents and tests Vixie-cron/OCPS behaviour across DST gaps and overlaps; 4.0 fixed overlap ordering.
    • The cron crate needs 6–7 fields (seconds) and has no documented DST policy. saffron is unmaintained.
  • chrono-tz 0.10.4 (chronotope, MIT/Apache-2.0): the IANA database compiled in, so a zone resolves the same in the release image without /usr/share/zoneinfo.
  • jiff (already in the lock through serde_with) would also do, but everything else, the DB rows included, is chrono.
• Decisions where the issue left room:
  • "Same or higher base version" is implemented as a listed release that compare_versions orders after the prerelease. For SemVer, NuGet and Maven ordering this is equivalent, and it needs no base-version hook. Snapshots count as superseded like prereleases.
  • max_versions_per_package counts listed and unlisted versions (what is stored). It is also allowed on proxy rules.
  • selector references a content selector by name. Names cannot change. A missing selector fails closed, and deletion is refused while a rule uses it.
  • A schedule's next run is at least one minute after the last start (the worker's floor).
  • A time missed while no worker ran (every instance down, or runs switched off) is made up once when the worker is back. It is documented, and max_run_seconds keeps such a run short.
• Acceptance "max_run_seconds stops and resumes a run": tested deterministically at the apply layer, which is where the stop happens. The worker computes the deadline from an Instant at claim time. The worker-level test only checks that a scheduled policy with a limit completes with deferred: 0. A worker test that actually stops would need a clock hook, which I did not add. Files and images share the same RunClock (unit-tested), without a DB-level deadline test of their own.
• Open questions in the issue (out of scope): shared policy templates (Retention: default policies per repository type, managed in the admin UI and applied to new repositories #418 was closed as not planned; per-type defaults would reuse this machinery) and a quota that blocks pushes (Retention: a size budget per repository that deletes the least recently used content first #417 only evicts).
• Search prerelease filter: the host's search filter (prerelease) still reads the -. Only retention asks version_class, and docs/format-plugins.md says so.
• Migration number: 0011, as assigned. It only adds a column and a trigger, so it applies in any order with the open PRs' 0008/0009.

---

Generated by Claude Code

[schmitch]

The merge queue removed this PR because the test job failed in run 41213497654. The failure is not this PR's:

• What failed: one test, dek_revocation_is_checked_at_commit_time_against_writers in crates/db/tests/encryption.rs:606. It expects set_upstream_password to fail with KeyRevoked when DEK v2 is revoked while the write waits for the row lock, and here the write succeeded. The other 12 encryption tests and the other 121 test binaries passed.
• This PR's diff never reaches that path. It changes no secret, encryption or crypto code. Its migration 0011 only adds a column and a trigger on component, while the test writes repository.upstream_password_enc and locks encryption_key.
• The same code passed this test. The next queue entry, Import: copy an npm registry's packages (Nexus, Verdaccio, another Forge) into a hosted npm repository #718's run 41213499599, was built on this PR's queue merge commit b2e11de8 and passed the full test job, including this test.

So this is a timing-dependent test that failed once, not something this PR can fix. No change is pushed for it, and the PR can go back into the queue as it is.

---

Generated by Claude Code