Skip to content

apt (Debian/Ubuntu) repositories: hosted and proxy, with signed indexes - #725

Open
schmitch wants to merge 4 commits into
mainfrom
schmitch/apt-repositories-176
Open

schmitch wants to merge 4 commits into
mainfrom
schmitch/apt-repositories-176

Conversation

@schmitch

@schmitch schmitch commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Fixes #176

apt repositories, like Nexus' apt-hosted and apt-proxy, served at /repository/<name>/ (the Nexus path, so a migrated sources.list only changes the host). User guide: docs/apt.md.

What it does

Hosted

  • One distribution per repository (the new distribution option). The component comes from the package's Section (contrib/net → contrib, else main); architectures come from the packages.
  • Uploads: Nexus' curl -H "Content-Type: multipart/form-data" --data-binary @pkg.deb, a form, or PUT. DELETE of a pool file deletes the version.
  • Identity: <name>:<arch> (hello:amd64) as the package id, so each architecture is its own package. Versions are ordered by dpkg's rules (epoch, ~, revision). Redeploy policy, selectors, retention and trusted publishing work as for the other package formats.
  • Indexes: dists/<dist>/{InRelease,Release,Release.gpg}, Packages{,.gz,.xz} with by-hash/SHA256 (Acquire-By-Hash: yes), and pool/<comp>/<prefix>/<name>/<file>.deb. They are made when a client asks and the packages, options or keys changed, stored in the CAS, and replaced in one transaction. by-hash copies stay a day after a newer index stops naming them.
  • Signing: Ed25519 keys made with rPGP, generated at creation or imported, sealed with [crypto]. Every key signs InRelease and Release.gpg, so a rotation needs no downtime. key.asc / key.gpg hold every key; keys are managed under /admin/repositories/{name}/signing-keys.

Proxy

  • The upstream's InRelease/Release/Packages are served unchanged and re-validated after metadata_ttl_seconds.
  • pool/ and by-hash/ files are immutable: cached on first pull, never re-validated. Pool files are checked against size and SHA-256 from the upstream's Packages index when it lists them; a mismatch is not cached (502).
  • Stale serving while the upstream is down, and the negative cache, work as for the other proxies.

Web UI

  • The format's options in the repository form.
  • A signing keys panel: list, generate, import, delete (never the last).
  • Connect snippets: key download, sources.list with signed-by, auth.conf.d, upload.
  • Browse by distribution, component and architecture (from Release and Packages).

Contract additions (forge-plugin-api)

  • Descriptor.options + FormatPlugin::check_option + Repository::options / Remote::options; admin API options on create/update; GET /admin/formats lists them.
  • Descriptor.indexes + Query::Index + FormatPlugin::index (IndexInput → IndexFiles with SignedCopys).
  • Package formats may use file / fetch-file / serve-file. FormatPlugin::file_package lets selectors decide on a file's package.
  • UpstreamAnswer::FileAt(FileLocation): immutable, plus size/SHA-256 checked before caching.
  • Descriptor.prereleases (Dash | None) for the host's search filter, so the web UI lists every apt version (1.0-1). Retention uses Retention: cron schedules in a time zone, superseded/unlisted/count triggers, version and tag patterns, and format retention hooks #721's version_class, which apt answers with Release for every version.
  • The issue's "listings are bounded" and "proxy reads whole documents" limits are avoided rather than lifted. Hosted indexes are made host-side from the database, and proxies stream through the file store.

For review

  • Advisories: rPGP links rsa 0.9 and crypto-bigint 0.5. .cargo/audit.toml now ignores RUSTSEC-2026-0387, and the RUSTSEC-2023-0071 rationale is rewritten, because rsa is no longer dev-only. Mitigation: Forge only generates and imports Ed25519 keys; RSA/ECDSA imports are refused, so those code paths never run with a secret key. The catch is that existing RSA archive keys can't be imported.
  • Groups are not included. The issue says they can come after hosted and proxy, and no acceptance criterion covers them. They are listed in the docs and in a limitation changelog fragment. If you'd rather keep apt (Debian/Ubuntu) repositories: hosted, proxy and group, Nexus-style #176 open for them, drop the Fixes line or open a follow-up issue.
  • Retention doesn't trim an apt proxy's cached files yet. Documented as a limitation.
  • New CI job apt-e2e. Added to release.yml's REQUIRED_JOBS, ci-changes.sh and DEVELOPMENT.md. It also needs adding to the branch protection's required checks, which only a maintainer can do.
  • A migration (0012_apt.sql, after main's 0009–0011) adds raw_file.immutable, repository_signing_key and repository_index. The DEK rotation sweep and revocation scan include the sealed keys.

Tests

  • crates/formats/apt: 36 unit tests (dpkg version order, control parsing, .deb in every compression, traversal, index rendering, proxy lookups).
  • crates/server/tests/apt.rs:
    • uploads, redeploy, signed indexes verified with rPGP, by-hash, pool, delete, traversal;
    • selectors, key rotation and import, options;
    • a proxy of a mock mirror, including digest refusal and serving while down.
  • .github/scripts/apt-e2e.sh: real apt-get in debian:bookworm. signed-by + auth.conf.d, update with no W:/E:, install then upgrade, gpgv across a key rotation, and a proxy of a signed mock mirror installing again while the mirror is down. Passed locally.
  • Locally green on the merge with current main: clippy (-D warnings), the db/core/plugin/format suites, the server lib and apt/raw/import/retention/browse suites, web tsc + 561 vitest tests, cargo audit, and the xtask changelog/docs checks.

Generated by Claude Code

Debian packages for apt, like Nexus' apt-hosted and apt-proxy, served at
/repository/<name>/.

- crates/formats/apt: the apt format plugin. Packages are
  <name>:<architecture>, versions are ordered as dpkg orders them, a
  .deb's control file (gzip, xz, zstd or plain control member) is read
  within bounds, the Section names the component, and the pool layout,
  Packages and Release are Debian's. Uploads: Nexus' raw-body POST, a
  form, or a PUT; a DELETE of a pool file deletes the version. Proxies
  pass the upstream's files on unchanged and check packages against the
  upstream's Packages index.
- The plugin contract grows: format options (apt's distribution) in the
  descriptor, checked by the plugin and set through the admin API; host-
  kept signed indexes (Query::Index, FormatPlugin::index), made when a
  client asks and the packages, options or keys changed, replaced in one
  transaction, by-hash copies kept a day; files of package formats
  (file, fetch-file, serve-file, file_package for selectors); file-at
  locations (immutable files, a size and digest checked before caching);
  and Prereleases::None, so apt's 1.0-1 is a release in browse and
  retention.
- Signing keys per repository (repository_signing_key): Ed25519 keys made
  with rPGP, generated at creation or imported, sealed with the secret
  encryption, every key signing InRelease and Release.gpg so a key is
  rotated without warnings; /admin/repositories/{name}/signing-keys.
- Web UI: the format's options in the repository form, a signing keys
  panel, the sources.list, key and auth.conf.d snippets, and the
  distribution browsed by component and architecture.
- Tests: the plugin's units (version order, traversal, indexes), server
  integration tests, and .github/scripts/apt-e2e.sh with the real apt-get
  in debian:bookworm and gpgv (CI job apt-e2e).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbxCkG53eCuDWqGGFy
@schmitch
schmitch enabled auto-merge October 31, 2026 17:20
@schmitch
schmitch disabled auto-merge October 31, 2026 17:20
schmitch and others added 3 commits October 31, 2026 17:21
main merged its own 0009 (#718), so the merge had two migrations of
version 9 and every job that migrates a database failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbxCkG53eCuDWqGGFy
…e apt branch

- The apt migration is 0012 now, after main's 0010_raw_import and
  0011_retention_unlisted_at.
- Retention asks the plugin's `version_class` (#721): apt answers
  `Release` for every version, and the host's `Format::is_prerelease` of
  this branch is dropped. `Descriptor.prereleases` stays for the search
  filter, which still reads the `-` (browse lists every apt version).
- The raw import's file store write sets `immutable: false`.
- docs: the Signed indexes, Format options and Retention sections side by
  side; apt's package ids already count keep-N per architecture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbxCkG53eCuDWqGGFy

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apt (Debian/Ubuntu) repositories: hosted, proxy and group, Nexus-style

1 participant