[schmitch]

Fixes #643

An import now copies an npm registry's packages into a hosted npm repository. Building on #642 (#712), it reads the source through npm's own protocol, so the source can be Nexus' …/repository/<name>, Verdaccio, Artifactory or another Forge. The npm crate decides where to ask and how to read the answers: a new forge_format_npm::import, built on the packument reader npm proxies already use (upstream::read). The server's import worker sends the requests through the shared UpstreamHttp client and copies through the hosted write path.

Source: an npm registry

• Names. npm has no listing of every package. Tried against real registries, the preflight picks the first of these that lists packages, and stores its choice in scan_listing:

  • /-/v1/search with an empty text, paged with from/size (100 a page). Another Forge lists every package this way. The walk ends at an empty page, and stops with an error if the search returns the same page again (it ignores from).
  • /-/all, read as a stream that keeps only the keys. Two registries need it:
    • Nexus 3.70.1: the search with an empty text finds nothing (total: 0). With any text it names a scoped package without its scope (core for @acme/core) and gives no scope field. /-/all lists every package by its full name.
    • Verdaccio 6.11.1: an empty text is refused (400 ERR_TEXT_MISSING). /-/all is served.
  • Neither: the preflight answers 400 and asks for names.

• Named packages or scopes. names takes packages (left-pad, @acme/ui) and scopes (@acme, @acme/ or @acme/*, stored as @acme). A scope's packages come from /-/all where the registry serves it, otherwise from the search for @acme. Only names in the scope are kept, and the next page starts after the last name returned.

• Missing packages. A named package or scope the registry does not have is an item that fails, saying so. So is a package the search or /-/all lists but has no packument for. This makes Nexus' scope-less search names show up instead of being dropped silently.

• Versions. Each package's full packument (GET /<name>, or @scope%2fname) is read as a stream. A new Keep::Release keeps only name, version, dist and deprecated of every version (Select::every). From it the import takes:

  • each version's dist.integrity (its SHA-512) and dist.shasum;
  • its publish time from time;
  • its deprecated message (an empty message is none);
  • the dist-tags.

  A full packument larger than proxy.max_metadata_bytes is read abbreviated instead, without publish times. A packument with no versions counts as missing.

• Copy.

  • Tarball URL. It is built from the import's URL as <url>/<name>/-/<file>, never taken from dist.tarball's host. <file> is the last segment of dist.tarball after /-/, or npm's <name>-<version>.tgz.
  • Verification. The tarball is checked against the SHA-512 in dist.integrity. For an old package listed without one, it is checked against dist.integrity/dist.shasum with the npm crate's Digests::verify.
  • Identity. New ImportOptions::expected, set for npm only: the tarball's package.json must be the package and version the packument names. Otherwise it is refused before anything is written.
  • Skip without download. A version Forge holds with the listed SHA-512 is identical without a download. A dry run decides from the checksums alone.
  • Publish times come from time through ImportOptions::published_at, as for NuGet.

dist-tags and deprecations

• Deprecations travel with their version item. They are applied to a version that is imported, replaced or identical (the dry run reports "would be deprecated: …").

• Dist-tags are items of their own: kind tag, name the package, version the tag. They are copied after every version, once the version they name is in Forge. A tag whose version failed fails with "version X is not in Forge".

• Conflicts. A dist-tag Forge points at another version, or a deprecation Forge gives otherwise, follows on_conflict:

  • skip: a conflict item with the reason;
  • overwrite: the source's value;
  • fail: the import fails.

  Refinement of the issue's rule, a judgment call to review: where Forge holds exactly what an earlier, non-dry-run import of the same repository set, the import follows the source without a conflict. The new import_dao::earlier_source reads the newest earlier import's item for the same kind, name and version that ended imported, replaced or identical. Without this, a cutover re-run with skip would report every latest the source moved as a conflict and leave Forge's latest stale. A tag or deprecation changed in Forge since then is still a conflict.

Schema

• Migration 0009_npm_import.sql:

  • repository_import.scan_listing also takes all;
  • repository_import_item.kind also takes tag.

  Nothing else changes, and no upgrade note is needed.

• New .sqlx entries: component_dao::deprecation, import_dao::earlier_source, and two test queries.

API / UI

• POST /admin/repositories/{name}/imports:
  • accepts npm hosted repositories (FORMATS is now nuget, npm, docker);
  • accepts scopes in names;
  • in the previous request form (url a Nexus server + repository), reads <url>/repository/<name> for npm.
• npm preflight:
  • GET /-/ping: a refusal of the credentials or a 5xx is an error, and a registry that does not serve the ping is let through;
  • then the listing choice above;
  • source_server comes from the ping's Server header;
  • source_total is the search's total, or how many names /-/all lists.
• Import view: the new listing field (search, catalog, all, names).
• Credentials: Basic (username + password or token), as for the other formats. A password without a username (a Bearer token) is not supported.
• Web UI (Admin → Repositories → npm hosted → Import):
  • IMPORT_FORMATS now includes npm, so npm hosted repositories link to the import screen;
  • npm wording for the URL (registry URL, Nexus path), the names ("Packages or scopes") and the conflict policy;
  • "The registry's /-/all lists N packages" or "The registry's search found N packages";
  • tag items are shown as <package> dist-tag <tag>.

Docs

• docs/import.md:
  • the npm row in the formats table;
  • url/names/on_conflict for npm;
  • the npm preflight (search, then /-/all, with what Nexus and Verdaccio answer), the scan, and the tarball copy (import URL, integrity/shasum, identity);
  • publish times;
  • a new section "npm: dist-tags and deprecations" with the earlier-import rule.
• docs/nexus-import.md: where Nexus serves npm, its search's behaviour, /-/all, and dist.tarball naming the configured Base URL.
• docs/npm.md: a new section "Importing another registry".
• README.md / docs/README.md: mention npm and Verdaccio.
• Changelog fragment: docs/changes/2.2.0/643.feature.md.
• CI: the e2e job pre-pulls mirror.gcr.io/verdaccio/verdaccio:6.11.1.

Tests

• crates/server/tests/import.rs. A mock npm registry joins the existing mock source (axum, Basic auth, Server: Nexus/…). Its search mode (lists everything like Forge / finds nothing and drops scopes like Nexus / refuses an empty text like Verdaccio) and /-/all can be switched. Its packuments name tarballs on an unreachable host (npm.internal.invalid). 4 new tests:
  • npm_import_keeps_bytes_times_tags_and_deprecations_and_a_second_run_copies_only_what_is_new:
    • first run: search paging; bytes, time → creation time, integrity, deprecation and dist-tags in Forge's packument;
    • second run: only the new version is downloaded (the shasum-only package is downloaded again to compare), latest is moved (replaced, "moved from 1.1.0 to 1.2.0"), and a deprecation added at the source arrives;
    • a tag moved and a deprecation lifted in Forge are conflicts with skip, and take the source's with overwrite.
  • npm_import_walks_all_where_the_search_lists_nothing_and_checks_every_tarball:
    • listing all with scoped packages;
    • a corrupt tarball fails on its SHA-512, a shasum-only one on dist.shasum, and the tag of a failed version fails;
    • a retry imports the versions, then the tag.
  • npm_preflight_names_scopes_conflicts_and_the_dry_run:
    • refusals: no listing, wrong credentials, an invalid name;
    • named packages and scopes, with the scope found through the search;
    • missing package and missing scope items;
    • the dry run downloads nothing;
    • a republished version is a conflict in the dry run (from checksums) and in the run, and is replaced with overwrite.
  • npm_import_names_a_nexus_repository_in_the_previous_request_form: the previous request form, and a tarball whose package.json names another version is refused ("the package is beta 9.9.9, not beta 1.0.0").
• Unit tests:
  • crates/formats/npm/src/import.rs: URLs, scopes, search names with and without scope, /-/all keys, and the packument read into releases (file names, SHA-512, shasum, deprecated, tags of versions it has);
  • crates/server/src/import/mod.rs: Protocol, nexus_source_url for npm, Listing::All, and how an item stores its source.
• e2e (crates/e2e/tests/import.rs), run locally against real containers. Both npm tests seed the registry over HTTP the way npm does it: publish, dist-tag add, and deprecate (a packument PUT).
  • npm_import_from_a_real_nexus (mirror.gcr.io/sonatype/nexus3:3.70.1):
    • seeds alpha 1.0.0 (deprecated), 1.1.0 and 2.0.0-beta.1 (next), @acme/ui 1.0.0 (stable) and 1.0.1, and @acme/core 0.1.0;
    • Forge walks /-/all;
    • 11 items are imported, and bytes, integrity, dist-tags, deprecation and publish times equal Nexus' packuments;
    • npm install alpha @acme/ui@stable @acme/core from Forge succeeds;
    • after publishing alpha 1.2.0, a second run gives 1 imported, 1 replaced (latest) and 10 identical, with bytes_copied equal to the new tarball's size;
    • npm install alpha@next gives 2.0.0-beta.1.
  • npm_import_from_verdaccio (mirror.gcr.io/verdaccio/verdaccio:6.11.1):
    • the config has no uplink, and @acme/* is readable only by authenticated users, so the import has to send its credentials;
    • Forge walks /-/all and imports 11 items, with the same checks and npm install;
    • a named @acme import comes out all identical with 0 bytes copied.
  • npm install: runs when npm is on the PATH and prints that it was skipped otherwise, like the dotnet smoke test. It ran here (node 22.23.4, npm 10.9.9).
  • Nexus tests take turns: a static tokio mutex is now held by both Nexus tests. This is the one-line change to the existing import_from_a_real_nexus.
• Commands run, and results.

  • Environment: a shared build machine, with per-package profile overrides that keep this branch's artifacts apart from other checkouts in the shared target dir, SQLX_OFFLINE=true (except prepare), CARGO_PROFILE_{DEV,TEST}_DEBUG=line-tables-only and CARGO_INCREMENTAL=0.

  • cargo test -p forge-format-npm: 61 + 15 passed.

  • cargo test -p forge-server --lib: 253 passed.

  • cargo test -p forge-server --test import: 16 passed (the 12 existing and the 4 new).

  • cargo test -p forge-server --test npm: 16 passed (the proxy reads the packument reader that changed).

  • cargo test -p forge-db --test migrations: 3 passed (including the 0006 → 0007 rename test, now migrating on through 0009). Re-run after the renumber: 3 passed.

  • cargo test -p forge-e2e --features forge-e2e --test import npm_import -- --nocapture: 2 passed, on the final code.

  • cargo test -p forge-e2e --features forge-e2e --test import: all 5 passed:

    • import_from_a_real_nexus (NuGet, and Docker with 105 images);
    • import_from_a_distribution_registry;
    • import_from_another_forge;
    • the 2 npm tests.

    This run was on the code before cargo fmt, the Verdaccio tag pin (6 → 6.11.1, the same image ID) and comment-only edits, with the same codegen override but incremental compilation on.

  • cargo clippy -p forge-format-npm -p forge-core -p forge-db -p forge-server -p forge-e2e --all-targets --features forge-e2e/forge-e2e -- -D warnings: clean, also re-run after the last comment edits.

  • cargo fmt --all -- --check: clean.

  • cargo xtask changelog --check: "32 changelog fragment(s) … all for milestones after 2.1.0".

  • cargo xtask docs --check: "16 page(s) of the site, every link resolves".

  • cargo sqlx prepare --workspace -- --all-targets --all-features against a Postgres 18 migrated through this migration: 4 new .sqlx files, none removed.

  • Web: pnpm install --frozen-lockfile && pnpm run tsc (clean), pnpm run test (40 files, 535 tests passed, including the new "offers the import on an npm hosted repository, with packages and scopes"), and pnpm run build (ok).

Notes

• Migration number: 0009, because Auth: device authorization grant (RFC 8628) with a /device approval page #717 (the device authorization grant, Device authorization grant (RFC 8628): device and token endpoints and a /device approval page #649) takes 0008. This migration only replaces two CHECK constraints, and the two apply in either order: 0001–0009, with Auth: device authorization grant (RFC 8628) with a /device approval page #717's 0008, apply cleanly in sequence on a fresh Postgres 18. If this merges before Auth: device authorization grant (RFC 8628) with a /device approval page #717, main has a gap at 0008 until Auth: device authorization grant (RFC 8628) with a /device approval page #717 lands; a database migrated in between then applies 0008 after 0009, which these two independent migrations allow.

• Why not the generic upstream conversation (Format::upstream)? That is how NuGet is read in Import through the format's own protocol (a NuGet feed, a registry's catalog) instead of Nexus' REST API, named neutrally #642, but for npm:

  • it would cost a packument fetch per operation and per version (Versions, Dating, Tags, then Package once per version, each re-reading the packument);
  • VersionMetadata carries no deprecation;
  • nothing in it can walk /-/all.

  So the npm crate gets a pure module of its own (URLs and parsing). The server sends one full packument request per package. UpstreamOp::Packages for npm still answers Search(None), with an updated comment.

• What is stored as a version's manifest: the tarball's package.json with the computed dist, as when a proxy downloads a tarball. The fields only the source's version object carries (_npmUser, gitHead, maintainers) are not copied. The publish time, deprecation and dist-tags are.

• Not covered:

  • a token-only source (npmjs.com-style _authToken without a username);
  • Nexus releases other than 3.70.1 (3.88's SQL search is untested; if its search lists every package with an empty text, a scoped name it gives without its scope would show up as a failed item, not be lost);
  • the abbreviated-packument fallback for packuments over proxy.max_metadata_bytes (no test);
  • Forge to Forge for npm (Verdaccio is the second registry, as the issue allows).

---

Generated by Claude Code