[schmitch]

Fixes #656

An import now copies a raw repository's files into a hosted raw repository. Raw has no protocol beyond a GET of a path and no standard listing, so the import reads one of three sources, through a new crates/server/src/import/raw.rs (RawSource), and stores each file through a new import path of the file store (files::store::import). It builds on #642 (#712)'s source interface: the leased worker, the scan cursor, resume and retry, the conflict policies and the dry run are the existing ones.

Sources

• Nexus' asset listing. GET <nexus>/service/rest/v1/assets?repository=<name>, paged with continuationToken, for a url of a Nexus repository's shape (<nexus>/repository/<name>, a context path allowed).
  • Each asset gives the path, size, content type, SHA-1, SHA-256, SHA-512, blobCreated (else lastModified), lastDownloaded and the uploader.
  • Nexus 3.70's repeated "blobCreated": null, "blobCreated": "<time>" is read with the lenient object reader the pre-Import through the format's own protocol (a NuGet feed, a registry's catalog) instead of Nexus' REST API, named neutrally #642 Nexus client had.
  • A leading / of an old Nexus' raw path goes.
• Another Forge's JSON listing. GET <url>/<dir>/?format=json&limit=100&after=… with Accept: application/json; each file's size, content type, SHA-256, last_modified and last_downloaded.
  • Walked a directory at a time, depth first in byte order. The scan's cursor is {"dir", "after"}: a page's files up to its first subdirectory are recorded, the walk goes down into it, and at a directory's end it goes on in the parent after it. No stack, so a stopped walk resumes in the directory it was in.
  • A subdirectory whose name Forge refuses is not walked and fails as an item. A page that would repeat (the same directory and after) stops the scan with an error.
• Named paths, for a server without a listing (a plain web server).
  • HEAD of each gives size, Content-Type and Last-Modified (a server answering 405/501 gives nothing, the download tells).
  • <path>.sha256, where served, gives the SHA-256: the first word, 64 hex digits (<hex> as Nexus writes it, or <hex> <name> as sha256sum and Forge do). A sidecar that holds no SHA-256 counts as none.
  • A named file the server does not have (404/410) is an item that fails. It is looked for again when the failed items are retried.

Preflight. Without names, the preflight tries another Forge's listing of url (?format=json&limit=1), then Nexus' asset listing, and records which in scan_listing (directories or assets). A source with neither is refused with 400: name the paths. With names, the first named file is asked for (HEAD), so wrong credentials or an unreachable server are reported before anything is stored. A read-only raw repository is refused. The request form of a Nexus server plus repository becomes <url>/repository/<name> for raw.

Copy

• Paths Forge refuses. An empty, . or .. segment, a backslash, a control character, a segment over 255 bytes, or more than 1,024 bytes in all.

  • The item fails with the reason: "Forge refuses the path: backslashes are not allowed".
  • It is decided at the scan, needs no request, and does not count towards the 25 failures in a row that stop an import (neither does a missing file). The import goes on.
  • The item's name is the path, with control characters escaped (Postgres text holds no NUL) and cut at 1,024 bytes.

• Files Forge already holds. Compared through the SHA-256 the source lists:

  • the same SHA-256 is identical, without a download;
  • another one follows on_conflict without a download: skip reports a conflict, overwrite replaces, fail fails the import.
  • Only a named file without a sidecar is downloaded to be compared.

• Download and checks. A file is downloaded from the import's URL and its path (segments percent-encoded), never from Nexus' downloadUrl. It is staged and checked against the listed size and every listed checksum (SHA-512, SHA-256, SHA-1; the message names a mismatching .sha256 sidecar as such).

• Store. files::store::import, the upload path's rules:

  • the path rules;
  • the write selectors of the administrator who started the import, and the delete level for a replace;
  • the upload limit (also checked against the listed size before a download);
  • the read-only policy;
  • CAS dedup;
  • the trusted-publishing ledger (a no-op for a user);
  • a forge::audit line action = "import" with the path, SHA-256, outcome, administrator, source and uploader.

  overwrite replaces even where the repository refuses redeploys, as for NuGet imports. The upload's staging is split into store::stage (staged and hashed) and the commit, so a file is checked before it reaches the object store.

• Content type. The source's (Nexus' contentType, Forge's content_type, else the response's Content-Type) is kept as an upload's declared type is (store::content_type): "bytes only" types give way to the file name's.

• Times.

  • blobCreated / last_modified / Last-Modified becomes the file's created_at and updated_at (the upload time Last-Modified, older_than_days and keep_newest_per_directory read).
  • lastDownloaded / last_downloaded becomes last_downloaded_at; without one, the import time, as Import through the format's own protocol (a NuGet feed, a registry's catalog) instead of Nexus' REST API, named neutrally #642 has it for packages.
  • A time at the epoch (a file unpacked without times) counts as none.
  • New raw_dao::set_imported_times and raw_dao::lock_updated_at (runtime queries, no new .sqlx files).

• Uploader. Nexus' uploader goes into the item's uploader and the audit line.

Schema

• Migration 0010_raw_import.sql. It only replaces two CHECK constraints:

  • repository_import.scan_listing also takes assets and directories;
  • repository_import_item.kind also takes file (its name is the path, its version empty).

  Import: copy an npm registry's packages (Nexus, Verdaccio, another Forge) into a hosted npm repository #718's 0009_npm_import.sql, on main now, replaces the same two constraints; 0010 runs after it, and its lists keep 0009's values (all, tag) beside raw's. No upgrade note is needed.

API / UI

• POST /admin/repositories/{name}/imports:
  • accepts raw hosted repositories (FORMATS is now nuget, npm, docker, raw);
  • url is the raw repository, names are file paths below it (normalized; a path Forge refuses or a directory is a 400; duplicates go).
• Import view: the listing field (Import: copy an npm registry's packages (Nexus, Verdaccio, another Forge) into a hosted npm repository #718 adds it too) also takes assets and directories. Items of a raw import have kind file.
• Web UI (Admin → Repositories → raw hosted → Import):
  • IMPORT_FORMATS includes raw, so raw hosted repositories link to the import screen;
  • raw wording for the URL ("Repository URL"), the paths field and the conflict policy;
  • paths are split one per line only, so a path with spaces or commas stays whole (the other formats still split on spaces and commas);
  • the card names the listing ("Nexus' asset listing names the files.").

Docs

• docs/import.md:
  • the raw row of the formats table;
  • url / names / on_conflict for raw;
  • the raw preflight and the raw dry run;
  • a new section "Raw files": the three sources, paths, download and checks, files Forge has, content type, times, uploader.
• docs/nexus-import.md: a "Raw" section: the asset listing (REST API), 10 assets a page, downloadUrl not used, and the privileges nx-repository-view-raw-<name>-browse and -read.
• docs/raw.md: a new section "Importing another server's repository", and the "No import" limitation goes.
• README.md / docs/README.md mention raw imports.
• Changelog fragment: docs/changes/2.2.0/656.feature.md.
• CI: the e2e job pre-pulls mirror.gcr.io/library/nginx:1.27.5-alpine.

Tests

• crates/server/tests/raw_import.rs (new, 4 tests) against a mock source (axum, Basic auth) that serves Nexus' asset listing (2 assets a page, blobCreated written as Nexus 3.70 does), a Forge's JSON listing (2 entries a page) and a plain web server with sidecars, with download and listing counters:
  • raw_import_from_nexus_keeps_bytes_types_times_and_uploaders:
    • paged asset listing; listing: assets, source_server Nexus';
    • bytes, content type, .sha256 sidecar; created_at/updated_at = blobCreated, last_downloaded_at = lastDownloaded or the import time; uploader on the item;
    • five refused paths (backslash, //, .., a control character, a 300-byte segment) each fail with the reason, and the import completes;
    • a file whose bytes do not match fails on its SHA-512;
    • a retention preview with not_downloaded_for_days: 30 selects only the file Nexus last served in 2022, not the ones downloaded "at import";
    • second run: only the new file is downloaded (counters), a changed file is a conflict without a download;
    • overwrite in the previous request form (Nexus server + repository) replaces it with Nexus' upload time;
    • fail fails the import, naming the path.
  • raw_dry_run_reports_files_sizes_and_conflicts_and_a_walk_resumes:
    • a Forge listing with nested directories and paging, one file identical and one conflicting in Forge;
    • the dry run reports 6 planned with their planned_size, 1 identical and 1 conflict, writes nothing and downloads nothing;
    • with overwrite it plans the replacement;
    • a walk stopped in dir2/ (its listing refuses the credentials) resumes there without listing dir1/ again;
    • times (microseconds), content types and last downloads carry over;
    • a second run downloads nothing.
  • raw_import_of_named_paths_checks_sha256_sidecars:
    • without names the plain server is refused ("lists no files"); duplicate and leading-/ names collapse;
    • a correct sidecar, none, a wrong one ("does not match the SHA-256 its .sha256 sidecar holds") and a missing file;
    • Last-Modified is the upload time;
    • second run: the sidecar file is identical without a download, the other one is downloaded and compared;
    • retrying the failed items imports the missing file once the server has it.
  • raw_import_preflight_refusals: wrong credentials (400), an unreachable source (502), paths Forge refuses and a directory in names (400), a named import whose first file refuses the credentials, a read-only repository, a proxy.
• Unit tests:
  • import::raw: Nexus assets with the repeated blobCreated, upper-case digests and an old leading /; the Forge listing; sidecar parsing; the walk down, across pages and back up, with a cursor round trip; refused directory names; URL building (percent-encoding, Nexus behind a context path, what is not a Nexus URL).
  • import: a listed file's item (times, the epoch, SHA-512 hex → base64, the stored source), every refused-path reason, the printable item name, and the new listings' round trip.
• e2e crates/e2e/tests/raw_import.rs (new, 3 tests), run here against real containers:
  • raw_import_from_a_real_nexus (mirror.gcr.io/sonatype/nexus3:3.70.1):
    • a raw-hosted repository with nested directories, docs/guides/a b ü.txt and 25 bulk files: 30 assets, which Nexus' listing gives in 4 pages;
    • a 300-byte segment Nexus stores and Forge refuses;
    • one file downloaded from Nexus first, so it has a lastDownloaded.
    • Dry run: 29 planned, planned_size = their sum, 1 failed, nothing written.
    • Import: 29 imported, the refused path failed with its reason, uploader admin on every item. Each file's bytes, content type (where Nexus' is not application/octet-stream), .sha256 sidecar = Nexus' SHA-256, upload time = blobCreated, last download = lastDownloaded or the import time.
    • Second run after a new and a changed file (Nexus writePolicy: allow): 1 imported, 1 conflict, 28 identical, bytes_copied = the new file's 3 bytes.
    • overwrite in the previous request form: 1 replaced.
  • raw_import_from_another_forge: 109 files on the same server (105 in one directory, more than the 100 a listing page holds, plus nested directories and a non-ASCII name with a space). Upload times, content types and last downloads equal the source's as the import found them; a second run is all identical with 0 bytes copied.
  • raw_import_of_named_paths_from_a_web_server (mirror.gcr.io/library/nginx:1.27.5-alpine, files copied in and touched to a fixed time):
    • without names it is refused;
    • with names: 2 imported, one failing on its sidecar, one missing;
    • upload time = nginx's Last-Modified;
    • a second run: 2 identical, only the sidecar-less file downloaded again.
• Web (imports.test.tsx): "offers the import on a raw hosted repository, with paths one per line": the link from the raw repository, the listing text, a failed refused-path item with its reason, the "Repository URL" prefilled from the last import, and paths with spaces and commas posted whole.

Checks run

Environment: the shared build machine, CARGO_PROFILE_{DEV,TEST}_DEBUG=line-tables-only, SQLX_OFFLINE=true, CARGO_BUILD_JOBS=2 (the last e2e and clippy runs with CARGO_INCREMENTAL=0 to save disk).

• cargo fmt --all -- --check: clean.
• cargo clippy -p forge-db -p forge-server -p forge-e2e --all-targets --features forge-e2e/forge-e2e -- -D warnings: clean (re-run after the last change).
• cargo test -p forge-server --lib -- import:: files::: 17 passed.
• cargo test -p forge-server --test raw_import: 4 passed (re-run on the final code).
• cargo test -p forge-server --test import --test raw: 12 + 9 passed (the existing NuGet/Docker import and raw suites; run before the last refactor, which only renamed a raw-only helper of the worker).
• cargo test -p forge-db --test migrations --test raw: 3 + 2 passed (migrations through 0010 on Postgres 18).
• cargo test -p forge-e2e --features forge-e2e --test raw_import -- --nocapture: 3 passed, on the final code ("Nexus' asset listing: 30 assets in 4 pages").
• cargo xtask changelog --check: "32 changelog fragment(s) under docs/changes, all for milestones after 2.1.0."
• cargo xtask docs --check: "16 page(s) of the site, every link resolves."
• Web: pnpm install --frozen-lockfile, pnpm run tsc (clean), pnpm run test (40 files, 535 tests passed), pnpm run build (ok).
• No new .sqlx files: the two new raw_dao queries use the runtime API.

Notes

• Decide whether an import from a Nexus also reads Nexus' REST API for checksums, last downloads and uploaders #645 (whether an import reads Nexus' REST API) is still open. For raw, Nexus' asset listing is the only listing with checksums, so this PR reads it, as the issue says a "no" in Decide whether an import from a Nexus also reads Nexus' REST API for checksums, last downloads and uploaders #645 would leave raw without a Nexus source. It is read only for a raw import, whose url has a Nexus repository's shape, and only when the source does not answer as another Forge. If Decide whether an import from a Nexus also reads Nexus' REST API for checksums, last downloads and uploaders #645 decides "no", the Nexus source here goes; the named-paths source still reaches a Nexus raw repository, which serves the files (but not their times).
• Deployment policy and on_conflict. As for NuGet imports, overwrite replaces a file even where the repository refuses redeploys (the import is the administrator's decision), but it needs the delete level on the path (selectors). A read-only repository refuses the import in the preflight, and each file in the worker.
• A replaced file's upload time is the source's, unless that is older than the time of the content it replaces. Then it is that time plus a microsecond, so If-Modified-Since never answers 304 for new content. A created file takes the source's time as it is.
• Times at the epoch (<= 1970-01-02, files unpacked without times, e.g. nginx serving a tar's mtime 0) count as none, so retention's age rules do not see such a file as 56 years old right after the import.
• Which failures stop an import. A refused path or a file the source does not have is decided without asking the source. It fails its item but does not count towards the 25 failures in a row that stop an import, so a large directory of refused names cannot stop it. A checksum mismatch, a refusal of the selectors or a transfer error counts, as for packages.
• Named paths of a web server: Last-Modified becomes the upload time (the issue names no source time for them). A directory in names is refused; naming a subtree of a listing source is not supported. An unparsable sidecar counts as no sidecar, and so does one the credentials may not read.
• The Forge walk asks for 100 entries a page and reads a page again from the subdirectory it stopped at once that is walked. That is a few extra listing requests per directory with subdirectories, and keeps the cursor a single position, with no stack.
• Import: copy an npm registry's packages (Nexus, Verdaccio, another Forge) into a hosted npm repository #718 (npm import) has merged. main is merged into this branch (3aa261e), keeping both formats side by side: FORMATS / IMPORT_FORMATS, Listing (npm's All beside Assets / Directories), one Found.kind for component, tag and file, both sides' ItemSource fields, the copy kinds component, manifest, tag, file, raw's Files target beside npm's arms in copy_item and Import: copy an npm registry's packages (Nexus, Verdaccio, another Forge) into a hosted npm repository #718's expand_one method, and a Raw variant of Import: copy an npm registry's packages (Nexus, Verdaccio, another Forge) into a hosted npm repository #718's Protocol (<url>/repository/<name>, as for npm). ImportView.listing and Import.tsx's format prop are one each; the docs, the CI pre-pulls (Verdaccio, nginx) and the web tests carry both. After the merge, fmt, clippy (with forge-format-npm), --lib import:: (13), --test import --test raw_import (16 + 4), forge-db --test migrations (3), the changelog and docs checks, the web checks (41 files, 550 tests) and the raw e2e (3) pass.
• Web UI ↔ admin API: the import's admin API change (raw) ships with its screen (the existing Import screen, with raw wording).
• Not covered:
  • Nexus releases other than 3.70.1;
  • a Nexus raw group or proxy (the asset listing of a proxy lists only what it cached; the docs say to point an import at a hosted repository);
  • a source whose HEAD is refused while its GET is served (the named-path scan treats a 405/501 as "nothing known", any other refusal as an error).

---

Generated by Claude Code