MENDELU/Security: per-collection "Allow external URLs" + ORE ingest egress hardening (#860)

[milanmajchrak]

Problem description

Issue: https://github.com/dataquest-dev/dspace-customers/issues/860

A collection harvesting with "metadata and bitstreams" downloads whatever URL the remote ORE record names, without any check. A rogue OAI provider can therefore point us at our own internal network, or at a local file, and get the result stored as a public bitstream.

Reproduced on this branch before fixing: http://dspacesolr:8983/solr/statistics/select?q=*:* ended up as a bitstream containing the Solr response, and file:///etc/passwd as a bitstream containing the container's password file. So it is a local file disclosure as well, not only SSRF.

Analysis

Adds the Allow external URLs checkbox per collection, off by default. It only widens which public hosts are allowed — internal addresses stay blocked in both states, including on redirects. With it off, files must come from the same host as the configured OAI provider.

Enforced regardless of the checkbox: http(s) only, no credentials in the URL, every resolved address checked against private/reserved ranges, redirects re-validated per hop, connect/read timeouts and a size cap. A rejected file now skips the whole record, instead of importing an item that silently has no files.

Two things worth knowing when reviewing:

• The migration sets allow_external_urls = TRUE for existing harvest_type = 3 collections so no running harvest breaks. If none of them fetch files from a host other than their OAI provider, FALSE is the safer default — one line in the migration. Worth checking first with SELECT collection_id, oai_source FROM harvested_collection WHERE harvest_type = 3;
• MENDELU/Security patches from vanilla DSpace 9.3 (CVE-2026-49830..49833 incl. LDN RCE) #1336 is still open and touches the same two files. Note that after it merges the SSRF still works, because validateUrlPrefix stays commented out there.

Needs the UI part: dataquest-dev/dspace-angular#1414. Merge this one first.

Manual Testing (if applicable)

• Reproduced the issue and verified the fix in docker — videos in the comment below
• Added to testing scenarios

Copilot review

• Requested review from Copilot

[Copilot]

Pull request overview

Security hardening for OAI-ORE ingestion to prevent SSRF and local file disclosure by introducing an explicit ORE egress policy (scheme/host/address validation, redirect re-validation, timeouts, and response size caps) and adding a per-collection “allow external URLs” toggle exposed via REST, CLI, and DB migrations.

Changes:

• Add a new org.dspace.harvest.ore policy/validation/fetching layer and wire it into OREIngestionCrosswalk + OAIHarvester to enforce safe egress and per-record rejection.
• Add harvested_collection.allow_external_urls (Flyway migrations for Postgres/H2), expose it via REST + integration tests, and support enabling it via harvest -x.
• Expand OAI module documentation (oai.cfg) describing the new behavior and related configuration knobs.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
dspace/config/modules/oai.cfg	Documents ORE URL validation behavior, internal-address blocking, redirects, timeouts, and size caps.
dspace-server-webapp/src/test/java/org/dspace/app/rest/CollectionHarvestSettingsControllerIT.java	Adds REST IT coverage for allow_external_urls defaulting, persistence, and “absent key leaves unchanged” semantics.
dspace-server-webapp/src/main/java/org/dspace/app/rest/repository/HarvestedCollectionRestRepository.java	Implements “absent Boolean leaves unchanged” behavior when updating harvest settings.
dspace-server-webapp/src/main/java/org/dspace/app/rest/model/HarvestedCollectionRest.java	Adds nullable allow_external_urls field to REST model to support partial-update semantics.
dspace-server-webapp/src/main/java/org/dspace/app/rest/converter/HarvestedCollectionConverter.java	Ensures REST responses include allow_external_urls (and defaults to false when no harvesting row exists).
dspace-api/src/test/java/org/dspace/harvest/ore/SafeResourceFetcherTest.java	Unit tests for safe fetching: redirects, downgrade prevention, address blocking, and response-size caps.
dspace-api/src/test/java/org/dspace/harvest/ore/OreUrlValidatorTest.java	Unit tests for URL parsing/validation and policy behaviors (scheme/host confinement/address blocking).
dspace-api/src/test/java/org/dspace/harvest/ore/BlockedAddressPredicateTest.java	Unit tests for blocked-range matching, including IPv6-embedded IPv4 forms.
dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres/V9.1_2026.07.31__harvested_collection_allow_external_urls.sql	Adds allow_external_urls column and seeds existing harvest_type=3 rows.
dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/h2/V9.1_2026.07.31__harvested_collection_allow_external_urls.sql	Same as Postgres migration for H2.
dspace-api/src/main/java/org/dspace/harvest/ore/SafeResourceFetcher.java	Implements redirect-safe, address-pinned fetching with timeouts and a streaming size cap.
dspace-api/src/main/java/org/dspace/harvest/ore/RejectionReason.java	Defines canonical rejection reasons for policy decisions vs transfer failures.
dspace-api/src/main/java/org/dspace/harvest/ore/OreUrlValidator.java	Validates scheme/authority/host confinement/DNS resolution/address blocking and returns validated addresses.
dspace-api/src/main/java/org/dspace/harvest/ore/OreResourceRejectedException.java	Crosswalk-compatible exception to signal per-record rejection with structured reason.
dspace-api/src/main/java/org/dspace/harvest/ore/OreEgressPolicy.java	Immutable policy derived from collection settings + config (timeouts, max bytes, redirects, allowlists).
dspace-api/src/main/java/org/dspace/harvest/ore/HarvestPolicyAware.java	Interface to inject per-collection egress policy into crosswalks that fetch remote content.
dspace-api/src/main/java/org/dspace/harvest/ore/BlockedAddressPredicate.java	Implements byte-level blocked-range checks (incl. IPv6 tunnels) for internal/reserved addresses.
dspace-api/src/main/java/org/dspace/harvest/OAIHarvester.java	Injects policy into ORE crosswalk, rejects failing records safely (rollback), and adjusts harvest status/watermark logic.
dspace-api/src/main/java/org/dspace/harvest/HarvestedCollection.java	Adds persisted allowExternalUrls flag to the harvest settings entity.
dspace-api/src/main/java/org/dspace/content/crosswalk/OREIngestionCrosswalk.java	Uses the safe fetcher + policy, adds null-guards, and converts transfer failures into per-record rejection.
dspace-api/src/main/java/org/dspace/app/harvest/HarvestScriptConfiguration.java	Adds -x/--allow-external-urls CLI option.
dspace-api/src/main/java/org/dspace/app/harvest/Harvest.java	Implements nullable CLI flag semantics so absence of -x leaves stored DB value unchanged.

Suppressed comments (1)

dspace-api/src/main/java/org/dspace/content/crosswalk/OREIngestionCrosswalk.java:213

• entryId can now be null (missing alternate link), so this message can end with ": null", which isn’t helpful for debugging. Build the message to omit the identifier when it’s absent.

            } else {
                throw new CrosswalkException("Entry did not contain link to resource: " + entryId);
            }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[milanmajchrak]

Copilot review addressed in eab245f.

Applied (1) — the suppressed comment on OREIngestionCrosswalk.java: entryId did become nullable with the null-guard this branch adds, so "Entry did not contain link to resource: " + entryId could end in ": null". The identifier is now named only when present. Correct catch. Recompiled, checkstyle clean, the 33 org.dspace.harvest.ore unit tests still pass.

Skipped (1) — the HTTP.TARGET_HOST comment on SafeResourceFetcher.java; replied inline with the evidence and resolved the thread. Short version: HTTP.TARGET_HOST is "Host" (the internal context key is the separate HttpCoreContext.HTTP_TARGET_HOST), and an explicit Host header does survive RequestTargetHost, verified on the wire — Host: localhost with the override vs Host: localhost:8099 without it.