[Port to dtq-dev] Issue dspace-customers#902: Add configurable crawler policy (robots.txt + SSR excludes)

[jr-rk]

infographics visualization:
pr1450-crawler-policy-infographic.html

Problem

dtq-dev has no crawler policy. On repozitar.mendelu.cz (DSpace 9.1) crawlers — Baiduspider and others — walked Discovery facet links. Every facet combination is a distinct URL, so the SSR page cache (keyed by URL) never hits and each request costs a full Angular render plus Discovery REST queries. That combination space is effectively unbounded, the PM2/SSR workers stayed saturated, and the whole site answered HTTP 504 — home page included — until the crawler backed off.

Four customer branches hardened robots.txt independently for this and adjacent reasons (mendelu, TUL, ZCU-PUB, VSB-TUO), so every new customer re-hits the same wall. This ports one configurable crawler policy to the integration branch instead of copying four variants.

Fixes dataquest-dev/dspace-customers#902 · relates to dataquest-dev/dspace-customers#846

Root cause

robots.txt had no rule for Discovery facet URLs (?f.author=…, &f.subject=…, …) on any path. Disallow: /search covers the base search route by prefix, but a crawler that ignores robots.txt entirely (or a search-engine bot that partially honours it) can still request an unbounded number of facet permutations on paths robots.txt doesn't name.

Correction vs. this PR's earlier revision: an earlier commit also added two universal.excludePathPatterns regexes (^/collections/[uuid]/search, ^/communities/[uuid]/search) and matching robots.txt lines, mirroring the mendelu 9.x source commit. GitHub Copilot's review caught that these are dead on this 7.6.5 branch: there is no :id/search child route on collection-page-routing.module.ts / community-page-routing.module.ts here (that route shape only exists on 9.x), and scoped search on 7.6.5 is /search?scope=<uuid>&f.* — already served by the pre-existing, unchanged ^/search SSR-exclude pattern. A follow-up commit removed both the dead SSR patterns and the dead robots.txt lines. universal.excludePathPatterns is therefore unchanged from dtq-dev base in this PR — verified: git diff origin/dtq-dev...HEAD -- src/environments/environment.ts src/environments/environment.production.ts config/config.yml config/config.example.yml touches the robots: config section only, none of the excludePathPatterns arrays.

Change set

Always on — src/robots.txt.ejs:

• Disallow: /*?f. and Disallow: /*&f. — blocks any URL carrying a Discovery facet filter, on any path, regardless of what comes before the query string. Two rules because the facet can be the first query parameter or a later one, and robots.txt has no "either" syntax.

Optional blocks — new robots config section, all default false:

• disallowHandle → Disallow: /handle (TUL / ZCU-PUB)
• disallowBrowse → Disallow: /browse (TUL / ZCU-PUB)
• disallowBitstreams → Disallow: /bitstream/ + /bitstreams/ (VSB-TUO)
• Wiring: src/config/robots-config.interface.ts (new), registered in app-config.interface.ts + DefaultAppConfig; server.ts builds the optional lines via buildOptionalRobotsDisallows() (src/config/robots.util.ts, unit-tested in robots.util.spec.ts) and passes the result as a single EJS local, so the template has no conditional logic of its own. Switchable per instance via config.yml or DSPACE_ROBOTS_* env vars — no template edit required.

Intentionally out of scope: a blanket Disallow: /handle as a default — it would de-index the persistent identifiers customers rely on for handle redirects, hence the opt-in toggle, off by default. Also out of scope: making the SSR decision (isExcludedFromSsr in server.ts) query-aware to catch facet params on non-/search paths (e.g. entity pages) for crawlers that ignore robots.txt entirely — the documented incident is already covered by the existing /search SSR exclusion; this would be a separate, larger change to core SSR routing. Tracked as a follow-up, not silently dropped.

Test evidence

Run in the worktree (E:/workspace/issue-902-crawler-policy), Yarn 1.22.22, Node 20, full CI order:

# lint (ng lint --quiet)                                            → PASS
# check-circ-deps (madge, 2996 files)  ✔ No circular dependency      → PASS
# build:prod (browser + SSR server bundle, dist/server/main.js)      → PASS
# test:headless (ng test --browsers=ChromeHeadless)
TOTAL: 5513 SUCCESS   0 failing   (5508 existing + 5 new in robots.util.spec.ts) → PASS

Served /robots.txt, defaults (rendered from the actual template in this branch):

User-agent: *
Disallow: /search
Disallow: /admin/*
Disallow: /processes
Disallow: /submit
Disallow: /workspaceitems
Disallow: /profile
Disallow: /workflowitems
Disallow: /entities/*?f
Disallow: /*?f.
Disallow: /*&f.

No /handle, /browse, or /bitstream(s) lines at default config — confirmed by booting the built server bundle twice: once with no env vars, once with DSPACE_ROBOTS_DISALLOWHANDLE=true DSPACE_ROBOTS_DISALLOWBROWSE=true DSPACE_ROBOTS_DISALLOWBITSTREAMS=true, and diffing the two live curl /robots.txt responses — the toggle set appeared in the second response with no template change.

Before / after evidence page (human-watchable): https://claude.ai/code/artifact/892966ee-3015-4146-b10d-d6f81cb700e9. Note some of that page's SSR-routing-table content reflects the earlier, now-reverted excludePathPatterns revision — treat this PR body and the diff as authoritative over that page for the SSR-exclude claims.

No screen-recording: this change has no rendered UI (it's the /robots.txt route + config plumbing), and the full backend stack wasn't started locally (Docker daemon down here). Per reference/pr-quality.md, the observable proof for a no-UI change is the served output above rather than a browser video.

Risk & rollback

• Over-blocking hurts indexing — mitigated by keeping the always-on default minimal (facet filter only); everything that de-indexes real content (/handle, /browse, /bitstream(s)) is opt-in and off by default.
• Residual gap, accepted: a crawler that ignores robots.txt entirely can still trigger full SSR on facet URLs outside /search (e.g. /entities/<type>/<uuid>?f.author=…), since isExcludedFromSsr matches on path only. Pre-existing on this branch, not introduced or worsened by this PR; see "Intentionally out of scope" above.
• Rollback: revert this PR. The robots config section is additive with safe false defaults; no data migration, no schema change; excludePathPatterns is untouched.

Notes / assumptions

• The issue asked for branch dtq-dev/fe-crawler-policy. That ref is impossible while a dtq-dev branch exists (git directory/file conflict), so this uses ufal/fe-crawler-policy — in customers/registry.yml, ufal's fe_branch/be_branch is itself dtq-dev (not customer/ufal), so a ufal/… work-branch prefix targets the integration branch by convention, same as the existing ufal/fe-admin-sidebar-ssr-padding. This change is not UFAL-specific.
• Ported from: mendelu 29b88bf36b (source of the facet-trap idea; its SSR-pattern half didn't port cleanly to 7.6.5's routing, see Root cause), TUL 0eaea26586, ZCU-PUB 11eb4d0f6e, VSB-TUO acae3b4532 (toggle sources).
• Cross-repo Fixes dataquest-dev/dspace-customers#902 creates a cross-reference but does not auto-close the issue on merge (different repo) — close it manually after merge.

Review-driven changes

• Removed the two scoped-search SSR excludePathPatterns and matching robots.txt lines — dead on this 7.6.5 branch (no :id/search route here; scoped search is /search?scope=... already covered by the existing ^/search exclusion). Kept the facet trap (/?f., /&f.) and toggles. Added robots.util.spec.ts. (addresses Copilot review)
• 2026-08-12: rewrote this PR body top-to-bottom — the previous "Change set" section still described the reverted SSR-pattern commit as current. Excerpted the actual served robots.txt output inline instead of relying solely on the external evidence page.

[jr-rk]

dspace-skills improvements

High impact

1. <name>/<slug> branch collides with dtq-dev ref — hard-fails integration backports.
setup-worktree.sh:55 builds branch="$fix/$suffix", no check that <name> is an existing branch. be-backport/SKILL.md:27 lists dtq-dev as a valid target → git worktree add -b dtq-dev/be-… fails (refs/heads/dtq-dev is a file, can't also be a dir). Affects every integration-branch backport + ufal.
Fix: in setup-worktree.sh, git rev-parse --verify -q refs/heads/$fix before creating branch; fail clear or prefix (bp/$fix/$suffix). Update be-backport/SKILL.md:27: use ufal, never dtq-dev, as <name>.

2. 12 unresolved verify:true markers in customers/registry.yml; skills route off them anyway.
8× theme (:41,55,65,75,85,95,105,115), 4× fe_backport_class (sav :64, vsb-tuo :74, zcu-data :104, jcu :114). fe-backport step 2 + recon.sh:47-53 pick fix-class from this — half are guesses.
Fix: recon.sh sav vsb-tuo zcu-data jcu derives class live; backfill and drop markers. Until then treat verify:true as untrusted.

3. fe-backport/verify.sh presented as CI gate, only runs a subset.
profiles/frontend.md:19-20 calls it the pre-done check, but verify.sh:36-44 only does install → build → one spec. Skips lint --quiet + check-circ-deps (both MUST-pass per frontend-stack.md:26-30) and full test:headless. Locally-"verified" branches can still fail CI.
Fix: add lint --quiet + check-circ-deps to verify.sh, or document it as partial.

4. node_modules donor-reuse is manual despite lockfile-identical clusters being known.
setup-worktree.sh:59-77 only junctions donor node_modules if [donor-worktree] passed manually. registry.yml:27 and recon.sh:60-70 already know identical-lockfile families (e.g. zcu-pub/zcu-data/sav).
Fix: auto-scan ../wt-* for byte-identical lockfile, junction automatically; manual arg becomes override.

Medium

5. Trigger phrases duplicated across frontmatter, SKILLS.md, profiles — already drifted. audit-upstream-security + assess-upstream-contribution missing from README.md:58-62 and CHANGELOG.md:10-13.
Fix: SKILLS.md = single source; add the two missing skills to README + CHANGELOG.

6. Demo-video capture recipe written 3x (docker-setup/SKILL.md:81-94, pr-quality.md:11-14, green-light.md:12).
Fix: canonical version in pr-quality.md; others link to it.

7. Registry parsed by two hand-rolled awk scripts (git-refs.sh:48-87, classify-issue.sh:113), both use GNU-only 3-arg match(). Silently returns empty on BSD/macOS awk → registry overrides (e.g. ufal → dtq-dev) silently dropped.
Fix: one shared registry_all_customers helper in git-refs.sh, reused by classify-issue.sh; gate on gawk or use POSIX-compatible match.

8. Three separate "3-iteration" caps, no shared counter: open-pr-from-issue/SKILL.md:74, respond-to-review/SKILL.md:122, plus separate Copilot loop cap (:106-118). No defined shared budget across handoff.
Fix: one iteration budget; state whether Copilot rounds count against it.

Low

9. Two vocabularies for same coverage concept — backport/SKILL.md:51-57 (✅/⏳) vs audit-backport-coverage/SKILL.md:57-63 (present/equivalent/missing/divergent/n-a). Standardize on audit's vocabulary.

10. dtq-dev/dtq-dev-9-base/dtq-dev-9/customer/* branch semantics documented in assess-upstream-contribution/SKILL.md:48-52, absent from registry.yml (the declared SoT). Move to registry.yml meta.

11. assess-upstream-contribution frontmatter description ~109 words, mixes triggers with policy prose. Trim to trigger-only; move policy to body.

12. setup-worktree.sh:43, audit-coverage.sh:46, recon.sh each fetch independently; no node_modules/.angular cache reuse in docker-setup. Could extend finding-4 donor-linking here.

[Copilot]

Pull request overview

Adds a configurable crawler policy to reduce crawler-triggered SSR load.

Changes:

• Blocks scoped search and facet URLs in robots.txt.
• Adds optional handle, browse, and bitstream restrictions.
• Adds SSR exclusion patterns and configuration wiring.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/robots.txt.ejs	Adds crawler directives and optional blocks.
server.ts	Passes crawler settings to EJS.
src/config/robots-config.interface.ts	Defines crawler configuration.
src/config/app-config.interface.ts	Registers crawler configuration.
src/config/default-app-config.ts	Provides safe defaults.
src/environments/environment.ts	Adds development SSR exclusions.
src/environments/environment.production.ts	Adds production SSR exclusions.
src/environments/environment.test.ts	Adds test defaults and exclusions.
config/config.yml	Adds runtime policy defaults.
config/config.example.yml	Documents available settings.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jr-rk]

Addressed the Copilot review.

Removed the dead SSR patterns. On 7.6.5 scoped search is /search?scope=<uuid>&f.* — there is no /collections|communities/:id/search route (that shape is 9.x), and isExcludedFromSsr tests req.path, so the existing ^/search exclusion already serves scoped/faceted search as CSR. Dropped the two dead excludePathPatterns (5 files) and the matching dead robots.txt lines; kept the effective generic facet trap (Disallow: /*?f. / /*&f.) and the optional toggles.

Added a regression test. Toggle-line assembly moved to buildOptionalRobotsDisallows() (server.ts renders it as a single EJS local), unit-tested in robots.util.spec.ts — each toggle plus the no-blank-line-in-group invariant.

Did not rewrite isExcludedFromSsr to inspect facet query keys — robots.txt plus the existing /search SSR exclusion already cover 7.6.5, so that change isn't warranted here.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[milanmajchrak]

Is this really a backport? I think this is a new feature.
From where is this backported?