[Port to dtq-dev] Issue dspace-customers#902: Add configurable crawler policy (robots.txt + SSR excludes) - #1450

Open

jr-rk wants to merge 2 commits into

ufal/fe-crawler-policy

Commits on Aug 11, 2026

feat(ssr): add configurable crawler policy

jr-rk
and
claude
committed
001de25

Commits on Aug 12, 2026

fix(ssr): drop dead scoped-search patterns, add robots.txt test

jr-rk
and
claude
committed
e82c952