UFAL/Metadata QA detectors, backend hardening and data curation for issues 1388/1387/169

[milanmajchrak]

Backend/data companion for three UFAL data-quality bugs. Frontend fixes are tracked separately:

• UFAL/local.sponsor OpenAIRE suggestion stores displayed value (EU) instead of stored value (euFunds) dspace-angular#1380 (funding displayed-vs-stored value — Storing displayed-value instead of stored-value ufal/clarin-dspace#1388)
• UFAL/Subject keywords autocomplete stores an unsplit ';'-separated list as a single value dspace-angular#1381 (unsplit dc.subject list — Subject fields ingested as a single unsplit string instead of repeated values ufal/clarin-dspace#1387)
• UFAL/Author name (clarin-name widget) stored without space after comma dspace-angular#1382 (author name comma spacing — Author name (firstname, lastname) spacing on details page ufal/dspace-angular#169)

This issue covers: (A) three new metadata-QA detectors, (B) backend hardening/normalization, (C) the one-off production data curation. Full root-cause analyses are in the linked issues.

A) Metadata QA detectors — one PR, ItemMetadataQAChecker

Target: dspace-api/src/main/java/org/dspace/ctask/general/ItemMetadataQAChecker.java, registered as metadataqa in dspace/config/modules/curate.cfg:19 (runnable via dspace curate -t metadataqa and the admin Curate UI). Implement against the dtq-dev variant (it diverges from ufal clarin-v7; dtq-dev already has the config-driven list infrastructure — getArrayProperty pattern at lines 82-96).

Implementation conventions (important):

• Aggregate findings into the results buffer instead of throwing CurateException per finding — the perform() try-block aborts at the first exception (lines ~158-175), so throwing detectors would mask each other and undercount.
• One shared property naming scheme lr.curation.metadata.*; severity via the class-local CURATE_WARNING (= -1000, line 47).
• Register new warning messages in metadata-check-patterns.json so MetadataCheck buckets them.
• Document limitation: curation only reaches archived items with handles — workspace/workflow items are not covered (the feature/aai-2-missing-idp-headers #169 curation SQL covers those).

Detectors:

1. Value-pairs stored-value membership (for 1380/Keep embed params out of subresource self links #1388): for every field bound to value-pairs (including complex sub-fields with value-pairs-name), verify the value (for complex fields: the relevant segment after splitting on ;) is one of the stored-values; flag when it equals a displayed-value ≠ stored-value (e.g. EU, nepoužitelné, Own funds, National, other — note other vs stored Other differs only by case). This also answers the original issue's "are there others?" empirically and will surface the known dirty dc.language.iso rows. Treat legacy DSpace5-era free-text values (e.g. dc.type facet noise: TEXT, IMAGE) as warnings, not failures.
2. Unsplit-separator scan (for 1381/MENDELU/Security: per-collection "Allow external URLs" + ORE ingest egress hardening (#860) #1387): flag values of multivalued free-text fields containing ;. Exclude complex fields (local.sponsor uses ; legitimately). Configurable field list (default at least dc.subject; consider dc.publisher, dc.language.iso, dc.relation*).
3. Person-name comma-spacing (for 1382/feature/aai-2-missing-idp-headers #169): flag person-field values matching ,\S (comma not followed by space). Configurable field list (default dc.contributor.author, dc.creator).

B) Backend hardening / normalization

1. Server-side split on ingest (co-primary for MENDELU/Security: per-collection "Allow external URLs" + ORE ingest egress hardening (#860) #1387 — the original issue says "split on ingest"): dspace-server-webapp/.../submit/step/DescribeStep.java doPatchProcessing (lines 201-245) — split repeatable free-text fields on ;, copy text_lang to all resulting rows, authority/confidence null. Honest scope note: this does NOT cover admin Edit-item / PATCH /api/core/items (DSpaceObjectMetadataAddOperation), SAF import, CSV metadata-import (splits only on ||), SWORD — the QA detector is the safety net there.
2. Value-pairs validation (for Keep embed params out of subresource self links #1388): dspace-server-webapp/.../submit/step/validation/MetadataValidation.java — validate submitted dropdown/list values against the stored-values from getPairs() (odd indices). Currently only regex + authority are validated (lines 277-296). Note: the existing openaire_id/euFunds exemption at lines 265-269 is dead code (openaire_id is readonly="true" with no required attribute — submission-forms.xml:3208).
   Sequencing: deploy only AFTER the data curation below — otherwise it blocks re-editing of the not-yet-cured items. Scope (strict vs warn for legacy values) to be agreed with UFAL.

C) One-off production data curation — ONE maintenance window, order is critical

1) deploy frontend fixes (1380 + 1381) → 2) run curations → 3) one full Solr reindex (including *_ac autocomplete cores) + OAI refresh → 4) only then enable strict validation (B2).
Rationale: bad values breed via suggestions (findByMetadataNameAndByValue and *_ac serve stored values verbatim); strict validation before curation would lock the affected items.

1. Funding (Keep embed params out of subresource self links #1388): derive the field id via SQL join over metadatafieldregistry/metadataschemaregistry (local.sponsor), do not hardcode 136. Inventory text_value LIKE 'EU;%' plus 'nepoužitelné;%', 'Own funds;%', 'National;%', 'other;%' (case-sensitive: other ≠ stored Other). Dry-run list → review → rewrite prefix EU; → euFunds;.
   Manual carve-out: OpenAIRE assigns info:eu-repo ids to national funders too (verified case 11234/1-6156, Czech MSM funder composed as EU) — such rows may belong to nationalFunds; curator decides.
   Also clean the known dirty dc.language.iso rows (11234/1-6153 "English; Czech", 20.500.12800/1-6040 "English", 20.500.12800/1-6204 "English" + "Egyptian Hieroglyphic Transliteration") — the code fix already landed via PR ZCU-DATA/Reduce warn logs noise - 404 dedicated logger #1275.
2. Subjects (MENDELU/Security: per-collection "Allow external URLs" + ORE ingest egress hardening (#860) #1387): detection SQL (registry join) for dc.subject values containing ; (extend to the other fields from the inventory); split preserving place order, copy text_lang, authority null. Mandatory human review (typos like a missing ; merging two keywords; duplicate keywords — the example item has "Digital Humanities" twice; dedup policy TBD with UFAL). Mechanism: CSV metadata-export/import (||) or a script.
3. Person names (feature/aai-2-missing-idp-headers #169): rewrite ([^,\s]),(\S) → \1, \2 in person fields; cover workspace/workflow items too (production archived data is currently clean — the reported value likely sits in an in-flight item). Dry-run review (commas in org names are not surname/firstname separators).

Open questions for UFAL

• Keep embed params out of subresource self links #1388: policy for non-EC info:eu-repo rows (rewrite to nationalFunds vs manual review); strict-validation scope vs legacy dc.type values.
• MENDELU/Security: per-collection "Allow external URLs" + ORE ingest egress hardening (#860) #1387: split on comma as well? (legitimate commas in phrases/names); dedup policy for curated splits; UX preference — multi-row split vs a validation error forcing one keyword per row.
• feature/aai-2-missing-idp-headers #169: which instance/item the reporter saw (not in production archived data); should dc.creator/dc.contributor.other join the detector defaults?
• Deployment: priority cherry-pick of the 1380/1381 frontend fixes into clarin-v7 vs the routine lindat-merge; production DB access for the curation SQL.

Suggested work order

1. Frontend fix 1380 (Keep embed params out of subresource self links #1388 — corruption ongoing, last bad row 2026-07-01)
2. Frontend+backend fix 1381 (MENDELU/Security: per-collection "Allow external URLs" + ORE ingest egress hardening (#860) #1387 — ongoing, user-visible on DH Data)
3. Frontend fix 1382 (feature/aai-2-missing-idp-headers #169 — archived data currently clean)
4. QA-detector PR (A)
5. Curation window (C) + reindex
6. Strict validation (B2)