WIP: OIE mapping pipeline + OWASP eval harness (not merge-ready)

[northdpole]

Status

Draft. Not ready for merging. This is a visibility snapshot of the living OIE librarian/harvester pipeline and the eval data the harnesses actually run on, so mentors, GSoC contributors, and reviewers can see the code, gold, and current numbers in one place.

It will be cleaned up (split, rebased, fixtures relocated, leftovers dropped) when it is actually ready to merge. Do not treat CI green/red on this PR as a merge signal. Do not write these decisions into the live OpenCRE graph.

Summary

• Post-GSoC librarian/harvester/orchestrator work currently sitting on a dirty local tree
• OWASP eval harness scripts under scripts/oie_owasp_eval/
• Harness testing data checked in at scripts/oie_owasp_eval/fixtures/:
  • b2_gold/ — section-level CRE gold (Top 10, API Top 10, LLM Top 10, AISVS, K8s 2022/2025, agentic stub)
  • b2_sources/ — section text the B2 harness scores against
  • reports/ — latest B1 / B2 / Module C accuracy JSON
• Librarian golden set remains at application/tests/librarian/fixtures/golden_dataset.json

How it is tested today

Vertical	What it measures	Latest snapshot
Librarian golden (evaluate_librarian.py)	C.0 boundary, C.0.5 explicit ids, C.1 recall@20, C.2 top-1, C.3 ECE, C.4 auto-link vs review	Retrieval ~98% recall@20; top-1 ~75%; decision accuracy at τ=0.80 ~57%; review recall 5/5
B1 (score_b1_opencre_api.py)	ASVS chunk / CheatSheet sheet vs live opencre.org CRE gold	33/37 (89%), gate >60% pass
B2 (run_b2_pr_mappings.py)	New-family section grain: ≥1 of union(rerank top-2 ∪ vector top-2) ∈ gold cre_ids	26/62 (42%), gate >60% fail

B2 by family (gated report b2_accuracy_report_gated.json): LLM Top 10 100%, K8s 2022 67%, Top 10 / API / K8s 2025 40%, AISVS 17%, Agentic stub 0% (no hub Links, expected weak).

The ~50% figure in Slack is the same story as conservative auto-link / B2 transfer, not “the pipeline is half broken.” Wrong auto-links are the dangerous direction; close calls go to review.

Reproduce

# B2 against the in-repo gold + sources
PYTHONPATH=. python scripts/oie_owasp_eval/run_b2_pr_mappings.py --score-only --run-id <id>

# Librarian golden (hermetic)
python scripts/evaluate_librarian.py \
  --dataset application/tests/librarian/fixtures/golden_dataset.json

Test plan

• Mentors / contributors: read the reports and comment on ranking, corpus (CRE descriptions), and B2 misses — not “please merge”
• Confirm gold/sources in scripts/oie_owasp_eval/fixtures/ match what the harness expects
• Before any real merge: split, rebase onto main, drop WIP leftovers, re-run B1/B2

Made with Cursor

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}