[hz-xiaxz]

Problem

When a ref has neither publisher JATS nor arXiv LaTeX, its body comes from pymupdf4llm. On math-heavy two-column papers that path loses the equations: display equations are cut out as PNG crops, and inline math is flattened into run-together text ([r^a_i, r^b_j]=0,aretherelative). In a KB built for reading physics, that is exactly the part that matters.

MinerU recovers it: inline and display math as LaTeX (with \tag{n} numbering), tables as HTML, figures as images, and a coherent reading order for two-column layouts.

Why remote, and why stdlib

Running MinerU locally needs torch plus GBs of models and, realistically, a CUDA GPU. Its hosted service exposes the same v1 parse API that the mineru 4.x client uses (/v1/parse/jobs), and it accepts anonymous jobs. So the new helper calls that API directly with urllib. Users install nothing and need no GPU, and every helper stays stdlib-only.

Two details make it fast:

• arXiv papers are submitted as URL sources (https://arxiv.org/pdf/<id>; the service advertises "sources": ["file_id", "url"] in /v1/health). The service fetches the PDF itself, so nothing is uploaded. This matters: from outside China, uploads to the service's presigned storage ran under 35 KB/s, and the mineru CLI timed out on a 4 MB PDF.
• Files are batched into one job and parsed in parallel. A 64-page and a 4-page paper both finished in 65 s end to end.

DOI PDFs still upload, since there is no public URL for them. The SKILL.md text tells the agent to confirm with the user before uploading paywalled PDFs to a third-party service. --api-url / MINERU_API_URL points at a self-hosted mineru-api instead; when that server allows local sources, it reads the PDF straight from .raw/.

What changed

• helpers/mineru_parse.py (new). Discovers .raw/{arxiv,doi}/<id>.pdf, submits batched jobs, polls them, and downloads each result zip. It writes .raw/<kind>/<id>.mineru.md and copies only the images the markdown links into .figures/<kind>__<id>/. MinerU's zip also contains equation crops that nothing references, and those are left out. Already-parsed PDFs are skipped (--force redoes them, --only <id> narrows the run). MINERU_API_KEY is optional and raises the quota. The key is sent only to the API origin, never to presigned storage URLs, and is never written to disk.
• Post-processing. MinerU drops the U+FB00 ff ligature (efect, efective), while fi/fl survive. A small stem list restores the common physics words (effect, differ, coeffic, off-diag, …). A (?!f) guard leaves correct words alone.
• render.py --mineru. Body priority is now JATS > LaTeX > MinerU > PDF, with full_text: mineru. Like --tex-source, it is opt-in: a cached .mineru.md changes nothing without the flag.
• index.py, kb_doctor.py and kb_sync.py accept mineru as a full_text value. kb_sync warns that it must re-run the parse, because it cannot regenerate MinerU figures offline.
• Docs: a SKILL.md Step 5 section, a dependencies.md#mineru section (quota, retention, self-hosting), two troubleshooting rows, and the full_text list in CLAUDE.md.

Testing

• pytest -q: 305 passed. tests/test_mineru.py is offline and covers body priority (with and without the flag, LaTeX beating MinerU), image relinking for both markdown and HTML <img> links, the ligature fix, copying only linked images out of a synthetic result zip, and discovery (skipping cached results, --force, old-style hep-th-9901001 → hep-th/9901001).
• scripts/validate_skills.py: OK.
• Live run on a real KB (1106.3375, Haldane PRL; 2610.35934, 64 pp.): both parsed via URL source in one 65 s job. The Haldane body went from image-only equations to all 30 numbered equations in LaTeX, with clean inline math. The 64-page paper gained 56 linked figures. kb_doctor.py reports 0 FAIL, 0 WARN.

Follow-up from an earlier proposal for the same backend in another tool: TDAA-Go/TDAA-Go#4.

🤖 Generated with Claude Code

[GiggleLiu]

Thank you!