Skip to content

Look up paper titles outside the catalog and say what a miss tried - #30

Merged
NielsRogge merged 1 commit into
mainfrom
claude/title-fallback-min-eval
Oct 16, 2026
Merged

NielsRogge merged 1 commit into
mainfrom
claude/title-fallback-min-eval

Conversation

@NielsRogge

Copy link
Copy Markdown
Collaborator

Why

In the 32 h to 2026-10-16, ~30 of 39 production pwc_get_paper_info errors were title lookups of papers missing from the catalog. public_paper sent titles to papers/{ref}/public-metadata, which rejects them (400). The MCP then said "no public Hugging Face Papers, arXiv, or Crossref record matched it", which was false.

Depends on huggingface/paperswithcode#552, which adds GET /papers/public-metadata/search?title=.

What

  • CatalogClient.public_paper returns a record or a PublicPaperMiss(reason). When the identifier route returns 400, the reference goes to the title search, and the record carries requested_title.
  • The miss wording depends on what was tried:
    • identifier: unchanged
    • title: "no arXiv or Crossref paper has exactly this title…"
    • unsupported: "not looked up publicly…"
    • unavailable: "lookup is unavailable right now…"
  • 429/503 are no longer reported as not found. Definitive misses are still cached for an hour; transient failures are not cached.
  • A paper matched only by its Name: prefix gets a note: No paper is titled exactly "…"; this is the paper with the same name, titled "…".
  • list_benchmarks bug: minimum_evaluations=0 (or a negative value) is omitted for the flat listing. Its API rejects 0 (ge=1), and omitting the value means no minimum. The grouped listing accepts 0 and keeps it.
  • pwc-mcp bumped to 0.2.9. SPEC, README and SKILL updated.

Verification

  • uv run --project mcp_server pytest mcp_server/tests: 79 passed. New tests cover the title fallback and caching, each miss wording, the renamed-title note, and minimum_evaluations 0/-3/2/grouped.
  • End to end: MCP get_paper_info against a local backend running #552 (port 8137), with live arXiv and Crossref:
    • "Why Should I Trust You? Explaining the Predictions of Any Classifier": the title search finds 1602.04938, which is in the catalog (id 312), so it returns the catalog paper. The MCP's exact-title check missed it because the catalog title has quotes.
    • "GlobalTrack: A Simple and Universal Tracker with Global Features": in_catalog=false, 1912.08531 (arXiv abs, PDF, code), with the renamed-title note. Chat's subtitle was wrong.
    • "Alleviating Over-segmentation Errors by Detecting Action Boundaries": in_catalog=false, arXiv 2007.06866 abs and PDF.
    • "Active Contours Without Edges": in_catalog=false, Crossref DOI 10.1109/83.902291.
    • Made-up title: not_found with the exact-title wording.
    • list_benchmarks(task=image-classification, minimum_evaluations=0): OK.

Chat-related: not merged or deployed without explicit approval. Merge after #552 is live, then redeploy the MCP.

🤖 Generated with Claude Code

About 30 of 39 production get_paper_info errors were titles of classics
missing from the catalog. The public-metadata route rejects titles, and
the MCP then claimed no Hugging Face Papers, arXiv, or Crossref record
matched. A title the identifier route rejects now goes to the backend's
exact-title search (papers/public-metadata/search). A paper matched only
by its "Name:" prefix says that the full title differs. Misses now say
whether an identifier or a title was tried, or that the lookup was down,
and transient failures are no longer reported as not found.

list_benchmarks with minimum_evaluations=0 sent min_eval_count=0, which
the flat listing rejects with a 422. Values below 1 are now omitted
there, which means no minimum; the grouped listing keeps 0.

Bump pwc-mcp to 0.2.9.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NielsRogge
NielsRogge merged commit 2474e8a into main Oct 16, 2026
2 checks passed
@NielsRogge
NielsRogge deleted the claude/title-fallback-min-eval branch October 16, 2026 08:45
NielsRogge added a commit that referenced this pull request Oct 16, 2026
PR #30 and PR #31 both set the version to 0.2.9, so the merged keyword
fallback would report the same version as the release already in production.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant