Record whether a problem has a forum thread

[Chessing234]

Closes #370.

Add a derived forum field to the problem export so readers can find which problems have discussion on erdosproblems.com. The updater reads the single-page forum index and records problem-thread posts, proof-claim counts, and replies to proof claims separately. Blog and named site-wide threads are excluded. Problems without discussion have no forum field.

The schema accepts positive counts only. The update workflow is manual (workflow_dispatch). Exact last-post timestamps are omitted because the index exposes relative dates; obtaining exact timestamps would require fetching individual threads.

Validation — 12 September 2026

Merged current main normally, preserving branch history, and refreshed the forum data. All 1,217 upstream problem records remain semantically identical after excluding the added forum field.

The live index contained 913 problem threads and 333 proof claims across 933 problem numbers, carrying 5,695 posts. The export contains forum fields for 931 of its 1,217 records; index-only problem numbers are not added to the database.

• scripts/validate.py: passed.
• scripts/derive_status.py --check: passed.
• scripts/update_forum_status.py --check: passed with no drift.
• The resolved merge tree exactly matched the independently validated rebase tree.

Limits

The parser depends on the current HTML structure and on the index remaining a complete single page. A zero-thread parse fails before modifying data; a partial markup failure or future pagination still requires an updater change. Counts are a dated snapshot and will change as discussion continues.

This was AI-assisted.

[teorth]

Reviewed. The engineering here is in good shape: the data change is purely additive (exactly 901 forum:/posts: pairs, placed between formalized and tags), validate.py passes, derive_status.py --check is clean, the README is untouched, the workflow is workflow_dispatch only, and the ruamel round-trip settings match the rest of the repo. I re-ran your parser against the live index and reproduced your figures exactly — 901 threads, 5189 posts. Thanks also for disclosing the AI assistance.

One thing to resolve before this goes in, about coverage rather than correctness.

The PR says non-problem threads are "the site's own blog:N codes". That undercounts what is being skipped. Of the 1152 <li class="thread-item"> blocks on the index, 251 are skipped, and only 7 of those are blog:. 234 are proof-claim threads, coded like:

411/proof-claims#proof-claim-217

Those are tied to a specific problem, so code.isdigit() drops them. The remaining ~10 are named topic threads (AI Contributions, Site feedback, and so on), which are correctly excluded.

What that costs, measured against today's index:

• 122 problems have proof-claim threads.
• 13 of them have no plain thread at all — so they gain no forum key, and the export still reports no discussion for a problem that has some. That is the case issue data: problems.yaml carries no signal that a problem has a forum thread (894 of 1217 do) #370 was filed about. They are: 117, 119, 131, 319, 416, 424, 450, 538, 634, 653, 882, 959, and one more.
• 109 more have both, so their posts is understated. 245 proof-claim posts go unrecorded overall.

I don't think this is obviously a bug — there's a reasonable position that a proof claim is a distinct object and shouldn't be folded into a single posts count. But a reviewer reading the current description would believe coverage was complete, and it isn't, so it's worth settling explicitly.

Three options, roughly in increasing order of work:

1. Leave the behaviour as-is and correct the description and the parse_forum_index docstring to say that proof-claim threads are excluded too, and why.
2. Count proof-claim posts into posts, so the field means "posts on this problem across the forum".
3. Record them separately, e.g. forum: {posts: N, proof_claims: M}.

My inclination is (1) or (3) — folding them into one number loses a distinction that looks worth keeping. Your call, and Terry's.

One smaller note: unlike update_formalization_status.py, this script deletes forum for any problem absent from the listing. The zero-thread guard catches a total parse failure, but not a partial one — if the index ever paginates, problems below the fold would silently lose the field. Worth a comment noting the single-page assumption that guard depends on.

[Chessing234]

Thanks — you were right that the description undercounted the skips, and I went with option (3) in 8b6701f.

forum now carries up to three keys, each omitted when zero:

forum:
  posts: 13
  proof_claims: 2
  proof_claim_posts: 2

One thing I had to settle to make (3) work: the index badges a proof-claim item only once it has replies — 78 of the 235 carry a title="N posts" badge, and the badged wording ("(1 post)", "to a claim of …") makes clear it counts replies to the claim, not the claim itself. So proof_claims counts claims and proof_claim_posts counts replies, and an unanswered claim contributes 1 and 0. That is why the two numbers are separate rather than one figure.

Reproducing your measurements on today's index: 1153 items, 901 problem threads, 235 proof claims over 123 problems, 7 blog:, 10 named topic threads. The 13 problems whose only discussion is a proof claim come out as your list plus 1059, which I take to be your "one more". They now gain a forum key with proof_claims and no posts, so the schema no longer requires posts — instead each key is integer >= 1 with minProperties: 1, and additionalProperties: false is unchanged.

On the smaller note: the delete path now carries a comment saying it assumes the index is a single page, and that the zero-thread guard only catches a total parse failure, not a partial one.

The description and the parse_forum_index docstring are both rewritten to describe all three kinds of item and how each is counted, so option (1)'s correction is in there too. validate.py, derive_status.py --check and a second update_forum_status.py --check are all clean after the regeneration; the data diff is +202/-2, the two deletions being post counts that moved with the live index since the first commit.

[teorth]

Thanks for taking the third option — splitting proof_claims and proof_claim_posts out from posts is the right call, and it keeps the distinction that folding everything into one number would have lost. The schema handling is good too: three optional keys with minProperties: 1, so a problem with only proof claims carries no posts rather than a misleading zero. Coverage is now 918 problems, up from 901.

Two things before this can land.

It needs a rebase. The branch is 28 commits behind and now conflicts in data/problems.yaml. A fair amount has moved since it was opened — problems 1, 74, 126, 548, 557 and 571 had their formal_status corrected in #398, #390 added oeis validation and fixed a duplicate on problem 121, #388 added OEIS identifiers to three problems, and #391 re-synced 48 tag assignments.

Please rebase rather than resolving the conflict in place. Diffing the branch against current main shows lines that would revert some of those newer changes, which a careless resolve would silently reintroduce — worth being deliberate about, given the size of the data diff.

Please re-run the script as the last step of the rebase. The counts in the branch were generated some days ago and the forum keeps moving, so the committed data will have drifted. Running update_forum_status.py --check afterwards should report no drift; if it does report some, that is the tell that the committed numbers are stale.

Worth also updating the docstring on parse_forum_index, which still describes the skipped set as blog threads — with proof claims now handled, the remaining exclusions are the site's blog:N threads and the named topic threads (AI Contributions, Site feedback, and so on).

One smaller point I raised before and still think is worth a line of comment: this script deletes forum from any problem absent from the listing. The zero-thread guard catches a total parse failure but not a partial one, so the whole design leans on the index being a single page. A comment recording that assumption would help whoever next touches the parser.