Skip to content

feat: scope the monthly refresh to the website author - #67

Merged
gabrielspadon merged 1 commit into
mainfrom
feat/monthly-refresh-single-author
Oct 26, 2026
Merged

gabrielspadon merged 1 commit into
mainfrom
feat/monthly-refresh-single-author

Conversation

@gabrielspadon

Copy link
Copy Markdown
Collaborator

Change

The scheduled monthly refresh now processes only Gabriel Spadon (Scholar bfdGsGUAAAAJ, DBLP 192/1659) instead of all 64 enabled authors in data/input.csv.

  • data/website-authors.csv: the input.csv header plus the Gabriel Spadon row, copied verbatim.
  • .github/workflows/monthly-refresh.yml: a workflow-level MONTHLY_CENSUS points at that file. python3 main.py takes --input "$MONTHLY_CENSUS" (line 137), and the manifest step derives enabled_authors and authors_with_bib from the same file (lines 205-217), so both counts describe one census.
  • data/input.csv, the CLI default, local runs and refresh-shadow.yml are unchanged.
  • The finalization tail and run_all are scoped to the authors in the given input (below), because --input against the standing output/ tree was not safe.

Data-safety analysis

What a one-author --input did to other authors' directories on main (7ee0ca5):

Pass main Effect on an out-of-scope author
orphan sweep io_utils.py:379, postrun.py:175 scanned every directory, deleted duplicate orphans
year window postrun.py:219 deleted any .bib below get_min_year()
post-run fixup postrun.py:269 rewrote and renamed any .bib not in serializer form
superseded preprints postrun.py:460 deleted preprints with a published twin
DOI conflict gate io_utils.py:87, postrun.py:363 raised FinalizationError on a conflict in a directory the run cannot re-enrich
forced re-enrichment scheduler.py:352, :361 counted out-of-scope copies, forcing passes that can never settle
a2i2 rebuild io_utils.py:~663 emptied and rewrote output/a2i2/ when at least half the members resolved

Not affected, read and confirmed per author directory: pipeline/article.py and merge_utils.py removals (author_dir is derived from the record being processed), the canonicalize.py post-run orphan repair (a per-entry rule inside _fixup_bib_entry, no file removal), and reconcile_summary_csv (drops only rows whose file is already gone).

After this change every pass above reads only the directories of the given records (iter_output_dirs(only=...), fsscan.py:276). baseline.json still mirrors the whole tree and is a derived file. build_a2i2_folder rebuilds a populated folder only when every member resolved (io_utils.py:661), replacing the half threshold.

Evidence, both from a copy of the committed output/ (3,693 files), one-author finalize_run:

  • main code: FinalizationError: conflicting copies of the same DOI remain in author directories: Haque (PMTOOSQAAAAJ), Oore (cI0dYX4AAAAJ), Trappenberg (EwkaTYEAAAAJ). The committed corpus already carries these conflicts, which only a full run's forced re-enrichment closes, so the monthly job would have failed finalization every run.
  • this branch: finalization completes, a2i2 rebuild skipped: only 1 of 21 members resolved ... which would delete 1105 existing file(s), and sha256 over every file under output/ is identical before and after (0 differences).
  • hazard tree (tests/test_finalize_run.py), main code with the conflict gate disabled so it reaches every pass: 3 out-of-scope files deleted (Roe2019-Old.bib, Roe2022-Detecting.bib, Roe2024-AlphaCopy.bib), 4 rewritten. Both new finalize tests fail on main and pass here.

Consequence to accept: a cross-author DOI conflict between Gabriel's copy and a co-author outside the census is no longer visible to a one-author run. The full run still gates it.

Website consumer evidence

MAPS-Lab/mapslab-website .github/workflows/import-publications.yml (blob d93ac20, main 1a26cbf) checks out MAPS-Lab/CiteForge with sparse-checkout: output/Spadon (bfdGsGUAAAAJ) and runs scripts/aggregate_bib.py "_citeforge/output/Spadon (bfdGsGUAAAAJ)/". No other directory is read. test_the_website_census_is_the_one_author_the_website_reads pins the census to that directory name.

Test record

  • Command: heavy .venv/bin/pytest -m 'not live' --cov=citeforge --cov=main --cov-fail-under=68 -q, the tests.yml selection plus the 3.12 coverage leg.
  • Commit 2687e52, host RTX-4090, Python 3.14.4.
  • 2091 passed, 0 failed, 2 skipped, 7 deselected (live). Coverage 84.15 percent against the 68 floor.
  • The 2 skips are the pre-existing cutover contracts at tests/test_workflow_contracts.py:308 and :339.
  • ruff check citeforge/ tests/ main.py clean, mypy citeforge/ main.py clean (56 files).
  • Not run locally: the 3.10 and 3.12 interpreters (both byte-compile the touched files), a live monthly run (needs provider keys), and uv pip compile lock freshness (no dependency changed).

New tests: test_finalize_run.py:923 (byte-identical tree, summary CSV unchanged, in-scope sweep still runs), :967 (a2i2 majority guard), test_workflow_contracts.py:203/224/230 (every main.py call and the manifest read the census; census is the website directory and a verbatim input.csv row), test_scheduler_parallel.py:238, test_regression.py:2432, test_fsscan.py:44. Existing callers updated for the required author_dirs argument, and the existing unresolved-conflict test now passes both authors so it still exercises a full-scope abort.

Left for the owner

  • output/a2i2/ is no longer refreshed by the monthly run, since 1 of its 21 members is in scope. It stays at its last full-run content until citeforge runs on the full census.
  • The other 63 authors' committed output stops moving monthly.

The monthly refresh enriched all 64 enabled authors in data/input.csv and
took 40 to 100 minutes. Its only consumer, MAPS-Lab/mapslab-website
import-publications.yml, sparse-checks out `output/Spadon (bfdGsGUAAAAJ)`
and aggregates that directory alone.

Add data/website-authors.csv (the input.csv header and the Gabriel Spadon
row) and pass it with --input to every `python3 main.py` call in
monthly-refresh.yml. The manifest reads the same file, so enabled_authors
and authors_with_bib describe one census.

A subset input was not safe against the standing output tree. finalize_run
walked every directory for the orphan sweep, year-window deletion, post-run
fixup and rename, superseded-preprint removal and the DOI conflict gate,
and run_all forced re-enrichment over conflicts it could not settle. All of
them now read only the directories of the given records through
iter_output_dirs(only=...). baseline.json still mirrors the whole tree.
build_a2i2_folder rebuilds a populated folder only when every member
resolved, where it used to accept half.

The committed corpus already holds DOI conflicts in Haque, Oore and
Trappenberg that only a full run's forced re-enrichment closes, so an
unscoped one-author run ended in FinalizationError.
@gabrielspadon
gabrielspadon merged commit 2a909a6 into main Oct 26, 2026
5 checks passed
@gabrielspadon
gabrielspadon deleted the feat/monthly-refresh-single-author branch October 26, 2026 13:55
gabrielspadon added a commit that referenced this pull request Oct 26, 2026
…68)

Rollback of #67 at the owner's request: no changes to the MAPS-Lab
release repositories. The resulting tree equals 7ee0ca5, main before
#67.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant