Repository navigation
feat: scope the monthly refresh to the website author - #67
Merged
Merged
Conversation
The monthly refresh enriched all 64 enabled authors in data/input.csv and took 40 to 100 minutes. Its only consumer, MAPS-Lab/mapslab-website import-publications.yml, sparse-checks out `output/Spadon (bfdGsGUAAAAJ)` and aggregates that directory alone. Add data/website-authors.csv (the input.csv header and the Gabriel Spadon row) and pass it with --input to every `python3 main.py` call in monthly-refresh.yml. The manifest reads the same file, so enabled_authors and authors_with_bib describe one census. A subset input was not safe against the standing output tree. finalize_run walked every directory for the orphan sweep, year-window deletion, post-run fixup and rename, superseded-preprint removal and the DOI conflict gate, and run_all forced re-enrichment over conflicts it could not settle. All of them now read only the directories of the given records through iter_output_dirs(only=...). baseline.json still mirrors the whole tree. build_a2i2_folder rebuilds a populated folder only when every member resolved, where it used to accept half. The committed corpus already holds DOI conflicts in Haque, Oore and Trappenberg that only a full run's forced re-enrichment closes, so an unscoped one-author run ended in FinalizationError.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
The scheduled monthly refresh now processes only Gabriel Spadon (Scholar
bfdGsGUAAAAJ, DBLP192/1659) instead of all 64 enabled authors indata/input.csv.data/website-authors.csv: theinput.csvheader plus the Gabriel Spadon row, copied verbatim..github/workflows/monthly-refresh.yml: a workflow-levelMONTHLY_CENSUSpoints at that file.python3 main.pytakes--input "$MONTHLY_CENSUS"(line 137), and the manifest step derivesenabled_authorsandauthors_with_bibfrom the same file (lines 205-217), so both counts describe one census.data/input.csv, the CLI default, local runs andrefresh-shadow.ymlare unchanged.run_allare scoped to the authors in the given input (below), because--inputagainst the standingoutput/tree was not safe.Data-safety analysis
What a one-author
--inputdid to other authors' directories on main (7ee0ca5):io_utils.py:379,postrun.py:175postrun.py:219.bibbelowget_min_year()postrun.py:269.bibnot in serializer formpostrun.py:460io_utils.py:87,postrun.py:363FinalizationErroron a conflict in a directory the run cannot re-enrichscheduler.py:352,:361io_utils.py:~663output/a2i2/when at least half the members resolvedNot affected, read and confirmed per author directory:
pipeline/article.pyandmerge_utils.pyremovals (author_diris derived from the record being processed), thecanonicalize.pypost-run orphan repair (a per-entry rule inside_fixup_bib_entry, no file removal), andreconcile_summary_csv(drops only rows whose file is already gone).After this change every pass above reads only the directories of the given records (
iter_output_dirs(only=...),fsscan.py:276).baseline.jsonstill mirrors the whole tree and is a derived file.build_a2i2_folderrebuilds a populated folder only when every member resolved (io_utils.py:661), replacing the half threshold.Evidence, both from a copy of the committed
output/(3,693 files), one-authorfinalize_run:FinalizationError: conflicting copies of the same DOI remain in author directories: Haque (PMTOOSQAAAAJ), Oore (cI0dYX4AAAAJ), Trappenberg (EwkaTYEAAAAJ). The committed corpus already carries these conflicts, which only a full run's forced re-enrichment closes, so the monthly job would have failed finalization every run.a2i2 rebuild skipped: only 1 of 21 members resolved ... which would delete 1105 existing file(s), and sha256 over every file underoutput/is identical before and after (0 differences).tests/test_finalize_run.py), main code with the conflict gate disabled so it reaches every pass: 3 out-of-scope files deleted (Roe2019-Old.bib,Roe2022-Detecting.bib,Roe2024-AlphaCopy.bib), 4 rewritten. Both new finalize tests fail on main and pass here.Consequence to accept: a cross-author DOI conflict between Gabriel's copy and a co-author outside the census is no longer visible to a one-author run. The full run still gates it.
Website consumer evidence
MAPS-Lab/mapslab-website.github/workflows/import-publications.yml(blobd93ac20, main1a26cbf) checks outMAPS-Lab/CiteForgewithsparse-checkout: output/Spadon (bfdGsGUAAAAJ)and runsscripts/aggregate_bib.py "_citeforge/output/Spadon (bfdGsGUAAAAJ)/". No other directory is read.test_the_website_census_is_the_one_author_the_website_readspins the census to that directory name.Test record
heavy .venv/bin/pytest -m 'not live' --cov=citeforge --cov=main --cov-fail-under=68 -q, thetests.ymlselection plus the 3.12 coverage leg.live). Coverage 84.15 percent against the 68 floor.tests/test_workflow_contracts.py:308and:339.ruff check citeforge/ tests/ main.pyclean,mypy citeforge/ main.pyclean (56 files).uv pip compilelock freshness (no dependency changed).New tests:
test_finalize_run.py:923(byte-identical tree, summary CSV unchanged, in-scope sweep still runs),:967(a2i2 majority guard),test_workflow_contracts.py:203/224/230(everymain.pycall and the manifest read the census; census is the website directory and a verbatiminput.csvrow),test_scheduler_parallel.py:238,test_regression.py:2432,test_fsscan.py:44. Existing callers updated for the requiredauthor_dirsargument, and the existing unresolved-conflict test now passes both authors so it still exercises a full-scope abort.Left for the owner
output/a2i2/is no longer refreshed by the monthly run, since 1 of its 21 members is in scope. It stays at its last full-run content untilciteforgeruns on the full census.