compact: build the chunk index once, not three times

[ThomasWaldmann]

Problem

borg compact builds the chunk index up to three times per run, and keeps two copies of it in memory at the same time:

1. get_repository_chunks() builds compact's own index — it needs the usage flags (init_flags=F_NONE, then analyze_archives() marks F_USED).
2. analyze_archives() reads the archive metadata objects. Resolving their pack locations goes through Repository.chunks, which lazily builds a second, identical index.
3. compact_packs() calls delete_chunkindex_from_repo(), which drops the repository's in-memory index (borg2: Interrupted borg compact can poison the cached chunk index and cause silent data loss #9748 crash-safety). The archive listing in cleanup_files_cache() afterwards then builds a third one.

The chunk index is the biggest structure borg keeps in memory — a ChunkIndex entry is 32 bytes of key plus a 48-byte value, plus hash table overhead — so on a large repository compact paid for that twice over in RAM and three times over in build time.

Fix

Share compact's index with the repository while the repository reads through it. get_repository_chunks() installs the index it builds as repository.chunks, so analyze_archives() resolves pack locations through that same index instead of lazily building a second one.

This is sound because the two indexes are the same thing:

• The repository only reads pack locations (pack_id, obj_offset, obj_size) and F_PENDING out of an index. It never looks at the F_USED flags or the size field that compact maintains for compaction and --stats.
• Stored index fragments zero the flags and sizes on write (write_chunkindex_to_repo), so both call sites load byte-identical entries anyway; init_flags only differs on the slow rebuild-from-packs path.

The sharing ends before the first store change, and is not restored. In compact_packs(), delete_chunkindex_from_repo() (crash-safety, #9748) drops the repository's reference together with the persisted fragments, and compact deliberately does not hand the index back. Nothing needs it there: from that point until save_chunk_index(), nothing resolves pack locations through the repository — compact_pack() and merge_packs() get chunks=self.chunks, and --stats and save_chunk_index() use self.chunks directly. So while the persisted index is gone, the repository holds no in-memory index either, and Repository.close() on an aborted run has nothing to persist. (Handing it back would only have been harmless because compact's index never carries F_NEW entries — an invariant nothing states or checks.) For the same reason save_chunk_index() can empty the index in place with clear=True: the repository no longer references it.

cleanup_files_cache() no longer lists the archives. It was the third build: it re-listed the archives after save_chunk_index() had cleared the index. It now reuses the names analyze_archives() already collected, so it needs no repository access at all. That set is still valid there — compaction only nukes soft-deleted archives, which were never in the non-deleted listing. As a side effect every archive's metadata is now read once per compact run instead of twice.

Note what this does not change: the call order in garbage_collect() is untouched, so cleanup_files_cache() still runs after the sig_int re-raise (skipped on Ctrl-C, as before), and it stays outside the window between delete_chunkindex_from_repo() and save_chunk_index() in which the repo has no persisted index.

Verification

Measured on the current head against its merge base with master. Built one repository, copied it, and ran compact on each copy with master and with this branch:

• the two repositories come out byte-identical (diff -r), including the content-addressed pack files and index fragments
• the logs are identical apart from the repo path
• borg check --verify-data passes on the compacted repository
• index builds per run: 3 on master, 1 on this branch

The persisted index/* namespace was compared separately across the runs that drive different index paths — completed compaction, --dry-run, no-op, no index/* present (slow rebuild), no index/* + dry run, and compact-twice. Identical fragment sets in every case (fragment names are the sha256 of their content), and in each case the persisted entries exactly match what the packs hold, checked against a fresh slow_rebuild walk of the pack headers.

An interrupted compaction was checked too. It first appeared to differ, but that is pre-existing nondeterminism, not this change: drop_packs is a set, so which pack is dropped before the interrupt depends on hash randomization — master differs from itself between runs. With PYTHONHASHSEED=0 master is reproducible and master == branch, whole repo and files caches included. (Completed runs match without pinning because write_chunkindex_to_repo sorts keys, so a finished index is order-independent.) test_compact_interrupted_does_not_poison_chunk_index covers the hard-abort case: no fragments are left behind.

Files-cache cleanup, the behaviour whose data source changed, was checked end to end on a run that really compacts: three archive series, one deleted entirely — exactly that series' cache file is removed and the two survivors are kept, identically on both sides.

New regression test test_compact_builds_the_chunk_index_only_once counts build_chunkindex_from_repo calls across a run that really compacts (it asserts store_changed, so it cannot silently degrade into a no-op run). It fails on master with assert 3 == 1 and passes here.

Full test suite: 3060 passed, 1011 skipped.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.05%. Comparing base (05debca) to head (99ce853).
⚠️ Report is 5 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10359      +/-   ##
==========================================
+ Coverage   88.02%   88.05%   +0.02%     
==========================================
  Files         103      103              
  Lines       18913    18917       +4     
  Branches     2919     2919              
==========================================
+ Hits        16649    16658       +9     
+ Misses       1572     1567       -5     
  Partials      692      692              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

@mr-raj12 please check.