Skip to content

[Issue #1415] Retire fully deleted GC groups with checkpoint protection - #1416

Draft
AntiO2 wants to merge 1 commit into
pixelsdb:masterfrom
AntiO2:fix/retina-retire-empty-gc
Draft

AntiO2 wants to merge 1 commit into
pixelsdb:masterfrom
AntiO2:fix/retina-retire-empty-gc

Conversation

@AntiO2

@AntiO2 AntiO2 commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fully deleted Storage GC groups currently delete their empty output and return without retiring the old files. Repeated updates therefore leave obsolete files permanently REGULAR.

Retire these groups atomically without a replacement file. Journal their safe visibility-fold timestamp, confirm retirement only after a covering recovery checkpoint, protect pending journal files from physical cleanup, and restore the old catalog if the selected checkpoint predates retirement. Equal-invalidity candidates now prefer older file IDs so a bounded group limit does not repeatedly select newly created groups before older ones.

Validation:

  • With parent Surefire skipTests temporarily set to false: mvn -pl pixels-retina -am -Dtest=TestStorageGcWal,TestEmptyStorageGc -Dsurefire.failIfNoSpecifiedTests=false test: 20 tests passed, none skipped.
  • JUnitCore selected atomicSwapFiles tests: 5/5 passed, including retirement without promotion and rejected empty requests. The daemon's default test provider discovers no JUnit 4 tests, so these were invoked explicitly.
  • Full DAO JUnitCore suite: 19/21; two insertBatch fixture failures were reproduced against unmodified master and are unrelated to this change.
  • Isolated 13-minute closed-loop replay with two-minute retention completed: 26,506,764 row events / 138 complete cycles, all eight tables recovered their initial active totals (870,600,000 overall), and 18 old files / 124,220,282 bytes were physically deleted. This validates retirement and cleanup only; it used a diagnostic native mutex and is not a final performance or steady-state measurement.
  • Integrated 15-minute validation without the diagnostic native mutex completed 10 partial-live rewrites, restored all table populations, and physically deleted 28 files / 125,261,498 bytes. Two-minute retention and warm-up pauses make this a correctness run only.

Closes #1415

Additional combined live validation after the separate sink same-key coalescing fix (pixelsdb/pixels-sink#22): HyBench SF100 original Proto replay ingested 22,945,933 rows and fully committed all 452,616 admitted transactions. Two GC cycles and three partial-live group rewrites completed. After draining, savingaccount/checkingaccount each had 30,200,000 active versions, customer 30,000,000 and company 200,000, exactly matching their initial populations; insert-growing tables are reported separately. Sink shutdown had no error entries. This is functional integration evidence, not a steady-state or isolated performance comparison. The repeated formal matrix has resumed. Results: gengdy1545/retina-graph#34.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[pixels-retina] Storage GC never retires fully deleted file groups

1 participant