Skip to content

fix: add .compaction-queue to handle in-mid compaction crash#453

Open
dkharms wants to merge 3 commits into
336-compactionfrom
336-atomic-substitute
Open

fix: add .compaction-queue to handle in-mid compaction crash#453
dkharms wants to merge 3 commits into
336-compactionfrom
336-atomic-substitute

Conversation

@dkharms

@dkharms dkharms commented Jun 30, 2026

Copy link
Copy Markdown
Member

Description

Compaction's substitute-and-delete of stale fractions isn't atomic, so a crash mid-compaction could leave source fractions behind and produce duplicates on restart.

This adds a .compaction-queue file recording the participating fractions, which is written before merging and used on startup to clean up any leftover sources from an interrupted compaction.


  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

@dkharms dkharms force-pushed the 336-atomic-substitute branch from 6c52a39 to bf68450 Compare July 1, 2026 12:19
@ozontech ozontech deleted a comment from seqbenchbot Jul 1, 2026
@ozontech ozontech deleted a comment from seqbenchbot Jul 1, 2026
@dkharms dkharms force-pushed the 336-atomic-substitute branch from bf68450 to c66f251 Compare July 1, 2026 13:06
@dkharms

dkharms commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot up main bulk

@seqbenchbot

seqbenchbot commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - 8eedacf8.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@dkharms

dkharms commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot down 8eedacf8

@seqbenchbot

seqbenchbot commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

The benchmark with identificator 8eedacf8 was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary
Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations
base comp diff base comp diff base comp diff base comp diff base comp diff base comp diff
bulk
warm 68.14 66.54 -2.34% 39.10 28.75 -26.46% 55.00 59.00 +7.27% 163.00 119.00 -26.99% 223.00 194.00 -13.00% 92689.00 92790.00 +0.11%

Have a great time!

@dkharms dkharms force-pushed the 336-atomic-substitute branch from 2168c9a to 452dfe3 Compare July 1, 2026 14:29
@dkharms

dkharms commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot up main bulk

@seqbenchbot

seqbenchbot commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - 1c283919.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@dkharms

dkharms commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot down 1c283919

@seqbenchbot

seqbenchbot commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

The benchmark with identificator 1c283919 was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary
Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations
base comp diff base comp diff base comp diff base comp diff base comp diff base comp diff
bulk
warm 89.78 83.67 -6.80% 58.54 42.06 -28.15% 67.00 70.00 +4.48% 225.00 167.00 -25.78% 298.00 252.00 -15.44% 1540251.00 1540362.00 +0.01%

Have a great time!

@dkharms

dkharms commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot up main bulk

@seqbenchbot

seqbenchbot commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - 36d0de9f.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@dkharms dkharms force-pushed the 336-atomic-substitute branch from eac5148 to cc819bd Compare July 2, 2026 12:55
@dkharms

dkharms commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot down 36d0de9f

@seqbenchbot

seqbenchbot commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

The benchmark with identificator 36d0de9f was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary
Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations
base comp diff base comp diff base comp diff base comp diff base comp diff base comp diff
bulk
warm 67.39 62.57 -7.16% 43.57 29.77 -31.69% 51.00 53.00 +3.92% 173.00 118.00 -31.79% 234.00 193.00 -17.52% 716648.00 716757.00 +0.02%

Have a great time!

@dkharms dkharms force-pushed the 336-atomic-substitute branch from cc819bd to a056057 Compare July 3, 2026 09:55
@eguguchkin eguguchkin requested review from cheb0 and eguguchkin July 3, 2026 11:47
@dkharms dkharms force-pushed the 336-atomic-substitute branch from a056057 to c8ce2fd Compare July 3, 2026 12:14
@dkharms

dkharms commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot up main mixed

@seqbenchbot

seqbenchbot commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - 29d571ac.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@dkharms

dkharms commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

@seqbenchbot down 29d571ac

@seqbenchbot

seqbenchbot commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Nice, @dkharms <(-^,^-)=b!

The benchmark with identificator 29d571ac was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary
Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations
base comp diff base comp diff base comp diff base comp diff base comp diff base comp diff
bulk
warm 48.99 50.87 +3.84% 11.90 13.74 +15.47% 47.00 48.00 +2.13% 70.00 76.00 +8.57% 93.00 102.00 +9.68% 45658.00 45585.00 -0.16%
service:payment-backend-eu
AND k8s_namespace:prod
AND level:[0 to 3]
AND (
    message:'failed'
    OR message:'timeout'
)
warm 53.72 58.03 +8.02% 20.62 24.77 +20.15% 45.00 48.00 +6.67% 97.00 113.00 +16.49% 117.00 139.50 +19.23% 9615.00 9596.00 -0.20%

Have a great time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants