Skip to content

feat: implement merge source for compaction#413

Open
dkharms wants to merge 2 commits into
mainfrom
336-merge-source
Open

feat: implement merge source for compaction#413
dkharms wants to merge 2 commits into
mainfrom
336-merge-source

Conversation

@dkharms

@dkharms dkharms commented Apr 30, 2026

Copy link
Copy Markdown
Member

Description

Yet another pull request of series #336.
This one introduces all buildings blocks required for compaction of fractions.


  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

@dkharms dkharms force-pushed the 336-blockbuilder branch from a1eae0f to 25539ea Compare May 7, 2026 08:36
@dkharms dkharms force-pushed the 336-blockbuilder branch from 25539ea to a7a8eb2 Compare May 13, 2026 17:29
@dkharms dkharms force-pushed the 336-blockbuilder branch 3 times, most recently from dab7993 to e0dfb13 Compare May 25, 2026 09:37
@dkharms dkharms force-pushed the 336-merge-source branch from c46a562 to 356fd8c Compare May 25, 2026 10:56
@dkharms dkharms marked this pull request as ready for review May 25, 2026 10:56
@dkharms dkharms force-pushed the 336-merge-source branch from 356fd8c to d3e1c9e Compare May 25, 2026 11:11
@eguguchkin eguguchkin requested review from cheb0 and eguguchkin May 25, 2026 11:25
@eguguchkin eguguchkin added this to the v0.73.0 milestone Jun 8, 2026
Comment thread compaction/merge_source.go Outdated
type (
Document = util.Pair[seq.ID, []byte]
DocBlockLocation = util.Pair[[]byte, uint64]
TokenPosting = util.Pair[[]byte, []uint32]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a strange feeling that we start to use two terms for same thing (lid and posting). I'd stick to a single term (which is a lid since we don't have much choise). At the same time docs/comments might mention that lids are postings.

Besides, a single posting == a single lid. So, util.Pair[[]byte, []uint32] is techically a posting list. But I'd suggest to stick to lids and rename to TokenLids

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, what I really meant here is TokenToPostingList, but for brevity I've dropped with part with To because this is Pair. Will rename it.

Comment thread compaction/merge_source.go
Comment thread frac/sealed_source.go Outdated
Comment thread frac/sealed_source.go
@dkharms dkharms force-pushed the 336-blockbuilder branch 2 times, most recently from 7689790 to 353316d Compare June 10, 2026 10:44
@dkharms dkharms force-pushed the 336-merge-source branch from d3e1c9e to 0f1955e Compare June 10, 2026 11:12
@dkharms dkharms changed the base branch from 336-blockbuilder to 336-call-info-once June 10, 2026 11:31
@eguguchkin eguguchkin modified the milestones: v0.73.0, v0.74.0 Jun 22, 2026
@dkharms dkharms force-pushed the 336-call-info-once branch from 5cbff00 to fe8b2e8 Compare June 24, 2026 10:44
@dkharms dkharms force-pushed the 336-merge-source branch 2 times, most recently from aa2aecf to 50256a3 Compare June 25, 2026 16:00
@dkharms dkharms force-pushed the 336-call-info-once branch 3 times, most recently from 326908f to dfa67ba Compare June 26, 2026 08:14
Base automatically changed from 336-call-info-once to main June 26, 2026 08:45
@dkharms dkharms force-pushed the 336-merge-source branch from 50256a3 to b017111 Compare June 26, 2026 08:54
@codecov-commenter

codecov-commenter commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.60664% with 200 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.27%. Comparing base (98c583d) to head (6ee2e2b).

Files with missing lines Patch % Lines
compaction/merge.go 0.00% 95 Missing ⚠️
frac/sealed_source.go 0.00% 78 Missing ⚠️
compaction/merge_source.go 89.86% 13 Missing and 9 partials ⚠️
frac/common/info.go 50.00% 1 Missing and 1 partial ⚠️
frac/sealed/token/table_entry.go 50.00% 1 Missing and 1 partial ⚠️
fracmanager/frac_manifest.go 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #413      +/-   ##
==========================================
- Coverage   70.67%   70.27%   -0.41%     
==========================================
  Files         226      229       +3     
  Lines       17611    18009     +398     
==========================================
+ Hits        12447    12655     +208     
- Misses       4238     4418     +180     
- Partials      926      936      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
MergeSource-4 ------ 5c1152
NaN B/op 74649824.00 B/op NaN 🔴
NaN allocs/op 2010802.00 allocs/op NaN 🔴
NaN ns/op 1550018039.00 ns/op NaN 🔴

Comment thread frac/sealed_source.go Outdated
Comment thread frac/sealed_source.go Outdated
Comment thread frac/sealed_source.go
Comment thread frac/sealed_source.go
Comment thread compaction/merge_source.go
Comment thread compaction/merge_source.go
Comment thread compaction/merge_source.go Outdated
Comment thread compaction/merge_source.go Outdated
Comment thread compaction/merge_source.go
Comment thread compaction/merge.go
@dkharms dkharms force-pushed the 336-merge-source branch from e75391a to 6ee2e2b Compare July 3, 2026 09:44
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
MergeSource-4 ------ fbf1fa
NaN B/op 74650272.00 B/op NaN 🔴
NaN allocs/op 2010803.00 allocs/op NaN 🔴
NaN ns/op 1532087124.00 ns/op NaN 🔴

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants