Skip to content

fix(executor): partition grouped global batches - #2645

Open
sylvesterkaczmarek wants to merge 1 commit into
NVIDIA:mainfrom
sylvesterkaczmarek:fix-global-batch-partitioning-2643
Open

fix(executor): partition grouped global batches#2645
sylvesterkaczmarek wants to merge 1 commit into
NVIDIA:mainfrom
sylvesterkaczmarek:fix-global-batch-partitioning-2643

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • keep grouped global-batch operators hash-partitioned even when actor concurrency is 1
  • retain at least the dataset's existing block count so unrelated source groups are not coalesced into one oversized Arrow block
  • preserve single-block behavior for truly ungrouped global-batch operators
  • add regression coverage for existing block counts, actor-pool concurrency, and ungrouped stages

Fixes #2643

Validation

  • python -m compileall -q nemo_retriever/src/nemo_retriever/graph/executor.py nemo_retriever/tests/test_global_batch_partitioning.py
  • git diff --check

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@greptile-apps

greptile-apps Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge with no actionable issues identified.

Summary

  • This PR updates Ray Data global-batch partitioning so grouped operators remain hash-partitioned even with actor concurrency of one. It preserves at least the dataset’s existing block count for grouped stages, expands partitions when actor concurrency is higher, and retains single-block behavior for ungrouped stages. Unit tests cover these three partitioning cases.

Reviews (1) · Last reviewed commit: "fix(executor): partition grouped global ..."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VideoFrameTextDedup global repartition overflows Arrow string offsets at multi-video scale

1 participant