Parallelize segment staging copy to deep store

[Vamsi-klu]

Why

Batch Spark/Hadoop segment generation jobs were spending a large share of wall-clock time on a single-threaded PinotFS copy/move of segment tars from the staging directory to deep store (S3/HDFS/…). For jobs that produce hundreds of segments, that serial hop dominated the SLA even after segment build and controller push were already parallelized.

This is the remaining bottleneck called out in #7510 (staging→deep-store copy), not tar push (which already supports pushParallelism).

Impact

• Faster batch ingestion jobs when staging is used (default path for Hadoop; optional for Spark-3).
• Lower time-to-deep-store for large segment counts on object stores where each move is a remote copy+delete.
• No change to segment contents, naming, overwrite semantics, or controller upload protocol.
• Default parallelism is bounded (not “one thread per segment”) to avoid thrashing remote FS connection pools.

How

• Extended SegmentGenerationJobUtils.moveFiles with a bounded thread-pool parallel implementation and an optional parallelism argument.
• Wired Spark-3 and Hadoop generation runners to pass configurable parallelism (via job extra configs / sensible default).
• Preserved existing overwrite / relative-path layout behavior; aggregate failures so a single failed move fails the job cleanly.

Test plan

• Unit tests in SegmentGenerationJobUtilsTest covering parallel vs serial layout equivalence, overwrite true/false, and failure aggregation (LocalPinotFS).
• ./mvnw -pl pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common -am -Dtest=SegmentGenerationJobUtilsTest -Dsurefire.failIfNoSpecifiedTests=false test
• Optional: Spark/Hadoop generation job tests if staging path is covered in module suites.

Related

closes: #7510

Reviewers

Suggested: xiangfu0, kkrugler (issue discussion / batch ingestion)

---

Was generative AI tooling used to co-author this PR?

• Yes — Grok Build (xAI)

Generated-by: Grok Build (xAI)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 76.00000% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.47%. Comparing base (2bcbdfe) to head (b8ce402).

Files with missing lines	Patch %	Lines
...estion/batch/common/SegmentGenerationJobUtils.java	76.00%	11 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #19082   +/-   ##
=========================================
  Coverage     65.47%   65.47%           
+ Complexity     1421     1415    -6     
=========================================
  Files          3426     3426           
  Lines        217315   217384   +69     
  Branches      34509    34525   +16     
=========================================
+ Hits         142283   142333   +50     
- Misses        63513    63527   +14     
- Partials      11519    11524    +5     

Flag	Coverage Δ
custom-integration1	?
integration	?
integration1	?
integration2	?
java-25	65.47% <76.00%> (+<0.01%)	⬆️
temurin	65.47% <76.00%> (+<0.01%)	⬆️
unittests	65.47% <76.00%> (+<0.01%)	⬆️
unittests1	56.82% <ø> (-0.01%)	⬇️
unittests2	37.90% <76.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Vamsi-klu]

Ready for review — all required CI checks are green on this PR.

Issue: #7510

Could the following folks take a look when convenient?
@xiangfu0
@kkrugler

Formal GitHub "Request review" is unavailable from a fork contributor account on apache/pinot (API returns 404/403), so this is the review ping.

Thank you!