Skip to content

Fix active LOAD files stranded when the last worker exits - #19076

Merged
jt2594838 merged 1 commit into
masterfrom
fix/active-load-pending-worker-wakeup
Nov 2, 2026
Merged

jt2594838 merged 1 commit into
masterfrom
fix/active-load-pending-worker-wakeup

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

A complete file published while the last active LOAD worker is retiring can remain pending until another file is enqueued. The scanner skips files already in the pending queue, leaving no runnable to process them.

Track submitted loader tasks under the executor lifecycle lock, release each task reservation on completion, and recheck the queue before leaving. Completions from a stopped executor cannot restart loading or alter its replacement's task count. No production test hooks are added.

Add focused regressions for the final empty-poll race, shutdown, and concurrent submissions, plus real-scanner integration tests for incomplete file publication, restart with mods, and listening-directory hot reload.

Validation:

  • Clean build of the required 37-module reactor.
  • ActiveLoadTsFileLoaderTest: 7 passed, no failures, errors, or skips.
  • IoTDBLoadTsFileActiveScanIT with ClusterIT: 2 passed, no failures, errors, or skips.
  • Spotless and integration-test discovery/shard checks passed.

Full test suites and Chinese-locale compilation were not run. Draft pending independent review.

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Initial independent review completed with no actionable findings in the full three-file diff or relevant context.

  • Reviewed local revision and confirmed remote source-branch/PR head: 2f8a392f7a1f68f48f5e90914a5095f8dd44c807 (base: 999721fcafd5e95ff5f3de931ad34175ea315946). No fixes or new commits were needed; no push was required.
  • Checked task reservations for concurrent submissions, the last-empty-poll handoff, stopped-executor completion, queue cleanup, and the scanner integration-test setup and assertions.
  • git diff --check passed. Existing CI reports Linux and Windows DataNode unit-test success and successful Chinese-locale compilation. The completed Linux DataNode job reports 5,662 unit tests, zero failures/errors and 48 skips, followed by BUILD SUCCESS. Builds and executable tests were not repeated in this review.
  • Some CI jobs are still running, and AINode is cancelled. The SonarCloud reliability failure points to unchanged PipeTaskInfo.java:1165, outside this PR's diff; it was not modified or independently confirmed as a defect here. The loader warnings concern the shared monitor protecting executor/task-count transitions and the rollback/rethrow when task submission fails; both have a correctness purpose.

The independent-review gate passes for the reviewed revision.

@Caideyipi
Caideyipi marked this pull request as ready for review November 2, 2026 07:34
@codecov

codecov Bot commented Nov 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.33333% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.13%. Comparing base (999721f) to head (2f8a392).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
...rageengine/load/active/ActiveLoadTsFileLoader.java 73.33% 8 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19076      +/-   ##
============================================
+ Coverage     48.11%   48.13%   +0.02%     
  Complexity      621      621              
============================================
  Files          5572     5572              
  Lines        410286   410299      +13     
  Branches      54129    54130       +1     
============================================
+ Hits         197408   197498      +90     
+ Misses       212878   212801      -77     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jt2594838
jt2594838 merged commit 1b2158f into master Nov 2, 2026
39 of 42 checks passed
@jt2594838
jt2594838 deleted the fix/active-load-pending-worker-wakeup branch November 2, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants