Skip to content

Preserve LOAD sources after unrecoverable query fallback - #19073

Merged
jt2594838 merged 4 commits into
apache:masterfrom
Caideyipi:fix/load-fallback-source-retention
Nov 2, 2026
Merged

jt2594838 merged 4 commits into
apache:masterfrom
Caideyipi:fix/load-fallback-source-retention

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Tree-model LOAD can skip unreadable measurements during query fallback and still report success. With on-success=delete, a partial load can then remove its source TsFile and companion files.

Retain the first unrecoverable measurement failure while recovering readable data, then return a LOAD error so the failed source is preserved. Fully recovered files still honor the deletion policy. Track timestamps per device and measurement across chunk groups, handle both timestamp limits, and use a throwing metadata iterator that preserves measurements across batch boundaries.

Regression coverage checks recovery without duplicate points, repeated aligned and non-aligned devices, timestamp limits, unreadable metadata, more than 4000 measurements, mixed-success directories under both source policies, companion-file lifetimes, and concurrent loading followed by repair and retry.

Validation: previously run targeted checks passed with 15 converter unit tests and 2 cluster tests (3 ConfigNodes, 3 DataNodes, 3 replicas), with no failures, errors, or skips. Spotless and IT selection checks passed. The DataNode clean install and targeted cluster verification also passed. This independent review inspected the complete current diff and relevant context without rerunning unchanged passing checks.

Validation used base 3a51384b2dcc8c29118b7d082c829450c4408b4c plus this patch. Full suites, Chinese-locale compilation, a build against the latest master, and remote CI were not run before publication. An earlier additional Windows EnvScriptIT run failed with a command syntax error; that failure remains unresolved. The focused install excluded unrelated Failsafe tests.

@jt2594838
jt2594838 merged commit 9463d01 into apache:master Nov 2, 2026
38 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/load-fallback-source-retention branch November 2, 2026 07:09
@luoluoyuyu

Copy link
Copy Markdown
Member

Thanks — turning a silent partial load into a failure is the right direction. A few concerns before merge:

  1. The checkpoint is a per-series time watermark. recordProgress advances every measurement in a tablet to the tablet's last row timestamp, and readDeviceMeasurementsInOrder treats emitted >= statistics.getEndTime() as "fully loaded". That only holds if each series' timestamps are non-decreasing across chunk groups, which TsFile writers don't enforce. If a later group holds an earlier timestamp, those points are skipped without recording a failure → SUCCESS + on-success=delete still deletes the source, i.e. the same bug class. Suggest advancing only to the last row where that measurement really has a value (nulls in aligned tablets over-advance it), plus a test with a non-monotonic file.

  2. No best-effort path. One unreadable measurement now makes the whole file permanently unloadable, even when the corrupt region contains no data. Worth an explicit decision (or an opt-in flag).

  3. Upstream retry heuristics key off message text (ActiveLoadFailedMessageHandler.isExceptionMessageShouldRetry). The surfaced message is the generic "Failed to iterate tablets while loading TsFile.", so even transient IO errors get moved to the fail dir. Including device/measurement and the cause (via a DataNodeQueryMessages constant) would make this diagnosable and retryable.

  4. Task count: one QueryTask per distinct checkpoint means one TsFileReader open per task; wide devices can turn fallback into thousands of file opens, and the split path can emit O(M) WARNs each printing the full measurement list.

  5. Minor: a split task whose halves are all filtered out by emitted >= endTime loses the failure entirely; only the first failure is reported; and waitUntilClusterAnswers() returning true unconditionally weakens the IT readiness check (unrelated to this fix).

VGalaxies pushed a commit to VGalaxies/iotdb that referenced this pull request Nov 2, 2026
* Fix LOAD fallback error reporting and scan-boundary resumption

* Fix LOAD fallback checkpoints across chunk groups

* Fix LOAD fallback metadata error propagation

* Fix LOAD fallback metadata batches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants