Repository navigation
Preserve LOAD sources after unrecoverable query fallback - #19073
Conversation
|
Thanks — turning a silent partial load into a failure is the right direction. A few concerns before merge:
|
* Fix LOAD fallback error reporting and scan-boundary resumption * Fix LOAD fallback checkpoints across chunk groups * Fix LOAD fallback metadata error propagation * Fix LOAD fallback metadata batches
Tree-model LOAD can skip unreadable measurements during query fallback and still report success. With
on-success=delete, a partial load can then remove its source TsFile and companion files.Retain the first unrecoverable measurement failure while recovering readable data, then return a LOAD error so the failed source is preserved. Fully recovered files still honor the deletion policy. Track timestamps per device and measurement across chunk groups, handle both timestamp limits, and use a throwing metadata iterator that preserves measurements across batch boundaries.
Regression coverage checks recovery without duplicate points, repeated aligned and non-aligned devices, timestamp limits, unreadable metadata, more than 4000 measurements, mixed-success directories under both source policies, companion-file lifetimes, and concurrent loading followed by repair and retry.
Validation: previously run targeted checks passed with 15 converter unit tests and 2 cluster tests (3 ConfigNodes, 3 DataNodes, 3 replicas), with no failures, errors, or skips. Spotless and IT selection checks passed. The DataNode clean install and targeted cluster verification also passed. This independent review inspected the complete current diff and relevant context without rerunning unchanged passing checks.
Validation used base
3a51384b2dcc8c29118b7d082c829450c4408b4cplus this patch. Full suites, Chinese-locale compilation, a build against the latest master, and remote CI were not run before publication. An earlier additional WindowsEnvScriptITrun failed with a command syntax error; that failure remains unresolved. The focused install excluded unrelated Failsafe tests.