[Caideyipi]

Description

Add integration coverage for recovery windows that can lose replicated or loaded data, leave stale metadata, or retain staging resources indefinitely.

Pipe recovery

• Exercise receiver outages, history/realtime handoff, multiple actual DataRegions and writers, rejected requests, lost successful responses, and queue/backpressure isolation.
• Cover runtime STOP/START, failed and successful ALTER, DROP and same-name recreation, role revocation, interrupted TsFile transfer, schema changes, Region redistribution, processor restart, forwarding loops, and ConfigNode restart during DROP.

Metadata recovery

• Cover live-node lease fencing/resynchronization, DDL procedure rollback or continuation across ConfigNode failover, cross-SchemaRegion deletion, asynchronous DROP and same-name recreation, delayed attribute batches, and SchemaRegion snapshot installation followed by log-tail recovery.
• Retain schema-log tag/alias recovery, database TTL and view-cache recovery, and ConfigNode catalog snapshot plus subsequent DDL recovery.
• Schema-log recovery uses a nonempty durable mlog and a forced restart. Catalog recovery observes snapshot loading on the restarted peer, then requires that peer to become leader before checking the complete catalog, data, and follow-up DDL.

LOAD TsFile recovery

• P0-04: keep the original committed task's staged TsFile/mods/progress and WAL payload references across the IoTConsensus writer's process restart while a real replica is behind. Observe the actual cleaner scan and retain the same file set and bytes until durable peer checkpoints permit cleanup, without additional writes. Check physical data on every replica, per-file mods, duplicate timestamps, and complete query values/types.
• P0-06: require original async Tree and Table pending handoffs to complete after a witnessed temporary failure is removed, including process restart, without resubmission. Check same-named Tree files and their distinct mods, source ownership, queue cleanup, explicit Table databases, null TAG/FIELD values, and complete ordered schema.
• Keep the Tree schema conflict until an original identity handoff is rejected for missing WRITE_DATA and moved to failed; then release the other handoffs. Stop owned cluster processes before directory cleanup.

The LOAD additions are limited to three IT files and add no production hooks or interfaces. Existing fault-injection agents remain in test sources; production metadata hooks removed earlier are not restored. Compatible scenarios share class-scoped clusters.

Deletion integrity

• Check measurement and device isolation within a physically shared TsFile across full deletion, rewrite, and forced restart.
• Check that deleted points stay absent and later rewrites survive sequence/unsequence cross-space compaction and forced restart, while preserving other measurements and devices and allowing continued writes/deletes.
• Keep ATTRIBUTE-filter deletion bound to the devices selected when DELETE completes, even after matching and nonmatching attributes change. Cover NULL versus empty TAGs and isolation from another table.
• Use test-only helpers to inspect closed-file membership and physical chunk values before checking the complete SQL results.

Write recovery and admission

• Recover a flushed baseline together with acknowledged unflushed Tree and Table writes, multiple sealed WAL segments, partition boundaries, sparse fields, and overwrites. Check complete values and TAG combinations before and after crash recovery and continued writes.
• Preserve partial successes and fully rejected fields across forced restart, distinguish explicit all-NULL Table rows from rejected-only requests, and reject phantom rows or devices.
• Route one multi-device batch through different single-replica DataRegions while one owner is READONLY; verify the successful and rejected subsets, retry only the failed target, and check values and NULLs after flush and restart.
• Retain the original Session and size-one SessionPool across loss and recovery of an actually cached Ratis write leader. Use fixed-key retries, a success ledger, and connection-pool recovery assertions.
• Trigger real write-memory admission rejection with bounded requests. Verify reservation rollback, recovery of the original writers, positive accounting for recovered writes in the rejected database, and release of both databases' accounting after flush.

The write additions are limited to six IT files and add no production hooks. Preserve the target branch's existing seed-fallback cases while adapting the new route-recovery fixture.

Validation

• All newly added write scenarios were run locally before transfer. Five files match the independently reviewed source byte for byte. The Session route-recovery addition preserves the reviewed write logic while omitting an audit setup helper unavailable in the target fixture; existing target seed-fallback tests remain unchanged.
• Spotless passed for the adapted Session file; exact-content, compatibility, whitespace, and six-file commit-scope checks passed. This transfer did not rerun executable tests on the target branch or validate the combined PR head. The adapted route-recovery method, complete write selection, Linux, and Chinese locale remain unverified at this head.
• The three added deletion scenarios were run locally and passed before transfer. Their test bodies, shared setup, and supporting utility code match the independently reviewed source at 28c5028bb462 plus the working changes. This transfer did not rerun executable tests or validate the combined PR head.
• Spotless passed for all four deletion files; whitespace, commit-scope, and exact-content checks passed. The deletion update changes test sources only.
• All newly added LOAD scenarios have been run locally on Apache IoTDB / JDK 17 with ClusterIT and TableClusterIT. A combined selection passed 4 tests, 0 failures, 0 errors, 0 skipped. After strengthening the ordered Table schema assertion, the latest focused run passed 1 test, 0 failures, 0 errors, 0 skipped, with Maven exit code 0.
• Those LOAD runs used source revision 28c5028bb462 plus the working changes. The exact reviewed three-file contents were transferred into this PR. Existing source-built dependencies and node templates were reused under recorded provenance; this was not a fresh uncached full build.
• Earlier metadata P0 verification at 030e5c543f9 passed 17 tests, 0 failures, 0 errors, 0 skipped. The retained adapted schema-log and catalog recovery cases subsequently passed individually. These historical results are not a full-PR-head validation.
• Spotless passed for the LOAD source changes; whitespace and exact file-content checks passed for this update. No executable test was rerun while transferring the unchanged files to the PR branch.
• Existing Pipe and metadata suites were not rerun in this update. The combined PR head, full IT suite, other editions/profiles, and RAT remain unverified. Directory-cleanup failures were not injected.
• Precise LOAD PIECE/PREPARE faults, lost COMMIT responses, and coordinator-orphan task recovery (P0-01/02/03/05) remain outside this addition. Ordinary restart/throttling and the empty cleaner scan probe are not counted as that coverage.

Review status

Keep this PR as a draft pending an independent review of the complete updated PR diff and context. The individual workspace reviews do not approve the complete combined PR diff. The write transfer includes a target-fixture adaptation and also requires independent review at the updated head.

The Pipe-focused workflow-dispatch input pipe_p0_only continues to select IoTDBPipeRecoveryIT under MultiClusterIT2DualTreeAutoEnhanced.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.11%. Comparing base (999721f) to head (32fd0ab).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##             master   #19051     +/-   ##
===========================================
  Coverage     48.11%   48.11%             
- Complexity      621      625      +4     
===========================================
  Files          5572     5572             
  Lines        410286   411488   +1202     
  Branches      54129    54166     +37     
===========================================
+ Hits         197408   197990    +582     
- Misses       212878   213498    +620     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Caideyipi]

I reviewed the P1 metadata recovery test additions in commit c5ebc9b1c15d. I found no concrete correctness issues in the covered scenarios; the partial items below are coverage gaps, not code defects.

• P1-1: Partial — concurrent same/conflicting auto-creation and ConfigNode leader failover are checked; SchemaRegion leader failover is not covered.
• P1-2: Partial — template extension, interrupted deactivation, and recovered results are checked, but the missed-broadcast DataNode is stopped and restarted, so recovery of a still-running node with stale cache is not covered.
• P1-3: Covered — schema-log recovery checks tag/alias indexes, complete SHOW/count results, and follow-up reads and writes.
• P1-4: Partial — database TTL inheritance, an overridden table TTL, expired-read filtering, and the recovered-node write boundary are checked; dynamic per-table or Tree parent/child TTL changes are not covered.
• P1-5: Partial — Table tree-to-table auto-sync is checked after rename/replace/disable and node recovery; Tree logical-view recovery is not covered.
• P1-6: Partial — a real SchemaRegion migration is verified with metadata changes and post-migration reads/writes; other migration/rebuild failure stages are not covered.
• P1-7: Covered — quota exhaustion, failed activation, deletion/restart, bounded quota refresh, retry, and subsequent writes are checked.
• P1-8: Covered — catalog snapshot, subsequent DDL, ConfigNode recovery and leadership transfer, catalog state, reads/writes, and continued DDL are checked.

Validation: the IT-enabled reactor compilation passed across 37 modules in the source worktree. The Failsafe report confirms only IoTDBSchemaLogRecoveryIT executed (1 test passed, 0 failures); the other new ITs were not executed. The PR remains a draft.

[Caideyipi]

Metadata P0 review/fix round for 030e5c543f9efe3b378e6e01c77fde35a4b875bb.

This round found and fixed test-logic and fixture precondition issues:

• Use supported equality predicates and complete device keys for cross-Region DELETE DEVICES; retain full metadata/data assertions and the unaffected-device checks.
• Make snapshot installation real by reducing the SchemaRegion log purge gap only inside the test agent, checking actual log truncation and a newer installed snapshot, selecting an actual follower distinct from the SQL coordinator, and addressing leader-transfer RPCs to the intended new leader.
• Give the PBTree fixture enough schema memory for its pinned Region roots; prepare template catalog extensions while all nodes are online; assert the alias query's actual column header.
• Wait for all actual IoTConsensus DataRegion replicas before the DROP fixture's flush/stop. Recreate only after the exact SUCCESS / written=1 / remaining=0 result while the stale DataNode remains offline.
• Wait for a direct RPC response from the restarted ConfigNode before reusing the shared cluster.

Final validation: Apache IoTDB on JDK 17, ClusterIT + TableClusterIT, 3 ConfigNodes/3 DataNodes. 17 tests passed, with 0 failures, 0 errors and 0 skips; Maven exited 0. Tree ran in Memory/PBTree and Table in Memory. The Failsafe class XML reports confirm the counts. Spotless and git diff --check passed. Required IoTDB dependencies were built from clean source before the provenance-matched retries.

The new P0 fault agent and controls stay under integration-test/src/test; no new production hook was added. Existing P1 production hooks are unchanged and remain a follow-up/full-PR review concern. The existing Pipe/P1 suites were not rerun in this round.

This round does not approve the full PR. It remains a draft pending a fresh independent review of the latest complete diff and context.

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE