[luoluoyuyu]

Description

TestUtils.restartCluster() returned as soon as every node reported a non-Unknown status and a
JDBC connection succeeded. A running node does not imply a serving Region: right after a restart the
ConfigNode LoadCache keeps the RegionGroups of a Database Disabled until all of their Regions
report Running, and any write issued in that window, including an active LOAD, is rejected with
NO_AVAILABLE_REGION_GROUP (906).

IoTDBLoadTsFileActiveScanIT#testIncompletePublicationSurvivesScannerRestartAndPreservesMods
restarts the cluster and immediately publishes a canary TsFile into the active LOAD listening
directory. ActiveLoadDirScanner rescans that directory every second, so the canary was picked up
inside the window described above, the LOAD failed permanently (the file was moved to
ext/load/failed and the status was rewritten to LOAD_FILE_ERROR) and awaitRows timed out after
60 seconds.

Timeline from a failing CI run:

16:00:38.549 DataNode17821 "DataNode started"
16:00:39.674 active load scanner picked 712-0-0-0.tsfile
16:00:39.794 ConfigNode: No available SchemaRegion ... {4=Disabled, 2=Disabled, 0=Disabled}
16:00:39.800 Failed to auto load ... File will be moved to fail directory
16:00:39.878 same RegionGroups: Disabled -> Running

The window is ~100 ms wide, so the test passes most of the time and fails rarely.

Fix

restartCluster() now also calls the new TestUtils.awaitAllRegionsRunning(env), which waits
(bounded by 2 minutes, polled every 100 ms) until every Region listed by SHOW REGIONS is
Running.

SHOW REGIONS reports LoadManager.getRegionStatus, read from the same LoadCache that computes
RegionGroupStatus (RegionGroupCache#caculateRegionGroupStatus: a group is Running only when all
of its Regions are Running). "All Regions Running" is therefore exactly the condition that
guarantees the next write cannot hit NO_AVAILABLE_REGION_GROUP — it waits on the same state that
rejects the write, not on an approximation, and it is an observable condition instead of a fixed
sleep. No assertion of the affected tests is changed, and the active LOAD test itself is untouched.

SHOW REGIONS only lists the Databases that belong to the dialect of the connection
(TreeConfigTaskVisitor.visitShowRegion() passes isTableModel=false and
PartitionInfo#getRegionInfoList filters the other model out), so the wait checks the tree model
and the table model through separate connections; otherwise a table-only cluster would wait for
nothing and a mixed cluster would only wait for its tree Regions.

Clusters without any RegionGroup

If the cluster has no Database yet, SHOW REGIONS returns no row and the wait returns immediately.
That is the intended behaviour: there is no RegionGroup that could be Disabled, and the statement
that creates a Database also creates its RegionGroups in the same plan, which
CreateRegionGroupsProcedure activates as Running (env.activateRegionGroup with a
RegionStatus.Running sample; Unknown only for a replica whose creation failed). So the state the
helper waits for is the one produced by a restart, where existing RegionGroups fall back to
Unknown until their DataNodes report again.

Tests

• IoTDBLoadTsFileActiveScanIT (existing): the flake this PR fixes; its assertions are unchanged.
• IoTDBRegionReadyAfterRestartTableIT (new, TableClusterIT): creates a table-model Database,
  calls restartCluster(), then asserts that the table connection lists Running Regions and that
  the first write after the restart succeeds. It fails with a tree-only readiness check, where
  SHOW REGIONS over the tree connection lists no Region of a table-only cluster.

Alternatives considered

• Probing the Database with an INSERT inside the affected test: works, but only covers one
  Database and has to be repeated in every restart-based IT.
• Waiting for all Regions inside checkClusterStatusWithoutUnknown(): would also cover the initial
  cluster start, but that helper is used by many tests that do not need the extra wait, so the wait
  is scoped to restartCluster() here.
• Making active LOAD retry instead of failing the file (the product-side counterpart of this race):
  kept out of this PR. A DataNode cannot observe the ConfigNode LoadCache view during startup, and
  blocking active LOAD globally until every Region is ready would stall healthy Databases whenever a
  single RegionGroup is unavailable.

---

This PR has:

• been self-reviewed.
• added comments explaining the "why" and the intent of the code wherever would not be obvious
  for an unfamiliar reader.
• added integration tests.

Not applicable: documentation, version/license information, concurrency of the changed helper (it
runs in the test JVM only, single-threaded via Awaitility).

---

Key changed/added classes (or packages if there are too many classes) in this PR

• integration-test/src/test/java/org/apache/iotdb/db/it/utils/TestUtils.java:
  restartCluster() calls the new awaitAllRegionsRunning(), which checks the tree and the table
  dialect.
• integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBRegionReadyAfterRestartTableIT.java:
  new coverage for a table-model-only cluster.

[Caideyipi]

Reviewed the full diff at a97ad9f, the restart callers, and the Region status query path. The new wait skips table-model Regions because it queries only the tree SQL dialect; see the inline comment. No additional tests were run. Completed CI checks are passing, with the DataNode unit-test job still in progress.

[Caideyipi]

[P2] Include table-model Regions in the readiness check

BaseEnv.getConnection() always opens a tree-dialect connection. In that dialect, TreeConfigTaskVisitor.visitShowRegion() passes isTableModel=false, and PartitionInfo.getRegionInfoList() filters out table-model databases. A table-only cluster therefore produces an empty result here and immediately passes the wait even while its RegionGroups are still Disabled; a mixed cluster only waits for its tree Regions. Existing table-model callers such as IoTDBPipeLifeCycleIT.testLifeCycleWithClusterRestart() consequently retain the restart/write race that this helper is intended to prevent. Please check both tree and table SHOW REGIONS results, or use a query that includes both models, and cover the table-only case.

[luoluoyuyu]

Good catch, fixed in ebe7ce9. awaitAllRegionsRunning() now runs the check on both dialects inside the same Awaitility condition: env.getConnection() (tree) and env.getTableConnection() (table), each asserting that every Region it lists is Running. A table-only cluster therefore waits for its table Regions, and a mixed cluster waits for both.

Also added IoTDBRegionReadyAfterRestartTableIT (TableClusterIT): it creates a table-model Database, calls TestUtils.restartCluster(), then asserts that SHOW REGIONS over the table connection lists Running Regions and that the first write after the restart succeeds. With a tree-only readiness check the regionCount > 0 assertion fails on a table-only cluster, so the regression is covered.

[jt2594838]

Maybe reduce the poll interval to avoid unnecessary wait.

[luoluoyuyu]

@jt2594838 thanks, reduced the poll interval from 1 s to 100 ms in ebe7ce9 (same order as the other Awaitility helpers in TestUtils). The wait still starts with a 0 ms poll delay and returns as soon as the Regions are Running, so the added cost per restartCluster() is now at most one poll tick plus the Region recovery time itself.

Both review points are in ebe7ce9:

• table-model Regions are checked through a second, table-dialect connection;
• regression coverage for a table-model-only cluster added as IoTDBRegionReadyAfterRestartTableIT.

CI is re-running on the new commit.