Skip to content

[C++][CI] arrow-gcsfs-test reports "failed to listen" when the testbench is just slow to start #51011

Closed

Description

@jonasdedden

Describe the bug, including details regarding any error messages, version, and platform.

arrow-gcsfs-test fails on CI with every test reporting:

Could not start GCS emulator 'storage-testbench' (failed to listen)

Seen on AMD64 macOS 15-intel C++, all three --repeat until-pass:3 attempts. The testbench installed fine and type storage-testbench succeeded.

GcsTestbench in cpp/src/arrow/filesystem/gcsfs_test.cc gives startup a 10 second budget:

auto ready_timeout = std::chrono::seconds(10);
end = now() + ready_timeout;
while (server_process->IsRunning() && now() < end) {
  auto client = gcs::Client(...
      .set<gcs::RetryPolicyOption>(gcs::LimitedTimeRetryPolicy(ready_timeout).clone()));
  auto metadata = client.GetBucketMetadata("nonexistent");
  ...
}

The per-attempt retry budget equals the overall budget, so the loop gets one attempt, not a poll.

That budget is tight because of how the testbench serves. Its entry point ends with run_simple(..., use_reloader=True), and Werkzeug binds the listening socket in the parent, then spawns a reloader child that re-imports grpcio, protobuf and flask before answering anything. Measured with werkzeug 3.0.4 and a 2.0s stand-in for those imports:

use_reloader=1:  TCP accept at 2.10s   first HTTP response at 4.17s
use_reloader=0:  TCP accept at 2.07s   first HTTP response at 2.07s

So the port accepts connections immediately, the first response arrives after a second full import, and the message is wrong: the testbench did listen.

server_process->IgnoreStderr() at line 80 discards Werkzeug's output, so there is nothing in the CI log to tell a slow start from a real crash.

Suggested: bound each attempt separately from the overall budget, raise the overall budget, keep stderr, and fix the message.

Component(s)

C++

Activity

  1. added a commit that references this issue on Sep 25, 2026
  2. added a commit that references this issue on Sep 29, 2026
  3. added this to the 26.0.0 milestone on Sep 29, 2026
  4. kou commented on Sep 29, 2026

    @kou
    Member

    Issue resolved by pull request 51012
    #51012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions