chore: whack a mole ReplicationConnection.ready? flaky tests (make sure replication connection shuts down)

[PragTob]

So, the first fix was good/helpful but a bit overeager.

I didn't realize that ReplicationConnection.ready? already waited 5 seconds so that + eventually could make the test time out. Meanwhile, according to the code base, connection establishing might also take 30 seconds. So, make the wait time adjustable.

That doesn't explain why it hung forever though - that'd mean the ReplicationConnection never really gets ready.

The leading theory here is that since ReplicationConnection is shut down async from Connection and in the test suite many other values are the same is that a "zombie" replication connection survives, confuses some check and we wait forever.

timed out test:

       1) test delete tenant deletes chosen tenant (RealtimeWeb.TenantControllerTest)
Error:      test/realtime_web/controllers/tenant_controller_test.exs:306
     ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:

       1. per test by setting "@tag timeout: x" (accepts :infinity)
       2. per test module by setting "@moduletag timeout: x" (accepts :infinity)
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. by running "mix test --timeout x" which sets timeout
       5. or by running "mix test --trace" which sets timeout to infinity
          (useful when using IEx.pry/0)

     where "x" is the timeout given as integer in milliseconds (defaults to 60_000).

     code: assert eventually(fn -> Realtime.Tenants.ReplicationConnection.ready?(tenant.external_id) end)
     stacktrace:
       lib/realtime/tenants/replication_connection.ex:138: Realtime.Tenants.ReplicationConnection.Mimic.Original.Module.ready?/1
       (realtime 2.129.8) test/support/test_helpers.ex:21: TestHelpers.eventually/2
       test/realtime_web/controllers/tenant_controller_test.exs:313: (test)
       (ex_unit 1.19.5) lib/ex_unit/runner.ex:528: ExUnit.Runner.exec_test/2
       (ex_unit 1.19.5) lib/ex_unit/capture_log.ex:121: ExUnit.CaptureLog.with_log/2
       (ex_unit 1.19.5) lib/ex_unit/runner.ex:477: anonymous fn/3 in ExUnit.Runner.maybe_capture_log/3
       (stdlib 7.3.0.1) timer.erl:599: :timer.tc/2
       (ex_unit 1.19.5) lib/ex_unit/runner.ex:450: anonymous fn/6 in ExUnit.Runner.spawn_test_monitor/4

[PragTob]

Aight, that's not it... now we're timing out on explicitly shutting down the ReplicationConnection in a couple of tests... which gives some credibility to the theory that they hang out after test exit but also isn't great.

i.e.:

  2) test shutdown Connect module for tenant shuts down Connect process when tenant exists (RealtimeWeb.TenantControllerTest)
Error:      test/realtime_web/controllers/tenant_controller_test.exs:433
     ** (ExUnit.TimeoutError) on_exit callback timed out after 60000ms. You can change the timeout:

       1. per test by setting "@tag timeout: x" (accepts :infinity)
       2. per test module by setting "@moduletag timeout: x" (accepts :infinity)
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. by running "mix test --timeout x" which sets timeout
       5. or by running "mix test --trace" which sets timeout to infinity
          (useful when using IEx.pry/0)

     where "x" is the timeout given as integer in milliseconds (defaults to 60_000).

     stacktrace:
       (stdlib 7.3.0.1) gen.erl:243: :gen.do_call/4
       (elixir 1.19.5) lib/gen_server.ex:1139: GenServer.call/3
       (realtime 2.129.8) test/support/test_tenant_db.ex:117: anonymous fn/3 in TestTenantDb.do_checkout_tenant/2
       (ex_unit 1.19.5) lib/ex_unit/on_exit_handler.ex:140: ExUnit.OnExitHandler.exec_callback/1
       (ex_unit 1.19.5) lib/ex_unit/on_exit_handler.ex:126: ExUnit.OnExitHandler.on_exit_runner_loop/0

Here's the fuller log: https://github.com/supabase/realtime/actions/runs/32485064385/job/96779610823?pr=2128