redshift/parser: retry first-set oracle container startup on slow CI runners - #405
Merged
Conversation
…runners The redshift-container-tests job intermittently failed (3 times in one day on PR #403) with "first-set oracle unavailable in CI: container start: ... create container: context deadline exceeded" across all four LeadTokensMatchPG tests. Root cause: startFirstSetOracle gave container creation only 15 seconds, which a busy GitHub runner can exceed on the postgres:17-alpine image pull alone. The sibling harnesses (paren_oracle_test.go, redshift/catalog/container_helper_test.go) were already bumped to 120s; this helper was left behind. - Extract one bootstrap attempt (create, conn string, open, ping) into startFirstSetOracleAttempt with the repo-standard 120s per-attempt timeout; on failure it tears down every partially-created resource, including the container tcpg.Run can return alongside its error, so retries don't leak containers on the runner. - Retry the attempt 3 times inside the sync.Once body, 2s apart, with a t.Logf per failed attempt so CI logs show the retries. The docker preflight stays outside the loop (deterministic failure, keeps the local skip path fast). - Keep the fail-in-CI / skip-locally policy: after 3 failures the errors.Join'd attempt errors flow into the existing t.Fatalf, so the tests still fail loudly rather than skip when the oracle is genuinely unavailable. Verified with `go test ./redshift/parser -run 'LeadTokensMatchPG' -count=1` against local Docker: all four tests pass. Worst case in CI is 3x120s plus pauses, well inside the job's 30-minute limit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
redshift-container-testsCI job intermittently fails (3 times in the last day on #403 alone) with the same signature across all four first-set oracle tests (TestSimpleTypenameLeadTokensMatchPG,TestTypenameLeadTokensMatchPG,TestAExprLeadTokensMatchPG,TestFuncTypeLeadTokensMatchPG):Root cause:
startFirstSetOraclegave container creation only 15 seconds. On a busy GitHub runner thepostgres:17-alpineimage pull alone can exceed that. The sibling harnesses in this repo (redshift/parser/paren_oracle_test.go,redshift/catalog/container_helper_test.go) were already bumped to 120s with the "Generous timeout" comment — this helper was left behind.Fix
startFirstSetOracleAttempt()with the repo-standard 120s per-attempt timeout. On failure it tears down every partially-created resource — including the containertcpg.Runcan return alongside its error (TerminateContaineris nil-safe) — so retries don't accumulate leaked containers on the runner.sync.Oncebody, 2s apart, with at.Logfper failed attempt so CI logs show retries happening. The docker preflight stays outside the loop: a missing daemon is deterministic, and this keeps the local skip path fast.errors.Join'd attempt errors flow into the existingt.Fatalf("first-set oracle unavailable in CI: ...")— the tests still fail loudly (never skip) when the oracle is genuinely unavailable, preserving the FIRST-set drift guardrail.Worst case in CI is 3×120s plus pauses (~6 min), well inside the job's 30-minute limit.
Verification
against local Docker: all four tests pass (shared container starts once, ~6.4s total).
go vet ./redshift/parserclean.🤖 Generated with Claude Code