Repository navigation
[hotfix][ci] Finalize PyArrow 11 on exit in Flink 1.20 tests - #1343
Merged
Merged
Conversation
Record both command statuses before normalization and run three independent Flink 1.20 Python integration jobs to investigate post-test CI failures. Generated-by: Codex CLI 0.153.4 (GPT-6) Co-authored-by: Codex <codex@openai.com>
Run ten independent Python integration samples and Ubuntu script checks. Temporarily disable unrelated jobs while the diagnostic PR remains draft. Generated-by: Codex CLI 0.153.4 (GPT-6) Co-authored-by: Codex <codex@openai.com>
wenjin272
changed the title
github-actions
Bot
added
doc-not-needed
Enable runner-local kernel core dumps without wrapping pytest, export native thread stacks and environment evidence after testing, and upload text-only diagnostics for each sample. Generated-by: Codex CLI 0.153.4 (GPT-6) Co-authored-by: Codex <codex@openai.com>
github-actions
Bot
added
doc-not-needed
Register the existing PyArrow cleanup function in the CI-installed wheel and verify the patch survives dependency setup. Keep native crash collection for ten Flink 1.20 samples. Generated-by: Codex CLI 0.153.4 (GPT-6) Co-authored-by: Codex <codex@openai.com>
wenjin272
changed the title
github-actions
Bot
added
doc-not-needed
Restore all normal CI jobs and version matrices. Remove raw exit logging, core collection and repeated samples while retaining the Flink 1.20-only cleanup patch and its checks. Generated-by: Codex CLI 0.153.4 (GPT-6) Co-authored-by: Codex <codex@openai.com>
wenjin272
changed the title
Remove standalone patch tests and post-test verification. Keep only the Flink 1.20 setup step and guarded cleanup registration. Generated-by: Codex CLI 0.153.4 (GPT-6) Co-authored-by: Codex <codex@openai.com>
wenjin272
marked this pull request as ready for review
October 30, 2026 08:55
github-actions
Bot
added
doc-not-needed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related upstream issue: apache/arrow#38364
Purpose of change
Flink 1.20 Python CI can finish all tests successfully and then crash while PyArrow 11 destroys AWS objects during process exit. Register PyArrow's existing
finalize_s3()function withatexitin the CI-installed wheel so pytest and independently launched Beam workers clean up AWS resources before C++ static destruction.Add one setup step to the Flink 1.20 Python integration job: install its dependencies and patch the shared PyArrow module before the existing test command. The shared installation covers workers that a pytest-only hook would miss. Flink 1.20's Beam dependency requires PyArrow <12, preventing an isolated upgrade to the upstream-fixed version. Remove this CI-only workaround when a compatible dependency upgrade is available.
Behavioral Semantics
The helper reads package metadata without importing native Arrow code. The normal CI matrices, test commands, and exit-code handling remain unchanged. No runtime package, public API, or dependency constraint changes. This is explicit shutdown cleanup, not a full backport of Arrow's lazy initialization.
Tests
Ten independent upstream Flink 1.20 jobs passed with the same cleanup registration. Each completed 82 E2E and 69 integration tests with both command exit codes 0. All ten core inventories and kernel crash logs were empty; dependency setup preserved the patch.
Ruff, shell syntax, and
git diff --checkpass. Version/source rejection was exercised during development; no standalone test harness or post-test verification step is retained. The final simplified revision still requires its normal CI run. This workaround is limited to CI and does not fix user-installed environments.API
No public API changes.
Documentation
doc-neededdoc-not-neededdoc-includedWas this patch authored or co-authored using generative AI tooling?
Generated-by: Codex CLI 0.153.4 (GPT-6)