Repository navigation
fix(scheduler): discard a rolled-back stage's partial outputs before it re-runs - #2677
Open
akshaychitneni wants to merge 1 commit into
Open
akshaychitneni wants to merge 1 commit into
akshaychitneni wants to merge 1 commit into
Conversation
…it re-runs A finished task's output locations are pushed to the stage's output links (or, for a final stage, to the job's output) as soon as it reports. When a running stage is rolled back - on a fetch failure, or on losing an executor whose output it reads - it re-runs every task, but the outputs its finished tasks had already pushed were kept. Those partitions were then produced twice, and the query returned duplicated rows. It shows whenever a task of the stage succeeds before another fails, which is rare when every task reads from the lost executor, but routine when each task reads a single upstream partition. rollback_running_stage now drops the stage's entries from its linked stages' inputs, or from the job's output locations for a final stage, before the stage re-runs. The adaptive graph keeps stage outputs in its planner and never re-runs a rolled-back stage yet, so it is unchanged. Tests: two execution-graph tests, a final stage and an intermediate stage rolled back after partial success, and a chaos scenario that kills a map-side executor after some final-aggregate tasks have finished.
Contributor
Author
|
cc @milenkovicm @andygrove @comphead |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
When a running stage is rolled back (a fetch failure, or losing an executor
whose output it reads), it re-runs every task. But each finished task had
already pushed its output locations to the next stage's inputs (or, for a final
stage, to the job's output) when it reported, and those were kept. After the
re-run those partitions are recorded twice, so the query returns duplicated
rows with no error.
It happens whenever a task of the stage succeeds before another fails: rare
when every task reads from the lost executor, but routine when each task reads a
single upstream partition.
What changes are included in this PR?
rollback_running_stagenow drops the stage's entries from its linked stages'inputs, or from the job's output locations for a final stage, before the stage
re-runs.
back after a partial success. Both fail without the fix.
have finished; the result must match the baseline.
The adaptive graph keeps stage outputs in its planner and does not re-run a
rolled-back stage yet, so it is unchanged.
Are there any user-facing changes?
No, other than queries that hit this case no longer return duplicated rows.