Enforce execution_timeout in deferrable DataprocSubmitJobOperator - #72030
Open
anor4k wants to merge 1 commit into
Open
Enforce execution_timeout in deferrable DataprocSubmitJobOperator#72030anor4k wants to merge 1 commit into
anor4k wants to merge 1 commit into
Conversation
DataprocSubmitJobOperator ignored execution_timeout once it deferred, so a stuck job could run forever. The operator now derives an absolute deadline from ti.start_date + execution_timeout, passes it to DataprocSubmitTrigger as execution_deadline, and also passes timeout= to defer() as a framework-level backstop. DataprocSubmitTrigger cancels the Dataproc job once the deadline passes (unless cancel_on_kill is False) and emits a TIMED_OUT event; execute_complete raises AirflowException on that event. Add tests for both the trigger and the operator, and record the new AirflowException raise in the known-exceptions allowlist.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DataprocSubmitJobOperator(deferrable=True)does not enforceexecution_timeout. Once the operator defers, the synchronousexecute()returns, and nothing bounds the deferral:self.defer()is called without atimeout, so the trigger polls the job forever. Even when a user passesdefer(timeout=...)through a subclass, a framework-level trigger timeout resumes the task only to fail it — the Dataproc job itself keeps running, because the triggerer does not run the trigger'son_killfor a timeout resume.In non-deferrable mode, exceeding
execution_timeoutraisesAirflowTaskTimeoutandon_kill()cancels the job. Deferrable mode silently loses both behaviors. The framework gap is acknowledged by# TODO: handle timeout in case of deferralintask-sdk/src/airflow/sdk/execution_time/task_runner.py.This is the same class of issue already fixed for
DbtCloudRunJobOperator(#61467 → #66449),AirbyteTriggerSyncOperator(#64048 → #64051), andKubernetesPodOperator(#67227 → #67229).What
Mirrors the approach merged in #67229:
dataproc.py): whenexecution_timeoutis set, derive an absolute deadline anchored onti.start_date(stable across re-deferrals) and pass it toDataprocSubmitTriggerasexecution_deadline. Also passtimeout=toself.defer()as a framework backstop, with one polling interval of slack so the trigger's own deadline handling (which cancels the job) fires first.triggers/dataproc.py):DataprocSubmitTriggeraccepts an optionalexecution_deadline(epoch seconds). When the deadline passes while the job is still running, the trigger cancels the Dataproc job (unlesscancel_on_kill=False) and emits aTIMED_OUTevent.execute_complete: raisesAirflowExceptionon theTIMED_OUTevent, failing the task.generated/known_airflow_exceptions.txtis bumped for the one newraise AirflowException(only the dataproc line; the file has unrelated drift on main that a full--generatewould also pick up).Behavior is unchanged when
execution_timeoutis not set (execution_deadline=None, no defer timeout).DataprocSubmitJobDirectTrigger(start-from-trigger path) has the same gap and could be handled in a follow-up.Tests
TIMED_OUT;cancel_on_kill=Falseskips the cancel; serialization includesexecution_deadline.execute_completeraises onTIMED_OUT.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, inairflow-core/newsfragments.