Repository navigation
Feat: automation dependencies - #2706
Open
BelhsanHmida wants to merge 36 commits into
Open
BelhsanHmida wants to merge 36 commits into
BelhsanHmida wants to merge 36 commits into
Conversation
…s end Context: - Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs. Change: - record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded. - The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job. - Plugin jobs carry automation_run_id in their trigger. - A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Schedule, report and plugin runs now record how their jobs ended. Change: - Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded. - A plugin run records its job's outcome, also when the job finished before the run recorded it. - Report run tests check the run's outcome: succeeded, nothing to do, and refused. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Schedule, report and plugin runs now record how their jobs ended, as forecast runs did. Change: - Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor. Change: - Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py. - Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor. - The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type. - The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue. Change: - Say execution state and job records are kept for every type of automation. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py. Change: - Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Issue #2586: an automation consuming another's results could only be ordered by staggering cron strings Change: - New automation_dependency table, and 'waiting' and 'skipped' run states - A run waits for the upstream runs scheduled for the same time, and is dispatched once they ended, or skipped if one failed (unless failure is allowed) - Waiting runs are released when the upstream run ends, and by the runner's tick as a backstop - Dependencies are validated: same schedule, no cycles, readable upstream automations Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…copy Context: - Issue #2586 asks what a copy does with a dependency Change: - A copy waits for the copied upstream automation where that was copied too, and for the original one otherwise Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Issue #2586 Change: - depends-on on creating, updating and reading automations, and upstream-runs on run stats - PATCH checks the user may read the upstream automations, and refuses a schedule that breaks a dependency Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Issue #2586 Change: - add/edit automation take --depends-on and --allow-failure-of, edit also --no-dependencies - show automation lists dependencies, run-automations reports waiting and skipped runs Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Issue #2586 Change: - A 'Waits for' picker on the create and edit forms, copying the picked automation's schedule into the create form - The details panel lists the dependencies Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Issue #2586 Change: - Runner tests with a real worker: wait then release, skip, allow-failure, cascade, missed release, passed-over and inactive upstreams - Validation, API permission and schedule checks, CLI options, and copied dependencies Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - Issue #2586 Change: - New section on dependencies, the waiting and skipped run states, and that a run on demand does not wait Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Context: - PR #2706 Change: - Entry in 'Automations, in detail' and in the CLI changelog Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
17 tasks
Resolve conflicts with the squash-merged #2670 and with #2657: take main's naming of schedule jobs and its job-failure path, keep the release of waiting runs where a run ends, and use AUTOMATION_RUN_RETRIED_TYPES, which now includes scheduling. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…tion bf22b21d0990 Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
A run now waits for each upstream automation's latest run due at or before its own scheduled time, by the upstream automation's own cron string and timezone, so for instance an hourly forecast can wait for meter data pulled every 15 minutes. The same-schedule check on creation and edit is dropped, and so is the UI copying the upstream schedule into the form. The run record names the time the upstream run was due at (due-at). Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
flexmeasures jobs run-automation --with-dependents, the trigger endpoint's with-dependents query parameter and the Automations page's Run with dependents start a run of the automation now, and a run for each active automation that waits for it, directly or through others, which waits for the runs started along with it, and is skipped if one of those failed. This is how a chain is re-run, for instance after backfilling the data at its start. A run records whether it was due by schedule or asked for on demand (trigger), so that on-demand runs can share a minute with scheduled runs, and a scheduled run only ever waits for scheduled upstream runs. The runner does not retry on-demand runs, as whoever started them is told when they fail. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
The details panel on the Automations page draws the automations listed there that this one waits for, on the left, and those that wait for it, on the right, directly or through others, with a dashed arrow where the waiting automation runs even when the one it waits for failed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
The Waits for field in the creation and edit forms was a multiple select, which took Ctrl-clicks to pick several automations and cut long names off. Each automation is now a row with a tick box, naming its asset and recurrence, and dragging the corner of the list enlarges it. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
v1.1.0 shipped without this feature, so its changelog entries move to new sections after it. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Clicking an automation in a chain opens its details on the page of its own asset. Hovering over an automation tells its type, asset and recurrence, and hovering over an arrow what a failure does. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
A killed job never recorded its outcome, which left its run unfinished, and everything waiting for that run waiting forever. On each tick, the runner now looks up the unfinished jobs in Redis, and records what became of them. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
The PATCH description said it could not, which was never the case. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
…ing for it The chain, the details and the edit form only had its ID, so they could not link to it. Automation responses now give its name and asset to whoever may read it. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Run with ascendants brings one automation up to date: what it waits for runs first, and what waits for it does not run. What may not be run is left out and reported, and a new run-plan endpoint tells what a run would come down to. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Run with ascendants joins the actions, and both ways of running a chain now show the IDs that run, in order, first. Branches above an automation can be left out there, and what will not run says why. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Refusing a dependency that would close a loop showed the raw response in the edit form. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
It then runs without it, which was easy to miss, so the page and the CLI name those automations first. They can be on other assets, so an automation's details now report waited-for-by. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
One selector offers those on the page, on this asset only, on public assets, or every automation you can see. The latter two come from a new listing across assets, of what the user may read. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Next to the chain of the automation being read, its details can now also draw the automations that have nothing to do with it. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
It used to be a blank box. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
An automation with nothing new to do still sets off what waits for it, which read as if nothing happened. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
It blamed the automation's parameters, which hold no sensor references: its flex config can. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Waiting for an automation gave 403 for another organisation's and 422 for an unknown ID, which told which IDs exist. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
An automation can now wait for the automations whose results it uses, instead of relying on staggered cron strings.
For example, a data pull feeds a report, which feeds a forecast.
allow-failure).--with-dependents,?with-dependents=true) re-runs a whole chain, for example after a backfill.--with-ascendants,?with-ascendants=true) brings one automation up to date: what it waits for runs first, it runs last, and what waits for it does not run.run-planendpoint. Branches above an automation can be left out there (leave-out), and what will not run says why, such as an automation on a public asset.GET /assets/automations?scope=public|all, which lists what you may read, 500 at most.waited-for-by.depends-on, CLI--depends-on/--allow-failure-of/--no-dependencies, and a UI Waits for list of tick boxesname,assetandasset-nameof each upstream automation the caller may read. One they may not read stays a bare ID.automation_dependencytable,waiting/skippedrun states, and a runtrigger(scheduleoron-demand)documentation/changelog.rstdocumentation/cli/change_log.rstdocumentation/api/change_log.rstUnlike the approach on the issue, runs wait in the database rather than through RQ's
depends_on, for two reasons:depends_onpassed through.A waiting run is released as soon as its upstream run ends.
A job reports its own outcome, which it cannot do when it is killed, so the runner asks Redis about the jobs that are still unfinished.
RQ fails a job whose work-horse died, and one whose worker disappeared once its heartbeat lapsed.
A job that left Redis without reporting counts as failed, and a job that is still queued or running is left alone, however long it takes.
A chain run on demand records a run per automation, so its runs can wait for each other.
They wait only for the runs started with them, and never for scheduled ones.
Running with ascendants can have several starting points, one per branch, and each starts right away.
An automation that does not run takes along what is above it, as nothing would use those results, and what waits for it goes by the results it already has.
No permission changes here: running needs permission to manage automations, so a normal user sees a public automation as one that won't run.
The unique index on runs now covers scheduled runs only, so an on-demand run can share a minute with a scheduled one.
The runner does not retry on-demand runs.
Look & Feel
How to test
pytest flexmeasures/data/tests/test_automation_dependencies_fresh_db.py flexmeasures/api/v3_0/tests/test_automation_dependencies_api_fresh_db.py flexmeasures/ui/tests/js/test_automation_chain.py, plus the dependency tests incli/tests/test_automations.pyandtest_asset_copy_automations_fresh_db.py.kill -9the work-horse of a running job, and runflexmeasures jobs run-automations: its run fails with the reason, and what waits for it is released.Further Improvements
skippedruns.Related Items
Closes #2586. Part of #2557.
Sign-off