Skip to content

schedule: a scheduled job has no timeout, no failure signal, and no declarable run policy #95

Description

@vishr

SyncSchedules installs a minimal pair of units per scheduled job — scheduleServiceUnit (internal/engine/schedule.go:148) and scheduleTimerUnit (internal/engine/schedule.go:176). The service is Type=oneshot with a single ExecStart; the timer is OnCalendar= plus Persistent=true. Nothing else is set, and nothing else is declarable in the contract.

That is the right starting point, and the reasoning in the file — the host owns the schedule, no scheduler process, no container whose job is starting containers — is sound. What follows is what the generated units do not say, ordered by how much it costs when a job misbehaves.

1. No timeout, so a wedged job blocks its own schedule silently

internal/engine/schedule.go:157 sets Type=oneshot and no TimeoutStartSec=. systemd disables the start timeout by default for Type=oneshot, so a hung docker compose run — an unresponsive registry on pull_policy: missing, a job waiting on a lock, a container ignoring SIGTERM — occupies the unit indefinitely. Because systemd will not run two instances of one unit, every later elapse queues behind it. The observable outcome is a job that stops running with no failure anywhere, which is exactly the state the file's own opening comment identifies as the worst one: "a nightly backup that silently never fires looks exactly like a nightly backup that works."

A default TimeoutStartSec= on the generated unit would convert that into a failed run. Making it declarable (schedule.timeout) would let an author whose job legitimately runs long opt out.

2. A failed run reaches nobody

The units carry no OnFailure=, and internal/engine/schedule.go never references internal/notify. Onebox is not in the loop when the timer fires, so the notifications block in the contract cannot observe a scheduled job at all — it only sees operation outcomes. Discovery today is systemctl list-timers or the journal, i.e. someone thinking to look.

This is the item worth solving even if nothing else here is: a schedule whose failures are invisible is a schedule whose reliability is unmeasured.

3. No retry

No Restart=, so a run that fails for a transient reason waits for the next cron tick. For a weekly job that is a week. Arguably correct for a cron-shaped contract — cron does not retry either — but it is currently an unstated consequence rather than a decision the author makes.

4. Persistent=true is not the author's choice

internal/engine/schedule.go:190 hardcodes it. It is the right default for a nightly backup on a box that was off, and the wrong behaviour for a job that means "at 06:00, or not at all" — a catch-up run of something time-sensitive can be worse than a skipped one. A schedule.catch_up: true|false would make the existing behaviour explicit and keep it as the default.

5. No overlap policy, and no jitter

Overlap is safe by construction — systemd will not start a second instance — so the effect of a long run is a late next run rather than a concurrent one. What is missing is a way to say "skip this elapse rather than queue it", and any signal that a run was late. Separately, no RandomizedDelaySec= means every job declared at the same hour fires on the same second; that matters little on a single host and more once several jobs share one.

Context

Found while migrating an application off a container-based cron runner (Ofelia) onto role: job + schedule.cron, which is a straight improvement — this is what the migration surfaced about the generated units, not an argument against them.

Worth noting alongside #35, which scores field-test coverage of role: job, data_effect and schedule at 0 / 11: this path has not yet been exercised on a real host, so items 1 and 2 are untested behaviour rather than observed failures.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions