Skip to content

Integration test suite against a real Redis server - #20

Open
Morry98 wants to merge 3 commits into
mainfrom
feature/integration-tests-redis
Open

Morry98 wants to merge 3 commits into
mainfrom
feature/integration-tests-redis

Conversation

@Morry98

@Morry98 Morry98 commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Integration test suite against a real Redis server

What changes

  • tests/integration/: a new suite that runs one or more TaskManager instances against a real Redis server.
    • conftest.py: REDIS_URL (default redis://localhost:6379/15), a clean skip when Redis does not answer (a failure instead when REDIS_REQUIRED is set, as in CI, so a missing service cannot turn the suite into silent skips), a random key prefix per test, teardown that deletes only that prefix's keys (SCAN + DEL, never FLUSHDB/FLUSHALL), a make_manager factory that starts instances with scaled-down timings and stops them all at teardown, and polling helpers (wait_until, wait_for_leader). Waits poll for a condition with a timeout. Fixed sleeps are not used.
    • test_scheduling.py: 8 safety-net tests for behaviour that works today: a triggered task runs exactly once (stats recorded, runtime keys cleared), a due cron task runs and next_run moves to the next occurrence, only one instance becomes leader, an instance that is not leader never schedules, leadership fails over, a triggered task runs once across two instances, allow_parallel=False waits for the running execution, and the consumer recovers after the app's keys are flushed (the 1.1.1 NOGROUP fix, previously uncovered).
    • test_known_bugs.py: one test per open finding, each asserting the correct behaviour and marked xfail(strict=True, raises=AssertionError, reason="FTM-xx: …"). Covered: FTM-01 (Reconciler requeue and startup XAUTOCLAIM, 2 tests), FTM-02 (dynamic task created on a follower never runs; deleted on a follower stays on the leader), FTM-03 (message for an unknown task is ACKed and lost in a rolling deploy), FTM-04 (no retry after backoff), FTM-06 (disabled flag expires), FTM-07 (yearly task runs at first startup), FTM-11 (tag filter matches untagged tasks), FTM-12 (group tags drop task tags). When a fix lands, the test reports XPASS(strict) and fails the run until the marker is removed, so the scenario becomes a regression test. Setup steps raise PreconditionError instead of AssertionError, so a broken setup is reported as a real failure and is never counted as the expected bug.
  • .github/workflows/integration.yml (new; ci.yml is untouched): integration job with a redis:7-alpine service, matrix Python 3.10 and 3.14; a separate lowest-direct job that runs both suites on Python 3.10 with UV_RESOLUTION=lowest-direct. permissions: contents: read at the top level, actions pinned to full SHAs with a # vX.Y.Z comment, persist-credentials: false, timeout-minutes: 15 per job (shutdown waits for running tasks without a timeout), REDIS_REQUIRED=1.
  • pyproject.toml:
    • integration marker;
    • addopts = "-m 'not integration'", so plain pytest (and the existing CI job) behaves as before and never contacts Redis; pytest -m integration overrides it;
    • filterwarnings = ["error", …] with one targeted exception: on Python 3.14 asyncio.iscoroutinefunction (still used by the stream consumer) emits a DeprecationWarning that, as an error, makes 3 unit tests fail. Remove that entry once the consumer switches to inspect.iscoroutinefunction;
    • [tool.coverage.run] omit = ["tests/integration/*"]: without it the deselected integration files count as uncovered and the reported total drops from 97% to 89%.
  • tests/test_statistics.py: the pipeline mock is now a MagicMock with only execute async, matching redis-py. This removes the 4 RuntimeWarning: coroutine ... was never awaited.
  • CONTRIBUTING.md: how to run the integration suite locally (throwaway Redis, REDIS_URL, marker) and what the strict xfail tests mean.

Why

All 251 existing tests mock Redis, and the distributed bugs listed above pass with a green suite. This suite reproduces them with real instances sharing one Redis, documents each one as an executable test, and gives the future fixes a ready-made regression test.

How to test

uv run pytest                      # 251 passed, 18 deselected, no Redis needed
redis-server --port 6399 --save "" --appendonly no --daemonize yes
REDIS_URL=redis://localhost:6399/0 uv run pytest -m integration   # 8 passed, 10 xfailed
REDIS_URL=redis://localhost:6399/0 uv run pytest -m integration --runxfail   # shows the 10 bug messages
redis-cli -p 6399 shutdown nosave

Results:

  • Python 3.13: 251 passed, 18 deselected; integration 8 passed, 10 xfailed. coverage report total 97%. ruff check, ruff format --check, ty check and pre-commit run --all-files pass.
  • Python 3.14: unit 251 passed, integration 8 passed, 10 xfailed.
  • UV_RESOLUTION=lowest-direct (fastapi 0.115.13, redis 6.2.0, pydantic 2.11.7, starlette 0.46.2, pytest 8.3.5): unit 251 passed, integration 8 passed, 10 xfailed.
  • With --runxfail, each known-bug test fails on its own assertion and the message names the bug, for example FTM-01: a running task with allow_parallel=False was started 2 times, at [0.0, 3.0] s, FTM-06: the disabled flag expires on its own (TTL 432000 s), FTM-03: … acknowledged and dropped (stream length now 0). The captured logs show the faulty code path (requeued stuck low priority message, Task not found … acknowledging).
  • Slowest test is about 8 s. The integration run takes 35–60 s (up to about 100 s on a heavily loaded machine). The final version passed 5 consecutive runs on 3.13 with a load average around 40 on 8 cores, with no keys left in Redis afterwards (DBSIZE 0). Waits for something that must happen use a 15 s upper bound (EVENTUALLY), which costs nothing on a passing run because wait_until returns as soon as the condition holds. Before that change, one run in five failed under this load (pending trigger never ran after the leader lock expired during a stall). Windows in which something must not happen, and the windows of the known-bug tests, stay short. Under the same load the FTM-01 (Reconciler) test once reported a false XPASS(strict): its window started when the task started, and the Reconciler only acts once the message has been idle for more than 2 s. For the three known-bug tests in which the bug is an event that has to happen on its own (FTM-01 ×2, FTM-07), the window (BUG_WINDOW, 5 s) now starts only after the bug's preconditions hold, so a slow machine cannot make the bug look fixed. The minimal fixes still give XPASS(strict), and for FTM-07 the Coordinator change alone still stays xfailed. On the final tree: Python 3.14 and lowest-direct both give 259 passed, 10 xfailed (unit and integration in one run).
  • With Redis down: skipped without REDIS_REQUIRED, setup error Failed: Redis not reachable at … with it.

Mutations

Each mutation was applied one at a time to src/. The run checked the occurrence count before and after, ran only the relevant tests, then restored the file and verified the checksum.

Safety-net tests (each must go red):

# file mutation test result
M1 leader_election.py nx=True removed from the lock SET test_only_one_instance_becomes_leader RED (two instances believe they are leader)
M2 coordinator.py continue removed after a failed acquisition (followers schedule too) test_instance_that_is_not_leader_never_schedules RED (an instance that is not leader rescheduled the task)
M3 stream_consumer.py setup_consumer_groups() removed from the NOGROUP handler test_consumer_recovers_after_redis_keys_are_flushed RED (consumer group was never recreated)
M4 coordinator.py if not effective_allow_parallel → if False test_allow_parallel_false_waits_for_running_execution RED (a second execution started while the first was running)
M5 coordinator.py _update_next_run call removed test_triggered_task_runs_exactly_once, test_cron_task_runs_when_due_… RED (['A', 'A'] == ['A'])
M6 coordinator.py next_run <= now → >= test_cron_task_runs_when_due_… RED (['A', 'A'] == ['A'])
M7 stream_consumer.py running key not deleted on success test_triggered_task_runs_exactly_once RED (running key left behind after success)

M7 first survived because the test waited up to 2 s for the running key to disappear, and the key's own 2 s TTL removed it anyway. The test now waits only for the consumer's last success step (ACK + XDEL of the message), then checks the running key immediately.

Waits never carry the assertion. Every fact polled inside a wait_until helper is asserted again directly after the wait, so a polling helper that becomes vacuous cannot hide a product bug. In each row below the helper is mutated together with the product bug it waits for:

# test-side mutation product mutation result
T1a none srem of the scheduled set removed on success RED (task still tracked as scheduled after success)
T1b ACK wait helper → return True none green, as expected: the helper only synchronises and every fact is asserted after it
T1c ACK wait helper → return True srem removed on success RED (task still tracked as scheduled after success)
T2 _next_run_changed → return False followers publish (M2) RED (an instance that is not leader rescheduled the task, '1798761600.0' == '0')
T3 started_again wait → lambda: False allow_parallel check off (M4) RED (a second execution started while the first was running)
T4 group_exists → return True no group recreation on NOGROUP (M3) RED (consumer group was never recreated, direct XINFO GROUPS check)

The known-bug tests need no extra direct assertion: they are strict xfail, so a vacuous wait makes the test pass and is reported as XPASS(strict), which fails the run.

M2 initially survived against test_triggered_task_runs_once_across_instances: its window is too short to catch a follower's 0.2 s publish cycle. I added test_instance_that_is_not_leader_never_schedules (a fake worker holds the lock), which catches it deterministically.

Known-bug tests: a minimal fix must turn each one into XPASS(strict):

finding minimal fix tried result
FTM-01 (Reconciler) skip requeue when the running key exists XPASS(strict)
FTM-01 (startup) skip claimed messages whose running key exists XPASS(strict)
FTM-03 do not ACK unknown tasks (leave them in the PEL) XPASS(strict)
FTM-04 also set next_run = retry_after on failure XPASS(strict)
FTM-06 SET disabled without ex XPASS(strict)
FTM-07 Coordinator and Reconciler: no publish without next_run XPASS(strict); the Coordinator change alone stays xfailed, as it should
FTM-11 not t.tags or tag not in t.tags XPASS(strict)
FTM-12 (self._tags or []) + (tags or []) XPASS(strict)
FTM-02 not tried: the fix is a synchronisation design, not a one-line change —

Limits and follow-ups

  • New race found while stabilising the suite (not covered by an xfail, because it is not deterministic): trigger_tasks stores next_run = 0. If a reconciliation cycle runs while the Coordinator is publishing that run (before SADD scheduled_set, or after the consumer has already finished and removed the entry, but before SET next_run), the Reconciler sees a task that is "overdue by 20762 days", not scheduled and not running, and republishes it. The task then runs twice. Observed once with reconciliation every 1 s: task ran 2 times: ['B', 'A'], log Reconciliation: task g/probe… is overdue by 20762 days …, republishing. It is the same family as FTM-08 (publish and next_run are not atomic). To keep the safety-net tests deterministic, the integration config keeps the Reconciler out of the test window (reconciliation_interval=3600), and only the scenarios about reconciliation (FTM-01, FTM-03, FTM-07) enable it.

  • The FTM-02 tests expect the instances to converge within 3–4 s. A fix with a slower sync interval must adjust the test window or the test config.

  • The FTM-03 test expects the run within 8 s with a 2 s stuck-message threshold. A fix that relies on the Reconciler's reclaim fits; a fix that is slower by design needs a longer window.

  • The Redis service image is pinned by tag (redis:7-alpine), not by digest.

  • On the lowest-direct versions pytest-asyncio prints a config-time PytestDeprecationWarning about asyncio_default_fixture_loop_scope being unset. It is not raised as an error. Setting asyncio_default_fixture_loop_scope = "function" would silence it.

  • The filterwarnings exception for asyncio.iscoroutinefunction should go away with the corresponding fix in stream_consumer.py.

  • The ruff in the dev dependencies (0.15.10) and the one pinned in the pre-commit hook (0.15.5) format a line that ends with a # ty: ignore[...] comment differently: the newer one treats it as a pragma and joins the expression onto one line. ruff format --check and the pre-commit hook could not both pass on such a line, so the test code avoids that shape. Aligning the two ruff versions is a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HVysDwSu3eNK7jNjrPxgQy

Run TaskManager instances against a real Redis (REDIS_URL, default
redis://localhost:6379/15) with a random key prefix per test and cleanup
limited to that prefix. Tests are marked `integration`, deselected by
default and skipped when Redis does not answer (failed when
REDIS_REQUIRED is set).

- Safety-net scenarios: single execution, cron rescheduling, leader
  election and failover, followers never scheduling, allow_parallel,
  consumer group recovery after a flush.
- Known bugs (FTM-01, 02, 03, 04, 06, 07, 11, 12) as strict xfail tests
  that assert the correct behaviour.
- New Integration workflow: Redis service, Python 3.10 and 3.14, and a
  lowest-direct dependency job.
- Treat warnings as errors and fix the statistics pipeline mock that
  emitted RuntimeWarnings.
- Every fact polled inside a wait_until helper is asserted again after
  the wait, so a vacuous helper can no longer hide a regression. The
  success-path test now waits for the message ACK and then checks
  statistics, scheduled-set cleanup and running key directly.
- Waits for something that must happen use a 15 s upper bound.
- Known-bug windows for events that must happen on their own start only
  once the bug's preconditions hold, avoiding false XPASS on slow
  machines.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant