Repository navigation
Conversation
Run TaskManager instances against a real Redis (REDIS_URL, default redis://localhost:6379/15) with a random key prefix per test and cleanup limited to that prefix. Tests are marked `integration`, deselected by default and skipped when Redis does not answer (failed when REDIS_REQUIRED is set). - Safety-net scenarios: single execution, cron rescheduling, leader election and failover, followers never scheduling, allow_parallel, consumer group recovery after a flush. - Known bugs (FTM-01, 02, 03, 04, 06, 07, 11, 12) as strict xfail tests that assert the correct behaviour. - New Integration workflow: Redis service, Python 3.10 and 3.14, and a lowest-direct dependency job. - Treat warnings as errors and fix the statistics pipeline mock that emitted RuntimeWarnings.
- Every fact polled inside a wait_until helper is asserted again after the wait, so a vacuous helper can no longer hide a regression. The success-path test now waits for the message ACK and then checks statistics, scheduled-set cleanup and running key directly. - Waits for something that must happen use a 15 s upper bound. - Known-bug windows for events that must happen on their own start only once the bug's preconditions hold, avoiding false XPASS on slow machines.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration test suite against a real Redis server
What changes
tests/integration/: a new suite that runs one or moreTaskManagerinstances against a real Redis server.conftest.py:REDIS_URL(defaultredis://localhost:6379/15), a clean skip when Redis does not answer (a failure instead whenREDIS_REQUIREDis set, as in CI, so a missing service cannot turn the suite into silent skips), a random key prefix per test, teardown that deletes only that prefix's keys (SCAN+DEL, neverFLUSHDB/FLUSHALL), amake_managerfactory that starts instances with scaled-down timings and stops them all at teardown, and polling helpers (wait_until,wait_for_leader). Waits poll for a condition with a timeout. Fixed sleeps are not used.test_scheduling.py: 8 safety-net tests for behaviour that works today: a triggered task runs exactly once (stats recorded, runtime keys cleared), a due cron task runs andnext_runmoves to the next occurrence, only one instance becomes leader, an instance that is not leader never schedules, leadership fails over, a triggered task runs once across two instances,allow_parallel=Falsewaits for the running execution, and the consumer recovers after the app's keys are flushed (the 1.1.1NOGROUPfix, previously uncovered).test_known_bugs.py: one test per open finding, each asserting the correct behaviour and markedxfail(strict=True, raises=AssertionError, reason="FTM-xx: …"). Covered: FTM-01 (Reconciler requeue and startupXAUTOCLAIM, 2 tests), FTM-02 (dynamic task created on a follower never runs; deleted on a follower stays on the leader), FTM-03 (message for an unknown task is ACKed and lost in a rolling deploy), FTM-04 (no retry after backoff), FTM-06 (disabledflag expires), FTM-07 (yearly task runs at first startup), FTM-11 (tag filter matches untagged tasks), FTM-12 (group tags drop task tags). When a fix lands, the test reportsXPASS(strict)and fails the run until the marker is removed, so the scenario becomes a regression test. Setup steps raisePreconditionErrorinstead ofAssertionError, so a broken setup is reported as a real failure and is never counted as the expected bug..github/workflows/integration.yml(new;ci.ymlis untouched):integrationjob with aredis:7-alpineservice, matrix Python 3.10 and 3.14; a separatelowest-directjob that runs both suites on Python 3.10 withUV_RESOLUTION=lowest-direct.permissions: contents: readat the top level, actions pinned to full SHAs with a# vX.Y.Zcomment,persist-credentials: false,timeout-minutes: 15per job (shutdown waits for running tasks without a timeout),REDIS_REQUIRED=1.pyproject.toml:integrationmarker;addopts = "-m 'not integration'", so plainpytest(and the existing CI job) behaves as before and never contacts Redis;pytest -m integrationoverrides it;filterwarnings = ["error", …]with one targeted exception: on Python 3.14asyncio.iscoroutinefunction(still used by the stream consumer) emits aDeprecationWarningthat, as an error, makes 3 unit tests fail. Remove that entry once the consumer switches toinspect.iscoroutinefunction;[tool.coverage.run] omit = ["tests/integration/*"]: without it the deselected integration files count as uncovered and the reported total drops from 97% to 89%.tests/test_statistics.py: the pipeline mock is now aMagicMockwith onlyexecuteasync, matching redis-py. This removes the 4RuntimeWarning: coroutine ... was never awaited.CONTRIBUTING.md: how to run the integration suite locally (throwaway Redis,REDIS_URL, marker) and what the strictxfailtests mean.Why
All 251 existing tests mock Redis, and the distributed bugs listed above pass with a green suite. This suite reproduces them with real instances sharing one Redis, documents each one as an executable test, and gives the future fixes a ready-made regression test.
How to test
Results:
251 passed, 18 deselected; integration8 passed, 10 xfailed.coverage reporttotal 97%.ruff check,ruff format --check,ty checkandpre-commit run --all-filespass.251 passed, integration8 passed, 10 xfailed.UV_RESOLUTION=lowest-direct(fastapi 0.115.13, redis 6.2.0, pydantic 2.11.7, starlette 0.46.2, pytest 8.3.5): unit251 passed, integration8 passed, 10 xfailed.--runxfail, each known-bug test fails on its own assertion and the message names the bug, for exampleFTM-01: a running task with allow_parallel=False was started 2 times, at [0.0, 3.0] s,FTM-06: the disabled flag expires on its own (TTL 432000 s),FTM-03: … acknowledged and dropped (stream length now 0). The captured logs show the faulty code path (requeued stuck low priority message,Task not found … acknowledging).DBSIZE0). Waits for something that must happen use a 15 s upper bound (EVENTUALLY), which costs nothing on a passing run becausewait_untilreturns as soon as the condition holds. Before that change, one run in five failed under this load (pending trigger never ranafter the leader lock expired during a stall). Windows in which something must not happen, and the windows of the known-bug tests, stay short. Under the same load the FTM-01 (Reconciler) test once reported a falseXPASS(strict): its window started when the task started, and the Reconciler only acts once the message has been idle for more than 2 s. For the three known-bug tests in which the bug is an event that has to happen on its own (FTM-01 ×2, FTM-07), the window (BUG_WINDOW, 5 s) now starts only after the bug's preconditions hold, so a slow machine cannot make the bug look fixed. The minimal fixes still giveXPASS(strict), and for FTM-07 the Coordinator change alone still staysxfailed. On the final tree: Python 3.14 and lowest-direct both give259 passed, 10 xfailed(unit and integration in one run).REDIS_REQUIRED, setup errorFailed: Redis not reachable at …with it.Mutations
Each mutation was applied one at a time to
src/. The run checked the occurrence count before and after, ran only the relevant tests, then restored the file and verified the checksum.Safety-net tests (each must go red):
leader_election.pynx=Trueremoved from the lockSETtest_only_one_instance_becomes_leadertwo instances believe they are leader)coordinator.pycontinueremoved after a failed acquisition (followers schedule too)test_instance_that_is_not_leader_never_schedulesan instance that is not leader rescheduled the task)stream_consumer.pysetup_consumer_groups()removed from theNOGROUPhandlertest_consumer_recovers_after_redis_keys_are_flushedconsumer group was never recreated)coordinator.pyif not effective_allow_parallel→if Falsetest_allow_parallel_false_waits_for_running_executiona second execution started while the first was running)coordinator.py_update_next_runcall removedtest_triggered_task_runs_exactly_once,test_cron_task_runs_when_due_…['A', 'A'] == ['A'])coordinator.pynext_run <= now→>=test_cron_task_runs_when_due_…['A', 'A'] == ['A'])stream_consumer.pytest_triggered_task_runs_exactly_oncerunning key left behind after success)M7 first survived because the test waited up to 2 s for the running key to disappear, and the key's own 2 s TTL removed it anyway. The test now waits only for the consumer's last success step (ACK +
XDELof the message), then checks the running key immediately.Waits never carry the assertion. Every fact polled inside a
wait_untilhelper is asserted again directly after the wait, so a polling helper that becomes vacuous cannot hide a product bug. In each row below the helper is mutated together with the product bug it waits for:sremof the scheduled set removed on successtask still tracked as scheduled after success)return Truereturn Truesremremoved on successtask still tracked as scheduled after success)_next_run_changed→return Falsean instance that is not leader rescheduled the task,'1798761600.0' == '0')started_againwait →lambda: Falsea second execution started while the first was running)group_exists→return TrueNOGROUP(M3)consumer group was never recreated, directXINFO GROUPScheck)The known-bug tests need no extra direct assertion: they are strict
xfail, so a vacuous wait makes the test pass and is reported asXPASS(strict), which fails the run.M2 initially survived against
test_triggered_task_runs_once_across_instances: its window is too short to catch a follower's 0.2 s publish cycle. I addedtest_instance_that_is_not_leader_never_schedules(a fake worker holds the lock), which catches it deterministically.Known-bug tests: a minimal fix must turn each one into
XPASS(strict):next_run = retry_afteron failureSET disabledwithoutexnext_runxfailed, as it shouldnot t.tags or tag not in t.tags(self._tags or []) + (tags or [])Limits and follow-ups
New race found while stabilising the suite (not covered by an
xfail, because it is not deterministic):trigger_tasksstoresnext_run = 0. If a reconciliation cycle runs while the Coordinator is publishing that run (beforeSADD scheduled_set, or after the consumer has already finished and removed the entry, but beforeSET next_run), the Reconciler sees a task that is "overdue by 20762 days", not scheduled and not running, and republishes it. The task then runs twice. Observed once with reconciliation every 1 s:task ran 2 times: ['B', 'A'], logReconciliation: task g/probe… is overdue by 20762 days …, republishing. It is the same family as FTM-08 (publish andnext_runare not atomic). To keep the safety-net tests deterministic, the integration config keeps the Reconciler out of the test window (reconciliation_interval=3600), and only the scenarios about reconciliation (FTM-01, FTM-03, FTM-07) enable it.The FTM-02 tests expect the instances to converge within 3–4 s. A fix with a slower sync interval must adjust the test window or the test config.
The FTM-03 test expects the run within 8 s with a 2 s stuck-message threshold. A fix that relies on the Reconciler's reclaim fits; a fix that is slower by design needs a longer window.
The Redis service image is pinned by tag (
redis:7-alpine), not by digest.On the lowest-direct versions pytest-asyncio prints a config-time
PytestDeprecationWarningaboutasyncio_default_fixture_loop_scopebeing unset. It is not raised as an error. Settingasyncio_default_fixture_loop_scope = "function"would silence it.The
filterwarningsexception forasyncio.iscoroutinefunctionshould go away with the corresponding fix instream_consumer.py.The ruff in the dev dependencies (0.15.10) and the one pinned in the pre-commit hook (0.15.5) format a line that ends with a
# ty: ignore[...]comment differently: the newer one treats it as a pragma and joins the expression onto one line.ruff format --checkand the pre-commit hook could not both pass on such a line, so the test code avoids that shape. Aligning the two ruff versions is a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_01HVysDwSu3eNK7jNjrPxgQy