Skip to content

fix(tests): reclaim completed pytest temporary directories - #9070

Open
cocolord wants to merge 1 commit into
loopx-project:mainfrom
cocolord:codex/pytest-temp-cleanup-20261119
Open

cocolord wants to merge 1 commit into
loopx-project:mainfrom
cocolord:codex/pytest-temp-cleanup-20261119

Conversation

@cocolord

Copy link
Copy Markdown
Member

Goal And Delivered Outcome

Long LoopX pytest sessions could retain temporary directories for completed cases until the session ended. The suite now reclaims case-owned scratch space after the final teardown report whenever the case completed without a failure. Setup, call, and teardown failures retain their evidence under the default policy.

  • The cleanup covers tmp_path, per-case isolated HOME, and named case-owned directories.
  • Before each case, pytest checks the filesystem used for its scratch space. The default reserve is 2 GiB and can be configured with --loopx-min-free-tmp-mib.
  • Target branch is loopx-project/loopx:main.

Author Declaration

  • Written by model_agent (OpenAI Codex).
  • Implemented against the maintenance request described in this PR.
Requested behavior Disposition Implementation Verification
Reclaim completed cases while the run is active Implemented tests/conftest.py case ownership and teardown reporting Nested pytest probe verifies that successful scratch directories are absent before the next observation case exits
Retain failed cases Implemented Collect setup, call, and teardown outcomes before cleanup Nested pytest probe checks that all three categories of failure retain their witness files
Prevent accumulated scratch use from exhausting the disk Implemented Pre-setup free-space check Serial and two-worker probes verify that a low-space condition interrupts the run before runtime fixtures start

Scope And Continuation

Complete within the main pytest suite. The change also checks that the touched runtime, sidecar, and short-path fixtures still work with the new lifecycle.

Validation

  • Tested revision f62c1f760e3274a2ee441e3cf0116937c9026df9
  • Run state finished
  • Input classes synthetic
Check kind Result Public-safe evidence
integration passed 19 targeted tests passed in 38.63 s on macOS with two pytest-xdist workers. The selection includes immediate cleanup, failure retention, low-space interruption in serial and parallel, managed runtime shutdown, sidecar scanning, monitor handoff, and a short-path gate case.
static passed Ruff on the five changed files, git diff --check origin/main...HEAD, and the RFC bilingual diff check.
integration not_run Full test suite and platform matrix locally.
python -B -m pytest -q -n 2 --dist loadfile \
  tests/test_managed_effect_runtime_fixture.py \
  tests/test_collaboration_sidecar_scan.py \
  tests/control_plane/test_monitor_gate_handoff.py \
  'tests/control_plane/test_user_gate_resume_cli.py::test_gate_first_link_rejects_invalid_target_without_consuming_link[missing-preview-sqlite]'

Frontend / Visual Evidence

UI impact is none.

Type of Change

  • Bug fix
  • Test update

LoopX Area

  • Build, packaging, installer, or CI

Shared-authority RFC fixture impact

N/A. This changes pytest scratch-space lifecycle.

Boundary Checklist

  • The diff and PR description contain public-safe information.
  • The change is scoped to pytest temporary-directory cleanup.
  • No paired RFC changed, and the local bilingual diff check passed.
  • There is no UI impact.
  • The commit includes a DCO sign-off.

Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, Gemini (Google), self_reported

动机 / Motivation

在本地、Worktree 或 CI Runner 上运行 LoopX 完整 pytest 套件的开发者与自动化流程,都会在执行测试时为每个用例创建独立的临时工作区与隔离的 HOME 目录。

改动前 pytest 默认使用 tmp_path_retention_policy=all,且自动隔离 HOME 使用 tmp_path_factory.mktemp,导致上千个已通过用例的临时目录在整场测试结束前持续堆积;改动后在每个通过用例的 teardown 报告阶段立即回收其登记的 tmp_path、case_mktemp 与隔离 HOME,同时在 setup 前校验临时目录剩余空间并在不足阈值时提前中止会话。

在 macOS 上实测已验证:通过用例的 tmp_path、case_mktemp 与隔离 HOME 会在下一条测试运行前被回收,setup/call/teardown 失败仍保留现场,多进程并发与 xdist 仅清理各自用例目录,低磁盘保护会在启动夹具前以 ExitCode.INTERRUPTED 中止;但当前 _remove_temp_dir 直接调用 shutil.rmtree 并在出错时在 pytest_runtest_makereport 内发出 PytestWarning,一旦通过用例留下只读目录(0o555)或 Windows Git 只读对象(0o444),清理会失败并可能在 -W error 下引发 INTERNALERROR 崩溃。

本 PR 不改动 LoopX 生产控制面或运行时的临时目录生命周期,也不跨越会话全局扫除其他并发 pytest 进程拥有的 pytest-of-/pytest-* 目录。

改动思路 / Approach

本次改动在 tests/conftest.py 内通过统一的 case_mktemp 夹具替代分散的 tmp_path_factory.mktemp 调用来登记单用例临时目录,而不是简单地在 pyproject.toml 中设置 tmp_path_retention_policy = "failed":一方面内置 tmp_path 不会回收 _isolated_default_runtime_routes(home)和短路径 case_mktemp 目录,另一方面 pytest 内置 tmp_path 最终清理器仅查看 call 阶段结果,容易在 teardown 失败时误删目录(具体机制详见下文)。

整体生命周期由三个点协同控制:

  1. 策略解析(pytest_configure):检查 --loopx-min-free-tmp-mib 非负,并通过 config.inicfg 与 -o 覆盖判断用户是否显式配置了 tmp_path_retention_policy;未显式配置时默认采用 "failed" 策略,且在 config.stash[_CASE_SEQUENCE] 维护进程内单调序号。
  2. 目录登记与按用例回收(case_mktemp / tmp_path / pytest_runtest_makereport):case_mktemp 创建 tmp_path_factory.mktemp(f"{name}-{serial}", numbered=False) 并记录在 request.node.stash[_CASE_TEMP_DIRS];tmp_path、_isolated_default_runtime_routes 以及 3 个短路径测试均委托给它。pytest_runtest_makereport 先汇总 setup / call / teardown 任意阶段的失败标记到 item.stash[_CASE_FAILED],在 report.when == "teardown" 时按策略删除当前用例登记目录(与内置 tmp_path 的最终清理解耦,从而确保所有 fixture finalizer 与 pytest_runtest_teardown 中的 stop_disposable_effect_runtimes 执行完后再删目录)。会话结束时若无默认路由泄漏、策略非 "all" 且 exitstatus == OK,pytest_sessionfinish 再回收 worker 级 _WORKER_TEMP(effect-runtime*)。
  3. 预启动磁盘保护(pytest_runtest_setup):在 tryfirst=True 阶段按 --basetemp -> PYTEST_DEBUG_TEMPROOT -> tempfile.gettempdir() 解析所在已存在挂载点,可用空间低于 --loopx-min-free-tmp-mib(默认 2048 MiB)时设置 item.session.shouldstop 并立即 pytest.fail(..., pytrace=False),在会话和用例 fixture 初始化前中断串行或 xdist 会话。

具体改动 / Changes

本次 PR 共修改 5 个测试文件(+202/-9),均处于 tests/ 范围内,仓库未就测试套件临时目录回收制定专门 RFC 或书面规范(spec_basis: no_spec),以下按改动文件逐一核查:

  • tests/conftest.py:新增 case_mktemp、重写 tmp_path、新增 _remove_temp_dir、pytest_configure、扩展 pytest_runtest_makereport 与 pytest_runtest_setup、扩展 pytest_sessionfinish 与 pytest_addoption(--loopx-min-free-tmp-mib),并将 _isolated_default_runtime_routes 切换至 case_mktemp("home")。
  • tests/test_managed_effect_runtime_fixture.py:新增 test_pytest_reclaims_finished_cases_and_keeps_failed_phases(校验 setup/call/teardown 失败保留与通过用例即时回收,及 finalizer 执行顺序)与 test_pytest_stops_before_fixtures_on_low_disk(串行与 -n 2 xdist 低磁盘中止)。
  • tests/control_plane/test_monitor_gate_handoff.py、tests/control_plane/test_user_gate_resume_cli.py、tests/test_collaboration_sidecar_scan.py:将原函数级短路径 tmp_path_factory.mktemp(...) 迁移为 case_mktemp(...),使短路径目录也能随用例结束而回收;同时仓库内剩余的两处 tmp_path_factory.mktemp(test_pr_review_git_file_inventory.py 与 test_runtime_stdio_host_utf8.py)均为 scope="module",未受误伤。

对主干的风险 / Risks

[P1 / Blocking] _remove_temp_dir 缺少只读权限恢复,且在 pytest_runtest_makereport 内发出 PytestWarning 会在 -W error 下触发 INTERNALERROR 崩溃

  • 触发路径:tests/conftest.py 的 _remove_temp_dir(temporary) 在 pytest_runtest_makereport(teardown 报告后)和 pytest_sessionfinish 中直接调用 shutil.rmtree(temporary),并在捕获 OSError 时调用 warnings.warn(pytest.PytestWarning(...))。
  • 实测证据(macOS):
    1. 当通过用例在 tmp_path 下留下 0o555 只读目录时,默认清理触发 PermissionError: [Errno 13],目录未被回收,且输出 PytestWarning。
    2. 在 pytest -W error 与单测标记 @pytest.mark.filterwarnings("error") 两种场景下实测,该 warnings.warn(pytest.PytestWarning(...)) 在 pytest_runtest_makereport 钩子内被转换为未捕获异常,导致 pytest 以 ExitCode.INTERNAL_ERROR(返回码 3)直接崩溃并中止后续测试,即便被测用例本身已经通过。
  • Windows 主干风险(基于仓库配置与现有修复记录分析,本地未运行 Windows 原生环境):.github/workflows/python-tests.yml 的 windows-powershell 任务会执行在 tmp_path 中运行 git init / git commit 的测试(如 tests/test_doctor_git_encoding.py、 tests/control_plane/test_cli_output_differential.py)。如 loopx/extensions/process_runtime.py(remove_owned_tree)及该工作流注释所记录,Windows 上 git 会将 loose object 写为只读(0o444),裸 shutil.rmtree 会以 PermissionError [WinError 5] 失败;由于本 PR 将默认回收策略从 "all" 改为 "failed",Windows CI 上相关通过用例将无法回收 .git 临时目录并产生清理警告(若带 -W error 则直接 INTERNALERROR)。
  • 最小修复:
    1. 在 _remove_temp_dir 删除树前或 onexc 回调中,仅针对 temporary 内部拥有的非符号链接/非 junction 目录与单硬链接文件(st_nlink == 1)补充写/执行权限后重试删除(可复用 loopx.extensions.process_runtime.remove_owned_tree 的只读清理思路并补充 POSIX 只读目录处理,或参考 _pytest.pathlib.rm_rf 的 on_rm_rf_error),绝不跟随符号链接或修改硬链接外部文件权限。
    2. 在 pytest_runtest_makereport / pytest_sessionfinish 清理钩子内避免发出可被 -W error 升级为 INTERNALERROR 的 PytestWarning(对齐上游 _pytest/tmpdir.py 的 best-effort 静默忽略或仅在安全通道处理不可删除目录)。
    3. 在 tests/test_managed_effect_runtime_fixture.py 补充对 0o444/0o555 只读树与 @pytest.mark.filterwarnings("error") 组合场景的回归测试。

其他已验证的并发、符号链接与边界项(均已通过,无额外阻塞)

  • 并发与兄弟运行安全性(PASS):未扫描或清理全局 pytest-of-<user>/pytest-* 目录,所有清理严格限定在当前进程的 item.stash[_CASE_TEMP_DIRS] 与 session.config.stash[_WORKER_TEMP]。实测两个进程共享同一 PYTEST_DEBUG_TEMPROOT/TMPDIR(无 --basetemp)并发运行,以及两组 pytest -n 2 在各自 --basetemp 下并发运行,均互相隔离、无误删且失败用例目录完整保留。
  • 符号链接与跨作用域 fixture(PASS):tmp_path 内部指向外部目录/文件的 symlink、悬空 symlink 以及 tmp_path 本身被替换为 symlink,均不会删除外部目标;scope="session" 与 scope="module" 的 tmp_path_factory.mktemp 不受按用例清理影响。
  • 次要非阻塞观察:当 -o tmp_path_retention_policy=none 且某 worker 有用例失败(exitstatus != 0)时,pytest_sessionfinish 仍保留该 worker 的 _WORKER_TEMP(effect-runtime*);另外 pytest_runtest_setup 若遇到前置测试泄露 os.chdir 且运行命令使用相对 --basetemp,会按当前工作目录解析路径,建议后续顺手以 invocation_params.dir 锚定。

我的整体评价 / Verdict

本 PR 针对长耗时 pytest 会话的磁盘膨胀问题切中要害,且在单进程与 xdist 并发隔离、三阶段失败保留、fixture finalizer 与 effect-runtime 停机顺序、tmp_path_factory 模块级目录隔离以及低磁盘中断等方面设计清晰、在 macOS 本地实测表现稳健。但 _remove_temp_dir 使用无只读恢复的裸 shutil.rmtree 并在 pytest_runtest_makereport 中发出 PytestWarning,已实测导致留有只读目录的通过用例在 -W error 或 @pytest.mark.filterwarnings("error") 下引发 INTERNALERROR 会话崩溃,且与仓库已有的 Windows Git 只读对象清理经验冲突,因此当前 head f62c1f760e3274a2ee441e3cf0116937c9026df9 需补齐只读清理与钩子警告安全后再合入。

本地验证摘要(exact head f62c1f760e3274a2ee441e3cf0116937c9026df9,未等待远程 CI):

  • PR 目标测试集(-n 2 --dist loadfile):PASS(86 passed in 164.69s,覆盖 tests/test_managed_effect_runtime_fixture.py、tests/test_collaboration_sidecar_scan.py、tests/control_plane/test_monitor_gate_handoff.py、tests/control_plane/test_user_gate_resume_cli.py)。
  • 相关仓库回归集(含 git、只读清理、filterwarnings、module 级 tmp_path_factory):PASS(94 passed in 28.55s)。
  • 基线 036a95fb1aac vs Head f62c1f760e32 对比与并发/策略/符号链接探针:PASS(基线保留通过用例 tmp_path 与 home;Head 即时回收通过用例目录并保留失败用例;并发与 xdist 互不干扰;symlink 外指安全)。
  • 只读 0o555 目录 + -W error / @pytest.mark.filterwarnings("error") 探针:FAIL(复现 ExitCode.INTERNAL_ERROR, rc=3)。
  • 静态检查与门禁:ruff check(5 个变更文件)PASS、mypy(17 个严格类型文件)PASS、git diff --check PASS、scripts/check_rfc_bilingual.py PASS、loopx check --scan-path(公私边界零告警)PASS、loopx canary premerge --tier quick(5 项直接检查 + 3 项 canary 全通过)PASS。
  • 视觉审查与未验证范围:visual_review 为 not_applicable(仅修改 tests/ 下测试夹具与测试文件,无前端 UI 渲染变化);unverified_scope 为 Windows 原生宿主机实际执行及全量完整测试套件(-W error 下另有 4 项既有用例因独立 Popen.__del__ ResourceWarning 失败,非本 PR 引入)。

English verdict: REQUEST_CHANGES at head f62c1f7 — _remove_temp_dir in tests/conftest.py lacks read-only permission recovery (0o555 dirs / Windows 0o444 git objects) and emits PytestWarning inside pytest_runtest_makereport, which crashes -W error and @pytest.mark.filterwarnings("error") runs with INTERNALERROR; PR touched suites (86 passed) and static/canary checks passed locally on macOS.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants