Skip to content

tests: test_every_tracked_name_exists_and_matches_git checks neither the index nor existence, so a hardcoded capture list would pass it #708

Description

@JarryShaw

tests/test_tier_guard.py:133 declares itself the check that the committed-capture set matches the git
index, and does not check that:

def test_every_tracked_name_exists_and_matches_git(self) -> None:
    """The set is the index's answer verbatim, prefix stripped."""
    tracked = _tiers.committed_captures()
    assert tracked is not None
    self.assertTrue(tracked, 'git tracks no capture at all, which cannot be right')
    for name in tracked:
        with self.subTest(capture=name):
            self.assertNotIn('/', name, 'names are relative to examples/captures/')

The body asserts only that the set is non-empty and that no name contains a /. It never
re-derives the index to compare against, and despite exists in its name it never stats a file. So:

  • a hardcoded list would pass it — any non-empty list of bare names satisfies both assertions;
  • a stale list would pass it — a name for a capture that has since been deleted is never checked to
    exist;
  • a wrong list would pass it — nothing ties the set to git ls-files output.

Its sibling test_capture_suggestions_are_captures (:142) does not close the gap either: it asserts
'in.pcap' in suggestions and 'out.txt' not in suggestions, which a hardcoded list containing
in.pcap also satisfies.

Why it matters

_tiers.committed_captures() at tests/_tiers.py:292 resolves the set live via
_git('ls-files', '-z', '--', relative_root), and that live resolution is the reason PR #697 needs no
change to the tier guard at all — the guard adapts to the tracked set shrinking from six to two on its
own. The one test named for that contract is the one test that would not notice if the live
resolution were replaced by a literal.

Suggested fix

Re-derive the index inside the test and compare: shell git ls-files -z -- examples/captures
independently, strip the prefix, and assert set equality against committed_captures(). Add a
Path.is_file() check per name so exists in the name is true. Both should skip gracefully outside a
git repo, as the existing suite does (tests/project/test_capture_tracking.py in #697 demonstrates the
pattern).

Found while cross-reviewing PR #703; confirmed independently by that PR's author and its reviewer, and
by reading the body directly. Judged out of scope for #703, which was prose-only.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugtestPull requests that add or correct tests (test: subject prefix)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions