tests/test_docs_examples.py runs every fenced Python block in the package docstrings and the site pages, but it never compares what a block prints with the # ... comment lines that show the expected result. Reviewers of #91, #93, and #97 found printed results that were wrong (# Torch 1 6 where the code prints Torches (6) 1 6; # EdgeKind.WALL where a StrEnum prints wall), and the harness passed them all.
Add a gate that executes each block, captures stdout, and compares it line by line with the comment lines that directly follow each completed print(...) call. Explanatory comments between print calls need to be told apart from expected output; the simplest rule is that only comment lines immediately after a print statement count. A working prototype exists from the pass (a script that does exactly this and reports mismatches per block); it should live under tests/ and run in the same parametrized shape as the examples harness so a failing block is named.
Found during the docstring pass (#87 to #99).
tests/test_docs_examples.pyruns every fenced Python block in the package docstrings and the site pages, but it never compares what a block prints with the# ...comment lines that show the expected result. Reviewers of #91, #93, and #97 found printed results that were wrong (# Torch 1 6where the code printsTorches (6) 1 6;# EdgeKind.WALLwhere aStrEnumprintswall), and the harness passed them all.Add a gate that executes each block, captures stdout, and compares it line by line with the comment lines that directly follow each completed
print(...)call. Explanatory comments between print calls need to be told apart from expected output; the simplest rule is that only comment lines immediately after a print statement count. A working prototype exists from the pass (a script that does exactly this and reports mismatches per block); it should live undertests/and run in the same parametrized shape as the examples harness so a failing block is named.Found during the docstring pass (#87 to #99).