Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 24 additions & 11 deletions docs/test_rule_contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,25 @@ an unplanted corpus.
- **php `public static function spy()`** — Mockery declaring its own `spy`
method fires the call-anchored form once; accepted (it is framework
source).
- **Bare-word menus the crucible cannot measure** (ruby's
`before`/`after`/`let`/`subject`, javascript's bare `describe`/`expect`/
`assert`, java's and groovy's `assert…(` runtime-statement exposure,
assembly's `(?i)` prose vocabulary): deferred with #2853 — no crucible
files, no corpus cell, nothing moves today. The audit records these as
conforming-by-absence-of-evidence, not validated.
- **Bare-word menus, resolved in #2853.** ruby's `before`/`after`/`let`/
`subject`/`context`, javascript's bare `describe`/`expect`/`assert`, java's
and groovy's `assert…(` runtime-statement exposure, and assembly's `(?i)`
prose vocabulary — all deferred at #2852 as "conforming-by-absence-of-
evidence" — were validated once the v1.2.0 corpus grew files for these
languages (ruby, javascript, groovy 312, assembly 239; java present but with
no test sites). C3 anchors the everyday word to its framework form: ruby's
words to their rspec/minitest call/block (`describe "x" do`, `before(:each)`,
`let(:u)`, `assert_equal`), javascript's `describe`/`expect` to `\s*\(` and
bare `assert` to the chai chain `assert.<x>`, assembly's menu dropped in
favour of the `testcase` macro. C1 hands the runtime guard to safety: java's
and groovy's `assert\w*\(` → `assert\w{1,40}\(` (the parenthesized power-
assert/JLS statement is safety's; `assertEquals(` stays; also closes a latent
`\w+\(` backtracking exposure), and js's bare `assert(`. Pipeline reprice:
javascript 5 → 0, ruby 2 → 0, assembly 284 → 275, groovy 954 → 954
(unchanged — no bare power-assert in corpus), java 0 → 0. **Residual:**
non-black/unspaced forms and languages still thin on test files (java has no
`@Test` sites in its springboot corpus) are validated by the contract test's
positive/negative pairs rather than a corpus cell.

## The 46-language audit

Expand All @@ -124,7 +137,7 @@ morphology.
| ada | — | 2 | conforms (`AUnit`, `Assert(` — Ada has no runtime assert keyword) |
| agc_assembly | 13 | 2 | conforms (SELFCHECK/ROPECHK are its self-test ops) |
| apex | 116 | 2 | conforms (@isTest/System.assert anchored forms) |
| assembly | 275 | 2 | conforms; `testcase` macro rows are case declarations; `(?i)` prose surface → #2853 |
| assembly | 284 → 275 | 2 | validated (#2853): kept the `testcase` macro + `it(`; dropped the `(?i)` prose menu and the linker `ASSERT(` guard (safety's) |
| c | 1052 → 97 | 3 → 2 | **C1**: `assert(` → safety's; uppercase `ASSERT_*` macros stay (framework form) |
| cobol | 51 → 0 | 2 | **C3**: hyphen guards; `UT-TEST-CASE-COUNT` was an identifier (#2622 shape) |
| cpp | 0 | 2 | conforms (framework macros only, no bare assert() alternative) |
Expand All @@ -135,11 +148,11 @@ morphology.
| embedded_python | 0 | 3 → 2 | **C1**: bare `assert` removed (#2626 applied to the twin) |
| fortran | 0 | 2 | conforms (pFUnit `@test`/`@assertEqual` directive-anchored) |
| go | 2 | 2 | conforms (`TestX`/`t.Run`/`assert.X(` anchored) |
| groovy | 948 | 2 | conforms; Spock labels line-anchored; `assert…(` exposure → #2853 |
| groovy | 954 → 954 | 2 | validated (#2853): C1 `assert\w{1,40}\(` drops the parenthesized power-assert (none in corpus); Spock labels + `assertEquals(` kept |
| haskell | 0 | 2 | conforms (hspec/QuickCheck names, `prop_` prefix) |
| html | 0 | 2 | conforms (`data-testid=` attribute form) |
| java | 0 | 2 | conforms; `assert…(` runtime-statement exposure → #2853 |
| javascript | 5 | 2 | conforms; anchored `it`/`test` halves already contract-shaped; bare `describe`/`expect`/`assert` → #2853 |
| java | 0 → 0 | 2 | validated (#2853): C1 `assert…{1,40}\(` keeps `assertEquals(`, drops the JLS runtime `assert(cond)`; no test sites in the springboot corpus, so pinned by the contract test |
| javascript | 5 → 0 | 2 | validated (#2853): C3 anchors `describe`/`expect` to `\s*\(`, bare `assert` to the chai chain `assert.`; the 5 were comment/string prose, now gone |
| kotlin | 1 | 2 | conforms (annotation + call-anchored forms) |
| livecode | 0 | 2 | conforms (`command test*`/`pass test` statement forms) |
| lua | 3471 → 134 | 2 | **C1**: bare `assert` → safety's; luassert chain `assert.<chain>` added (the 134 are all chains) |
Expand All @@ -151,7 +164,7 @@ morphology.
| php | 53 → 7 | 2 | **C3**: everyday words call-anchored, `(?<!->)` on `test`/`it`; remainder Mockery calls |
| powershell | 203 | 2 | conforms (Pester `Should`/`Describe`/`It` forms) |
| python | 1221 | 2 | conforms (#2626 already removed `assert`; framework names + `def test_`) |
| ruby | 0 | 2 | conforms-by-absence-of-evidence; bare-word menu → #2853 |
| ruby | 2 → 0 | 2 | validated (#2853): C3 anchors describe/context/it/before/after/let/subject/expect and minitest assert_*/refute_* to their framework forms; prose (`# context`, `assertions`) no longer fires |
| rust | 464 | 2 | conforms (`#[test]`/`assert!` macros — the macro *is* the framework form) |
| scala | 0 | 2 | conforms (framework names + `it should`) |
| scheme | 0 | 2 | conforms (paren-anchored srfi-64 forms) |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,12 @@
# #2882 contract C4: doc counts the block, not the author tag -- @author is ownership's alone.
"doc": re.compile(r"^[;#@/|]+\s*@(?:param|return|brief|note)", re.M | re.I),
# 14. test (Testing & Assertions)
"test": re.compile(r"(?i)\b(?:describe|expect|assert|TestCase)\b|\bit[ \t]*\("),
# #2853 contract C3/C1: dropped the `(?i)` bare prose menu
# `describe|expect|assert` (comment vocabulary, and a linker `ASSERT(`
# guard is safety's, not a test). The real test surface in the corpus is
# the nasm/masm `testcase` macro table (kept, case-insensitively) and a
# call-anchored `it(`.
"test": re.compile(r"(?i)\bTestCase\b|\bit[ \t]*\("),
# --- PHASE 3: ARCHITECTURE & DOMAIN SENSORS ---
# 15. concurrency (Asynchronous Execution)
"concurrency": re.compile(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,11 @@
# Spock-label anchor could stretch across blank lines. Bounded to
# `[ \t]*` (same-line whitespace only).
"test": re.compile(
r"@(?:Test|Before|After|BeforeEach|AfterEach|Mock)|assert\w*\s*\(|^[ \t]*(?:given|when|then|expect|setup|cleanup|where):",
# #2853 contract C1: `assert\w{1,40}\(` (was `\w*`) keeps `assertEquals(`
# but drops groovy's parenthesized power-assert `assert(cond)` -- a
# runtime guard, safety's. Spock block labels are untouched. Bounded so
# a long `\w` run with no `(` can't catastrophically backtrack.
r"@(?:Test|Before|After|BeforeEach|AfterEach|Mock)|assert\w{1,40}\s*\(|^[ \t]*(?:given|when|then|expect|setup|cleanup|where):",
re.M,
),
# --- PHASE 3: ARCHITECTURE & DOMAIN SENSORS ---
Expand Down
8 changes: 7 additions & 1 deletion gitgalaxy/standards/language_standards/languages/java.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,8 +267,14 @@
r"/\*\*[\s\S]{0,15000}?\*/|@param|@return|@throws|@deprecated|@see|@since|@apiNote|@implSpec|@Operation|@Schema"
),
# 14. test (Testing & Assertions)
# #2853 contract C1: `assert[...]{1,40}\(` (was `*`) requires >=1 char after
# `assert`, so a JUnit `assertEquals(`/`assertThat(` matches but the JLS
# runtime `assert(cond)` statement -- a production guard, safety's -- does
# not (the runtime `assert cond;` form was never matched here anyway). The
# count is bounded (a real assertion name is short) so a long run of word
# chars with no `(` can't catastrophically backtrack the `\w+\s*\(` tail.
"test": re.compile(
r"@(?:Test|ParameterizedTest|Before|After|BeforeEach|AfterEach|Mock|InjectMocks)|assert[A-Za-z0-9_]*\s*\(|\b(?:verify|expect|given|when)\s*\("
r"@(?:Test|ParameterizedTest|Before|After|BeforeEach|AfterEach|Mock|InjectMocks)|assert[A-Za-z0-9_]{1,40}\s*\(|\b(?:verify|expect|given|when)\s*\("
),
# --- PHASE 3: ARCHITECTURE & DOMAIN SENSORS ---
# 15. concurrency (Asynchronous Execution)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -258,8 +258,15 @@
# already carries this guard so `myRegex.test('x')` (a regex method
# call) isn't miscounted as a test-framework call -- JavaScript's own
# rule never got the same fix despite the identical ambiguity.
# #2853 contract C3/C1: the bare menu `describe|expect|assert` fired on
# comment/string prose (`methodName === 'assert'`, `/* assert */`). Anchor
# the everyday words to their call form (`describe(`, `expect(`), keep the
# unambiguous framework names bare, and reduce bare `assert` to the chai
# matcher chain `assert.<x>` -- Node/`console.assert` runtime guards are
# safety's (C1). The `(?<!\.)\b(?:it|test)\s*\(` half is already
# contract-shaped. (`\bcy\.` anchored so it can't match `transparency.`.)
"test": re.compile(
r"\b(describe|expect|assert|beforeEach|afterEach|jest|mocha|vitest|cy\.)\b|(?<!\.)\b(?:it|test)\s*\("
r"\b(?:jest|mocha|vitest|beforeEach|afterEach)\b|\bcy\.|(?<!\.)\b(?:describe|it|test)\s*\(|\bexpect\s*\(|\bassert\s*\."
),
# --- PHASE 3: ARCHITECTURE & DOMAIN SENSORS ---
# 15. concurrency (Asynchronous Execution)
Expand Down
17 changes: 16 additions & 1 deletion gitgalaxy/standards/language_standards/languages/ruby.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,23 @@
re.M | re.I,
),
# 14. test (Testing & Assertions)
# #2853 contract C3: the bare menu (`context`, `before`, `after`, `setup`,
# `let`, `subject`, `assert[a-z_]*` matching the noun `assertions`) fired on
# ordinary prose/comments. Anchor every everyday word to its rspec/minitest
# form: describe/context to a description string or `do`, it/specify to a
# description string, before/after to a hook call/`do`/`:each`/`:all`,
# let/subject to `(`/`{`, expect to `(`, minitest assert/refute to the
# `assert_<name>` or `assert(` call form, and setup/teardown to a `def`.
# Ruby has no runtime `assert` keyword, so `assert*` stays test's (C1 n/a).
"test": re.compile(
r'\b(describe|context|expect|assert[a-zA-Z_]*|refute[a-zA-Z_]*|setup|teardown|before|after|let|subject)\b|\b(?:it|test)\s+[\'"]'
r"\bRSpec\b"
r"|\b(?:describe|context)\s*(?:['\"(]|do\b)"
r"|\b(?:it|specify)\s+['\"]"
r"|\b(?:before|after)\s*(?:\(|do\b|:each|:all)"
r"|\b(?:let|subject)\s*(?:\(|\{)"
r"|\bexpect\s*\("
r"|\b(?:assert|refute)(?:_\w+|\s*\()"
r"|\bdef[ \t]+(?:setup|teardown)\b"
),
# --- PHASE 3: ARCHITECTURE & DOMAIN SENSORS ---
# 15. concurrency (Asynchronous Execution)
Expand Down
6 changes: 3 additions & 3 deletions tests/extraction/languages/test_assembly_strict.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@
("state_mutation", "\txchg eax, ebx", "\tmov eax, ebx"),
("dead_code", "; mov eax, 5", "; just a note"),
("doc", "; @param x", "; just a note"),
("test", "\tassert eax", "\tmov eax, ebx"),
("test", "testcase {", "\tassert eax"), # #2853: nasm testcase macro counts; bare `assert` prose does not
("concurrency", "\tlock xadd eax, ebx", "\tmov eax, ebx"),
("globals", "\tbuf: resd 4", "\t.data"), # #2858: labeled storage, not the section switch
("comprehensions", "\trep movsb", "\tmov eax, ebx"),
Expand Down Expand Up @@ -286,14 +286,14 @@ def test_assembly_test_vs_regex_execution_no_false_collision():
"""
Known ambiguity pattern from the issue template (TypeScript's
`myRegex.test('x')` colliding with `test`). assembly's `test` maps to
describe/expect/assert/TestCase/`it(`; `regex_execution` maps to
the `testcase` macro and `it(`; `regex_execution` maps to
`call`/`bl` into POSIX regex functions -- disjoint token vocabularies,
no realistic overlap.
"""
test_rule = ASM_RULES["test"]
regex_execution = ASM_RULES["regex_execution"]

assertion = "\tassert eax"
assertion = "testcase {" # #2853: assembly's test surface is the nasm `testcase` macro (bare `assert` prose dropped)
assert test_rule.search(assertion)
assert not regex_execution.search(assertion)

Expand Down
65 changes: 65 additions & 0 deletions tests/extraction/languages/test_test_contract_2852.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,60 @@ def _rule(lang, signal="test"):
["ASSERT RESULT-ONE.", "ZUNIT RESULT-TWO.", "READY TRACE."],
["MOVE UT-TEST-CASE-COUNT TO UT-TEST-CASE-NUMBER", "UT-TEST-CASE-NUMBER '. '"],
),
# --- #2853: bare-word menus in languages the crucible now measures -----------
"ruby": ( # C3: anchor rspec/minitest words; ruby has no runtime assert (C1 n/a)
[
'describe "x" do',
'context "when y" do',
'it "does" do',
"before(:each)",
"before do",
"let(:user)",
"subject { described_class }",
"expect(x).to be",
"assert_equal a, b",
"refute_nil x",
"def setup",
],
[
"# the context.",
'require "bundler/setup"',
"Struct.new(:name, :failures, :assertions)",
"named after the method",
"before the request",
],
),
"javascript": ( # C3 anchors + C1 (bare `assert` runtime is safety's; keep chai `assert.`)
[
'describe("x", () => {',
"expect(x).toBe(1)",
"it('adds', () => {",
"assert.equal(a, b)",
"beforeEach(() => {",
"afterEach(() => {",
"jest.fn()",
"cy.visit('/')",
],
[
"if (methodName === 'assert') {",
"/* assert on the output */",
"myRegex.test('x')",
"transparency. Blending",
"const describer = 1",
],
),
"java": ( # C1: `assert...(` keeps JUnit `assertEquals(`, drops the runtime `assert(cond)` statement
["@Test", "@ParameterizedTest", "assertEquals(a, b)", "assertThat(x).isNull()", "verify(mock).run()", "when(mock.get())"],
["assert(cond);", "assert x > 0;", "assert cond : message;"],
),
"groovy": ( # C1: keeps `assertEquals(`, drops groovy's parenthesized power-assert; Spock labels stay
["@Test", "given:", "when:", "then:", "expect:", "assertEquals(a, b)", "assertTrue(ok)"],
["assert(x)", "assert x == y", "assert (result > 0)"],
),
"assembly": ( # C3: keep the nasm `testcase` macro + call-anchored `it(`, drop the `(?i)` prose menu
["%macro testcase 2", "testcase {", "it (foo)", "TESTCASE macro"],
["; describe the loop", "assert clarity of intent", "; expect failure here", "ASSERT(sizeof == 8)"],
),
}

# One statement is one hit (C2), and the plant shapes stay pinned.
Expand All @@ -112,6 +166,14 @@ def _rule(lang, signal="test"):
("lua", "function probe_test(kit)\n busted(kit)\n luassert(kit)\nend", 2),
("cobol", " PROBE-TEST.\n ASSERT RESULT-ONE.\n ZUNIT RESULT-TWO.", 2),
("php", "public function probe_test($kit) {\n PHPUnit::run($kit);\n assertTrue($kit);\n}", 2),
# #2853
("ruby", "def probe\n assert_equal a, b\n refute_nil x\nend", 2),
("ruby", "x = respond_to?(:foo)\n# context here", 0),
("javascript", "describe('x', () => {\n expect(a).toBe(b);\n});", 2),
("javascript", "if (name === 'assert') doThing();", 0),
("java", "@Test\nvoid t() { assertEquals(a, b); assert(raw); }", 2), # framework assert counts, runtime assert(raw) does not
("groovy", "def 'spec'() {\n expect:\n assertEquals(a, b)\n assert(raw)\n}", 2), # label + framework assert; power-assert(raw) excluded
("assembly", "%macro testcase 2\n; describe the flow\ntestcase {", 2), # macro decl + invocation; comment prose excluded
]

PAYLOADS = [
Expand Down Expand Up @@ -153,6 +215,9 @@ def test_test_runtime_guard_is_safetys_hit_alone():
("embedded_python", "assert isinstance(value, int)"),
("lua", "assert(os.remove(file))"),
("python", "assert x == 1"),
# #2853: java's and groovy's parenthesized runtime assert is safety's now
("java", "assert(cond);"),
("groovy", "assert(x)"),
]:
assert _rule(lang, "safety").search(guard), f"{lang}: safety lost {guard!r}"
assert not _rule(lang).search(guard), f"{lang}: test still claims {guard!r}"
Expand Down
Loading
Loading