Repository navigation
test(tests): rebuild contract coverage and remove runtime test hooks - #193
Merged
Merged
Conversation
Replace scalar presence, type, range, enum and representability checks with a named matrix and direct typed outcomes. Nested parameter and ParameterSet responsibilities await their own replacement. Validation: cmake --build build --target edgeflow_test_nodes_runner -j 4; nodes runner ScalarParametersTest.*:ParameterBindingTest.* passed 23 tests. Full phase gate follows the three pilot checkpoints.
Replace the entire prior text_chunk suite with codepoint windows, overlap coverage stopping, per-request child numbering, parent counts, empty values and invalid UTF-8 failure. Shared parameter and author mechanisms have separate owners. Validation: built edgeflow_test_nodes_runner; TextChunkNodeTest.* passed all 5 tests.
Run the actual keyword converters, Pipeline and rule node through Operator for three distinct rows. Compare full literal JSON, service and status, return all leases and refill the pool, then prove whole-rule replacement through public Control. Validation: built edgeflow_test_adapter_runner; KeywordResponseTest.* passed its complete business case.
Add a literal 101-codepoint request under empty configuration to distinguish defaults 100/0 from nearby window and overlap mutations. Record the fresh pilot size without treating untouched responsibility groups as completed. Validation: TextChunkNodeTest.* passed 6 tests; scalar lower-bound mutation in an isolated header probe was detected (exit 1). Earlier e160c29 canonical passed 106/106 before this review fix; current phase gate follows.
Fresh typed stock data verifies struct defaults and preparation, primitive and packed boolean containers, escaped nested pointers, and non-null JSON payloads. Remove the replaced recursive matrix and its dedicated declarations. Validation: built nodes runner; NestedParametersTest.*:ParameterBindingTest.* passed 14 tests.
…cy suite Verify flat included fields, inside-out hooks, move-only prepared state, immutable effective values, actual binding facts, callback barriers and integer projection. Replace all remaining legacy parameter tests; scalar and recursive matrices now have one owner each. Validation: built nodes runner; ScalarParametersTest.*:NestedParametersTest.*:ParameterStateTest.* passed 12 tests.
Replace the prior suites with owned payload and stable-view evidence plus a synchronized competition for one immutable key. Provenance transformations belong to batch algorithms; input binding and diagnostic isolation have their own owners. Validation: standalone compilation of this exact source against production headers and GoogleTest; both TypedBlackboardContractsTest cases passed. Combined runner rebuild is pending completion of fresh planning/author files.
RegistrationStartupTest: one native case passed in its isolated process. Checks first-entry retention and repeated Operator startup rejection after an actual duplicate registration.
LayerGuardTest passed, including native compiler checks of actual per-layer target include views and prohibited upward dependencies.
ThirdPartyCacheMetadataTest passed using actual CMake/cache scripts, including pinned metadata and llama source reuse checks. This is build transport evidence, not vendor runtime inference.
SpecSignatureDiagnosticsTest passed with independently compiled inline public authoring consumers. Retires the old signature fixture after its cases were replaced.
ArchitectureDocsDriftGateSelfTest and DiagramRenderGateSelfTest passed against actual gate scripts with isolated diagnostic perturbations.
ScriptGeneratorDetectionTest passed against actual configure/build transport and generator selection; final canonical gate remains pending.
Fresh native runtime diagnostic passed 368 required tests with one explicit optional GGUF skip. Official minimal ASan/UBSan focused checks passed 99/99 required native cases. Retires old scenario harnesses, DTO examples and unused validation fixtures after their responsibilities moved to fresh native tests. No failure swallowing or per-thread allocation ledger remains. Final canonical gate is pending.
Current native tests compile without these legacy test-only helpers and synthetic benchmark scenarios. Actual Model semantics and authoring contracts are proved by their fresh owning suites; no historical test copies are retained. Unreferenced paths were checked in the working tree before retirement.
Default focused collection checks passed 32/32, including actual DemoSmokeTest, DevRecipeTest and Studio consumers. One OBJECT target now registers five minimal Model capabilities and two neutral kFixture resource holders only in Demo/test CLI. Runtime semantic suites use their own strict probes. Removes the old split fake inference implementation and its headers. No real vendor inference claim is made.
Default native ONNX Runtime tests passed with independent literal outputs from a public-schema small rerank graph and actual Sequence-output load rejection. Final collection focused checks passed 32/32, including cross-rerank Demo success and missing-model failure. Missing-model fixture is one generated mutation of the valid configuration; unused embedding graph and duplicate handwritten negative configuration are retired. Generator/template maintenance is counted.
Fresh compiled GoogleTest cases are discovered directly, without historical suite inventory. Real-main negative checks prove required skip, zero selection/repeat, first-iteration skip and optional cleanup failure cannot succeed; explicit optional asset skip uses 77. Default focused collection diagnostic passed 32/32 including QualityGateScriptsContractTest and TestLabelsContractTest. Official minimal ASan/UBSan99/99 and TSan10/10 resource/concurrency checks passed. Public SDK target stays isolated. Complete final gate remains pending.
BgeEmbeddingModelTest and BgeRerankerModelTest passed all 12 native cases in the default focused diagnostic. Shared helper only records session calls, creates host tensors and supplies a temporary vocabulary; local defaults and independent literal expectations remain. Total helper plus consumers decreases from584 to551 manual lines.
Adds braces around conditional GoogleTest assertions to avoid dangling-else macro warnings and formats the native registry helper. Seven BusinessResponsesTest native cases passed in the final focused diagnostic. Optional E2E is not real-weight acceptance in this environment.
Fresh five-case CLI suite passed; DocLinksSelfTest, DocLinksTest, LayerGuardSelfTest and TestLabelsContractTest passed4/4 in current CTest. Old inline test body and its retired flag are removed; actual production checker behavior remains unchanged.
Fresh six-case CLI suite passed with26 actual invocations and no skips. Native source-ownership, vendor-header boundaries and shell failure transfer are exercised without SDK data. CTest focused engineering checks passed4/4. Removes old inline script self-tests and retired flags.
Removes retired test environment flags and follows native preflight checks with actual Demo execution of text, generated embeddings and vision configurations. YAML parsed with all six jobs retained; read-only review verified config/data/resource paths and nonzero failure propagation. Remote CI and private SDK execution were not run in this external environment.
Applies the required clang-format18 style to the new thin allocation helpers; no behavior changes. Latest focused collection and engineering checks passed.
Documents fresh native responsibilities, independent expectations, actual discovery and skip policy, current helper references and explicit verification limits. Complete manual cost is22311 lines including Demo mock scenarios, generators/templates, runner scripts and allCI. The20200 upper budget is not met; the working plan remains open and its original budget is unchanged. Full canonical verification follows at this exact source revision.
Removes the never-set Pipeline test_internal_hook_/friend and uncalled IoConverterRegistry ResetConflictForTesting. Required native registry clear/access seams remain. References were audited; default affected build succeeded and52 focused native checks passed. Cleanup ASan/UBSan118/118 required cases passed against the resulting private class layout. TSan10-case revalidation is pending; full canonical follows after it. No public Operator ABI changes.
Adds150 previously uncounted manual lines: external test CMake wiring112, current source test interfaces34, and UBSan suppressions4. Complete maintained cost is22461, exceeding the unchanged20200 upper budget by2261. Size acceptance remains unmet and the plan stays open. Markdown links passed; final canonical will bind this revision.
Reject DISABLED entries and enabled GoogleTest listing flags before counting CTest coverage. Reuse the existing compiled guard for normal commands, explicit listing=false/0, disabled entries and both listing flag forms. Validation: old checker failed all three new negative regressions. ctest --test-dir build -R "^(QualityGateScriptsContractTest|TestLabelsContractTest)$" --output-on-failure --no-tests=error passed 2/2 after the fix, including explicit listing=false/0. Maintenance cost at this checkpoint is 22,489 lines, still above the unchanged budget. Full delivery gate follows the separately scoped mock parameter cleanup.
Remove max_seq_len from the mock LLM Definition and creation state, and from its five maintained Demo configurations. The mock now consumes NoParameters. Update the full maintenance cost to 22,465 lines; the original budget remains unmet. Validation: rebuilt alg_demo, alg_pipeline_tool_test and the tooling runner. All five edited JSON files passed actual validate/plan and their Demo runs checked nine literal responses. describe-model exposes no mock LLM fields; reintroducing max_seq_len fails with UNKNOWN_CONFIG_FIELD. Sixteen affected Demo, Catalog, CLI, Studio, recipe and JSON prompt CTests passed. Independent review passed. The first canonical attempt stopped on macro formatting; clang-format-18 fixed and checked that file. Canonical revalidation follows this frozen commit.
Run CTest from its target directory instead of using --test-dir, so the minimum supported CMake 3.19 actually reads the intended registrations and nested skip fixtures. Keep all execution, diagnostic and skip assertions intact; update the documented command. Validation: QualityGateScriptsContractTest and TestLabelsContractTest passed 2/2 on the default toolchain and on checksum-verified official CMake/CTest 3.19.8 for aarch64. The minimum-version run used the actual build directory and selected child CTest executable from that official release. Independent source review passed. Canonical revalidation will run through the authorized delivery script after the separate E2E carrier correction.
Initialize the GGUF translation and Whisper speech carriers with the current public platform_mock service constants. This lets physical E2E requests reach the actual Converter and Model instead of failing input validation with service_type zero. Production code, response assertions and optional asset conditions are unchanged. Validation: independent review confirmed the current native binding expects translation 8 and audio intent 6. The corrected E2E translation unit compiled without warnings using the actual Whisper variant command with HAVE_WHISPERCPP=1. Physical execution will be verified by the authorized GitHub CI jobs; no local full gate or extra weight downloads were performed for this checkpoint.
Remove ClearForTesting from the production class and use the existing RegistryTestAccess helper for isolated converter setup and restoration. Swap registrations and normalized parameters under the existing lock order, then destroy their owners after unlocking. Validation: 27 focused adapter cases and 27 ASan/UBSan cases passed. The final combined working tree passed the canonical gate (389 passed, one optional GGUF skipped, zero failures).
Remove the production waiting counter, its guard and the pool test friend. The adapter test executable observes the real standard-library wait with a one-shot thread-local signal, then forwards to the original function. Return and Close take the held pool mutex before notifying. Keep necessary Session and Runtime observers and update the maintenance scope and unverified Apple execution boundary. Validation: 27 focused cases, 27 ASan/UBSan cases and 3 TSan cases passed. Temporary production-source mutations that removed either wakeup were rejected by the corresponding tests. The canonical gate passed: 389 CTest entries passed, one optional GGUF skipped, zero failures. Production SDK symbol checks found no test wait or reset implementation.
chamsechan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuild the test system around current production contracts, replacing legacy suites and scenario harnesses with direct component, lifecycle, tooling and complete Operator response tests. Remove production test hooks and verify concurrency through normal interfaces and test-side barriers.
max_seq_lenparameter from five Demo configurations.Validation for
51000af74c7acbd1107a7385a142601f794f286a:bd3a4460dd0f3ffecb28bb991ca03f94203c01e0; all six main push CI checks passed for that exact merge SHA: main CI.The maintenance budget remains unmet: 22,465 lines versus the original upper budget of 20,200, exceeding it by 2,265. The budget is unchanged and
plans/TEST_SYSTEM_REDESIGN.mdremains active. This change delivers the current implementation and test-hook cleanup without claiming completion of the full redesign plan.