Skip to content

feat: add Tutor Quality Stage 2A real-provider evaluation foundation - #115

Merged
ttbombadil merged 22 commits into
mainfrom
feature/tutor-quality-stage-2a
Sep 28, 2026
Merged

ttbombadil merged 22 commits into
mainfrom
feature/tutor-quality-stage-2a

Conversation

@ttbombadil

@ttbombadil ttbombadil commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

  • Real-Provider Evaluation Foundation using the normal TutorService/LLMProvider path
  • Versioned Anchor Corpus with 10 scenarios, including TQ-REG-001
  • Separate executionStatus and invariantViolations axes
  • Raw capture of the parsed provider result before TutorService validation/repair
  • Secret-free allow-list transcript and report artifacts
  • evaluationIdentity covering Git, Course, Corpus, Provider, Model, Prompt revision and digest metadata
  • Fixed model identifier; auto is rejected
  • Provider-call budget counting listModels and generation calls
  • Prompt revision and digest metadata
  • Manual workflow_dispatch only; no required CI dependency

Stage 1 safety

  • Existing deterministic Tutor Quality hard gates remain unchanged
  • Stage 2A is not a pull-request hard-gate dependency
  • No direct provider HTTP calls
  • Existing Tutor and provider trust boundaries remain authoritative
  • Shared diagnostic validation is used instead of duplicated rules

Explicitly not included

  • LLM-as-Judge
  • Semantic learning-support scoring
  • Learning-effect measurement
  • Adaptive Tutor strategies
  • Fact Extractor expansion
  • Learner profiles

Verification

All local verification was run under Node 24.20.0:

  • npm run test:tutor-quality: 29 files, 216 tests passed
  • npm run test:unit: 2477 tests passed, 1 existing skip
  • npm run check passed
  • npm run check:docs passed
  • git diff --check passed
  • Fake-Provider Anchor Run over all 10 scenarios: 0 technical failures, 0 invariant violations
  • No-credential CLI: not-run / missing-credential / 0 provider calls / report.json only
  • Sonar Quality Gate passed: 0 new violations, 100% reviewed security hotspots, 82.5% new-code coverage, no open issues

Real Provider Smoke Test

  • Provider: KI:connect
  • Model: mistralai-mistral-small-4-119b-2603
  • Scenario: TQ-REG-001
  • Status: completed
  • Returned model matched the requested model
  • No technical failures
  • invariantViolations = []
  • Secret scan passed
  • Credential removed after the run
  • No secrets or evaluation artifacts committed

No credential value or token fragment is included in this description.

@ttbombadil
ttbombadil merged commit 39c0bb9 into main Sep 28, 2026
5 checks passed
@ttbombadil
ttbombadil deleted the feature/tutor-quality-stage-2a branch September 28, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant