Skip to content

docs(tutor-quality): plan stage 2b implementation - #117

Merged
ttbombadil merged 2 commits into
mainfrom
docs/tutor-quality-stage-2b-implementation-plan
Sep 28, 2026
Merged

ttbombadil merged 2 commits into
mainfrom
docs/tutor-quality-stage-2b-implementation-plan

Conversation

@ttbombadil

Copy link
Copy Markdown
Collaborator

Summary

Adds the non-normative implementation plan for Tutor Quality Stage 2B, derived from the merged Stage-2B SSOT.

The plan splits implementation into six independently reviewable PRs and prioritizes an early absolute semantic-evaluation vertical slice before multi-model benchmarking, pairwise evaluation, or ranking.

Architecture

  • Keeps the Semantic Corpus Stage-2B-owned.
  • Uses a narrow Stage-2A Case Adapter as the primary Tutor execution path.
  • Keeps runTutorQualityEvaluation and the Stage-2A execution contract unchanged.
  • Supports compatible existing Stage-2A transcripts as a secondary ingestion path.
  • Keeps candidate-specific CaseExposureRecords separate from Semantic Corpus versioning.
  • Keeps the existing Tutor LLMProvider contract unchanged and plans a separate structured-output capability for Judge calls.
  • Computes Stage-2B transcript digests without modifying Stage-2A artifacts.

Delivery sequence

  1. Semantic contracts, Corpus, and Stage-2A adapter
  2. Absolute Semantic Judge foundation
  3. Human Reference and calibration evidence
  4. Repeated multi-model benchmark and raw reporting
  5. Blinded pairwise evaluation
  6. Conditional reporting/ranking

FIRST_USEFUL_STAGE_2B requires PR 2 plus DG-1 and DG-10 for the bounded real-Judge smoke.

MODEL_BENCHMARK_MVP is targeted after PR 4 once sufficient complete reviewed cases exist.

Decision gates

The plan deliberately leaves unresolved normative decisions explicit rather than encoding them as implementation defaults, including rubric scale, calibration thresholds, statistics, held-out governance, ranking, cost normalization, pairwise protocol, and operational limits.

Scope

Documentation/planning only.

  • No Stage-2B implementation.
  • No Tutor runtime change.
  • No Stage-1 or Stage-2A contract change.
  • No provider calls.
  • No production model selection.
  • No Stage 2C work.

Verification

  • npm run check:docs (Node 24.20.0)
  • git diff main...HEAD --check
  • Only docs/plan-tutor-quality-stage-2b.md changed.

@ttbombadil
ttbombadil merged commit 74a2739 into main Sep 28, 2026
5 checks passed
@ttbombadil
ttbombadil deleted the docs/tutor-quality-stage-2b-implementation-plan branch September 28, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant