Skip to content

Establish the seasonal live-client Workshop validation and structured-diff workflow #21

Description

@Teakowa

Parent: #10
Depends on: #18, #19

Goal

Establish a repeatable seasonal validation workflow that takes the canonical Workshop census through the live Overwatch client, records provenance-rich captures, and reports structured semantic/catalog drift against prior evidence.

Current status

The repository-owned/offline portion is implemented and merged via #30: provenance-rich LiveCapture, canonical census/catalog identity linkage, structured diff classification, CLI support, and the documented maintainer procedure are in main.

This issue remains open for the real-client acceptance boundary only. The remaining work is to run the procedure against an actual current Overwatch client, record an en-US baseline and zh-CN locale sweep with pinned provenance, review the resulting feature-attributed structured diff, and preserve those captures as real evidence. No fabricated or synthetic capture should be used to close this issue.

Context

Workshop changes are partly black-box and may not appear in patch notes. Offline catalog/parser tests therefore cannot by themselves establish that the current client still accepts and exports the declared surface. #19 provides deterministic census shards; this issue turns them into repeatable live-client evidence without making live validation the runtime dependency of workshop-rs.

Scope

  • Define the manual/semi-automated workflow for assembling census shards into client-importable probes, importing them into the current Overwatch Workshop client, exporting/copying the resulting Workshop text, and recording the capture.
  • Record capture provenance including game/client version or season identifier where observable, capture date, client locale, catalog identity, census identity/version, and relevant environment notes.
  • Use en-US as the primary semantic baseline and define an explicit zh-CN locale sweep for localization/catalog drift; keep additional locales optional/evidence-driven.
  • Compare new captures with prior evidence using stable Define the Workshop conformance result contract and stable feature identities #18 feature identities and semantic/catalog-aware classifications rather than only whole-text diffs.
  • Classify observed changes such as lexical/locale/content/schema/signature/serialization drift where the evidence supports that distinction; keep runtime-behavior claims separate unless a reproducible runtime experiment exists.
  • Preserve raw client captures as evidence while generating machine-readable structured diff reports suitable for review and later catalog updates.
  • Document the review path from observed drift to a focused canonical catalog/semantic issue or data update; do not silently rewrite expected output to match a new capture.
  • Keep the workflow usable without requiring a network service or runtime online catalog lookup.

Non-goals

  • Automating Overwatch client startup, login, locale switching, or gameplay through unsupported mechanisms.
  • Claiming exhaustive runtime/gameplay correctness from import/export acceptance.
  • Making seasonal client validation mandatory for every normal code PR.
  • Automatically accepting client drift as the new canonical truth without review.
  • Expanding all client locales in the first implementation.

Acceptance criteria

  • A maintainer can follow the documented procedure from Build the canonical sharded Workshop feature census #19 census artifacts to a provenance-recorded live-client capture and structured comparison report.
  • en-US baseline and zh-CN locale sweep produce feature-attributed evidence rather than only raw text snapshots.
  • Prior/new capture differences are classified without conflating locale drift, catalog/content drift, semantic/schema drift, and runtime-only uncertainty.
  • Raw captures and machine-readable reports are reproducible/pinned to census/catalog identities.
  • Observed drift leads to explicit reviewed follow-up work rather than silent expectation updates.
  • The normal workshop-rs runtime remains fully offline with bundled reviewed catalog data.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    Backlog

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions