Skip to content

Support independently versioned full Japanese pronunciation resources - #250

Merged
Lucas1479 merged 12 commits into
mainfrom
codex/japanese-resource-iterations
Oct 21, 2026
Merged

Lucas1479 merged 12 commits into
mainfrom
codex/japanese-resource-iterations

Conversation

@Lucas1479

@Lucas1479 Lucas1479 commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Result and acceptance boundary

Keep the complete 134,201-row Japanese pronunciation dictionary as an independently versioned optional resource, with reviewed corrections and explicit runtime deferrals. This PR fixes the personal-dictionary, casing and numeric-hyphen regressions from independent review. The maintainer authorized merging the application code after the final head passed CI. Resource publication, remote data-repository creation and live installation remain outside this authorization.

Base: main 879a8fbd, containing #252, #249 and #253. Integration head 3e06932a includes #253 without conflicts; Japanese implementation hashes remain identical to qualified 518475d2. Inherited Qwen/provenance conflicts use exact main versions; no Qwen, AEC, root dependency or lockfile differences remain.

Follow-up to independent review

Independent review confirmed the previous four findings on 916fa472 and the scaling/width fixes on 69368c19. The final follow-up addresses the remaining title-case personal-entry mismatch:

  • Collects personal keys and subtracts them once from the official vocabulary, avoiding one full-set copy per personal row.

  • Normalizes personal surfaces with the same case rule as text input, then native width normalization, so Menu/Existing/Okabe entries match lower/title/uppercase inputs under official aliases and formats 1/2 native rows. Case-owned Disney remains uppercase. Source CSV bytes remain intact; native/audio identity is revised to v11. Previously ineffective personal entries can start taking effect after upgrade.

  • Removes dead numeric-prefix handling without changing the reviewed casing policy.

Windows fresh-process configuration measured for the prior scaling fix (single measurements, not end-to-end TTS):

| Personal rows | Before | After |

|---|---:|---:|

| 200 | 2.92 s | 2.21 s |

| 1,000 | 5.91 s | 2.22 s |

| 3,000 | 14.93 s | 2.20 s |

The regression suite uses a deterministic vocabulary-copy work bound instead of a machine-speed timeout. The new tests reproduce three failures on the unmodified old implementation (halfwidth overrides in both formats and repeated copies), and pass after the fix.

The final case-alignment patch passes six additional native cases that all fail on 69368c19, covering half/fullwidth entries, alias/native partitions, exact base names and cached reloads. With the final code, rc2 plus 10,000 synthetic personal rows configured in 2.25 seconds in a fresh Windows process with precompiled dictionaries (one sample).

Hand remains a low-priority base-segmentation exception. General lowercasing of equal-reading base spellings was tested and not included: base equivalence alone does not preserve selection with the legacy CSV loaded. There is no new word-specific exception or unverified resource mapping.

Resource accounting and selection

Unchanged candidate: 2026.10.22.1-rc2, 37,386,096 bytes.

ZIP SHA-256: a935b3adfded9022afe3b2384cebc2ef1e4023f54f48df1f9ded9ff45b896989.

  • 134,201 original/master rows retained: 462 reviewed corrections, 133,739 original-unverified, zero source deletion.

  • Supplement: 4,539 rows / 4,537 words. Runtime: 121,912 native rows and 4,411 aliases, not unique newly corrected-word counts.

  • 7,989 source exclusions: 4,920 unresolved alternatives, 1,300 CMU flags, 1,038 base-owned rows, 399 pending pronunciation normalization, 236 short tokens, 77 contextual rows and 19 known rejected rows. Overlay replacement is separate.

  • Reviewed defaults supersede unverified alternatives. Existing CMU final-phone screening is reused. Personal keys take precedence over official aliases and native-word boundary insertion.

  • Exact base spelling ownership applies with no pack, format 1, legacy CSV and format 2; unknown spellings use the established lowercase path. Numeric-adjacent hyphens remain boundaries; alphabetic compounds remain whole.

  • JMdict attribution is included; Alkana comparison readings stay in research data outside runtime metadata. The unused format-1 alias prototype is removed.

Unchanged, single-reading, unflagged original rows remain active and unverified. This policy (#244) still requires explicit maintainer acceptance. Independent review reported remaining errors and deferred common words; the follow-up receipt records a frequency-priority review backlog without approving speculative readings.

Build inputs remain data commit 4f490be3f96aabde99885e3eea0e17d7e0a39d04. Full source ZIP and history bundle remain at data snapshot 6afbd8a8583046b3150cc578aec104a753452ddf. All resources, source archives, corpus and recordings remain local and outside the application diff.

Validation

  • After integrating fix(vn): qualify macOS Tk/ImageTk and document interpreter migration #253, 154 related local tests passed, including its six VN environment contracts.

  • 148 related tests passed (145 Japanese/resource tests and 3 main Qwen contracts), with two existing deprecation warnings; Ruff and diff checks passed.

  • Four fresh production profiles, 26,552 records each: no pack, format 1, legacy CSV and full candidate. Every phone/target record equals reviewed 69368c19 (and unchanged 916fa472); the prior seven-profile comparison against main 8de4c958 remains reproducible. This is regression evidence, not daily-error-rate measurement.

  • All 127 published readings retain their three original contexts (381/381). The previously disclosed Main reading difference and unscored accompaniments spans are unchanged.

  • The prior actual format-1 installation, full-candidate update, rollback and restoration remain recorded. The final related suite again passes lifecycle, protected personal/legacy files, identity restoration, idempotence and failed-update recovery contracts. Builder/installer code and the reviewed resource ZIP remain unchanged.

  • Existing four GSV A/B pairs remain historical listening evidence, not whole-vocabulary certification. No new audio or ASR-count claims. The current changes do not alter the reviewed no-personal corpus sequences.

All 17 checks passed on personal-case head 518475d2155607f07632c7d835eeade966ce1788. All 17 fresh CI checks passed on integration head 3e06932a3ed38fe127be6b4d5d66e0dd356b01af against main 879a8fbd. The maintainer authorized code merge on this qualified head; the resource candidate remains unpublished. Full resource/corpus and performance validation is Windows-local; native model CI verifies dictionary hashes and regression contracts on Windows/Linux/macOS, not full acoustic validation on every platform.

References: resource contract, current case-alignment receipt, prior scaling receipt, prior integration receipt.

@Lucas1479
Lucas1479 changed the base branch from main to codex/headless-inference-security October 21, 2026 12:21
Base automatically changed from codex/headless-inference-security to main October 21, 2026 13:51
@Lucas1479
Lucas1479 marked this pull request as ready for review October 21, 2026 16:39
@Lucas1479
Lucas1479 merged commit 5382b32 into main Oct 21, 2026
17 checks passed
@Lucas1479
Lucas1479 deleted the codex/japanese-resource-iterations branch October 21, 2026 16:58
@Lucas1479
Lucas1479 restored the codex/japanese-resource-iterations branch October 21, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant