Raw vs derived transcripts: speaker-labeled transcript.txt from WhisperX json#15
Merged
Conversation
transcript.json is the immutable raw diarization layer (SPEAKER_NN forever); transcript.txt is a derived view regenerated from the json plus the journal-owned parts[].speakers mapping. Provenance is encoded by file shape (see the new table) — no metadata flag needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoVcFP7bBu92mK6RqM5nPG
Regenerate transcript.txt for every item whose transcript.json carries WhisperX diarization (apply_speaker_names.py, per the raw/derived design in docs/SCHEMA.md): - 421 items upgraded from YouTube-caption text to speaker-labeled WhisperX text (SPEAKER_NN until parts[].speakers maps names); the previous caption text is preserved as captions/youtube_captions.txt - 8 items with existing speaker text reformatted to the canonical rendering (consecutive same-speaker segments merged) - GuestStream_128 + Session_023 transcript.json reverted to raw SPEAKER_NN labels (names had been baked in; they now live only in parts[].speakers and the derived txt) - legacy AssemblyAI-shaped transcript.json files left untouched - INDEX.json / INDEX.md regenerated; validate_journal.py passes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UoVcFP7bBu92mK6RqM5nPG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the raw vs derived transcript design now documented in
docs/SCHEMA.md:transcript.json= immutable raw diarization layer. Speaker labels staySPEAKER_NNforever, so re-mapping is always possible (two labels identified as the same person stay distinguishable).parts[].speakersinmetadata.jsonis the only place human speaker identifications live (journal-owned, hand-edited).transcript.txt= derived view, regenerated from the json + mapping byJournal-Utilities/scripts/apply_speaker_names.py(idempotent; unmapped labels staySPEAKER_NN).What changed
transcript.json(carried over by the June refactor) — only the txt was still captions. Previous caption text is preserved ascaptions/youtube_captions.txtper item (no data loss in the working tree; also still in git history).transcript.jsonreverted to rawSPEAKER_NN(names had been substituted in-place; they now live only inparts[].speakersand the derived txt — txt output is unchanged in content).transcript.jsonfiles untouched (frozen legacy raw layer, per schema).INDEX.json/INDEX.mdregenerated;validate_journal.pypasses (missing_manifest_videos: 0, no forbidden files).Verification
.simple.txtoutputs (checked against commit9da9266) for unmapped items.Speaker-name mapping remains the manual step: record
parts[].speakersin an item's metadata and re-run the script.🤖 Generated with Claude Code
https://claude.ai/code/session_01UoVcFP7bBu92mK6RqM5nPG