Skip to content

fix: the remaining Unbounded workflow histories sub-items - #449

Open
KillerX wants to merge 7 commits into
fix/incremental-ingest-signal-selectorfrom
fix/unbounded-histories-payloads
Open

fix: the remaining Unbounded workflow histories sub-items#449
KillerX wants to merge 7 commits into
fix/incremental-ingest-signal-selectorfrom
fix/unbounded-histories-payloads

Conversation

@KillerX

@KillerX KillerX commented Aug 13, 2026

Copy link
Copy Markdown
Member

10/n of a stack. Base: fix/incremental-ingest-signal-selector (#448). One commit per sub-item — the last five of the Unbounded workflow histories finding, after the two loops in #447 and #448.

MoveFilesWorkerFlow continues as new — 0498ba9

The finding calls it an eternal fixed-ID signal workflow whose history is never reset. It is not eternal: ReceiveWithTimeout gives up after ten seconds and the run completes, restarted by the next SignalWithStartWorkflow.

The real exposure is narrower. Requests arriving less than ten seconds apart keep one run alive indefinitely, and every move adds a GetShapes plus one MoveFileWait per shape to the same history — a bulk move never lets the run end. It now continues as new when the server suggests it, and only while the signal channel is empty so no queued request is dropped. A request landing after the check is handled by the server, which turns an unhandled signal at completion into a new workflow task.

MergeExportData stops persisting a marker — 4ca34f9

exportDataToMergeInputs ran inside a workflow.SideEffect, which writes its return value into the history as a marker: every clip path, language and offset, recorded once as the marker and again in the activity inputs derived from it. The function reads no clock, file or global, so the SideEffect bought nothing.

It does range over the per-clip audio and subtitle maps, so those keys are now sorted — each language accumulates into its own MergeInput so the order was already immaterial, but running in workflow code it has to be visibly so, to workflowcheck and to the next reader.

Guarded with workflow.GetVersion, same reasoning as #448: an export started before this recorded a marker, and replaying it without one fails the workflow task, which Temporal retries forever.

CleanupTemp returns counts — fdfd164

It sweeps ~60 folders and returned every deleted path as its result, which goes into the completion event. DeletedFiles is replaced by DeletedCountPerRoot, which is what the result is actually read for — seeing a folder that is not being cleaned. Paths remain in the activity results and worker logs.

Also moved the per-folder log line below the error check, where it can report the folder it just finished: as written it logged the running total, so it printed 0 for the first folder however much it had deleted.

ImportSubtitles can take a path — d04b386

A word-level transcription of a long programme is megabytes, and a workflow argument lives in the WorkflowExecutionStarted event. A big enough one does not merely bloat the history, it exceeds Temporal's payload limit and the workflow cannot start.

SubtitlesFile is an alternative input read by an activity. Subtitles is kept, not replaced — nothing in this repository starts ImportSubtitles, so the field cannot go until whatever produces the transcription has moved over.

VXExport sends children only what they read — 3aa26bc

Each destination child gets the whole ExportData, written into the parent's history once per destination and into each child's again. Most of it is per-language material no child reads: the only thing a child does with a clip is hand it to GetTimedMetadataChaptersActivity, which needs the VXID and the in/out points.

clipsForChapterLookup narrows the copy to five fields. The hazard is a child later reading a field that is now silently zero, so TestClipsForChapterLookup pins the surviving set as a contract.

🤖 Generated with Claude Code

KillerX and others added 7 commits August 13, 2026 15:24
…etes

stopPreviewFunc cancels the growing preview the instant the copy loop ends, so
the lowres shape the asset keeps covers only as much as ffmpeg had reached by
then. The tail of a live recording — the part most likely to be checked right
after the event — is missing from the preview, and nothing later replaces it:
this workflow never runs TranscodePreviewVX.

Once the file is complete and closed in Vidispine, it now transcodes a preview
of the whole thing and imports that over the same shape tag with Replace set.
The growing import deliberately does not set Replace, because it is the first
file to claim the tag; this one is the second and is meant to supersede it.

Started right after CloseFile and awaited at the very end, so a full transcode
does not hold up the reaper audio imports, the transcription or the duration
fix. A failure is logged and notified rather than returned: by that point the
audio and video are already in place, and an incomplete preview is not worth
failing an ingest over.

The aux output folder needed a name of its own for this. previewPath held the
folder and was then reassigned to the file inside it, so there was nothing left
pointing at the directory.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The finding calls this an eternal fixed-ID signal workflow whose history is
never reset. It is not eternal: ReceiveWithTimeout gives up after ten seconds
and the run completes, to be started again by the next SignalWithStartWorkflow.

The exposure is narrower and real. Requests arriving less than ten seconds
apart keep one run alive indefinitely, and every move adds a GetShapes call
plus one MoveFileWait per shape to the same history — so a bulk move never lets
the run end and never resets the history.

It now continues as new when the server suggests it, which is the signal that
takes the actual event count and size into account rather than a threshold
guessed here. The check runs only while the signal channel is empty, so no
queued request is dropped; a request that arrives after the check but before
the server applies the continue-as-new is handled by the server, which turns an
unhandled signal at completion into a new workflow task.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MergeExportData computed its merge inputs inside a workflow.SideEffect, which
writes the returned value into the history as a marker. For a long export that
value is every clip path, every language and every offset — recorded once as
the marker and again in the activity inputs derived from it.

exportDataToMergeInputs takes an ExportData and two paths and returns a value;
nothing in it reads a clock, a file or a global, so the SideEffect bought
nothing. It does range over the per-clip audio and subtitle maps, which is why
it now sorts those keys: each language accumulates into its own MergeInput so
the order was already immaterial, but running in workflow code it has to be
visibly so — to workflowcheck and to the next reader.

Guarded with workflow.GetVersion for the same reason as the live ingest: an
export started before this recorded a marker, and replaying it without one
fails the workflow task, which Temporal retries forever.

Four tests, including one that calls the function fifty times and compares, so
a future edit that lets map order leak into the result is caught rather than
being caught by a customer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CleanupTemp sweeps around sixty folders and returned the full list of deleted
paths as its result. A workflow result is written into the completion event, so
a fortnight of temp files landed in the history and in every caller that
fetches the result — to say something a number says.

DeletedFiles is replaced by DeletedCountPerRoot, which is more useful for the
thing the result is actually read for: seeing that a folder is not being
cleaned. The paths are still in the activity results and the worker logs.

The per-folder log line moves below the error check, where it can report the
folder it just finished rather than the running total — as written it logged 0
for the first folder no matter how much it had deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ption

The transcription arrives as a workflow argument, and a workflow argument is
stored in the WorkflowExecutionStarted event. A word-level transcription of a
long programme runs to megabytes, so every import writes that into the history
— and a big enough one does not merely bloat it, it exceeds Temporal's payload
limit and the workflow cannot be started at all.

SubtitlesFile is an alternative input: point it at the same JSON on shared
storage and an activity reads it, keeping the payload out of the history. When
it is set, Subtitles is ignored.

Subtitles is kept rather than replaced. Nothing in this repository starts
ImportSubtitles — it is registered and triggered from outside — so the field
cannot be removed until whatever produces the transcription has moved over. The
comment on it says why to prefer the path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VXExport hands each destination child the whole ExportData as a child-workflow
argument, so it is written into the parent's history once per destination and
into each child's history again. Most of that payload is per-language material
— for every clip, the video file, a map of language to audio file with its
stream list, a map of language to subtitle file, and the transcript path.

No child reads any of it. The only thing a child does with a clip is hand it to
GetTimedMetadataChaptersActivity, which resolves chapters from the VXID and the
in/out points; the per-language files are consumed by MergeExportData, which
takes the parent's own copy.

clipsForChapterLookup copies the clips down to those five fields. The parent is
unaffected — it keeps the full data it was given.

The obvious hazard is that a child later reads a field that is now zero, and
does so silently. TestClipsForChapterLookup pins the surviving set as a
contract, so the place to change is named in the failure.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Roughly a quarter of the lines added by this branch were comment, and most of
the excess was the commit message written into the source a second time — why
the change was made rather than what a reader of the code needs.

What stays is the part that is not recoverable from the code: why the version
gate exists, why the channel has to be empty before continuing as new, why the
clip fields are narrowed and where to add one, and why the cleanup result
carries counts. What goes is the retelling around it.

One comment went entirely rather than shrinking: the note explaining that the
per-folder log line had moved below the error check described the change, not
the code, and the line reads correctly on its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@KillerX
KillerX force-pushed the fix/unbounded-histories-payloads branch from f5aa249 to a353c1d Compare August 13, 2026 13:26
@KillerX

KillerX commented Aug 13, 2026

Copy link
Copy Markdown
Member Author

Trimmed in 1dd3280. You were right about the density — roughly a quarter of the lines this branch added were comment, and most of the excess was the commit message written into the source a second time.

What stays is the part that is not recoverable by reading the code: why the version gate exists, why the channel has to be empty before continuing as new, why the clip fields are narrowed and where to add one, and why the cleanup result carries counts instead of paths.

One went entirely rather than shrinking: the note explaining that the per-folder log line had moved below the error check described the change rather than the code, and the line reads fine on its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant