Avoid a Buildbot bug that can hang CTest steps forever

[alexreinking]

Summary

• We've had buildbot workers get stuck for days, occupying a build slot and holding performance_lock forever on that worker, starving any sibling step needing exclusive access (e.g. performance tests). This happened even when the superseding-commit cancellation service tried to cancel the offending build -- there was nothing left to cancel, because the step wasn't running a command or waiting on a lock; it was permanently parked inside a hung file transfer.
• Root cause (confirmed against production master logs and DB, byte-for-byte): CTest.run() fetches the CTest XML results file from the worker via getFileContentFromWorker(), which uses buildbot's StringFileWriter. That class decodes each 32KB transfer chunk as UTF-8 independently (buildbot/process/remotetransfer.py), so a multi-byte character split across a chunk boundary raises UnicodeDecodeError inside a Twisted PB remote-message handler -- not through buildbot's normal command-failure path -- so the step is never notified and hangs forever.
• This is a long-standing, still-open upstream bug: buildbot/buildbot#3982 (filed 2018). A 2019 fix attempt, buildbot/buildbot#4621, stalled in review and was closed by stale-bot without landing. It's still present in the pinned buildbot~=4.0 (currently resolving to 4.3.0), so there's no version bump that avoids it.
• Fix: since Xml.fromstring() accepts bytes directly, we don't need decoded text at all here. Added a small _BytesFileWriter (a FileWriterImpl that keeps raw bytes) and a local getFileBytesFromWorker() in custom_steps.py, sidestepping the bug entirely instead of working around its symptoms.

Test plan

• Reproduced the exact production failure (UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 32767: unexpected end of data) standalone by feeding a chunk-boundary-split multi-byte character through buildbot's real StringFileWriter, and confirmed the new _BytesFileWriter parses the same input cleanly via Xml.fromstring.
• uv run ruff check / ruff format --check on master/custom_steps.py
• uv run ty check master/custom_steps.py
• buildbot checkconfig .
• After merge/deploy, watch for any CTest step that previously would have hung to complete normally instead

🤖 Generated with Claude Code