Skip to content

quest: delete mkv-fragments, the MKV importer no longer replays input - #7515

Merged
kixelated merged 4 commits into
mainfrom
quest/m2/performance-survey/mkv-fragments
Nov 3, 2026
Merged

quest: delete mkv-fragments, the MKV importer no longer replays input#7515
kixelated merged 4 commits into
mainfrom
quest/m2/performance-survey/mkv-fragments

Conversation

@kixelated

@kixelated kixelated commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Claims and deletes quest/m2/performance-survey/mkv-fragments.md. The maintainer chose deletion on 2026-11-01 (see Decisions).

Summary

The quest targeted Import::drain cloning the retained BytesMut and replaying a fresh WebmIterator over it on every decode (audit commit 1d43c096e044, import.rs lines 227-252). A B-byte block arriving in c-byte chunks copied O(B²/c) bytes. That code no longer exists:

  • moq-dev/moq#6908 (25cd95ea64) replaced the replay with ebml-iterable's incremental TagDecoder. It still buffered each whole block, but did not replay it.
  • moq-dev/moq#7234 (bdd045f361) replaced that with the importer's own EBML reader (rs/moq-mux/src/container/mkv/ebml.rs). Each byte is consumed once. A block's payload streams into its frame as it arrives, and only a split element header (12 bytes at most) or a field being read is retained.

I measured all three revisions. The slope is present at the audit commit and absent on both later ones. On main, copy volume and peak memory are the same at every chunk size, and instructions scale with the number of decode calls, not with block size. Following the survey's "accept a measured no-win" contract, this PR deletes the quest. It also rewrites the epic's stale "MKV fragmentation below concerns parser replay" sentence to record the outcome for the epic's reconciliation step.

API and wire impact: none, since the PR only changes quest files.

Measurements

Method. I built one harness (mkv_measure, below) unchanged against three source snapshots in release mode: audit = 1d43c096e044, pre = b652b3e11d (parent of #7234, the #6908 decoder), and head = 304ee3316a (main). Each run generates a WebM of about 4 MiB in memory: a single block for simple-4096, otherwise 128 blocks per Cluster. It then feeds the WebM through Import::decode in c-byte chunks and calls finish().

  • Instructions are user-space retired instructions read in-process from perf_event_open on the performance-core PMU (cpu_core, type 4). The process is pinned with taskset -c 0-7. Counting covers only the decode/finish window, so input generation and importer setup are excluded. The cold count covers the header-through-Tracks prefix (71–107 B). The steady count covers the Clusters plus finish().
  • Copy volume and peak memory come from a counting global allocator over the same window: bytes allocated (a realloc counts its new size) and peak live bytes. The retained output frames are included and are the same in every variant. Instruction counts undercount bulk copies, since rep movsb retires as a few instructions, so the allocation column is the copy measure.
  • Cases: simple-N = VP9 keyframe SimpleBlocks of N KiB; group-1024 = 1 MiB Blocks in BlockGroups; laced-64 = Opus SimpleBlocks EBML-laced with 8 × 8 KiB frames; live-64 = 64 KiB blocks in unknown-size Clusters inside an unknown-size Segment.
  • I ran every cell twice in alternating revision order. The A/A spread between rounds was ≤ 0.42% on steady instructions, and allocation bytes were identical. Host: Intel Core Ultra 7 270K Plus, Linux 7.0.0-42, rustc 1.98.1, cargo build --release --locked, load average ~26 (instruction counts do not depend on load). "—" marks an audit cell skipped because its predicted replay copy, total·B/(2c), exceeded 8 GiB.

Bytes allocated per input byte (copy volume):

case revision chunk 1 chunk 64 chunk 1500 chunk 16384 chunk 65536 chunk whole
simple-1 audit 131779.04 2102.29 101.22 20.20 14.11 13.05
simple-1 pre 5.01 5.01 5.01 5.02 5.06 6.01
simple-1 head 3.60 3.60 3.60 3.60 3.60 3.60
simple-64 audit — 8682.84 374.91 39.10 13.68 5.60
simple-64 pre 3.38 3.38 3.37 3.38 3.36 4.32
simple-64 head 1.33 1.33 1.33 1.33 1.33 1.33
simple-1024 audit — — 2890.17 271.21 73.18 5.69
simple-1024 pre 4.29 4.29 4.06 4.30 4.28 4.29
simple-1024 head 1.30 1.30 1.30 1.30 1.30 1.30
simple-4096 audit — — 7086.38 655.95 169.93 6.44
simple-4096 pre 7.29 7.29 6.36 5.30 5.28 4.29
simple-4096 head 1.29 1.29 1.29 1.29 1.29 1.29
group-1024 audit — — 2890.37 271.23 73.19 5.69
group-1024 pre 4.29 4.29 4.06 4.30 4.28 4.29
group-1024 head 1.30 1.30 1.30 1.30 1.30 1.30
laced-64 audit — 8651.44 375.06 39.24 13.83 5.76
laced-64 pre 3.50 3.50 3.49 3.50 3.49 4.44
laced-64 head 1.47 1.47 1.47 1.47 1.47 1.47
live-64 audit — 8682.64 374.91 39.10 13.68 5.60
live-64 pre 3.38 3.38 3.37 3.38 3.36 4.32
live-64 head 1.33 1.33 1.33 1.33 1.33 1.33

Steady instructions per input byte:

case revision chunk 1 chunk 64 chunk 1500 chunk 16384 chunk 65536 chunk whole
simple-1 audit 4128.41 108.31 26.21 22.19 21.96 21.76
simple-1 pre 519.18 25.91 18.48 18.01 17.98 17.97
simple-1 head 1041.36 37.13 22.04 21.42 21.37 21.36
simple-64 audit — 252.01 11.34 1.56 0.82 0.52
simple-64 pre 512.31 8.27 0.78 0.37 0.34 0.33
simple-64 head 1026.27 16.35 1.17 0.44 0.39 0.38
simple-1024 audit — — 33.56 3.29 1.01 0.21
simple-1024 pre 512.09 8.03 0.54 0.13 0.10 0.10
simple-1024 head 1109.09 17.36 0.94 0.17 0.12 0.10
simple-4096 audit — — 7.98 0.84 0.31 0.13
simple-4096 pre 522.08 8.18 0.54 0.12 0.09 0.08
simple-4096 head 1109.08 17.35 0.93 0.15 0.10 0.08
group-1024 audit — — 34.46 3.38 1.03 0.21
group-1024 pre 554.09 8.69 0.57 0.13 0.11 0.10
group-1024 head 1109.09 17.36 0.94 0.17 0.12 0.10
laced-64 audit — 250.00 11.97 2.23 1.50 1.18
laced-64 pre 512.70 8.66 1.17 0.75 0.73 0.72
laced-64 head 1026.79 17.01 1.84 1.11 1.06 1.05
live-64 audit — 253.03 11.38 1.56 0.82 0.53
live-64 pre 704.30 11.27 0.91 0.38 0.35 0.33
live-64 head 1026.27 16.34 1.17 0.44 0.39 0.38

Reading the tables:

  • audit shows the quest's slope. With 1 MiB blocks, going from 64 KiB to 16 KiB to 1500 B chunks multiplies copy volume by ~3.7× and then ~10.7×, reaching 2890 B/B, about 12 GB copied for 4 MiB of input. Instructions rise from 0.21/B (whole input) to 33.6/B.
  • pre has no slope (4–7 B/B), but it holds each block whole. Peak memory is 20 MiB for a 4 MiB block versus 4.8 MiB on head.
  • head allocates the same bytes and reaches the same peak memory at every chunk size, from 1 byte to whole. Instructions are about 0.1/B plus about 1,100 per decode call: 1109/B at c=1, 17.4/B at c=64, and 0.94/B at c=1500, identical for 64 KiB, 1 MiB and 4 MiB blocks. That cost is linear in calls and does not depend on block size.
Peak live MiB and cold catalog startup instructions
case revision chunk 1 chunk 64 chunk 1500 chunk 16384 chunk 65536 chunk whole
simple-1 audit 10.5 10.5 10.5 10.6 10.7 18.6
simple-1 pre 10.7 10.7 10.7 10.8 10.9 14.8
simple-1 head 10.9 10.9 10.9 10.9 10.9 10.9
simple-64 audit — 5.8 5.8 5.8 5.8 13.0
simple-64 pre 4.9 4.9 4.9 4.9 4.9 8.6
simple-64 head 4.9 4.9 4.9 4.9 4.9 4.9
simple-1024 audit — — 12.8 13.8 13.8 15.8
simple-1024 pre 8.4 8.4 7.9 8.4 8.4 10.4
simple-1024 head 4.8 4.8 4.8 4.8 4.8 4.8
simple-4096 audit — — 26.9 24.8 24.8 24.8
simple-4096 pre 20.0 20.0 18.2 16.1 16.0 16.0
simple-4096 head 4.8 4.8 4.8 4.8 4.8 4.8
group-1024 audit — — 12.8 13.8 13.8 15.8
group-1024 pre 8.4 8.4 7.9 8.4 8.4 10.4
group-1024 head 4.8 4.8 4.8 4.8 4.8 4.8
laced-64 audit — 5.8 5.7 5.8 5.8 13.4
laced-64 pre 4.8 4.8 4.7 4.8 4.8 8.7
laced-64 head 4.6 4.6 4.6 4.6 4.6 4.6
live-64 audit — 5.8 5.8 5.8 5.8 13.0
live-64 pre 4.9 4.9 4.9 4.9 4.9 8.6
live-64 head 4.9 4.9 4.9 4.9 4.9 4.9
case revision chunk 1 chunk 64 chunk 1500 chunk 16384 chunk 65536 chunk whole
simple-1 audit 487.6k 93.4k 82.5k 82.6k 82.5k 82.6k
simple-1 pre 74.4k 55.4k 55.1k 55.1k 55.3k 55.1k
simple-1 head 65.9k 53.1k 52.9k 52.9k 52.9k 52.9k
simple-64 audit — 93.6k 82.6k 82.6k 82.6k 82.6k
simple-64 pre 74.4k 55.4k 55.1k 55.1k 55.1k 55.1k
simple-64 head 65.9k 53.1k 52.9k 52.9k 52.9k 52.9k
simple-1024 audit — — 82.6k 82.6k 82.6k 82.6k
simple-1024 pre 74.4k 55.4k 55.1k 55.1k 55.1k 55.1k
simple-1024 head 65.9k 53.1k 52.9k 52.9k 52.9k 52.9k
simple-4096 audit — — 82.6k 82.6k 82.8k 82.6k
simple-4096 pre 74.3k 55.2k 55.0k 55.0k 55.0k 55.0k
simple-4096 head 65.8k 53.1k 52.8k 52.8k 52.8k 52.8k
group-1024 audit — — 82.6k 82.6k 82.7k 82.6k
group-1024 pre 74.4k 55.4k 55.1k 55.1k 55.1k 55.1k
group-1024 head 65.9k 53.1k 52.9k 52.9k 52.9k 52.9k
laced-64 audit — 96.0k 85.5k 85.5k 85.5k 85.5k
laced-64 pre 95.7k 58.6k 58.4k 58.4k 58.4k 58.4k
laced-64 head 77.2k 57.9k 57.7k 57.7k 57.7k 57.7k
live-64 audit — 93.6k 82.6k 82.6k 82.6k 82.6k
live-64 pre 74.4k 55.4k 55.1k 55.1k 55.1k 55.1k
live-64 head 65.9k 53.1k 52.9k 53.0k 52.9k 52.9k

First-frame latency. I did not measure this separately. The audit and pre revisions publish a block only once the whole block has arrived. Head publishes each payload byte as it arrives: a_block_streams_into_its_frame_before_it_completes (rs/moq-mux/src/container/mkv/import.rs:1601) reads the first 1 KiB of a 1 MiB block before the block completes, with at most 12 bytes retained.

Split equivalence on head (mkv_verify, below), 21,789 comparisons, all equal. For each input, the reads published per track (group, timestamp, duration, keyframe flag, payload length and hash, gaps, ends, read errors) and the decode/finish errors at each chunk size match unsplit input:

  • Each measured shape at 256 KiB (block sizes 1–256 KiB, BlockGroup, EBML lacing, unknown-size live), at chunk sizes 1, 7, 64, 1500 and 16384.
  • The real live opus.webm (1813 B) truncated at every length, and with every byte XORed by 0x01, 0x80 and 0xff, at chunk sizes 1, 7 and 64. The outcomes covered 5031 clean imports, 1866 Truncated at finish, 265 MatroskaParse, and 42 other distinct error outcomes (Opus head, lacing, timestamp rewind, negative timestamps, missing fields).

Nightly coverage already exists. The nightly Criterion run and CodSpeed's simulated instruction counts both run rs/moq-mux/benches/containers.rs. Its mkv-chunks (1/7/64/whole on opus.webm) and mkv-payloads (1 KiB/64 KiB/1 MiB payloads × 1500/64 KiB/whole chunks) series would show a reintroduced replay: at payload 1 MiB, chunk 1500, the audit revision costs ~36× head's instructions and ~2200× its allocation. I added no new bench.

Harness: mkv_measure.rs (copied into each tree's rs/moq-mux/examples/) 
//! Deterministic cost of MKV import across block and input chunk sizes.
//!
//! Usage: mkv_measure <case> <chunk|whole>
//! Prints one TSV row: case chunk bytes cold_bytes cold_instr steady_instr alloc_bytes allocs peak_live
//!
//! Instructions are user-space retained instructions on the performance-core PMU, counted only
//! around `decode`/`finish` (setup and input generation are excluded). Allocation figures come
//! from a counting global allocator over the same window.
use std::alloc::{GlobalAlloc, Layout, System};
use std::io::Read;
use std::os::fd::FromRawFd;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering::Relaxed};

use webm_iterable::WebmWriter;
use webm_iterable::matroska_spec::{BlockLacing, Frame, Master, MatroskaSpec, SimpleBlock};
use webm_iterable::WriteOptions;

struct Counting;
static ON: AtomicBool = AtomicBool::new(false);
static BYTES: AtomicU64 = AtomicU64::new(0);
static COUNT: AtomicU64 = AtomicU64::new(0);
static LIVE: AtomicU64 = AtomicU64::new(0);
static PEAK: AtomicU64 = AtomicU64::new(0);

fn grow(size: u64) {
	if ON.load(Relaxed) {
		BYTES.fetch_add(size, Relaxed);
		COUNT.fetch_add(1, Relaxed);
		let live = LIVE.fetch_add(size, Relaxed) + size;
		PEAK.fetch_max(live, Relaxed);
	}
}
fn shrink(size: u64) {
	if ON.load(Relaxed) {
		// Frees of memory allocated before the window would underflow; saturate instead.
		let _ = LIVE.fetch_update(Relaxed, Relaxed, |live| Some(live.saturating_sub(size)));
	}
}

unsafe impl GlobalAlloc for Counting {
	unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
		grow(layout.size() as u64);
		unsafe { System.alloc(layout) }
	}
	unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
		shrink(layout.size() as u64);
		unsafe { System.dealloc(ptr, layout) }
	}
	unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
		grow(layout.size() as u64);
		unsafe { System.alloc_zeroed(layout) }
	}
	unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new: usize) -> *mut u8 {
		// A realloc copies up to the old size and allocates the new one.
		shrink(layout.size() as u64);
		grow(new as u64);
		unsafe { System.realloc(ptr, layout, new) }
	}
}

#[global_allocator]
static GLOBAL: Counting = Counting;

unsafe extern "C" {
	fn syscall(number: i64, ...) -> i64;
	fn ioctl(fd: i32, request: u64, ...) -> i32;
}

/// A user-space instruction counter on the hybrid part's performance-core PMU.
struct Instructions(std::fs::File, i32);

impl Instructions {
	fn open() -> Self {
		let pmu: u64 = std::fs::read_to_string("/sys/devices/cpu_core/type")
			.map(|t| t.trim().parse().unwrap())
			.unwrap_or(0);
		let mut attr = [0u64; 16];
		// type = PERF_TYPE_HARDWARE, size = 128
		attr[0] = 128 << 32;
		// config = PERF_COUNT_HW_INSTRUCTIONS on the extended PMU type
		attr[1] = (pmu << 32) | 1;
		// disabled | exclude_kernel | exclude_hv
		attr[5] = 1 | (1 << 5) | (1 << 6);
		let fd = unsafe { syscall(298, attr.as_ptr(), 0i32, -1i32, -1i32, 0u64) };
		assert!(fd >= 0, "perf_event_open failed");
		let fd = fd as i32;
		Self(unsafe { std::fs::File::from_raw_fd(fd) }, fd)
	}
	fn start(&self) {
		unsafe {
			ioctl(self.1, 0x2403, 0); // RESET
			ioctl(self.1, 0x2400, 0); // ENABLE
		}
	}
	fn stop(&mut self) -> u64 {
		unsafe { ioctl(self.1, 0x2401, 0) }; // DISABLE
		let mut buf = [0u8; 8];
		self.0.read_exact(&mut buf).unwrap();
		u64::from_ne_bytes(buf)
	}
}

const TOTAL: usize = 4 * 1024 * 1024;

fn opus_entry() -> MatroskaSpec {
	let mut head = Vec::new();
	head.extend_from_slice(b"OpusHead");
	head.push(1);
	head.push(2);
	head.extend_from_slice(&0u16.to_le_bytes());
	head.extend_from_slice(&48_000u32.to_le_bytes());
	head.extend_from_slice(&0i16.to_le_bytes());
	head.push(0);
	MatroskaSpec::TrackEntry(Master::Full(vec![
		MatroskaSpec::TrackNumber(1),
		MatroskaSpec::TrackUID(1),
		MatroskaSpec::TrackType(2),
		MatroskaSpec::CodecID("A_OPUS".into()),
		MatroskaSpec::CodecPrivate(head),
		MatroskaSpec::DefaultDuration(20_000_000),
		MatroskaSpec::Audio(Master::Full(vec![
			MatroskaSpec::SamplingFrequency(48_000.0),
			MatroskaSpec::Channels(2),
		])),
	]))
}

fn vp9_entry() -> MatroskaSpec {
	MatroskaSpec::TrackEntry(Master::Full(vec![
		MatroskaSpec::TrackNumber(1),
		MatroskaSpec::TrackUID(1),
		MatroskaSpec::TrackType(1),
		MatroskaSpec::CodecID("V_VP9".into()),
		MatroskaSpec::Video(Master::Full(vec![
			MatroskaSpec::PixelWidth(1920),
			MatroskaSpec::PixelHeight(1080),
		])),
	]))
}

/// Build `(prefix, body)`: the header through Tracks, then every Cluster, about `TOTAL` bytes.
fn input(case: &str) -> (Vec<u8>, Vec<u8>) {
	input_sized(case, TOTAL)
}

/// Build `(prefix, body)` of about `total` bytes, or one block if larger.
pub fn input_sized(case: &str, total: usize) -> (Vec<u8>, Vec<u8>) {
	let (kind, size) = case.split_once('-').unwrap();
	let size: usize = size.parse::<usize>().unwrap() * 1024;
	let entry = if kind == "laced" { opus_entry() } else { vp9_entry() };
	let unknown = WriteOptions::is_unknown_sized_element;

	let blocks = total.max(size) / size;
	// Laced blocks carry 8 frames of 20 ms; the others one frame of 1 ms.
	let step: i16 = if kind == "laced" { 160 } else { 1 };
	let per_cluster = 128;
	let block = |index: usize| -> MatroskaSpec {
		let rel = (index % per_cluster) as i16 * step;
		match kind {
			"laced" => {
				let frames = vec![0x5au8; size / 8];
				let mut block = SimpleBlock::new_uncheked(&[], 1, rel, false, Some(BlockLacing::Ebml), false, true);
				block.set_frame_data(&(0..8).map(|_| Frame { data: &frames[..] }).collect());
				block.into()
			}
			"group" => {
				let payload = vec![0x5au8; size];
				let MatroskaSpec::SimpleBlock(raw) =
					SimpleBlock::new_uncheked(&payload, 1, rel, false, None, false, false).into()
				else {
					unreachable!()
				};
				MatroskaSpec::BlockGroup(Master::Full(vec![MatroskaSpec::Block(raw)]))
			}
			_ => {
				let payload = vec![0x5au8; size];
				SimpleBlock::new_uncheked(&payload, 1, rel, false, None, false, true).into()
			}
		}
	};

	let mut data = Vec::new();
	{
		let mut writer = WebmWriter::new(&mut data);
		writer
			.write(&MatroskaSpec::Ebml(Master::Full(vec![MatroskaSpec::DocType("webm".into())])))
			.unwrap();
		writer.write_advanced(&MatroskaSpec::Segment(Master::Start), unknown()).unwrap();
		writer
			.write(&MatroskaSpec::Info(Master::Full(vec![MatroskaSpec::TimestampScale(1_000_000)])))
			.unwrap();
		writer.write(&MatroskaSpec::Tracks(Master::Full(vec![entry]))).unwrap();
		let clusters = blocks.div_ceil(per_cluster);
		for cluster in 0..clusters {
			let range = cluster * per_cluster..((cluster + 1) * per_cluster).min(blocks);
			let ts = cluster as u64 * per_cluster as u64 * step as u64;
			if kind == "live" {
				// An unknown-size Cluster, as a live WebM muxer writes it.
				writer.write_advanced(&MatroskaSpec::Cluster(Master::Start), unknown()).unwrap();
				writer.write(&MatroskaSpec::Timestamp(ts)).unwrap();
				for index in range {
					writer.write(&block(index)).unwrap();
				}
				writer.write(&MatroskaSpec::Cluster(Master::End)).unwrap();
			} else {
				let mut children = vec![MatroskaSpec::Timestamp(ts)];
				children.extend(range.map(block));
				writer.write(&MatroskaSpec::Cluster(Master::Full(children))).unwrap();
			}
		}
		writer.write(&MatroskaSpec::Segment(Master::End)).unwrap();
		writer.flush().unwrap();
	}
	let split = data
		.windows(4)
		.position(|id| id == [0x1f, 0x43, 0xb6, 0x75])
		.expect("a Cluster");
	let body = data.split_off(split);
	(data, body)
}

fn main() {
	let args: Vec<String> = std::env::args().collect();
	let case = &args[1];
	let (prefix, body) = input(case);
	let total = prefix.len() + body.len();
	let chunk = match args[2].as_str() {
		"whole" => total,
		value => value.parse().unwrap(),
	};

	let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap();
	let _enter = runtime.enter();
	let mut broadcast = moq_net::broadcast::Info::new().produce();
	let _consumer = broadcast.consume();
	let catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap();
	let mut import = moq_mux::container::mkv::Import::new(broadcast, catalog.reserve());
	let mut counter = Instructions::open();

	ON.store(true, Relaxed);
	counter.start();
	for piece in std::hint::black_box(&prefix[..]).chunks(chunk) {
		import.decode(piece).unwrap();
	}
	let cold = counter.stop();
	counter.start();
	for piece in std::hint::black_box(&body[..]).chunks(chunk) {
		import.decode(piece).unwrap();
	}
	import.finish().unwrap();
	let steady = counter.stop();
	ON.store(false, Relaxed);

	assert!(!catalog.snapshot().video.renditions.is_empty() || !catalog.snapshot().audio.renditions.is_empty());
	println!(
		"{case}\t{}\t{total}\t{}\t{cold}\t{steady}\t{}\t{}\t{}",
		args[2],
		prefix.len(),
		BYTES.load(Relaxed),
		COUNT.load(Relaxed),
		PEAK.load(Relaxed),
	);
	drop(import);
	drop(catalog);
}
Harness: mkv_verify.rs (head only, next to mkv_measure.rs) 
//! Split equivalence of MKV import: every chunking yields the reads and errors of unsplit input.
//!
//! Usage: mkv_verify
//! Valid inputs (the mkv_measure cases at reduced size, plus the real live opus.webm) and
//! malformed ones (opus.webm truncated at every length and with every byte corrupted) are each
//! imported unsplit and at several chunk sizes; the published reads (group, timestamp, duration,
//! keyframe, payload hash, gaps, ends, read errors) and the decode/finish errors must match.
use std::collections::BTreeMap;
use std::hash::{DefaultHasher, Hash, Hasher};
use std::time::Duration;

#[path = "mkv_measure.rs"]
#[allow(dead_code)]
mod measure;

#[derive(Debug, PartialEq)]
struct Outcome {
	decode: Option<String>,
	finish: Option<String>,
	reads: BTreeMap<String, Vec<String>>,
}

async fn import(data: &[u8], chunk: usize) -> Outcome {
	let mut broadcast = moq_net::broadcast::Info::new().produce();
	let consumer = broadcast.consume();
	let catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap();
	let mut import = moq_mux::container::mkv::Import::new(broadcast, catalog.reserve());
	let mut decode = None;
	for piece in data.chunks(chunk.max(1)) {
		if let Err(err) = import.decode(piece) {
			decode = Some(format!("{err:?}"));
			break;
		}
	}
	let finish = import.finish().err().map(|err| format!("{err:?}"));
	let snapshot = catalog.snapshot();
	let mut tracks = Vec::new();
	for (name, config) in &snapshot.video.renditions {
		tracks.push((name.clone(), moq_mux::catalog::hang::Container::try_from(config).unwrap()));
	}
	for (name, config) in &snapshot.audio.renditions {
		tracks.push((name.clone(), moq_mux::catalog::hang::Container::try_from(config).unwrap()));
	}
	let mut reads = BTreeMap::new();
	for (name, container) in tracks {
		let replay = moq_net::track::Subscription::default().with_max_delay(Duration::from_secs(3600));
		let track = consumer.track(&name).unwrap().subscribe(replay).await.unwrap();
		let mut reader = moq_mux::container::Consumer::new(track, container);
		let mut items = Vec::new();
		loop {
			let Ok(read) = tokio::time::timeout(Duration::from_millis(20), reader.read()).await else {
				items.push("still open".into());
				break;
			};
			match read {
				Ok(Some(moq_mux::container::Read::Frame { group, frame })) => {
					let mut hasher = DefaultHasher::new();
					frame.payload.hash(&mut hasher);
					items.push(format!(
						"frame g{group} {:?} {:?} key={} len={} {:x}",
						frame.timestamp,
						frame.duration,
						frame.keyframe,
						frame.payload.len(),
						hasher.finish()
					));
				}
				Ok(Some(other)) => items.push(format!("{other:?}")),
				Ok(None) => break,
				Err(err) => {
					items.push(format!("error {err:?}"));
					break;
				}
			}
		}
		reads.insert(name, items);
	}
	drop(import);
	Outcome { decode, finish, reads }
}

fn main() {
	let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap();
	runtime.block_on(async {
		let mut checked = 0;
		// Valid inputs: the measured shapes at 1/16 size (block sizes up to 256 KiB).
		for case in ["simple-1", "simple-64", "simple-256", "group-64", "laced-16", "live-64"] {
			let (prefix, body) = measure::input_sized(case, 256 * 1024);
			let data = [prefix, body].concat();
			let whole = import(&data, data.len()).await;
			assert!(whole.decode.is_none() && whole.finish.is_none(), "{case}: {whole:?}");
			let frames: usize = whole.reads.values().map(|items| items.iter().filter(|i| i.starts_with("frame")).count()).sum();
			assert!(frames > 0);
			for chunk in [1, 7, 64, 1500, 16384] {
				let split = import(&data, chunk).await;
				assert_eq!(split, whole, "{case} chunk {chunk}");
				checked += 1;
			}
			println!("{case}: {} bytes, {frames} frames, chunks 1/7/64/1500/16384 match unsplit", data.len());
		}

		// Malformed inputs: the real live WebM truncated at every length and with each byte flipped.
		let webm = include_bytes!("../src/container/mkv/test_data/opus.webm");
		let mut errors = BTreeMap::<String, usize>::new();
		for len in 0..=webm.len() {
			let data = &webm[..len];
			let whole = import(data, data.len()).await;
			if whole.reads.values().any(|items| items.last().is_some_and(|i| i == "still open")) {
				println!("truncated at {len}: a track stays open: {whole:?}");
			}
			*errors.entry(format!("{:?}/{:?}", whole.decode, whole.finish)).or_default() += 1;
			for chunk in [1, 7, 64] {
				assert_eq!(import(data, chunk).await, whole, "truncated at {len}, chunk {chunk}");
				checked += 1;
			}
		}
		for at in 0..webm.len() {
			for mask in [0x01u8, 0x80, 0xff] {
				let mut data = webm.to_vec();
				data[at] ^= mask;
				let whole = import(&data, data.len()).await;
				if whole.reads.values().any(|items| items.last().is_some_and(|i| i == "still open")) {
					println!("byte {at} ^ {mask:#x}: a track stays open: {whole:?}");
				}
				*errors.entry(format!("{:?}/{:?}", whole.decode, whole.finish)).or_default() += 1;
				for chunk in [1, 7, 64] {
					assert_eq!(import(&data, chunk).await, whole, "byte {at} ^ {mask:#x}, chunk {chunk}");
					checked += 1;
				}
			}
		}
		println!("opus.webm ({} bytes): every truncation and 3 flips per byte match unsplit at chunks 1/7/64", webm.len());
		for (outcome, count) in &errors {
			println!("  {count:5} x {outcome}");
		}
		println!("{checked} split comparisons, all equal");
	});
}
Driver: sweep.py

Run as quest gate python3 -I sweep.py sweep.tsv audit pre head --rounds 2, followed by a pre-only pass over the small-chunk cells the first pass had skipped.

"""Run the MKV import cost sweep across revisions; append TSV rows to the output file.

Usage: python3 -I sweep.py OUT REV... [--cases a,b] [--chunks 1,64] [--rounds N]
A baseline run whose predicted replay copy (total * block / (2 * chunk)) exceeds 8 GiB is skipped.
"""

import subprocess
import sys

ROOT = "/home/kixelated/.cache/mkv-fragments-7515"
CASES = ["simple-1", "simple-64", "simple-1024", "simple-4096", "group-1024", "laced-64", "live-64"]
CHUNKS = ["1", "64", "1500", "16384", "65536", "whole"]
TOTAL = 4 * 1024 * 1024
LIMIT = 8 * 1024**3


def main():
    args = sys.argv[1:]
    out = args.pop(0)
    cases, chunks, rounds, revs = CASES, CHUNKS, 1, []
    while args:
        arg = args.pop(0)
        if arg == "--cases":
            cases = args.pop(0).split(",")
        elif arg == "--chunks":
            chunks = args.pop(0).split(",")
        elif arg == "--rounds":
            rounds = int(args.pop(0))
        else:
            revs.append(arg)
    with open(out, "a") as f:
        for round in range(rounds):
            for case in cases:
                block = int(case.split("-")[1]) * 1024
                for chunk in chunks:
                    # Alternate revisions within each cell so load drift affects them alike.
                    order = revs if round % 2 == 0 else list(reversed(revs))
                    for rev in order:
                        if rev == "audit" and chunk != "whole":
                            predicted = max(TOTAL, block) * block / (2 * int(chunk))
                            if predicted > LIMIT:
                                row = f"{rev}\t{round}\t{case}\t{chunk}\tskipped (predicted replay copy {predicted / 1024**3:.0f} GiB)"
                                print(row, flush=True)
                                f.write(row + "\n")
                                continue
                        binary = f"{ROOT}/target-{rev}/release/examples/mkv_measure"
                        try:
                            result = subprocess.run(
                                ["taskset", "-c", "0-7", binary, case, chunk],
                                capture_output=True,
                                text=True,
                                timeout=900,
                            )
                            line = result.stdout.strip() or f"{case}\t{chunk}\tfailed: {result.stderr.strip()[-300:]}"
                        except subprocess.TimeoutExpired:
                            line = f"{case}\t{chunk}\ttimeout 900s"
                        row = f"{rev}\t{round}\t{line}"
                        print(row, flush=True)
                        f.write(row + "\n")
                        f.flush()


main()
Raw samples (revision round case chunk bytes cold_bytes cold_instr steady_instr alloc_bytes allocs peak_live) 
audit	0	simple-1	1	4223397	71	487558	17435640980	556555190244	21205543	11050623
pre	0	simple-1	1	4223397	71	74397	2192664010	21165287	72011	11255783
head	0	simple-1	1	4223397	71	65883	4397990509	15223947	84324	11423104
audit	0	simple-1	64	4223397	71	93432	457410599	8878826030	495215	11053761
pre	0	simple-1	64	4223397	71	55367	109417239	21165231	72008	11255783
head	0	simple-1	64	4223397	71	53134	156792359	15223947	84324	11423104
audit	0	simple-1	1500	4223397	71	82533	110712286	427512739	117899	11057009
pre	0	simple-1	1500	4223397	71	55101	78056387	21165983	72005	11256877
head	0	simple-1	1500	4223397	71	52891	93093348	15223947	84324	11423104
audit	0	simple-1	16384	4223397	71	82592	93706204	85310726	102589	11098275
pre	0	simple-1	16384	4223397	71	55102	76062926	21210635	72005	11286645
head	0	simple-1	16384	4223397	71	52894	90458283	15223947	84324	11423104
audit	0	simple-1	65536	4223397	71	82536	92738523	59595746	101433	11246334
pre	0	simple-1	65536	4223397	71	55281	75951511	21358091	72005	11384949
head	0	simple-1	65536	4223397	71	52890	90257661	15223947	84324	11423104
audit	0	simple-1	whole	4223397	71	82606	91880866	55105537	101019	19494523
pre	0	simple-1	whole	4223397	71	55108	75902024	25384667	72004	15477132
head	0	simple-1	whole	4223397	71	52899	90192251	15223947	84324	11423104
audit	0	simple-64	1	skipped (predicted replay copy 128 GiB)
pre	0	simple-64	1	skipped (predicted replay copy 128 GiB)
head	0	simple-64	1	4194898	71	65884	4305035362	5567431	1574	5104079
audit	0	simple-64	64	4194898	71	93564	1057147972	36423618930	590833	6110949
pre	0	simple-64	64	4194898	71	55418	34696934	14180747	1391	5101782
head	0	simple-64	64	4194898	71	53142	68564543	5567431	1574	5104079
audit	0	simple-64	1500	4194898	71	82604	47590171	1572711783	27068	6049893
pre	0	simple-64	1500	4194898	71	55102	3270667	14118255	1387	5101782
head	0	simple-64	1500	4194898	71	52897	4893941	5567431	1574	5104079
audit	0	simple-64	16384	4194898	71	82628	6540102	164016142	4244	6112085
pre	0	simple-64	16384	4194898	71	55114	1533586	14165563	1384	5101782
head	0	simple-64	16384	4194898	71	52893	1853217	5567431	1574	5104079
audit	0	simple-64	65536	4194898	71	82591	3452056	57378241	2517	6111233
pre	0	simple-64	65536	4194898	71	55102	1432200	14115559	1382	5101782
head	0	simple-64	65536	4194898	71	52893	1653968	5567431	1574	5104079
audit	0	simple-64	whole	4194898	71	82586	2198614	23487723	1940	13665644
pre	0	simple-64	whole	4194898	71	55097	1393346	18113636	1381	9038866
head	0	simple-64	whole	4194898	71	52897	1587468	5567431	1574	5104079
audit	0	simple-1024	1	skipped (predicted replay copy 2048 GiB)
pre	0	simple-1024	1	skipped (predicted replay copy 2048 GiB)
head	0	simple-1024	1	4194418	71	65879	4651907151	5433008	359	5008984
audit	0	simple-1024	64	skipped (predicted replay copy 32 GiB)
pre	0	simple-1024	64	skipped (predicted replay copy 32 GiB)
head	0	simple-1024	64	4194418	71	53141	72833832	5433008	359	5008984
audit	0	simple-1024	1500	4194418	71	82616	140748887	12122572190	22144	13461793
pre	0	simple-1024	1500	4194418	71	55104	2275628	17034902	336	8325560
head	0	simple-1024	1500	4194418	71	52898	3956537	5433008	359	5008984
audit	0	simple-1024	16384	4194418	71	82607	13800753	1137575849	2456	14456865
pre	0	simple-1024	16384	4194418	71	55106	539182	18015090	333	8823096
head	0	simple-1024	16384	4194418	71	52898	703839	5433008	359	5008984
audit	0	simple-1024	65536	4194418	71	82605	4226460	306956057	966	14443233
pre	0	simple-1024	65536	4194418	71	55109	437256	17952306	331	8816280
head	0	simple-1024	65536	4194418	71	52891	487786	5433008	359	5008984
audit	0	simple-1024	whole	4194418	71	82617	881602	23873442	464	16536238
pre	0	simple-1024	whole	4194418	71	55113	400073	18013483	326	10911274
head	0	simple-1024	whole	4194418	71	52891	412704	5433008	359	5008984
audit	0	simple-4096	1	skipped (predicted replay copy 8192 GiB)
pre	0	simple-4096	1	skipped (predicted replay copy 8192 GiB)
head	0	simple-4096	1	4194395	71	65756	4651847116	5427002	293	5005002
audit	0	simple-4096	64	skipped (predicted replay copy 128 GiB)
pre	0	simple-4096	64	skipped (predicted replay copy 128 GiB)
head	0	simple-4096	64	4194395	71	53063	72778391	5427002	293	5005002
audit	0	simple-4096	1500	4194395	71	82604	33450058	29723056102	19975	28210559
pre	0	simple-4096	1500	4194395	71	54970	2257332	26683138	286	19053976
head	0	simple-4096	1500	4194395	71	52768	3898897	5427002	293	5005002
audit	0	simple-4096	16384	4194395	71	82620	3516411	2751313370	2191	25988223
pre	0	simple-4096	16384	4194395	71	55019	495668	22223582	282	16831640
head	0	simple-4096	16384	4194395	71	52820	648113	5427002	293	5005002
audit	0	simple-4096	65536	4194395	71	82779	1281551	712733018	845	25974591
pre	0	simple-4096	65536	4194395	71	54963	390926	22147166	280	16818008
head	0	simple-4096	65536	4194395	71	52768	431969	5427002	293	5005002
audit	0	simple-4096	whole	4194395	71	82624	528246	26996860	391	25970402
pre	0	simple-4096	whole	4194395	71	54970	351989	18009472	274	16813555
head	0	simple-4096	whole	4194395	71	52768	356431	5427002	293	5005002
audit	0	group-1024	1	skipped (predicted replay copy 2048 GiB)
pre	0	group-1024	1	skipped (predicted replay copy 2048 GiB)
head	0	group-1024	1	4194434	71	65878	4651912752	5433008	359	5008984
audit	0	group-1024	64	skipped (predicted replay copy 32 GiB)
pre	0	group-1024	64	skipped (predicted replay copy 32 GiB)
head	0	group-1024	64	4194434	71	53142	72834771	5433008	359	5008984
audit	0	group-1024	1500	4194434	71	82606	144531031	12123474454	27752	13462440
pre	0	group-1024	1500	4194434	71	55103	2397102	17035414	340	8325688
head	0	group-1024	1500	4194434	71	52893	3956194	5433008	359	5008984
audit	0	group-1024	16384	4194434	71	82614	14163379	1137660233	2984	14472624
pre	0	group-1024	16384	4194434	71	55110	554914	18015602	337	8823224
head	0	group-1024	16384	4194434	71	52899	706635	5433008	359	5008984
audit	0	group-1024	65536	4194434	71	82674	4323110	306978617	1110	14508144
pre	0	group-1024	65536	4194434	71	55106	445296	17952818	335	8816408
head	0	group-1024	65536	4194434	71	52902	489820	5433008	359	5008984
audit	0	group-1024	whole	4194434	71	82610	893809	23878146	482	16537262
pre	0	group-1024	whole	4194434	71	55109	406314	18014011	330	10911418
head	0	group-1024	whole	4194434	71	52893	415536	5433008	359	5008984
audit	0	laced-64	1	skipped (predicted replay copy 128 GiB)
pre	0	laced-64	1	skipped (predicted replay copy 128 GiB)
head	0	laced-64	1	4195510	107	77167	4307796667	6188310	4574	4820150
audit	0	laced-64	64	4195510	107	95961	1048852362	36297210019	594040	6049887
pre	0	laced-64	64	4195510	107	58607	36322594	14686907	3426	5008949
head	0	laced-64	64	4195510	107	57924	71364525	6188273	4572	4820150
audit	0	laced-64	1500	4195510	107	85521	50230145	1573557936	31224	5993439
pre	0	laced-64	1500	4195510	107	58355	4893154	14629023	3422	4980725
head	0	laced-64	1500	4195510	107	57666	7700489	6188273	4572	4820150
audit	0	laced-64	16384	4195510	107	85492	9363802	164631520	8400	6051599
pre	0	laced-64	16384	4195510	107	58355	3157136	14672299	3419	5009805
head	0	laced-64	16384	4195510	107	57661	4675232	6188273	4572	4820150
audit	0	laced-64	65536	4195510	107	85508	6275146	58040329	6673	6050315
pre	0	laced-64	65536	4195510	107	58366	3054954	14621863	3417	5009163
head	0	laced-64	65536	4195510	107	57668	4465358	6188273	4572	4820150
audit	0	laced-64	whole	4195510	107	85501	4958870	24166029	6096	14047806
pre	0	laced-64	whole	4195510	107	58356	3016571	18620444	3416	9073387
head	0	laced-64	whole	4195510	107	57665	4395490	6188273	4572	4820150
audit	0	live-64	1	skipped (predicted replay copy 128 GiB)
pre	0	live-64	1	skipped (predicted replay copy 128 GiB)
head	0	live-64	1	4194902	71	65886	4305034696	5567431	1574	5104079
audit	0	live-64	64	4194902	71	93576	1061407276	36422832438	590825	6110949
pre	0	live-64	64	4194902	71	55376	47279383	14180747	1391	5101782
head	0	live-64	64	4194902	71	53139	68562818	5567431	1574	5104079
audit	0	live-64	1500	4194902	71	82586	47741796	1572700603	27068	6049893
pre	0	live-64	1500	4194902	71	55109	3809872	14118255	1387	5101782
head	0	live-64	1500	4194902	71	52897	4889286	5567431	1574	5104079
audit	0	live-64	16384	4194902	71	82624	6540884	164015122	4244	6112085
pre	0	live-64	16384	4194902	71	55119	1586903	14165563	1384	5101782
head	0	live-64	16384	4194902	71	52958	1851117	5567431	1574	5104079
audit	0	live-64	65536	4194902	71	82590	3454491	57377989	2517	6111233
pre	0	live-64	65536	4194902	71	55106	1447303	14115559	1382	5101782
head	0	live-64	65536	4194902	71	52898	1650965	5567431	1574	5104079
audit	0	live-64	whole	4194902	71	82564	2207954	23487731	1940	13665652
pre	0	live-64	whole	4194902	71	55108	1396644	18113640	1381	9038870
head	0	live-64	whole	4194902	71	52898	1584319	5567431	1574	5104079
head	1	simple-1	1	4223397	71	65880	4397991884	15223947	84324	11423104
pre	1	simple-1	1	4223397	71	74387	2192661147	21165287	72011	11255783
audit	1	simple-1	1	4223397	71	487716	17435654478	556555190244	21205543	11050623
head	1	simple-1	64	4223397	71	53138	156792666	15223947	84324	11423104
pre	1	simple-1	64	4223397	71	55360	109415491	21165231	72008	11255783
audit	1	simple-1	64	4223397	71	93428	457407657	8878826030	495215	11053761
head	1	simple-1	1500	4223397	71	52889	93093265	15223947	84324	11423104
pre	1	simple-1	1500	4223397	71	55104	78055842	21165983	72005	11256877
audit	1	simple-1	1500	4223397	71	82531	110714810	427512739	117899	11057009
head	1	simple-1	16384	4223397	71	52890	90456803	15223947	84324	11423104
pre	1	simple-1	16384	4223397	71	55102	76061459	21210635	72005	11286645
audit	1	simple-1	16384	4223397	71	82539	93701923	85310726	102589	11098275
head	1	simple-1	65536	4223397	71	52892	90259053	15223947	84324	11423104
pre	1	simple-1	65536	4223397	71	55099	75949817	21358091	72005	11384949
audit	1	simple-1	65536	4223397	71	82550	92734452	59595746	101433	11246334
head	1	simple-1	whole	4223397	71	52889	90190553	15223947	84324	11423104
pre	1	simple-1	whole	4223397	71	55122	75900471	25384667	72004	15477132
audit	1	simple-1	whole	4223397	71	82538	91881828	55105537	101019	19494523
head	1	simple-64	1	4194898	71	65883	4305034477	5567431	1574	5104079
pre	1	simple-64	1	skipped (predicted replay copy 128 GiB)
audit	1	simple-64	1	skipped (predicted replay copy 128 GiB)
head	1	simple-64	64	4194898	71	53135	68564162	5567431	1574	5104079
pre	1	simple-64	64	4194898	71	55356	34696841	14180747	1391	5101782
audit	1	simple-64	64	4194898	71	93536	1057146931	36423618930	590833	6110949
head	1	simple-64	1500	4194898	71	52899	4895252	5567431	1574	5104079
pre	1	simple-64	1500	4194898	71	55100	3269664	14118255	1387	5101782
audit	1	simple-64	1500	4194898	71	82653	47590504	1572711783	27068	6049893
head	1	simple-64	16384	4194898	71	52891	1853353	5567431	1574	5104079
pre	1	simple-64	16384	4194898	71	55100	1533319	14165563	1384	5101782
audit	1	simple-64	16384	4194898	71	82601	6542138	164016142	4244	6112085
head	1	simple-64	65536	4194898	71	52891	1653456	5567431	1574	5104079
pre	1	simple-64	65536	4194898	71	55108	1432016	14115559	1382	5101782
audit	1	simple-64	65536	4194898	71	82587	3453345	57378241	2517	6111233
head	1	simple-64	whole	4194898	71	52893	1586576	5567431	1574	5104079
pre	1	simple-64	whole	4194898	71	55110	1393421	18113636	1381	9038866
audit	1	simple-64	whole	4194898	71	82635	2197181	23487723	1940	13665644
head	1	simple-1024	1	4194418	71	65883	4651908149	5433008	359	5008984
pre	1	simple-1024	1	skipped (predicted replay copy 2048 GiB)
audit	1	simple-1024	1	skipped (predicted replay copy 2048 GiB)
head	1	simple-1024	64	4194418	71	53135	72834877	5433008	359	5008984
pre	1	simple-1024	64	skipped (predicted replay copy 32 GiB)
audit	1	simple-1024	64	skipped (predicted replay copy 32 GiB)
head	1	simple-1024	1500	4194418	71	52901	3954570	5433008	359	5008984
pre	1	simple-1024	1500	4194418	71	55111	2275314	17034902	336	8325560
audit	1	simple-1024	1500	4194418	71	82632	140748244	12122572190	22144	13461793
head	1	simple-1024	16384	4194418	71	52895	703781	5433008	359	5008984
pre	1	simple-1024	16384	4194418	71	55104	538207	18015090	333	8823096
audit	1	simple-1024	16384	4194418	71	82668	13800819	1137575849	2456	14456865
head	1	simple-1024	65536	4194418	71	52894	487858	5433008	359	5008984
pre	1	simple-1024	65536	4194418	71	55105	436698	17952306	331	8816280
audit	1	simple-1024	65536	4194418	71	82660	4226290	306956057	966	14443233
head	1	simple-1024	whole	4194418	71	52894	412586	5433008	359	5008984
pre	1	simple-1024	whole	4194418	71	55104	401162	18013483	326	10911274
audit	1	simple-1024	whole	4194418	71	82643	880131	23873442	464	16536238
head	1	simple-4096	1	4194395	71	65752	4651846179	5427002	293	5005002
pre	1	simple-4096	1	skipped (predicted replay copy 8192 GiB)
audit	1	simple-4096	1	skipped (predicted replay copy 8192 GiB)
head	1	simple-4096	64	4194395	71	53003	72777135	5427002	293	5005002
pre	1	simple-4096	64	skipped (predicted replay copy 128 GiB)
audit	1	simple-4096	64	skipped (predicted replay copy 128 GiB)
head	1	simple-4096	1500	4194395	71	52767	3900704	5427002	293	5005002
pre	1	simple-4096	1500	4194395	71	54968	2256149	26683138	286	19053976
audit	1	simple-4096	1500	4194395	71	82607	33452010	29723056102	19975	28210559
head	1	simple-4096	16384	4194395	71	52773	648094	5427002	293	5005002
pre	1	simple-4096	16384	4194395	71	54964	493601	22223582	282	16831640
audit	1	simple-4096	16384	4194395	71	82655	3514956	2751313370	2191	25988223
head	1	simple-4096	65536	4194395	71	52770	431288	5427002	293	5005002
pre	1	simple-4096	65536	4194395	71	54985	389817	22147166	280	16818008
audit	1	simple-4096	65536	4194395	71	82615	1281772	712733018	845	25974591
head	1	simple-4096	whole	4194395	71	52767	356386	5427002	293	5005002
pre	1	simple-4096	whole	4194395	71	54981	353056	18009472	274	16813555
audit	1	simple-4096	whole	4194395	71	82620	527495	26996860	391	25970402
head	1	group-1024	1	4194434	71	65888	4651912286	5433008	359	5008984
pre	1	group-1024	1	skipped (predicted replay copy 2048 GiB)
audit	1	group-1024	1	skipped (predicted replay copy 2048 GiB)
head	1	group-1024	64	4194434	71	53138	72835548	5433008	359	5008984
pre	1	group-1024	64	skipped (predicted replay copy 32 GiB)
audit	1	group-1024	64	skipped (predicted replay copy 32 GiB)
head	1	group-1024	1500	4194434	71	52891	3957558	5433008	359	5008984
pre	1	group-1024	1500	4194434	71	55112	2398667	17035414	340	8325688
audit	1	group-1024	1500	4194434	71	82616	144532138	12123474454	27752	13462440
head	1	group-1024	16384	4194434	71	52892	705616	5433008	359	5008984
pre	1	group-1024	16384	4194434	71	55106	555897	18015602	337	8823224
audit	1	group-1024	16384	4194434	71	82617	14162292	1137660233	2984	14472624
head	1	group-1024	65536	4194434	71	52898	490362	5433008	359	5008984
pre	1	group-1024	65536	4194434	71	55109	445572	17952818	335	8816408
audit	1	group-1024	65536	4194434	71	82605	4325023	306978617	1110	14508144
head	1	group-1024	whole	4194434	71	52899	416156	5433008	359	5008984
pre	1	group-1024	whole	4194434	71	55106	405470	18014011	330	10911418
audit	1	group-1024	whole	4194434	71	82610	894239	23878146	482	16537262
head	1	laced-64	1	4195510	107	77070	4307796649	6188310	4574	4820150
pre	1	laced-64	1	skipped (predicted replay copy 128 GiB)
audit	1	laced-64	1	skipped (predicted replay copy 128 GiB)
head	1	laced-64	64	4195510	107	57990	71364574	6188273	4572	4820150
pre	1	laced-64	64	4195510	107	58608	36322664	14686907	3426	5008949
audit	1	laced-64	64	4195510	107	95943	1048852023	36297210019	594040	6049887
head	1	laced-64	1500	4195510	107	57662	7700222	6188273	4572	4820150
pre	1	laced-64	1500	4195510	107	58370	4892273	14629023	3422	4980725
audit	1	laced-64	1500	4195510	107	85489	50229553	1573557936	31224	5993439
head	1	laced-64	16384	4195510	107	57663	4674754	6188273	4572	4820150
pre	1	laced-64	16384	4195510	107	58358	3156907	14672299	3419	5009805
audit	1	laced-64	16384	4195510	107	85492	9362256	164631520	8400	6051599
head	1	laced-64	65536	4195510	107	57672	4465689	6188273	4572	4820150
pre	1	laced-64	65536	4195510	107	58349	3053966	14621863	3417	5009163
audit	1	laced-64	65536	4195510	107	85495	6275064	58040329	6673	6050315
head	1	laced-64	whole	4195510	107	57672	4395371	6188273	4572	4820150
pre	1	laced-64	whole	4195510	107	58368	3016659	18620444	3416	9073387
audit	1	laced-64	whole	4195510	107	85485	4957425	24166029	6096	14047806
head	1	live-64	1	4194902	71	65892	4305035619	5567431	1574	5104079
pre	1	live-64	1	skipped (predicted replay copy 128 GiB)
audit	1	live-64	1	skipped (predicted replay copy 128 GiB)
head	1	live-64	64	4194902	71	53199	68563012	5567431	1574	5104079
pre	1	live-64	64	4194902	71	55447	47280052	14180747	1391	5101782
audit	1	live-64	64	4194902	71	93514	1061405746	36422832438	590825	6110949
head	1	live-64	1500	4194902	71	52903	4889540	5567431	1574	5104079
pre	1	live-64	1500	4194902	71	55113	3810129	14118255	1387	5101782
audit	1	live-64	1500	4194902	71	82599	47742765	1572700603	27068	6049893
head	1	live-64	16384	4194902	71	52900	1851258	5567431	1574	5104079
pre	1	live-64	16384	4194902	71	55142	1584680	14165563	1384	5101782
audit	1	live-64	16384	4194902	71	82652	6542692	164015122	4244	6112085
head	1	live-64	65536	4194902	71	52904	1651109	5567431	1574	5104079
pre	1	live-64	65536	4194902	71	55163	1447194	14115559	1382	5101782
audit	1	live-64	65536	4194902	71	82582	3454184	57377989	2517	6111233
head	1	live-64	whole	4194902	71	52902	1584353	5567431	1574	5104079
pre	1	live-64	whole	4194902	71	55116	1396300	18113640	1381	9038870
audit	1	live-64	whole	4194902	71	82569	2208665	23487731	1940	13665652
pre	0	simple-64	1	4194898	71	74383	2149064477	14180803	1394	5101782
pre	0	simple-64	64	4194898	71	55377	34696890	14180747	1391	5101782
pre	0	simple-1024	1	4194418	71	74387	2147903387	18013290	343	8814008
pre	0	simple-1024	64	4194418	71	55423	33698389	18013234	340	8814008
pre	0	simple-4096	1	4194395	71	74251	2189791134	30592214	293	21007768
pre	0	simple-4096	64	4194395	71	55232	34305909	30592158	290	21007768
pre	0	group-1024	1	4194434	71	74382	2324070995	18013802	347	8814136
pre	0	group-1024	64	4194434	71	55370	36455127	18013746	344	8814136
pre	0	laced-64	1	4195510	107	95724	2150982379	14686963	3429	5008949
pre	0	laced-64	64	4195510	107	58617	36322462	14686907	3426	5008949
pre	0	live-64	1	4194902	71	74377	2954422658	14180803	1394	5101782
pre	0	live-64	64	4194902	71	55375	47281055	14180747	1391	5101782

Decisions

  1. ✅ 2026-11-01, maintainer: delete the quest, as this PR does. The Goal ("costs remain reasonable as the same input is split into smaller reads") holds on main: copy volume and peak memory are the same at every chunk size, and instructions grow only with the number of reads. The alternative was a narrowed quest on head's fixed ~1,100 instructions per decode call. It was not chosen: that cost is the "byte-at-a-time is ~1.9× slower" trade moq-dev/moq#7234 accepted, and it stays under 1 instruction per byte at a 1500-byte read.

Choices made without asking

  • I measured three revisions (audit, pre, head) instead of only the audit baseline and main. This shows that moq-dev/moq#6908 had already removed the slope and that moq-dev/moq#7234 additionally removed whole-block buffering.
  • The measurement harness stays in this PR body and is not committed. It duplicates the existing mkv-payloads nightly series, and the survey only asks for retained evidence.

Follow-ups (suggested, not planned)

  • When finish() returns Mkv(Truncated), the importer leaves its tracks neither finished nor aborted until it is dropped. A subscriber's read() then waits. In the verification this happened for 1462 truncation lengths and 29 byte flips of opus.webm, with the same outcome at every chunk size. A refused decode does end the tracks in finish() (rs/moq-mux/src/container/mkv/import.rs:838-852). Callers that drop the importer after an error are unaffected. Consider aborting the tracks with the truncation error in finish().
  • On 1 KiB keyframe blocks fed whole, head costs ~19% more instructions than pre (21.4 vs 18.0 per byte, ~22k per block, each opening a group). This is small in absolute terms. The recurring performance audit could note it under importer coverage.

Checks

  • 2026-11-02: merged origin/main at 8c2f04c972 into 13aa7ca699. GitHub reported a conflict in the survey README's ## Required link list, which quest's merge driver resolved locally. The PR's added and removed lines are unchanged from the reviewed head; only surrounding context moved.
  • quest check --base origin/main on 13aa7ca699: 553 documents ok.
  • Still accurate on main: rs/moq-mux/src/container/mkv/ uses its own ebml.rs reader (no WebmIterator in the importer). mkv-chunks and mkv-payloads are in rs/moq-mux/benches/containers.rs, which the shared Rust bench nightly runs. Main's only link to the quest is the README entry this PR removes.
  • moq-dev/moq#7440 (open) does not touch this quest or the survey README. Its new quest/m3/mkv-input-coalesce.md tracks the per-decode cost from byte-at-a-time input, which matches the alternative Decision 1 left out of this quest. It does not conflict with deleting this one.
  • Earlier, quest gate nix develop -c just ci origin/main on 2334f47387: passed (markdown, gh, quest, shell, large_files, toml, nix, justfile).

Review

Head 13aa7ca699: no new local review. The only change since f8218f3520 is a merge from main that leaves the PR's own added and removed lines identical. The PR touches only quest files and carries no behavior risk, so the skip under the simple-review rule (moq-dev/quest#364) still applies.

Online reviews on f8218f3520: the OpenAI automated review (review) and pullfrog (review) found no issues. CodeRabbit was rate-limited and did not review. There are no unresolved threads.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 2 commits November 1, 2026 00:53
@kixelated
chore(quest): claim quest/m2/performance-survey/mkv-fragments.md
555d666 
Quest-Claim: 935e227d3b6442f8
@kixelated @claude
quest: delete mkv-fragments, the MKV importer no longer replays input
2334f47 
#6908 replaced the clone-and-replay drain with an incremental
decoder, and #7234 streams blocks through its own EBML reader.
A deterministic sweep (block size x chunk size, instructions and
allocation) shows the O(B^2/c) slope at the audit commit and none since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Fragmented MKV import avoids replaying accumulated input quest: delete mkv-fragments, the MKV importer no longer replays input Nov 1, 2026
@kixelated kixelated mentioned this pull request Nov 1, 2026
Merged
@kixelated
kixelated marked this pull request as ready for review November 1, 2026 22:38
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

The performance survey README now records that accumulated-input replay was removed, cites the related pull requests and measured result, and names the nightly benchmark series. The Required list no longer includes the fragmented MKV import quest. The MKV fragmentation proposal was deleted.

Merge Risk: Minimal · up to 13aa7

The update records the MKV replay findings and removes the obsolete proposal without changing runtime behavior. No merge-blocking concern is identified.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title describes the main change: deletion of the mkv-fragments quest, with the stated reason that the MKV importer no longer replays input. The PR deletes that quest file and updates the survey RE…
Description check Passed The description states that it claims and deletes the mkv-fragments quest and explains why. It cites the commits and PRs that replaced the replay behavior, reports measurements, and records the decisi…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title describes the main change: deletion of the mkv-fragments quest, with the stated reason that the MKV importer no longer replays input. The PR deletes that quest file and updates the survey README, and the objectives describe the same change. The title is short and specific.

Full details: Description check

Explanation

The description states that it claims and deletes the mkv-fragments quest and explains why. It cites the commits and PRs that replaced the replay behavior, reports measurements, and records the decision. It matches the changeset and the objectives.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

Finishing Touches
Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: db37ead3-fc46-4f37-8cfc-ee778d9715c5
Commits

Reviewing files that changed from the base of the PR and between 8c2f04c and 13aa7ca.

Files selected for processing: 2
  • quest/m2/performance-survey/README.md
  • quest/m2/performance-survey/mkv-fragments.md
Files with no reviewable changes: 1
  • quest/m2/performance-survey/mkv-fragments.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

kixelated
kixelated commented Nov 1, 2026
View reviewed changes

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: f8218f3

No actionable bugs found in this documentation-only change. Deleting the obsolete replay quest, removing its index entry, and retaining the outcome in the survey is simpler than adding another parser optimization or duplicate benchmark. The current EBML reader consumes input incrementally, and the existing mkv-chunks/mkv-payloads benchmarks are selected by the nightly container benchmark run. No runtime, API, or wire changes are introduced.

Verification: inspected the changed quest files, relevant importer/framing code, benchmark cases and nightly selection. I did not run tests or reproduce the PR's performance measurements; those results remain author-reported.

(Written by OpenAI)

pullfrog[bot]
pullfrog Bot reviewed Nov 1, 2026
View reviewed changes

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Quest deleted — quest/m2/performance-survey/mkv-fragments.md removed; the maintainer chose deletion on 2026-11-01 (recorded under Decisions), so this matches the quest-completion convention.
  • Survey index updated — the ## Required bullet for the quest is removed and the existing-ownership bullet now records that the MKV importer's replay of accumulated input was removed by #6908 and #7234, with the measured no-win in this PR and nightly mkv-chunks / mkv-payloads covering the slope.

I verified the claims rather than taking them on faith:

  • No dangling references. git grep mkv-fragments at the head resolves only to the edited README; the deletion removes its last reference.
  • The cited PRs are merged and relevant: #6908 (merged 2026-10-27, 25cd95ea64) replaced the replay with ebml-iterable's incremental TagDecoder; #7234 (merged 2026-11-01, bdd045f361) replaced that with the importer's own EBML reader.
  • The behavior matches the prose: Import::drain now takes &[u8] (rs/moq-mux/src/container/mkv/import.rs:344) and drives a streaming reader, with no retained-BytesMut/WebmIterator replay.
  • The nightly-coverage claim holds: mkv-chunks and mkv-payloads are Criterion groups in rs/moq-mux/benches/containers.rs, and nightly rust-bench discovers every bench target through cargo metadata (bench/run.sh:242), so they are scheduled.

Quest files only, no behavior risk, and the report carrying the measurement lives in the PR body as expected for a report-shaped quest.

Pullfrog  ｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏

@kixelated
kixelated enabled auto-merge (squash) November 1, 2026 22:43
@kixelated
kixelated disabled auto-merge November 3, 2026 06:15
@kixelated @claude
Merge remote-tracking branch 'origin/main' into quest/m2/performance-…
13aa7ca 
…survey/mkv-fragments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 8121cf6 Nov 3, 2026
9 of 10 checks passed
@kixelated
kixelated deleted the quest/m2/performance-survey/mkv-fragments branch November 3, 2026 07:11
This was referenced Nov 3, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pullfrog pullfrog[bot] pullfrog[bot] left review comments

Assignees

No one assigned

Labels

quest:merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kixelated