Skip to content

bench(ytdlp): composite benchmark, first result FAIL (1.36 to 1.47) - #796

Draft
espetro wants to merge 43 commits into
work/planesplitfrom
work/ytbench
Draft

espetro wants to merge 43 commits into
work/planesplitfrom
work/ytbench

Conversation

@espetro

@espetro espetro commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

yt-dlp composite benchmark, defined before measuring (.agents/notes/2026-10-30-ytdlp-composite.md) and a runnable script (benchmarks/ytdlp_composite/run.py, docs in the same dir). Stacked on work/planesplit.

Result: FAIL. Geometric mean over 11 dimensions (lower is better, pass needs below 1.0 against both baselines and no dimension worse than 2x):

  • V-bundle: 1.375 vs B1 (uv venv), 1.467 vs B2 (PyInstaller onefile)
  • V-minimal: 1.357 vs B1, 1.447 vs B2

Min of 7 interleaved samples, VPS, 3.14.9:

--version s --list-extractors s extract s RSS MiB files installed bytes
V-bundle 1.079 1.513 2.325 163.0 152 87.2 MB
V-host 0.971 1.540 2.294 159.6 1 11.3 MB (needs host libpython)
B1 uv venv 0.270 0.383 1.131 49.8 2,999 163.7 MB
B2 PyInstaller 1.277 1.492 2.288 74.1 1 41.2 MB

Output is byte-identical to CPython (timestamp fields masked) and the bundle runs in a clean debian:trixie-slim container with no Python. It beats B1 on files and disk and ties B2 on speed, but loses on RSS (3.3x B1) and startup (4x B1).

Cause: the island seed carries all 1,044 module sources as one 10.6 MB blob that is compiled and executed at startup (about 0.96 s of the 1.0 s --version). Levers tried: upstream's lazy extractors, minimal stdlib trim, deflated stdlib zip, strip. zstd and upx are excluded by the note since no baseline ships a compressed executable.

Next lever: precompile the seed to bytecode and load modules lazily on first import instead of exec-ing everything at startup.

Agent added 11 commits October 30, 2026 16:10
One island class in a widely imported module dragged every consumer's
top level island-side (Decision A): the export table carried only
lowered classes, so the cross-module import fenced at module level with
'not exported by this package'.

The export table now carries island classes with the island flag, the
way it carries island functions (Refs #641). A consumer binds the class
across the boundary as an island-floor class: instantiation and
class-as-value go through the namespaced seam
(ps_island_mod_def_handle), a native subclass and an except clause
naming the class fence their own definition instead of the module
level.

In split mode a module's own island floor also rides the namespaced
seam: the single-namespace seed does not exist, so the calls and
handles name the owning module. The per-module seeds run in the order
CPython finishes the module bodies, so a seed that imports a sibling
finds that sibling's namespace in sys.modules.
…orting one

A package re-exports a class (`from ._utils import *`), so the module
the import names is not the module that holds the class object. The
cross-module island binding used the importing module's name, so the
handle and the call looked in a namespace that does not exist. The
export entry carries the owner, and the binding uses it.

The island class instantiation also returns the class's own handle
type, so an annotated destination agrees with the call site.
… native

A Stage 2 seed re-executed every top-level import, so a fenced entry
module whose import target lowered native died at startup with
ModuleNotFoundError: the split seeds one namespace per fenced module and
the native sibling has none. The seed now keeps an import only when its
target rides the island too (or is not a program module at all). The
import is not load-bearing: an island definition that used the native
sibling's definition already refused at the island-to-compiled seam.
The module-owns-its-floor test now exercises the seam with a real
native-to-island call in the entry.
…ntime

The bundle compile of a program now stages the runtime pack's CPython
headers (--embed-headers), the runtime lock file and the pack's own stdlib
zip, so the compiled binary can compile island programs at run time with no
Python install, no runtime pack cache and no network. A bundle dependency
root can be declared extern (--bundle-exclude), and a missing header
directory names the remedy.
tests/test_selfbin.py (marked slow) builds the self-compiled pyscriptc
through the cli and runs a hello world compile with no Python on PATH, and
mise run selfbin wraps the same build. The runtime-pack reference documents
the two-file bundle layout and the external tools the binary still needs.
…emedy

The -m pyrefly fallback in a self-compiled binary launches the compiler
itself, so the failure surfaced as an argparse usage error. The failure now
names PYREFLY_BIN, pip install pyrefly or --skip-check, and the doubled
Install one of: in PS0001 is deduplicated.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Deploying pyscriptc with  Cloudflare Pages  Cloudflare Pages

Latest commit: ded3ed0
Status: ✅  Deploy successful!
Preview URL: https://16845011.pyscriptc.pages.dev
Branch Preview URL: https://work-ytbench.pyscriptc.pages.dev

View logs

Agent added 18 commits November 2, 2026 07:27
…o the self-compiled bundle

The self-compiled bundle keeps its own carried_modules helper (a pack
whose stdlib travels as a zip carries the zip's names), so the merge
keeps it and adds the runtime set on top: the check answers the runtime
the bundle links, not the compiler host's build.
…e yt-dlp composite lane

Both sides repinned the self-host census, so the merge takes the sum:
the composite runner and the island seed import drop plus the
runtime-linked manifest check land at 3294 self-host fences.
The --bundle-exclude and --embed-headers flags landed without a docs
regeneration, so the generated-docs seam reported the CLI reference as
out of date.
…compiled bundle

The runtime-linked stdlib manifest check adds 7 self-host fences in
stdlibzip.py, so the bundle branch's census pin (3278) is stale on the
merged tree. The merged tree measures 3285 fences and the same package
sum; the pin moves with a comment that names both deltas.
…eed import drop

The runtime-linked stdlib manifest check adds 7 self-host fences in
stdlibzip.py, so the island-seed branch's census pin (3284) is stale on
the merged tree. The merged tree measures 3291 fences and the same
package sum; the pin moves with a comment that names both deltas.
…mposite lane

The runtime-linked stdlib manifest check adds 7 self-host fences in
stdlibzip.py, so the composite lane's census pin had to be recomputed
against the merged tree. The merged tree measures 3294 fences and the
same package sum; the pin moves with a comment that names every delta.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

1 participant