Add R04, frozen modules

README.md

@@ -162,6 +162,7 @@ A lesson can also end with a boss fight, which is a problem the text does not so
 | R01 | [Before your first line](lessons/r01-before-your-first-line/r01.ipynb) | Everything the interpreter has already done by the time your first statement runs. A fresh interpreter has a few dozen modules in sys.modules and, with site out of the way, not one of them was read from a file, because they are either C compiled into the binary or Python bytecode frozen into it. Startup comes in two halves on purpose, and the first half has no import system and no sys.path, which is why a configuration mistake comes out as a fatal error with a plain C string rather than as a traceback. Settings arrive from the command line, the environment and the embedder, and the answer is the highest number anybody asked for rather than whichever one is nearest, so PYTHONOPTIMIZE=2 is not undone by passing -O. sys.path itself is produced by a Python program that is frozen into the binary and handed eleven C functions to stand in for the os.path it cannot import, and its output always includes a zip file that usually does not exist. The front of sys.path is pushed on last, after startup is over, which is the whole mechanism behind a file called random.py shadowing the standard library, and -P turns it off. Two recordings then compare a release build against a debug one and find that the extra time is mostly not the assertions, it is the fourteen modules a debug build reads off the disk because it turns frozen modules off | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r01-before-your-first-line/r01.ipynb) |
 | R02 | [Where the state lives](lessons/r02-where-the-state-lives/r02.ipynb) | A running Python is three nested things, a runtime with interpreters inside it and threads inside those, and every fact in this book belongs to exactly one of the three levels. Two interpreters in one process return the same id for None, for the booleans, for the small ints, for one character strings and for the built in type objects, because all of those are fields of the runtime struct rather than allocations, and different ids for everything else. Where the small int range ends is a #define rather than a policy, and it moved from 256 to 1024 in 3.15, which the notebook measures by asking both interpreters for the address of every int in a range. An interpreter owns its own sys.modules, its own builtins, its own import lock, its own warnings filters and its own recursion limit, so changing any of them is invisible next door. Signals are the exception, one handler table for the whole process with the main thread of the main interpreter as the only writer, which is two conditions in one line of C and is exactly what the error message says. A thread owns the exception it is handling, and two threads inside an except block at the same instant each see their own. Two recordings then price it: an interpreter costs about fifty times what an operating system thread costs to create, and a build without the GIL charges more for both because mimalloc heaps and obmalloc pools do not price an extra interpreter the same way | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r02-where-the-state-lives/r02.ipynb) |
 | R03 | [What import does](lessons/r03-what-import-does/r03.ipynb) | An import statement compiles to one IMPORT_NAME plus zero or more IMPORT_FROM, and IMPORT_NAME looks the name __import__ up in builtins every single time, which is why replacing it works. Compiling the four spellings shows that import a.b binds a rather than a.b, and that a relative import is the empty string at a level above zero. A finder put on the front of sys.meta_path that answers nothing and writes down every question shows a dotted import searching for each part in turn from the outside in, with everything after the first part looked for in the parent package's __path__. A fresh interpreter has three finders, and import os stops at the second of them, so os.py is never opened. The module object goes into sys.modules before its body runs, which is what makes circular imports work and what decides how much of a half loaded module the other side can see, and if the body raises the entry is taken back out. Three caches sit in the way of a repeat import, and a directory created after it was first looked for stays invisible until importlib.invalidate_caches is called. Two recordings then settle the import lock: it is one lock per module name, so four threads importing four different modules keep one core busy on a build with the GIL and three and a half on a build without | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r03-what-import-does/r03.ipynb) |
+| R04 | [Frozen modules](lessons/r04-frozen-modules/r04.ipynb) | The import system is written in Python, so it cannot be imported, and the way out of that is to compile a handful of modules during CPython's own build and write the bytecode into the binary as C arrays. The module body of _frozen_importlib contains zero IMPORT_NAME opcodes, which is what makes it loadable with no import system running, and init_importlib in C hands sys and _imp to it as arguments. Thirty three names are frozen in a stock 3.15 build, in three groups that the flag treats differently: three for the import system that can never be switched off, nineteen for what a bare startup needs, and eleven hello world modules for the test suite. A frozen module still knows where it came from, because the loader puts the original path in loader_state and copies it onto __file__, which is why inspect.getsource and tracebacks still work while co_filename says <frozen os>. Switching the flag off in process with _imp._override_frozen_modules_for_tests moves os from FrozenImporter to SourceFileLoader and leaves the same constants behind. What freezing actually buys is the finder search and the file read, not the unmarshal, which is identical on both paths, and two recordings put it at about a tenth of a startup on a release build and a tenth on a debug build, where it is off by default | M8 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/r04-frozen-modules/r04.ipynb) |
 
 More are landing in order. [lessons/README.md](lessons/README.md) explains how one is put together and how to run them locally.
 

citations.lock.json

@@ -1210,6 +1210,26 @@
       "first_line": "def _parse_exception_table(code):",
       "lines": 15
     },
+    "Lib/importlib/_bootstrap.py:1000-1011@v3.15.0rc1": {
+      "digest": "6ccd8a9547a7d8c2",
+      "first_line": "class FrozenImporter:",
+      "lines": 12
+    },
+    "Lib/importlib/_bootstrap.py:1106-1133@v3.15.0rc1": {
+      "digest": "21e719984e0eaeb5",
+      "first_line": "def find_spec(cls, fullname, path=None, target=None):",
+      "lines": 28
+    },
+    "Lib/importlib/_bootstrap.py:1148-1153@v3.15.0rc1": {
+      "digest": "63fc1959306d2042",
+      "first_line": "@staticmethod",
+      "lines": 6
+    },
+    "Lib/importlib/_bootstrap.py:1155-1165@v3.15.0rc1": {
+      "digest": "04de1062682f64e4",
+      "first_line": "@classmethod",
+      "lines": 11
+    },
     "Lib/importlib/_bootstrap.py:1198-1223@v3.15.0rc1": {
       "digest": "10232b40fdaf68e3",
       "first_line": "def _find_spec(name, path, target=None):",
@@ -1230,6 +1250,16 @@
       "first_line": "def __import__(name, globals=None, locals=None, fromlist=(), level=0):",
       "lines": 35
     },
+    "Lib/importlib/_bootstrap.py:1501-1539@v3.15.0rc1": {
+      "digest": "175a08818ad49605",
+      "first_line": "def _setup(sys_module, _imp_module):",
+      "lines": 39
+    },
+    "Lib/importlib/_bootstrap.py:1541-1546@v3.15.0rc1": {
+      "digest": "a8ccfaac3e1ef22d",
+      "first_line": "def _install(sys_module, _imp_module):",
+      "lines": 6
+    },
     "Lib/importlib/_bootstrap.py:226-240@v3.15.0rc1": {
       "digest": "54c579dcaffdc305",
       "first_line": "class _ModuleLock:",
@@ -1290,11 +1320,21 @@
       "first_line": "def getsourcelines(object):",
       "lines": 20
     },
+    "Lib/inspect.py:3386-3400@v3.15.0rc1": {
+      "digest": "1460bc9e4c86a95e",
+      "first_line": "reported_target = reported_module_name",
+      "lines": 15
+    },
     "Lib/inspect.py:886-897@v3.15.0rc1": {
       "digest": "108c22eee97b22a8",
       "first_line": "def getsourcefile(object):",
       "lines": 12
     },
+    "Lib/linecache.py:122-140@v3.15.0rc1": {
+      "digest": "80a479410cfbbe4e",
+      "first_line": "entry = cache.pop(filename, None)",
+      "lines": 19
+    },
     "Lib/opcode.py:16-23@v3.15.0rc1": {
       "digest": "e4e79270a74738f6",
       "first_line": "from _opcode_metadata import (_specializations, _specialized_opmap, opmap,  # noqa: F401",
@@ -4135,6 +4175,21 @@
       "first_line": "static void",
       "lines": 22
     },
+    "Python/frozen.c:127-137@v3.15.0rc1": {
+      "digest": "1992fe1133dc64e4",
+      "first_line": "static const struct _module_alias aliases[] = {",
+      "lines": 11
+    },
+    "Python/frozen.c:140-144@v3.15.0rc1": {
+      "digest": "50e0058411e2f488",
+      "first_line": "",
+      "lines": 5
+    },
+    "Python/frozen.c:74-79@v3.15.0rc1": {
+      "digest": "6c90a431b887b976",
+      "first_line": "static const struct _frozen bootstrap_modules[] = {",
+      "lines": 6
+    },
     "Python/gc.c:1011-1039@v3.15.0rc1": {
       "digest": "63bcba6da6794191",
       "first_line": "/* Handle uncollectable garbage (cycles with tp_del slots, and stuff reachable",
@@ -4390,6 +4445,31 @@
       "first_line": "static PyObject *",
       "lines": 14
     },
+    "Python/import.c:3104-3129@v3.15.0rc1": {
+      "digest": "d8230b80e5b80f4e",
+      "first_line": "static const struct _frozen *",
+      "lines": 26
+    },
+    "Python/import.c:3130-3150@v3.15.0rc1": {
+      "digest": "1af64cf35596eb29",
+      "first_line": "// Frozen stdlib modules may be disabled.",
+      "lines": 21
+    },
+    "Python/import.c:3152-3159@v3.15.0rc1": {
+      "digest": "51c97e549dbb2884",
+      "first_line": "struct frozen_info {",
+      "lines": 8
+    },
+    "Python/import.c:3210-3229@v3.15.0rc1": {
+      "digest": "96bd53170c95c4ee",
+      "first_line": "static PyObject *",
+      "lines": 20
+    },
+    "Python/import.c:3389-3428@v3.15.0rc1": {
+      "digest": "cc9ed48c09b8fb3d",
+      "first_line": "static int",
+      "lines": 40
+    },
     "Python/import.c:4190-4227@v3.15.0rc1": {
       "digest": "40a1544a3fd82833",
       "first_line": "PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals,",
@@ -4830,6 +4910,11 @@
       "first_line": "int",
       "lines": 36
     },
+    "Tools/build/freeze_modules.py:37-76@v3.15.0rc1": {
+      "digest": "9d354f9c964e4b4f",
+      "first_line": "FROZEN = [",
+      "lines": 40
+    },
     "Tools/cases_generator/README.md:14-34@v3.15.0rc1": {
       "digest": "fb06409a976b7a51",
       "first_line": "- `tierN_generator.py`: a couple of driver scripts to read `Python/bytecodes.c` and",

experiments/README.md

@@ -42,6 +42,8 @@ So those programs run somewhere else. They run in the images this project publis
 | [r02-what-a-second-interpreter-costs-without-the-lock](tier1/r02-what-a-second-interpreter-costs-without-the-lock.md) | R02 | freethreaded | Does an interpreter cost the same to make and to keep on a build with no lock? |
 | [r03-how-much-of-an-import-is-parallel](tier1/r03-how-much-of-an-import-is-parallel.md) | R03 | release | Does the import lock stop two threads importing at once, or does something else? |
 | [r03-how-much-of-an-import-is-parallel-without-the-lock](tier1/r03-how-much-of-an-import-is-parallel-without-the-lock.md) | R03 | freethreaded | With the GIL out of the way, do four imports on four threads finish in the time of one? |
+| [r04-what-freezing-saves-at-startup](tier1/r04-what-freezing-saves-at-startup.md) | R04 | release | What does compiling the standard library into the binary actually save at startup? |
+| [r04-what-freezing-saves-on-a-debug-build](tier1/r04-what-freezing-saves-on-a-debug-build.md) | R04 | debug | Does a debug build behave the same way, and does freezing still pay for itself there? |
 
 ## The commands
 

experiments/tier1/r04-what-freezing-saves-at-startup.md

@@ -0,0 +1,165 @@
+# A startup with the frozen standard library, and the same startup without it
+
+Generated by `just build-tier1`. Do not edit by hand, the change will be overwritten.
+
+What does compiling the standard library into the binary actually save at startup?
+
+- Lesson: R04
+- Build: release
+- Image: ghcr.io/tamnd/cpython-internals/cpython:release@sha256:fb55d6afcf053c974de6447fafbd2be6af20cdb9f596e25a0445607b8af981e3
+- Interpreter: 3.15.0rc1 (37e98da:37e98da, Aug 29 2026, 09:24:54) [GCC 14.2.0]
+- Recorded: 2026-09-06
+
+Why this needs the release build: it wants a machine that is not busy with anything else, because half of what it reports is wall clock for a process that only lives for a few milliseconds.
+
+## The program
+
+```python
+"""What freezing the standard library into the binary is worth at startup.
+
+Import is written in Python, so it cannot be imported. CPython gets around that by compiling a
+handful of modules during its own build and writing the bytecode into the binary as C arrays. The
+first three are the import system itself, and they are what lets the interpreter get going at all.
+
+Everything after those three is a speed decision rather than a correctness one, and it can be
+switched off with -X frozen_modules=off, so the cost of it can be measured rather than guessed.
+Which way the switch sits by default is a build choice, so this program never relies on the
+default. It asks for on and off explicitly and reports what the default happens to be.
+
+Two numbers matter. The first is how many code objects a bare startup reads off disk, which -v
+prints one line for, so it is a count rather than a timing and it is the same on any machine. The
+second is wall clock, measured with the two cases alternating, because anything that runs one case
+forty times and then the other forty times is measuring the state of the page cache instead.
+"""
+
+import _imp
+import statistics
+import subprocess
+import sys
+import time
+
+ROUNDS = 40
+ON = ["-X", "frozen_modules=on"]
+OFF = ["-X", "frozen_modules=off"]
+ORIGIN = "import os; print(os.__spec__.origin)"
+BOOTSTRAP = "import sys; print(sys.modules['_frozen_importlib'].__spec__.origin)"
+COUNT = (
+    "import sys\n"
+    "specs = [getattr(m, '__spec__', None) for m in sys.modules.values()]\n"
+    "print(len(sys.modules), sum(1 for s in specs if s is not None and s.origin == 'frozen'))\n"
+)
+
+
+def child(flags, args):
+    """Run this same interpreter again with those flags and hand back the finished process."""
+    return subprocess.run(
+        [sys.executable, *flags, *args], capture_output=True, text=True, check=True
+    )
+
+
+def origin(flags):
+    """Ask a fresh interpreter where the os module it just imported came from."""
+    return child(flags, ["-c", ORIGIN]).stdout.strip()
+
+
+def loaded(flags):
+    """Ask a fresh interpreter how many modules it loaded and how many came out of the binary."""
+    return [int(part) for part in child(flags, ["-c", COUNT]).stdout.split()]
+
+
+def files_read(flags):
+    """Count the code objects a fresh interpreter reads off disk, which -v prints one per line."""
+    printed = child(flags, ["-v", "-c", "pass"]).stderr
+    return sum(1 for line in printed.splitlines() if line.startswith("# code object from"))
+
+
+def import_work(flags):
+    """Add up the self time -X importtime reports, in microseconds."""
+    printed = child(["-X", "importtime", *flags], ["-c", "pass"]).stderr
+    total = 0
+    for line in printed.splitlines():
+        if line.startswith("import time:"):
+            first = line.removeprefix("import time:").split("|")[0].strip()
+            if first.isdigit():
+                total += int(first)
+    return total
+
+
+def startup_ms(flags):
+    """Wall clock milliseconds for one whole interpreter startup that does nothing at all."""
+    started = time.perf_counter()
+    child(flags, ["-c", "pass"])
+    return (time.perf_counter() - started) * 1000
+
+
+def alternating(measure, rounds):
+    """Measure both cases once each per round, so neither one gets the cold cache every time."""
+    got = {"on": [], "off": []}
+    for _ in range(rounds):
+        got["on"].append(measure(ON))
+        got["off"].append(measure(OFF))
+    return got
+
+
+_imp._override_frozen_modules_for_tests(1)
+names = _imp._frozen_module_names()
+frozen_os = _imp.get_frozen_object("os")
+_imp._override_frozen_modules_for_tests(0)
+
+print("names compiled into this binary:", len(names))
+print("the three that cannot be turned off:", ", ".join(names[:3]))
+print("filename on the frozen code object for os:", frozen_os.co_filename)
+print("bytes of bytecode in it:", len(frozen_os.co_code))
+print("this build uses them unless told otherwise:", origin([]) == "frozen")
+print("where os comes from with the flag on:", origin(ON))
+print("where os comes from with the flag off:", origin(OFF).rpartition("/")[2])
+print("_frozen_importlib with the flag off:", child(OFF, ["-c", BOOTSTRAP]).stdout.strip())
+
+on_total, on_frozen = loaded(ON)
+off_total, off_frozen = loaded(OFF)
+print("modules a bare startup loads, flag on:", on_total)
+print("modules a bare startup loads, flag off:", off_total)
+print("of those, frozen with the flag on:", on_frozen)
+print("of those, frozen with the flag off:", off_frozen)
+print("code objects read off disk with the flag on:", files_read(ON))
+print("code objects read off disk with the flag off:", files_read(OFF))
+
+work = alternating(import_work, 10)
+print(f"~ import work reported by importtime, frozen on: {min(work['on'])} us")
+print(f"~ import work reported by importtime, frozen off: {min(work['off'])} us")
+
+runs = alternating(startup_ms, ROUNDS)
+fast_on, fast_off = min(runs["on"]), min(runs["off"])
+print(f"~ fastest startup with the flag on: {fast_on:.1f} ms")
+print(f"~ fastest startup with the flag off: {fast_off:.1f} ms")
+print(f"~ middle startup with the flag on: {statistics.median(runs['on']):.1f} ms")
+print(f"~ middle startup with the flag off: {statistics.median(runs['off']):.1f} ms")
+share = (1 - fast_on / fast_off) * 100
+print(f"~ share of a startup that freezing gives back: {share:.1f} percent")
+```
+
+## What it printed
+
+```text
+names compiled into this binary: 33
+the three that cannot be turned off: _frozen_importlib, _frozen_importlib_external, zipimport
+filename on the frozen code object for os: <frozen os>
+bytes of bytecode in it: 3360
+this build uses them unless told otherwise: True
+where os comes from with the flag on: frozen
+where os comes from with the flag off: os.py
+_frozen_importlib with the flag off: frozen
+modules a bare startup loads, flag on: 33
+modules a bare startup loads, flag off: 33
+of those, frozen with the flag on: 17
+of those, frozen with the flag off: 3
+code objects read off disk with the flag on: 0
+code objects read off disk with the flag off: 13
+~ import work reported by importtime, frozen on: 12884 us
+~ import work reported by importtime, frozen off: 19167 us
+~ fastest startup with the flag on: 30.5 ms
+~ fastest startup with the flag off: 35.6 ms
+~ middle startup with the flag on: 33.9 ms
+~ middle startup with the flag off: 39.5 ms
+~ share of a startup that freezing gives back: 14.2 percent
+```