Skip to content

Commit d652323

Browse files
miss-islingtonitamaroencukou
authored
[3.15] gh-157384: Document and test PyImport_CreateModuleFromInitfunc() semantics (GH-157389) (#159791)
gh-157384: Document and test PyImport_CreateModuleFromInitfunc() semantics (GH-157389) Document that single-phase init modules are added to sys.modules by the call itself (unlike multi-phase init modules), and that the spec name identifies the module, so a later call with the same name and a different init function reuses the cached module without calling the new init function. Add regression tests for both behaviors. (cherry picked from commit 0c4ca17) Co-authored-by: Itamar Oren <itamarost@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>
1 parent 8da32df commit d652323

4 files changed

Lines changed: 139 additions & 18 deletions

File tree

‎Doc/c-api/extension-modules.rst‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -313,16 +313,32 @@ This is typically done using :c:func:`PyModule_Create` and functions like
313313
Single-phase initialization differs from the :ref:`default <multi-phase-initialization>`
314314
in the following ways:
315315
316-
* Single-phase modules are, or rather *contain*, “singletons”.
316+
* Single-phase modules with :c:member:`PyModuleDef.m_size` set to -1 are,
317+
or rather *contain*, “singletons”.
317318
318-
When the module is first initialized, Python saves the contents of
319+
When such a module is first initialized, Python saves the contents of
319320
the module's ``__dict__`` (that is, typically, the module's functions and
320321
types).
321322
322323
For subsequent imports, Python does not call the initialization function
323324
again.
324325
Instead, it creates a new module object with a new ``__dict__``, and copies
325326
the saved contents to it.
327+
Modules are matched to the saved contents by a (*module_name*, *file_name*)
328+
tuple where:
329+
330+
- *module_name* is the module's full dotted name, taken from
331+
:py:attr:`ModuleSpec.name <importlib.machinery.ModuleSpec.name>`
332+
(or the equivalent argument of the importing function).
333+
Note that names that the initialization function uses to create the module
334+
(for example, :c:member:`PyModuleDef.m_name`) are not taken into account.
335+
- *file_name* is the extension's file name.
336+
It is unset for modules not loaded from a file, such as built-ins and
337+
modules added with :c:func:`PyImport_CreateModuleFromInitfunc` or
338+
:c:func:`PyImport_AppendInittab`.
339+
340+
As part of the first initialization, Python also adds the module to
341+
:data:`sys.modules` under its name.
326342
For example, given a single-phase module ``_testsinglephase``
327343
[#testsinglephase]_ that defines a function ``sum`` and an exception class
328344
``error``:
@@ -344,6 +360,16 @@ in the following ways:
344360
345361
The exact behavior should be considered a CPython implementation detail.
346362

363+
* For single-phase modules with non-negative :c:member:`PyModuleDef.m_size`,
364+
Python may cache a pointer to the initialization function under the
365+
(*module_name*, *file_name*) key as for ``m_size == -1``.
366+
367+
Subsequent imports that specify a different initialization function
368+
(for example, using :c:func:`PyImport_CreateModuleFromInitfunc`)
369+
may call either the cached function or the provided "new" one.
370+
371+
The exact behavior should again be considered a CPython implementation detail.
372+
347373
* To work around the fact that ``PyInit_modulename`` does not take a *spec*
348374
argument, some state of the import machinery is saved and applied to the
349375
first suitable module created during the ``PyInit_modulename`` call.

‎Doc/c-api/import.rst‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -408,7 +408,7 @@ Importing Modules
408408
409409
*spec* must be a :class:`~importlib.machinery.ModuleSpec` object.
410410
411-
*initfunc* must be an :ref:`initialization function <extension-export-hook>`,
411+
*initfunc* must be an :ref:`initialization function <extension-pyinit>`,
412412
the same as for :c:func:`PyImport_AppendInittab`.
413413
414414
On success, create and return a module object.
@@ -417,10 +417,27 @@ Importing Modules
417417
(Custom importers should do this in their
418418
:py:meth:`~importlib.abc.Loader.exec_module` method.)
419419
420-
If the spec's name is dotted and *initfunc* creates the module using only
421-
the last component of the name, the module's :attr:`~module.__name__`
422-
is set to the full name from the spec.
423-
424420
On error, return NULL with an exception set.
425421
422+
For modules using :ref:`legacy single-phase initialization <single-phase-initialization>`,
423+
most of the import process is done in this function, rather than in
424+
extra steps that importers typically take after
425+
:py:meth:`~importlib.abc.Loader.create_module`.
426+
Specifically:
427+
428+
- The module is added to :data:`sys.modules` under the *spec*'s
429+
:py:attr:`~importlib.machinery.ModuleSpec.name` as part of initialization.
430+
- If a single-phase module was previously imported under the
431+
same :py:attr:`~importlib.machinery.ModuleSpec.name` by this function
432+
or as a built-in module (for example, one registered with
433+
:c:func:`PyImport_AppendInittab`), Python may reuse the previous module's
434+
contents instead of calling *initfunc* again.
435+
- The module will be fully initialized by *initfunc*.
436+
Calling :c:func:`PyModule_Exec` on it will have no effect, though it is
437+
still recommended, in case the module later switches to
438+
:ref:`multi-phase initialization <multi-phase-initialization>`.
439+
- Also, if the *spec*'s :py:attr:`!name` contains dots, this name is used
440+
for prepending a parent package to the new module's
441+
:attr:`~module.__name__`.
442+
426443
.. versionadded:: 3.15

‎Lib/test/test_embed.py‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -250,16 +250,27 @@ def test_create_module_from_initfunc(self):
250250
out, err = self.run_embedded_interpreter("test_create_module_from_initfunc")
251251
self.assertEqual(self._nogil_filtered_err(err, "embedded_ext"), "")
252252
self.assertEqual(out,
253-
"<module 'my_test_extension' (static-extension)>\n"
254-
"my_test_extension.executed='yes'\n"
255-
"my_test_extension.exec_slot_ran='yes'\n"
256-
"<module 'embedded_ext' (static-extension)>\n"
257-
"embedded_ext.executed='yes'\n"
258-
"<module 'sp_pkg.sp_submod' (static-extension)>\n"
259-
"sp_pkg.sp_submod.__name__='sp_pkg.sp_submod'\n"
260-
"sys.modules[\"sp_pkg.sp_submod\"] is sp_pkg.sp_submod=True\n"
261-
"\"sp_submod\" in sys.modules=False\n"
262-
)
253+
# multi-phase init: not in sys.modules until importlib adds it
254+
"created my_test_extension: in sys.modules=False\n"
255+
"<module 'my_test_extension' (static-extension)>\n"
256+
"my_test_extension.executed='yes'\n"
257+
"my_test_extension.exec_slot_ran='yes'\n"
258+
# single-phase init: added to sys.modules by the init function
259+
"created embedded_ext: in sys.modules=True\n"
260+
"<module 'embedded_ext' (static-extension)>\n"
261+
"embedded_ext.executed='yes'\n"
262+
# same name, different initfuncs: the cached module is
263+
# returned and the second initfunc is never called
264+
"a.which='A' b.which='A' a is b=True\n"
265+
"create_static_module.initfunc_calls()=(1, 0)\n"
266+
# Static extension
267+
"created sp_pkg: in sys.modules=False\n"
268+
"created sp_pkg.sp_submod: in sys.modules=True\n"
269+
"<module 'sp_pkg.sp_submod' (static-extension)>\n"
270+
"sp_pkg.sp_submod.__name__='sp_pkg.sp_submod'\n"
271+
"sys.modules[\"sp_pkg.sp_submod\"] is sp_pkg.sp_submod=True\n"
272+
"\"sp_submod\" in sys.modules=False\n"
273+
)
263274

264275
def test_inittab_submodule_multiphase(self):
265276
out, err = self.run_embedded_interpreter("test_inittab_submodule_multiphase")

‎Programs/_testembed.c‎

Lines changed: 68 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2422,6 +2422,58 @@ static int test_repeated_init_and_inittab(void)
24222422
return 0;
24232423
}
24242424

2425+
// Two different single-phase init functions for the same module name.
2426+
static int cmfi_initfunc_a_calls = 0;
2427+
static int cmfi_initfunc_b_calls = 0;
2428+
2429+
static PyModuleDef cmfi_same_name_a_def = {
2430+
PyModuleDef_HEAD_INIT,
2431+
.m_name = "same_name",
2432+
.m_size = -1,
2433+
};
2434+
2435+
static PyModuleDef cmfi_same_name_b_def = {
2436+
PyModuleDef_HEAD_INIT,
2437+
.m_name = "same_name",
2438+
.m_size = -1,
2439+
};
2440+
2441+
static PyObject*
2442+
PyInit_cmfi_same_name_a(void)
2443+
{
2444+
cmfi_initfunc_a_calls++;
2445+
PyObject *mod = PyModule_Create(&cmfi_same_name_a_def);
2446+
if (mod == NULL || PyModule_AddStringConstant(mod, "which", "A") < 0) {
2447+
Py_XDECREF(mod);
2448+
return NULL;
2449+
}
2450+
return mod;
2451+
}
2452+
2453+
static PyObject*
2454+
PyInit_cmfi_same_name_b(void)
2455+
{
2456+
cmfi_initfunc_b_calls++;
2457+
PyObject *mod = PyModule_Create(&cmfi_same_name_b_def);
2458+
if (mod == NULL || PyModule_AddStringConstant(mod, "which", "B") < 0) {
2459+
Py_XDECREF(mod);
2460+
return NULL;
2461+
}
2462+
return mod;
2463+
}
2464+
2465+
static PyObject*
2466+
create_same_name_b(PyObject* self, PyObject* spec)
2467+
{
2468+
return PyImport_CreateModuleFromInitfunc(spec, PyInit_cmfi_same_name_b);
2469+
}
2470+
2471+
static PyObject*
2472+
initfunc_calls(PyObject* self, PyObject* Py_UNUSED(args))
2473+
{
2474+
return Py_BuildValue("(ii)", cmfi_initfunc_a_calls, cmfi_initfunc_b_calls);
2475+
}
2476+
24252477
// A single-phase init submodule that only knows its short name,
24262478
// like pybind11's PYBIND11_MODULE(sp_submod, m) would.
24272479
static PyModuleDef cmfi_sp_submod_def = {
@@ -2471,6 +2523,10 @@ create_module(PyObject* self, PyObject* spec)
24712523
Py_DECREF(name);
24722524
return PyImport_CreateModuleFromInitfunc(spec, PyInit_embedded_ext);
24732525
}
2526+
if (PyUnicode_EqualToUTF8(name, "same_name")) {
2527+
Py_DECREF(name);
2528+
return PyImport_CreateModuleFromInitfunc(spec, PyInit_cmfi_same_name_a);
2529+
}
24742530
if (PyUnicode_EqualToUTF8(name, "sp_pkg")) {
24752531
Py_DECREF(name);
24762532
return PyImport_CreateModuleFromInitfunc(spec, PyInit_cmfi_sp_pkg);
@@ -2495,6 +2551,8 @@ exec_module(PyObject* self, PyObject* mod)
24952551

24962552
static PyMethodDef create_static_module_methods[] = {
24972553
{"create_module", create_module, METH_O, NULL},
2554+
{"create_same_name_b", create_same_name_b, METH_O, NULL},
2555+
{"initfunc_calls", initfunc_calls, METH_NOARGS, NULL},
24982556
{"exec_module", exec_module, METH_O, NULL},
24992557
{NULL}
25002558
};
@@ -2526,6 +2584,12 @@ test_create_module_from_initfunc(void)
25262584
L"import embedded_ext;"
25272585
L"print(embedded_ext);"
25282586
L"print(f'{embedded_ext.executed=}');"
2587+
// Same name, different initfuncs: the first one wins
2588+
L"spec = spec_from_loader('same_name', StaticExtensionImporter);"
2589+
L"a = create_static_module.create_module(spec);"
2590+
L"b = create_static_module.create_same_name_b(spec);"
2591+
L"print(f'{a.which=} {b.which=} {a is b=}');"
2592+
L"print(f'{create_static_module.initfunc_calls()=}');"
25292593
// Single-phase submodule whose init only knows its short name
25302594
L"import sp_pkg.sp_submod;"
25312595
L"print(sp_pkg.sp_submod);"
@@ -2558,7 +2622,10 @@ test_create_module_from_initfunc(void)
25582622
" return None\n"
25592623
" @staticmethod\n"
25602624
" def create_module(spec):\n"
2561-
" return create_static_module.create_module(spec)\n"
2625+
" mod = create_static_module.create_module(spec)\n"
2626+
" print(f'created {spec.name}: '\n"
2627+
" f'in sys.modules={spec.name in sys.modules}')\n"
2628+
" return mod\n"
25622629
" @staticmethod\n"
25632630
" def exec_module(module):\n"
25642631
" create_static_module.exec_module(module)\n"

0 commit comments

Comments
 (0)