Skip to content

gh-156780: Emscripten: support building CPython with static linking - #156781

Open
clementperon wants to merge 4 commits into
python:mainfrom
clementperon:emscripten-static-linking
Open

clementperon wants to merge 4 commits into
python:mainfrom
clementperon:emscripten-static-linking

Conversation

@clementperon

@clementperon clementperon commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #156780.

Allow Emscripten libpython.a to be embedded without -sMAIN_MODULE or wrapping the application's main(). The wrapper no longer depends on resolveGlobalSymbol().

Keep the JavaScript setup in Python/emscripten_syscalls.c and invoke it from a constructor in Programs/emscripten_beforemain.c, linked only into the interpreter and _testembed. Preserve the existing _emscripten_exit_with_live_runtime() and Module.onExit behavior. Omit the constructor when Emscripten syscall overrides are disabled.

fd_read and poll suspend only when Module.Py_EmscriptenStackSwitching is set. The interpreter sets this flag when entering its promising main(); embedders can opt in when providing their own promising entry point.

Add CI smoke tests for static embedding and for an Emscripten callback completing while main() is suspended. The latter checks that main() resumes and its exit status is preserved.

Validation with Emscripten 6.0.9 and Node 26.8.1:

  • Rebuilt the dynamic-linking interpreter; test_capi.test_emscripten, test_builtin, test_json, test_sys, and test_functools passed. test_select was skipped because socket support was unavailable.
  • Static embedding and suspended-main smoke tests passed. Also checked the suspended-main test with -sMAIN_MODULE; it fails against the previous wrapper and passes with this change.
  • Verified that --disable-emscripten-syscalls omits the constructor and syscall objects from the configuration.
  • Regenerated configure with CPython's regeneration script.

@clementperon

Copy link
Copy Markdown
Contributor Author

@hoodmane 👀 maybe ?

@hoodmane

hoodmane commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Maybe let's start with a PR that adds the EM_JS_DEPS() everywhere? That seems like an easy improvement.

I think rather than using Module.Py_EmscriptenPromisingMain as a runtime flag, it would be more morally upstanding to move that logic into python.o or some other object file we only link into the interpreter program, that way embedders wouldn't get it at all. Maybe we can rename _emscripten_promising_main_js to _PyEmscripten_BeforeMain() or something and move the following to python.c or to its own object file next to it:

#ifdef __EMSCRIPTEN__
__attribute__((constructor)) void _PyEmscripten_Constructor(void) {
   _PyEmscripten_BeforeMain();
}

python.c is currently very short, so I'm not sure how people will feel about adding this there but we should find out.

And of course as traditional we need much less prose since Claude is as rambly as usual. Please rewrite all the comments and the commit message by hand. I find that helps cut them down to a more reasonable length.

Of course, it's likely that any of this will bit rot without test coverage.

cc @freakboy3742 WDYT?

@clementperon

Copy link
Copy Markdown
Contributor Author

@hoodmane as you suggest move to a dedicated object file + reword commit + add a test

Comment thread Misc/NEWS.d/next/Build/2026-09-01-23-32-22.gh-issue-156780.Qw3nRt.rst Outdated
Comment thread Python/emscripten_syscalls.c Outdated
@freakboy3742

Copy link
Copy Markdown
Contributor

Maybe let's start with a PR that adds the EM_JS_DEPS() everywhere? That seems like an easy improvement.

+1 to this as an approach to landing the changes.

I think rather than using Module.Py_EmscriptenPromisingMain as a runtime flag, it would be more morally upstanding to move that logic into python.o
...
python.c is currently very short, so I'm not sure how people will feel about adding this there but we should find out.

I agree that linking into the mainline is a preferable approach to a runtime flag.

As for where to put it - I can definitely see the argument for using python.c, but the fact that it is so minimal at present is at least a cause for a raised eyebrow. Is Modules/main.c perhaps a better candidate?

And of course as traditional we need much less prose since Claude is as rambly as usual. Please rewrite all the comments and the commit message by hand. I find that helps cut them down to a more reasonable length.

+1 to this as well. Claude's predilection for extraneous verbositude is usually contrary to the underlying goals of effective communication :-)

@clementperon
clementperon force-pushed the emscripten-static-linking branch from fa3261b to ee90cfa Compare September 8, 2026 09:00
@clementperon

clementperon commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor Author

@hoodmane EM_JS_DEPS has been merged thanks to @freakboy3742.

I will rework a bit this PR + clean commit message & comment.

@clementperon
clementperon marked this pull request as draft September 8, 2026 10:23
@clementperon
clementperon force-pushed the emscripten-static-linking branch 3 times, most recently from 5417e3e to 7c1f523 Compare September 8, 2026 14:30
@clementperon
clementperon marked this pull request as ready for review September 8, 2026 14:30
@clementperon
clementperon force-pushed the emscripten-static-linking branch from 7c1f523 to 18d7011 Compare September 8, 2026 14:32
@clementperon

clementperon commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor Author

@hoodmane new version after discussion with AI

  • No more Py_EMSCRIPTEN_DYNAMIC_LINKING or -sALLOW_TABLE_GROWTH. The wrapper rebinds _main and, when present, wasmImports.main, so a single code path covers both configurations. The trampoline now takes over a placeholder's table slot instead of calling addFunction(), so the table never grows. The previous static branch also wrapped wasmExports["main"], which does not exist (the export is __main_argc_argv); fixed.
  • Cleaner exit. After main() completes the wrapper goes through exitJS() instead of Module.onExit directly, so atexit handlers run and trailing output is flushed.
  • Module.Py_EmscriptenStackSwitching is set when main() is actually entered rather than in the constructor, has its own commit, and is documented in emscripten_syscalls.c for embedders.
  • FS.createAsyncInputDevice() stays in libpython, so _testinternalcapi keeps working for any libpython consumer.
  • Build fixes: emscripten_beforemain.o depends on the headers, is linked into _testembed, and is installed by libainstall.
  • Embed test links libpython.a directly, is built by run_test.sh instead of every make, checks node's exit status, and is linked with -O2 -g0 (wasm went from 41 MB to 8 MB).

Tested with the CI configuration, a --disable-wasm-dynamic-linking interpreter, and the embed test. PR description updated.

@hoodmane

hoodmane commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

The changes to Python/emscripten_trampoline.c look good to me, let's next split out those into a separate PR.

@clementperon

Copy link
Copy Markdown
Contributor Author

Rebase on top of #157238

@clementperon
clementperon force-pushed the emscripten-static-linking branch from f96de99 to afd1a38 Compare September 16, 2026 20:07
@clementperon

Copy link
Copy Markdown
Contributor Author

@hoodmane can I have again your feedback on this PR please 🙏

Thanks

clementperon and others added 3 commits October 1, 2026 15:14
Running main() under WebAssembly.promising only makes sense when Python
is the program. Move the wrapper into Programs/emscripten_beforemain.c,
linked into the interpreter but not into libpython, and let the runtime
shut down normally afterwards so atexit handlers run and buffered output
is flushed. FS.createAsyncInputDevice() stays in libpython.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…() wrapper

Suspending in fd_read or poll needs a promising entry point, which
libpython no longer sets up itself. Check Module.Py_EmscriptenStackSwitching,
set by the interpreter's wrapper, instead of WebAssembly.promising alone.
An embedder with its own promising entry point sets the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Link libpython.a into a program whose main() is not Python's, without
-sMAIN_MODULE or the interpreter's link flags, and run a script that
imports from the stdlib, calls through the trampoline and polls.
Platforms/emscripten/web_embed_test/run_test.sh builds and runs it in CI.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@clementperon

Copy link
Copy Markdown
Contributor Author

@freakboy3742 what do you think of this PR ? Do you want me to update something ?

Comment thread Programs/emscripten_beforemain.c Outdated
Comment thread Programs/emscripten_beforemain.c Outdated
Comment thread Programs/emscripten_beforemain.c Outdated
Keep the JavaScript setup in emscripten_syscalls.c and only the constructor in the interpreter object. Restore the existing live-runtime exit behavior and omit the constructor when syscall overrides are disabled.

Add a smoke test that resumes main after an Emscripten callback and verifies its exit status.
@clementperon

Copy link
Copy Markdown
Contributor Author

@hoodmane thanks for your review, I hope i understand correctly what you wanted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emscripten: static (non-MAIN_MODULE) builds fail at startup on dynamic-linking-only JS helpers

3 participants