Mobile packaging fixes: iOS interdependent-dylib frameworks + Android native-__init__ packages

src/serious_python/CHANGELOG.md

@@ -1,5 +1,7 @@
 ## 4.3.2
 
+* **iOS:** interdependent bundled dylibs (e.g. pyarrow, llama-cpp-python) no longer crash the app at launch with `dyld: Library not loaded: @rpath/lib<X>.dylib` — the framework install-ids and sibling `@rpath` references are reconciled to the relocated framework paths ([#223](https://github.com/flet-dev/serious-python/issues/223)). See `serious_python_darwin` 4.3.2.
+* **Android:** fix `import <pkg>` yielding an empty module for a package whose `__init__` is itself the native extension (e.g. apsw) — the extension is now resolved from its `<pkg>/__init__.soref` marker. See `serious_python_android` 4.3.2.
 * Bump the bundled python-build snapshot to `20260712`, fixing two on-device startup crashes affecting Python **3.13/3.14** apps (3.12 is unaffected). Bundled Python versions (**3.12.13 / 3.13.14 / 3.14.6**) and `dart_bridge` (**1.5.0**) are unchanged from 4.3.1.
 * **Android:** fix 3.13/3.14 apps crashing with `SIGSYS` before any app code runs on x86_64 (and other ABIs with a 32-bit-style `SYS_open`) — mimalloc's raw `open(2)` syscall is forbidden by Android's seccomp policy. See `serious_python_android` 4.3.2.
 * **iOS/Android:** `_pyrepl` is no longer pruned from the mobile stdlib — Python 3.14's `pdb` imports it at module load, so anything importing `pdb` (e.g. pytest's debugging plugin) failed with `ModuleNotFoundError: No module named '_pyrepl'`. See `serious_python_darwin` / `serious_python_android` 4.3.2.

src/serious_python_android/CHANGELOG.md

@@ -1,5 +1,6 @@
 ## 4.3.2
 
+* **Resolve a package whose `__init__` is itself the native extension.** `_SorefFinder` only probed `<dotted>.soref`, so a package that ships its extension as `<pkg>/__init__.<abi>.so` — e.g. apsw (import name `apsw`), whose relocation marker lands at `apsw/__init__.soref` — was never resolved: `find_spec` returned `None`, the synthesized empty `apsw/__init__.py` won, and `import apsw` yielded an empty module (`AttributeError: module 'apsw' has no attribute 'Connection'`). `find_spec` now falls back to `<dotted>/__init__.soref`, loads the extension under the correct top-level name via `ExtensionFileLoader`, and marks the result a package (with `submodule_search_locations`) so pure-Python submodules (`apsw.ext`, …) still resolve.
 * Bump the bundled python-build snapshot to `20260712`, fixing two on-device crashes on Python **3.13/3.14** (3.12 is unaffected; Python/`dart_bridge` versions are unchanged):
   * Apps died with `SIGSYS` at `dlopen()` of `libpython` — before the interpreter even started — on **x86_64/x86/armeabi-v7a**: mimalloc (bundled with CPython since 3.13) reads `/proc/sys/vm/overcommit_memory` during allocator init via a bare `open(2)` syscall, which Android's bionic seccomp policy forbids (only `openat(2)` is allowed). python-build now patches the call to `SYS_openat(AT_FDCWD, …)`. `arm64-v8a` was latently unaffected (no `SYS_open` there, so mimalloc already went through libc `open()` → `openat`), but emulators are typically x86_64.
   * `_pyrepl` is no longer pruned from the bundled stdlib: Python 3.14's `pdb` imports it at module load, so anything importing `pdb` (e.g. pytest's debugging plugin) died with `ModuleNotFoundError: No module named '_pyrepl'`.

src/serious_python_android/python/_sp_bootstrap.py

@@ -1,23 +1,23 @@
 """serious_python Android import bootstrap.
 
-Installed by the dart-bridge embedder *before* ``site`` runs, via the fixed call::
+Installed by the dart-bridge embedder *before* `site` runs, via the fixed call::
 
     import _sp_bootstrap; _sp_bootstrap.install()
 
-It registers a ``sys.meta_path`` finder that resolves native CPython extension
-modules which the build relocated into ``jniLibs/<abi>/`` as real ``lib<mangled>.so``
+It registers a `sys.meta_path` finder that resolves native CPython extension
+modules which the build relocated into `jniLibs/<abi>/` as real `lib<mangled>.so`
 files (loaded by basename through the Android linker namespace, exactly like
-``libdart_bridge.so``). Pure ``.py``/``.pyc`` modules are left to ``zipimport`` /
-``FileFinder`` — this finder returns ``None`` for them.
+`libdart_bridge.so`). Pure `.py`/`.pyc` modules are left to `zipimport` /
+`FileFinder` — this finder returns `None` for them.
 
-For every relocated extension the build leaves a ``.soref`` marker at the module's
-original path; its content is the ``lib<mangled>.so`` filename. The marker is read
-**lazily** in ``find_spec`` via the frozen ``zipimport`` ``get_data`` API (for zip
-entries) or a plain ``open`` (for entries extracted to disk, e.g. ``extract.zip``).
+For every relocated extension the build leaves a `.soref` marker at the module's
+original path; its content is the `lib<mangled>.so` filename. The marker is read
+**lazily** in `find_spec` via the frozen `zipimport` `get_data` API (for zip
+entries) or a plain `open` (for entries extracted to disk, e.g. `extract.zip`).
 
 CRITICAL: this module must load and run *before any native module is resolvable*,
-so it imports **only builtin/frozen** machinery — ``sys``, ``zipimport``,
-``importlib.machinery`` — and never ``zipfile``/``struct``/``zlib`` (which would be
+so it imports **only builtin/frozen** machinery — `sys`, `zipimport`,
+`importlib.machinery` — and never `zipfile`/`struct`/`zlib` (which would be
 a chicken-and-egg: those are themselves native).
 """
 
@@ -48,7 +48,7 @@ def _apk_native_prefix():
 
 
 class _SorefFinder:
-    """meta_path finder: dotted name -> jniLibs lib via its ``.soref`` marker."""
+    """meta_path finder: dotted name -> jniLibs lib via its `.soref` marker."""
 
     def __init__(self):
         # Cache one zipimporter per zip sys.path entry. Value is None for entries
@@ -69,34 +69,44 @@ def _zipimporter(self, entry):
             return zi
 
     def _read_marker(self, member):
-        """Return the soname recorded in ``member`` (.soref), or None if absent.
-
-        Probes every current ``sys.path`` entry: zip entries via the frozen
-        ``zipimport.get_data`` (known member, no native deps), directory entries
-        via a plain ``open`` (covers packages unpacked from ``extract.zip``).
+        """Return `(soref_bytes, sys.path entry)` for `member`, or
+        `(None, None)` if absent.
+
+        Probes every current `sys.path` entry: zip entries via the frozen
+        `zipimport.get_data` (known member, no native deps), directory entries
+        via a plain `open` (covers packages unpacked from `extract.zip`). The
+        winning entry is returned too so a package whose `__init__` is the
+        native extension can locate its pure-Python submodules beside it.
         """
         for entry in sys.path:
             if not entry:
                 continue
             zi = self._zipimporter(entry)
             if zi is not None:
                 try:
-                    return zi.get_data(member)  # archive-relative member path
+                    return zi.get_data(member), entry  # archive-relative member
                 except Exception:
                     continue
             else:
                 # Directory entry: try the marker as a real file on disk.
                 path = entry + "/" + member
                 try:
                     with open(path, "rb") as f:
-                        return f.read()
+                        return f.read(), entry
                 except OSError:
                     continue
-        return None
+        return None, None
 
     def find_spec(self, fullname, path=None, target=None):
-        member = fullname.replace(".", "/") + _MARKER_SUFFIX
-        data = self._read_marker(member)
+        base = fullname.replace(".", "/")
+        # A plain extension module: its marker is "<dotted>.soref".
+        data, entry = self._read_marker(base + _MARKER_SUFFIX)
+        is_package = False
+        if data is None:
+            # A package whose __init__ IS the native extension (e.g. apsw ships
+            # apsw/__init__.<abi>.so): the marker sits at "<dotted>/__init__.soref".
+            data, entry = self._read_marker(base + "/__init__" + _MARKER_SUFFIX)
+            is_package = data is not None
         if data is None:
             return None  # not a relocated native module -> let others handle it
         soname = data.decode("utf-8").strip()
@@ -115,20 +125,31 @@ def find_spec(self, fullname, path=None, target=None):
         if origin == soname and self._apk_prefix:
             origin = self._apk_prefix + soname
         loader = ExtensionFileLoader(fullname, origin)
-        return ModuleSpec(fullname, loader, origin=origin)
+        spec = ModuleSpec(fullname, loader, origin=origin)
+        if is_package and entry is not None:
+            # The native __init__ lives in jniLibs, but the package's pure-Python
+            # submodules (e.g. apsw.ext) sit at "<entry>/<dotted>/" in the winning
+            # sys.path entry (sitepackages.zip or an extract.zip dir). Point
+            # __path__ there so `import <pkg>.<sub>` resolves via the normal
+            # zipimport/FileFinder machinery.
+            spec.submodule_search_locations = [entry + "/" + base]
+        return spec
 
 
 def install():
-    """Insert the finder at the front of ``sys.meta_path`` (idempotent)."""
+    """Insert the finder at the front of `sys.meta_path` (idempotent)."""
     global _installed
     if _installed:
         return
     # Bootstrap audit: any extension module (.so) already imported at this point was
     # loaded during interpreter core-init, BEFORE the finder existed — which only works
     # if it was builtin/frozen. A non-empty list means that module must be made static
     # (PyImport_AppendInittab) or it will fail under modern packaging. Expected: empty.
-    pre = [n for n, m in sys.modules.items()
-           if getattr(m, "__file__", None) and str(m.__file__).endswith(".so")]
+    pre = [
+        n
+        for n, m in sys.modules.items()
+        if getattr(m, "__file__", None) and str(m.__file__).endswith(".so")
+    ]
     if pre:
         sys.stderr.write("SP_BOOTSTRAP pre-finder native modules: %r\n" % (pre,))
     for f in sys.meta_path:

src/serious_python_darwin/CHANGELOG.md

@@ -1,5 +1,7 @@
 ## 4.3.2
 
+* **iOS: reconcile framework install-names for interdependent bundled dylibs** ([#223](https://github.com/flet-dev/serious-python/issues/223)). Site-package `.so`/`.dylib`s are wrapped into frameworks named by their dotted relative path (`opt/lib/libarrow.dylib` → `opt.lib.libarrow.framework/opt.lib.libarrow`), but the Mach-O install-id and every interdependent `@rpath` reference were left at their original bare name (`@rpath/libarrow.dylib`). Because each framework is a `Package.swift` binaryTarget linked at launch, dyld could not resolve `@rpath/libarrow.dylib` (it looks for `Frameworks/libarrow.dylib`, which does not exist) and the app crashed **before Python started** — hitting any package that bundles a chain of interdependent libs (pyarrow's `libarrow`/`libarrow_compute`/`libarrow_python`, llama-cpp-python's `libggml*`/`libllama`). A new reconcile pass, run after `sync_site_packages` frameworks the libs, sets each framework's own install-id to `@rpath/<fw>.framework/<fw>` and rewrites every dependency pointing at a sibling's old id to that framework path, then re-signs. The Python/stdlib xcframeworks are left untouched. (This supersedes the 4.2.1 approach of preserving `.dylib` install-names, which only worked when every sibling happened to be loaded first.)
+* The reconcile pass fails the build — rather than silently swallowing — on a genuine `install_name_tool`/`codesign` error, notably insufficient Mach-O header space to grow a load command (which would otherwise leave a bare `@rpath` ref and reproduce the launch crash), and records every framework slice's old install-id so a lib with divergent per-slice install names has all slices rewritten.
 * Bump the bundled python-build snapshot to `20260712`: `_pyrepl` is no longer pruned from the bundled stdlib. Python **3.14**'s `pdb` imports `_pyrepl` at module load, so anything importing `pdb` (e.g. pytest's debugging plugin) died with `ModuleNotFoundError: No module named '_pyrepl'` on iOS (3.13's `pdb` doesn't import it). Python/`dart_bridge` versions are unchanged.
 
 ## 4.3.1

src/serious_python_darwin/darwin/sync_site_packages.sh

@@ -49,6 +49,19 @@ if [[ -n "$SERIOUS_PYTHON_SITE_PACKAGES" && -d "$SERIOUS_PYTHON_SITE_PACKAGES" ]
         done
         done
 
+        # After every .so/.dylib is framework-ized, reconcile the Mach-O
+        # install names so the interdependent @rpath refs point at the new
+        # framework paths (serious-python #223). Without this, dyld cannot
+        # resolve e.g. @rpath/libarrow.dylib at launch and the app crashes
+        # before Python starts. Exclude the python/stdlib xcframeworks copied
+        # in above -- they are already correct.
+        # A reconcile failure (e.g. a Mach-O with no header space to grow a load
+        # command, or a signing error) would otherwise ship an app that crashes
+        # at launch -- abort the build instead. sync_site_packages.sh has no
+        # set -e; prepare_spm.sh runs it under set -euo pipefail, so a non-zero
+        # exit here fails `flet build` loudly.
+        reconcile_framework_install_names "$dist/site-xcframeworks" "$dist/python-xcframeworks" || exit 1
+
         rm -rf $dist/site-packages
         mkdir -p $dist/site-packages
         cp -R $tmp_dir/${archs[0]}/* $dist/site-packages

src/serious_python_darwin/darwin/xcframework_utils.sh

@@ -104,4 +104,98 @@ create_xcframework_from_dylibs() {
     # cleanup
     popd >/dev/null
     rm -rf "${dylib_tmp_dir}" >/dev/null
+}
+
+# Reconcile install names across the newly-created site-package frameworks.
+#
+# create_xcframework_from_dylibs renames each lib to a framework named by its
+# dotted relative path (opt/lib/libarrow.dylib -> opt.lib.libarrow.framework/
+# opt.lib.libarrow), but leaves the Mach-O install-id and every interdependent
+# @rpath reference at their ORIGINAL bare names (e.g. @rpath/libarrow.dylib):
+# it only rewrites the install-id, and only for ext=so. dyld links every one of
+# these frameworks at app launch (each is a Package.swift binaryTarget the
+# plugin depends on), so a bare @rpath/libarrow.dylib resolves to
+# Frameworks/libarrow.dylib -- which does not exist -- and the app crashes
+# BEFORE Python starts. See serious-python #223.
+#
+# This pass makes the install names match the framework layout:
+#   1. set every framework binary's own install-id to @rpath/<fw>.framework/<fw>
+#   2. rewrite every dep that pointed at a sibling's OLD id to that sibling's
+#      framework path, so interdependent libs (libarrow_python -> libarrow, or a
+#      C-extension -> its bundled .dylib) resolve at launch.
+# Only the frameworks created from site-packages are touched; the Python /
+# stdlib xcframeworks (passed as $2) are already correct and left untouched.
+reconcile_framework_install_names() {
+    local xcframeworks_dir=$1
+    local exclude_dir=$2
+
+    local -a map_old=()
+    local -a map_new=()
+
+    # Pass 1: set each framework's own install-id to @rpath/<fw>.framework/<fw>,
+    # and record old-id -> new-id for the dep rewrite below. Read the old id from
+    # EVERY slice, not just the first: a lib whose slices carry divergent install
+    # names (e.g. an upstream build that baked an arch-specific absolute path
+    # instead of an @rpath id) would otherwise leave the other slices' deps
+    # unrewritten. Distinct old ids all map to the same new id.
+    local xcf fw newid bin oldid raw err j found
+    for xcf in "$xcframeworks_dir"/*.xcframework; do
+        [ -d "$xcf" ] || continue
+        fw=$(basename "$xcf" .xcframework)
+        [ -n "$exclude_dir" ] && [ -e "$exclude_dir/$fw.xcframework" ] && continue
+        newid="@rpath/$fw.framework/$fw"
+        for bin in "$xcf"/*/"$fw.framework/$fw"; do
+            [ -f "$bin" ] || continue
+            # Buffer otool output before filtering: piping otool straight into
+            # `head -1` lets head close the pipe early, and the SIGPIPE race
+            # intermittently drops the first read (empty oldid).
+            raw=$(otool -D "$bin" 2>/dev/null)
+            oldid=$(printf '%s\n' "$raw" | grep -v ':$' | grep -vi 'Architectures in' | head -1 | sed 's/^[[:space:]]*//')
+            # -id must succeed; a failure means the binary is unwritable/corrupt
+            # and the app would crash at launch, so surface it instead of hiding
+            # it behind `|| true` (stderr is captured and only printed on error,
+            # to keep the expected "will invalidate the code signature" warning
+            # off the build log).
+            if ! err=$(install_name_tool -id "$newid" "$bin" 2>&1); then
+                echo "reconcile_framework_install_names: install_name_tool -id failed for $bin: $err" >&2
+                return 1
+            fi
+            if [ -n "$oldid" ] && [ "$oldid" != "$newid" ]; then
+                found=0
+                for j in ${map_old[@]+"${map_old[@]}"}; do
+                    [ "$j" = "$oldid" ] && { found=1; break; }
+                done
+                [ "$found" -eq 0 ] && { map_old+=("$oldid"); map_new+=("$newid"); }
+            fi
+        done
+    done
+
+    # Pass 2: rewrite each binary's deps that point at a sibling's old id to that
+    # sibling's framework path, then re-sign (install_name_tool invalidates the
+    # ad-hoc signature). `install_name_tool -change` is a no-op returning 0 when
+    # the dep is absent, so a NON-zero rc means a dep that IS present could not be
+    # rewritten -- usually no Mach-O header space to grow the load command. That
+    # is fatal: it would leave a bare @rpath ref and reproduce the exact launch
+    # crash this pass exists to prevent, so fail the build rather than ship it.
+    local i n=${#map_old[@]}
+    for xcf in "$xcframeworks_dir"/*.xcframework; do
+        [ -d "$xcf" ] || continue
+        fw=$(basename "$xcf" .xcframework)
+        [ -n "$exclude_dir" ] && [ -e "$exclude_dir/$fw.xcframework" ] && continue
+        for bin in "$xcf"/*/"$fw.framework/$fw"; do
+            [ -f "$bin" ] || continue
+            i=0
+            while [ $i -lt $n ]; do
+                if ! err=$(install_name_tool -change "${map_old[$i]}" "${map_new[$i]}" "$bin" 2>&1); then
+                    echo "reconcile_framework_install_names: install_name_tool -change '${map_old[$i]}' -> '${map_new[$i]}' failed for $bin (no Mach-O header space?): $err" >&2
+                    return 1
+                fi
+                i=$((i+1))
+            done
+            if ! err=$(codesign --force --sign - "$bin" 2>&1); then
+                echo "reconcile_framework_install_names: codesign failed for $bin: $err" >&2
+                return 1
+            fi
+        done
+    done
 }