Mobile packaging fixes: iOS interdependent-dylib frameworks + Android native-__init__ packages#232
Merged
Merged
Conversation
_SorefFinder only probed "<dotted>.soref" for a relocated native module, so a package whose __init__ is itself the extension (e.g. apsw ships apsw/__init__.<abi>.so, dotted import name `apsw`) was never resolved: the marker for it is written at "<dotted>/__init__.soref", not "<dotted>.soref". find_spec returned None, the synthesized empty apsw/__init__.py won, and `import apsw` yielded an empty module -> AttributeError on apsw.Connection / apsw.apswversion(). find_spec now falls back to "<dotted>/__init__.soref" when the plain marker misses, loads the extension under the correct top-level name via ExtensionFileLoader (its PyInit_<name> matches, e.g. PyInit_apsw), and marks the result a package with submodule_search_locations pointing at the package dir in the winning sys.path entry (sitepackages.zip or an extract.zip dir) so pure- Python submodules (apsw.ext, apsw._unicode, ...) resolve normally. _read_marker now also returns the winning entry for that purpose. Verified on-device (Android arm64, Flet 0.86): apsw 3.53.2.0 imports and runs a full in-memory SQLite CREATE/INSERT/SELECT round-trip (2/2 recipe tests pass); previously both failed with the empty-__init__ AttributeError.
…223) After sync_site_packages framework-izes each site-package .so/.dylib into a framework named by its dotted relative path (opt/lib/libarrow.dylib -> opt.lib.libarrow.framework/opt.lib.libarrow), the Mach-O install-ids and every interdependent @rpath reference were left at their original bare names (@rpath/libarrow.dylib). Because each framework is a Package.swift binaryTarget the plugin depends on, dyld links them all at launch, and a bare @rpath/libarrow.dylib resolves to Frameworks/libarrow.dylib -- which does not exist -- so the app crashes before Python starts (e.g. pyarrow, llama-cpp). Add reconcile_framework_install_names(): after the conversion loop, set each framework binary's own install-id to @rpath/<fw>.framework/<fw> and rewrite every dep that pointed at a sibling's old id to that sibling's framework path, then re-sign. The python/stdlib xcframeworks are already correct and excluded. Verified locally: pyarrow 24.0.0 on iOS simulator now launches and passes 4/4 tests (was a dyld launch crash: Library not loaded @rpath/libarrow.dylib).
…ce ids) Two robustness fixes to the #223 reconcile pass (both from code review): 1. Stop swallowing real failures. The install_name_tool/-id/-change and codesign calls used `2>/dev/null || true`, which also hid genuine failures — most importantly install_name_tool's 'larger updated load commands do not fit' (a Mach-O with no header space to grow a load command). A swallowed -change leaves a bare @rpath ref and reproduces the exact dyld launch crash this pass exists to prevent, silently. Now: -id, a *present-dep* -change (absent deps are still a rc-0 no-op), and codesign failures are fatal — the function returns non-zero with a clear message, and sync_site_packages.sh propagates it (`|| exit 1`) so `flet build` fails loudly instead of shipping a broken app. stderr is captured and only printed on error, so the expected 'will invalidate the code signature' warning stays off the build log. 2. Record every slice's old install-id, not just the first. Pass 1 read oldid only from the first slice; a lib whose slices carry divergent install names (e.g. an arch-specific absolute build path instead of an @rpath id) would leave the other slices' deps unrewritten. Now each distinct old id is mapped (deduped) to the same new framework id. Verified: happy path (padded dylibs) + the real llama ggml->llama chain both reconcile to rc 0 with no bare refs; a divergent-slice-id pair rewrites both slices; an unpadded binary now fails loudly (rc 1) instead of silently.
…o the unreleased 4.3.2 4.3.2 is not yet released, so these packaging fixes ship in it rather than a new version: darwin gets the interdependent-dylib framework reconcile (#223) + its fail-loud/all-slice hardening; android gets the native-__init__ soref resolution (apsw); the main package aggregates both. Versions unchanged (already 4.3.2).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent Flet 0.86 mobile-packaging fixes, surfaced while building native-heavy wheels (pyarrow, llama-cpp-python, apsw).
1. iOS — reconcile framework install-names for interdependent dylibs · fixes #223
create_xcframework_from_dylibsrenames each bundled.so/.dylibto a framework named by its dotted relative path (opt/lib/libarrow.dylib→opt.lib.libarrow.framework/opt.lib.libarrow) but leaves every Mach-O install-id and interdependent@rpathreference at its original bare name (@rpath/libarrow.dylib) — it only runsinstall_name_tool -id, and only for.so. Each framework is aPackage.swiftbinaryTarget the plugin links at launch, so dyld can't resolve@rpath/libarrow.dylib(it looks forFrameworks/libarrow.dylib, which doesn't exist) and the app crashes before Python starts:Adds
reconcile_framework_install_names()(xcframework_utils.sh), called fromsync_site_packages.shafter the conversion loop: it sets each framework binary's own install-id to@rpath/<fw>.framework/<fw>and rewrites every dependency that pointed at a sibling's old id to that framework path, then re-signs. The Python/stdlib xcframeworks are already correct and excluded.Affects any package bundling a chain of interdependent dylibs — pyarrow (
libarrow←libarrow_compute←libarrow_python+ thepyarrowC-extensions), llama-cpp-python (libggml-base←libggml-cpu←libggml←libllama).2. Android — resolve a package whose
__init__IS a native extension_SorefFinderonly probed<dotted>.soref. A package whose__init__is itself the extension — apsw shipsapsw/__init__.<abi>.so, import nameapsw— has its marker at<dotted>/__init__.soref, sofind_specreturnedNone, the synthesized emptyapsw/__init__.pywon, andimport apswyielded an empty module (AttributeError: module 'apsw' has no attribute 'Connection').find_specnow falls back to<dotted>/__init__.soref, loads the extension under the correct top-level name viaExtensionFileLoader, and marks it a package withsubmodule_search_locationsso pure-Python submodules (apsw.ext, …) still resolve.Testing
Verified through the mobile-forge recipe-tester (Flet 0.86, py3.12), pinning
serious_pythonto this branch via the app's Flutterdependency_overrides:.dyliblibs so they load on the iOS simulator #223 dyld launch crash)test_native_lib_callablepasses