feat(rust) add native link option - #1685
Conversation
|
Could you address the question I had in the other PR? This seems like a lot to take on in terms of maintainership here because only codegen is tested, not any runtime parts, and since that comment the component model has continued to add features like async which I'm not sure how would map to native counterparts. |
|
Sure! I'll respond to the previous comment.
Yes. This has been demonstrated in the Pumpkin PR I shared. It allows for native and wasm targets to be compiled interchangeably with no code changes from the plugin developer.
The reason a direct native call doesn't work here is that the guest is compiled separately from the host and loaded with dlopen. Rust has no stable ABI, so passing native types like String or Vec across that boundary, potentially across different rustc versions, or from a non-Rust host won't work. The canonical ABI is already implemented by wit-bindgen and tapping into that is easy. We wouldn't really have to "serialize/deserialize." Nothing is encoded to a buffer. Scalars pass as-is and strings cross as pointer+length, so simple calls are effectively direct calls already. Reusing the existing ABI also keeps the change small. The existing codegen already emits the full ABI representation on native targets (the unreachable!() stubs sit after all of that), so this PR just swaps the stub for a dispatch through a registered function pointer and hex-encodes the symbol names. Only around +130 lines of actual generator code. Most of that is the register hooks, which exist so the host doesn't have to re-export every import symbol into the dlopen'd library (the -rdynamic problem) and can implement only the subset of imports it needs. And because it's the same underlying C ABI emission, async, resources, and every other WIT construct should work without a parallel code path. Which is why I didn't really see the need to test the runtime part, since whatever ABI works on wasm should just work with native. A direct-call mode (traits with idiomatic signatures) is imaginable, but I don't think it fits wit-bindgen's shape. There's no language-neutral contract for it to target, so it would be a separate parallel bindings mode per generator, none of which could interoperate across the dlopen or language boundary. The canonical ABI is the one representation every generator here already shares. Which is what lets, e.g., a C++ host load a Rust plugin under this scheme. Theoretically this could be added to the other generators as well. |
|
@cpetig I know you've done similar work, so it'd be great if you could weigh in here as well. |
|
Wow, good to see growing interest in applying the component model to native binaries. I have a long living fork of wit-bindgen at https://github.com/cpetig/wit-bindgen/ which adds
The symmetric ABI is what makes the real difference in usability as the (also mental) overhead for (maintaining) a host mode (mesh) code generator is significant. I guess we should have a discussion about standardizing native names in a |
cpetig
left a comment
cpetig
left a comment
There was a problem hiding this comment.
I feel that the amount of code changes can be reduced by making the native symbol name generation and target-conditional linker attributes the default behavior. This is how I would solve this.
| /// // still does; use `type_section_suffix` to tell them apart. See | ||
| /// // `wit_bindgen_rust::Opts::link_native_symbols` for the full list of | ||
| /// // symbols a host can expect. | ||
| /// link_native_symbols: true, |
There was a problem hiding this comment.
(written before I saw that your code is likely the same when the native flag is turned on)
I think my approach of using conditional compilation to choose between both naming schemes at compilation time of identical generated code is preferable. See e.g. https://github.com/cpetig/wit-bindgen/blob/work-in-progress/crates/cpp/tests/symmetric_future/future/src/future_world.rs#L17-L24 for an example.
| /// `#`, `[` and `]` characters that canonical names contain. Names that | ||
| /// survive encoding unchanged (`$root` exports, for instance) are emitted | ||
| /// once with no `cfg` rather than twice. | ||
| fn core_export_symbols(&self, export_name: &str) -> Vec<(&'static str, String)> { |
There was a problem hiding this comment.
ok, I see, this is additive, so the generated code is unchanged in the default case.
I will leave this decision to Alex, but for my branch I decided that it should become the default, simplifying the generation logic and making the flag unnecessary. (It only adds visual clutter to the generated code, no runtime effects)
There was a problem hiding this comment.
I pushed some changes that now make it the default as well.
|
@cpetig thank you for the review! I hadn't realized you had a fork tackling this issue as well, and I would love to combine our work. We're already aligned on naming (the hex encoding this PR moves into wit-bindgen-core is the same scheme your C++ generator uses). I've also taken your suggestion. Native symbol name generation and target-conditional linker attributes are now the default behavior, and the link_native_symbols option is gone entirely. Exports are emitted once with paired cfg_attr(target_arch = "wasm32", ...) export names. The one place I deliberately differ from your fork is imports. Instead of undefined externs, the old unreachable!() slot is filled with a dispatch through a _wit_bindgen_register(func: unsafe extern "C" fn(...)) hook. That keeps the default free for existing users (bindings crates still build and link everywhere on host targets. No import stub libraries needed, and calling an unregistered import aborts with a message, same as the stub did), and it's what makes runtime-discovered plugins work: a dlopen'd cdylib can't resolve import symbols against the host executable without -rdynamic on Linux, and can't on Windows at all, which is why I abandoned #1565. The hooks are convention-agnostic. They just carry a core-signature function pointer, so a symmetric mode could register through the same mechanism later. On the symmetric ABI, I had considered something similar for this PR, but I was worried about whether something requiring maintenance at that scale would be accepted, which is why I settled on the existing wasm32 ABI. It also helped that wasmtime's bindgen existed as a reference for making a fully working native host runtime (native-wit). A lot of our changes are shared regardless. Would it make sense to land this PR as the common substrate (standardized names, no stubs, a portable dispatch mechanism), and then adopt the symmetric ABI as an opt-in mode once the BuildTargets.md discussion stabilizes? Happy to help draft the naming section there. The encoding and hook naming in this PR could serve as input. |
|
Hmm seems like a single unit test fails due to a naming conflict after my changes. I just had an idea how to fix this that might make naming resolution never a problem and also simplify the rest of the code. I'll probably push some changes tomorrow, but for now I would love to hear your input! |
4 participants
Addresses bytecodealliance/wit-bindgen#1062. Currently the Rust bindings stub with
unreachable!()when compiling to wasm. This PR makes bindings usable on native targets, so people can test and run component code in ordinary native environments.This is a new approach to what was initially done in #1565.
The reason this PR needs to exist is described by a comment in the original PR
To solve this issue I've added a register function which can add a pointer to the host function. This fixes the above issue.
Testing
The Pumpkin PR is interesting because the approach shows how one can make very minimal changes and get wasmtime and native runtimes working at the same time.