Implement canonical interface names in wit-component

[chenyan2002]

Follow-up to #2556.

• Add resolve.use_canonical_names in wit-parser to merge interfaces with the same canonical version.
• Propagate versionsuffix to wit-component, so that we can link two interfaces via the canonical version name.
• Add a config flag semver_compat = none|merge|canonical to wasm-tools component new to control the merging behavior. Deprecate the old merge_imports_based_on_semver flag (equivalent to semver_compat=merge).

This is a breaking change. Notably,

• import a:b/c@0.1.1; import a:b/c@0.1.2; would fail under the new feature flag, because of duplicate imports. Users can use the implements feature to update the wit file if needed. See the merge-import-versions test.
• When printing the wit file, the minor version may change depending on which package version gets linked. For example import a:b/c@0.1.1 may become import a:b/c@0.1.2 if the wit package from deps/ is versioned at 0.1.2. See the canon-names-merge test.

Detailed changes

• Implements PackageName::canon_version_split to split canonical version and its version suffix.
• Defined a new wrapper type wit_parser::PackageKey, so that we can control whether to use the canonical version or not during construction time. Resolve will use PackageKey as the map key, instead of PackageName. Gradually, we can migrate all uses of PackageName to PackageKey.
• sort_unresolved_packages merges packages that share a canonical name, keeping the larger version in Resolve. This behavior needs to be made explicit in the spec.
• encode_interface() and encode_world() consistently use canonical names with version suffixes in the binary encoding.
• Tests are mostly generated by Claude with the following refactor:
  • Existing tests run in both modes (with and without canon-names feature)
  • Tests that produce different output under canon-names feature use *.canon-names.* alternate blessed files
  • In wit-component, test names with canon-names prefix are only run when canon-names feature is enabled.

Things left for future PRs

• Validate a given prefix/suffix combo is well-formed.
• In Resolve, when merging two canonical versions, check if the interfaces really conform to the subtyping relation.
• When both implement and versionsuffix are present, the versionsuffix should refer to the version from implement, instead of the main package name. Need to clarify this from the spec as well.
• Some ergonomic feature to auto-convert legacy wasm modules into the canon-names format.

[alexcrichton]

Thanks for the PR! Before diving too deep into review though I want to clarify a few things first. Primarily I don't think this'll work if it's a crate or a runtime flag feature unfortunately. We need a way to roll this out gradually which means that the previous implementation has to live side-by-side with the new implementation, and then eventually we can slowly transition everything over. Could you dig in a bit more to see if this is possible? If it's not possible that'd be somewhat surprising to me, so could you explain a bit more?

Second is that I'm a bit confused by the breaking change you mentioned here -- I would expect being able to import 0.0.1 and 0.0.2 at the same time to work out. This is similar to importing both WASIp2 and WASIp3 APIs which is intended to work.

[chenyan2002]

We need a way to roll this out gradually which means that the previous implementation has to live side-by-side with the new implementation, and then eventually we can slowly transition everything over.

You mean we need a runtime/CLI flag, instead of a feature flag? That's possible, but less cleaner. Primarily, we would like to define a custom Hash/Eq/Ord for PackageName like this: https://github.com/bytecodealliance/wasm-tools/pull/2602/changes#diff-5b7c6c5cfc5aaf75afd5c7532b27923eaabd7ef9a2a8a4df5636ef1796c927a0R261. Instead of a feature flag, I guess I can use a env var, or a global state that indicates which flag we pick?

If we opt to a runtime flag, like --enable-canon-names, we probably cannot define a custom Hash/Eq trait for PackageName. Then the merging logic would have to live inside wit-component and wit-parser somewhere. The down side is that the upstream libraries, like wac, would have to implement their own canonical version logic, instead of relying on the default Hash/Eq trait. Not sure how important is it to bake the canonical version logic into the PackageName natively in Hash/Eq .vs. let the upstream libraries do their own thing. My intuition is that the goal of canonical names is to make the upstream users' life easier, so that we want to implement all the canonicalization logic inside wit-parser and wit-component.

import 0.0.1 and 0.0.2

My bad. It should be 0.1.1 and 0.1.2, which has the same canonical version. Updated the description as well.

[chenyan2002]

I think the question comes down to this: after we fully switch over, what's the expected behavior of HashMap<PackageName, _>. Do we expect the key to be the canonicalized version or the full version?

During the transition period, we can probably wrap the PackageName struct and use a flag to decide its behavior. Something like this:

impl PackageKey {
  fn new(pkg: &PackageName, use_canonical: bool) -> Self
}

Update: this new function is actually better. It decides the behavior at construction time and we can rely on the regular Hash/Eq trait impl. I will go ahead and implement this.

[alexcrichton]

Personally what I'd expect is a configuration option on ComponentEncoder which would get reflected as a configuration flag on wasm-tools component new to use canonical names. This would be off-by-default because it's still an unstable feature, but the same wasm-tools binary or wit-component library would be capable of building both artifacts. This CLI flag would similarly then get reflected into wasm-component-ld to be available for languages like Rust/wasi-sdk.

Internally though I don't think that updating Hash/Eq on PackageName is the right way to go. We will still want the ability to handle a situation where the Rust standard library uses WASI 0.3.1 and the wasip3 crate manually used in an application uses WASI 0.3.2 or something like that. As-written where this rejects that in Resolve I don't think that's the way to go.

What I would roughly envision for this is that wit-component would get more complicated (I realize it's already quite complicated), but largely wit-parser wouldn't change. With wac I'm not 100% sure what the impact or best implementation route would be, however.

[chenyan2002]

I add a new CLI flag semver_compat = none|merge|canonical in wasm-tools component new reflect this. No more feature flags now.

I defined a new wrapper type PackageKey, so that we can control whether to use the canonical version or not during construction time. Resolve will use PackageKey as the map key, instead of PackageName. Gradually, we can migrate all uses of PackageName to PackageKey.

We will still want the ability to handle a situation where the Rust standard library uses WASI 0.3.1 and the wasip3 crate manually used in an application uses WASI 0.3.2 or something like that. As-written where this rejects that in Resolve I don't think that's the way to go.

In this case, Resolve will merge the version to the higher full version, which is 0.3.2. It won't reject the insertion. See here. The only place we reject it is when the user imports both 0.3.1 and 0.3.2 in a single world, which results in duplicate imports. Merging two worlds in Resolve will only unify the package to the larger version.

[alexcrichton]

Reading over this, how come there needs to be a flag in Resolve? That is semantically a very big change for Resolve since flipping that bool drastically changes loading/lookup behavior. What I would have otherwise expected is that wit-component gets a flag to emit canonical names, and perhaps encoding wit-to-wasm additionally gets a flag to emit canonical names, but I wouldn't expect Resolve to need to change all that much. The changes here seem like they're sort of half-reimplementing the merge-based-on-semver method and that seems ideal to keep that separate. Additionally it seems like that would enable loading two versions in one world, which seems like something we'll want to preserve to me.

[chenyan2002]

Perhaps it's easier to understand from bytecodealliance/wit-bindgen#1686, with the flag in Resolve, wit-bindgen automatically gets the canonical version support. I would expect we can do similar things for wac as well.

From the spec's point of view, canonical version essentially renames the package name to exclude the version suffix. This means that the Resolve behavior is going to change, because we are now loading a wit file with a different name, e.g., from a:b@1.0.0 to a:b@1, and a:b@1.0.1 to a:b@1. This necessarily means it's a new behavior in Resolve.

The changes here seem like they're sort of half-reimplementing the merge-based-on-semver method and that seems ideal to keep that separate.

With the changes in Resolve, merge-based-on-semver becomes a no-op, because there are no multiple packages with the same canonical version in Resolve, so there is nothing to merge, and this simplifies the logic in wit-component, as well as the upstream libraries like wit-bindgen and wac.

Additionally it seems like that would enable loading two versions in one world, which seems like something we'll want to preserve to me.

Not sure what you mean. We can do import a:b@1.0.0; import a:b@2.0.0; in one world. But we cannot do import 1.0.0 and 1.0.1 in one world. Do you mean we want to preserve that ability? If we follow the spec, I think 1.0.0 and 1.0.1 would be considered as the same package under the canonical version, so it's essentially duplicate imports?

[alexcrichton]

Personally at least I'm viewing this feature of the component model specification differently (I think) which I believe is what leads to a lot of my comments. I'm going to explain my thinking here before answering some of your points as I think it'll help convey better where I'm coming from.

I see this feature of canonical names in the component model as simplifying the story for dealing with situations like composition, writing a runtime, and knowing what to supply as imports during instantiation. Right now Wasmtime sort of silently matches versions in its component::Linker where if the host provides wasi:cli/exit@0.2.0 and the guest wants wasi:cli/exit@0.2.1 those get linked up just fine despite having differing strings. This causes headaches for any tools other than Wasmtime, like wac, where now it has to replicate all of the same logic for dealing with linking components together. In essence this change will mean that import strings are just wasi:cli/exit@0.2 meaning there's no need to do version matching. In particular though WITs aren't changing and we're not dropping information from components. The full version is always still there it's just that the actual literal import string name isn't always the full version. Notably WIT packages still have the full version, there's still point releases of WASI, etc.

This then leaves the question of where all this is handled from a unifying perspective. Today in wit-component it's already merging everything together based on semver. This handles the case where wasi-libc, for example, binds against WASI 0.2.1, the Rust standard library binds against WASI 0.2.2, and then a user could use the wasip2 crate which links against WASI 0.2.3. In this situation when wit-component runs it'll have have type information for all of these versions of WASI. Then wit-component will implicitly unify these all together into one WASI 0.2.3 import by default (but that's a flag). With the canonical names feature wit-component is basically required to do this unification to emit canonical names as opposed to having it be optional.

One important point though is that this base situation isn't changing. When creating a component there's still type information from many sources at possible different versions, but all still semver-compatible versions. This means that something in the componentization process is still going to need to do some union-ing, no matter what. For bindings generators this technically doesn't need to affect anything. They're still emitting core wasm imports with whatever names wit-component recognizes, and already wit-component's matching is far beyond what the spec writes down and includes the full version right now.

With all that in mind, I want to clarify that the way I'm approaching this canonical names feature is that most of the process of handling WITs, bindings generation, etc, doesn't need to change. AFAIK everything is already pretty robust and working as intended, and the main feature that canonical-names is changing is the "last mile" of sorts where the actual literal names in the component are differing. This'll enable hosts or component-handling tools like Wasmtime and wac to start ignoring semver entirely and only think about string matching.

---

Ok so with that background to answer some of your points:

Perhaps it's easier to understand from bytecodealliance/wit-bindgen#1686, with the flag in Resolve, wit-bindgen automatically gets the canonical version support. I would expect we can do similar things for wac as well.

From the spec's point of view, canonical version essentially renames the package name to exclude the version suffix. This means that the Resolve behavior is going to change, because we are now loading a wit file with a different name, e.g., from a:b@1.0.0 to a:b@1, and a:b@1.0.1 to a:b@1. This necessarily means it's a new behavior in Resolve.

This is not what I would have imagined for wit-bindgen. Internally a Resolve to me represents distinct WIT packages and that has nothing to do with canonical names. WASI 0.2.1 and WASI 0.2.2 are distinct publications of packages which, in my opinion, should be simultaneously representable in WIT. I don't think that Resolve sould be changing at all because the purpose of canonical names is to, in theory, only change this "last mile" of handling components as opposed to every layer of the stack.

By putting this in Resolve it has a much larger affect on version resolution in WITs, for example, which is quite different than what happens today. Put another way what I'd expect is that there'd be a bindgen option to emit canonical names in the core wasm imports, for example shortening import names, but that's by-and-large entirely optional since core wasm import names are effectively orthogonal from the component import names. By unifying it'd make things a little bit simpler, but not a ton. Overall emitting a component with canonical names shouldn't require any bindgen changes at all.

The changes here seem like they're sort of half-reimplementing the merge-based-on-semver method and that seems ideal to keep that separate.

With the changes in Resolve, merge-based-on-semver becomes a no-op, because there are no multiple packages with the same canonical version in Resolve, so there is nothing to merge, and this simplifies the logic in wit-component, as well as the upstream libraries like wit-bindgen and wac.

I understand that merge-on-semver isn't required when this flag is enabled, but that was a very tricky pass to get write and something that was intentionally left off-by-default. I don't think this simplifies wit-component since it's basically just a single function call, and I'd also need to do a complete audit of the WIT-loading-implementation to ensure it handles the myriad cases that the current pass already handles. For tools like wac I'd expect them to work with the component model type system, not WIT, but is that not the case?

... we cannot do import 1.0.0 and 1.0.1 in one world. Do you mean we want to preserve that ability? If we follow the spec, I think 1.0.0 and 1.0.1 would be considered as the same package under the canonical version, so it's essentially duplicate imports?

Yes, I would like to perserve the ability to import 1.0.0 and 1.0.1 in the same world. I disagree with the interpretation of the spec here where a generated component cannot simultaneously import 1.0.0 and 1.0.1 but that doesn't mean that the WIT packages don't exist and implicitly get unified. WIT, as a layer on the specification to assist with distributing/managing type information, has things like include which aren't natively in components. Reflecting all of this to component types is where merging behavior comes in, but that doesn't mean the WIT concepts don't exist.

The main use case I have for preserving this is the situation where different libraries are using different WASI versions. When linking them together this all needs to be rationalized because all of bindings genertation and componentization is world-centric, and that means that there needs to be a way to merge two worlds together (bindings for two different libraries) and get another world out of that. The current implementation of how things are separate I feel is correct, notably a world simultaneously imports two different interfaces at semver-compatible-but-different-versions, and then there's an opt-in pass to mutate packages/etc to merge those semver-compatible versions as necessary.