unified: Add Swift parser based on `swift-syntax` by tausbn · Pull Request #22195 · github/codeql

tausbn · 2026-07-15T11:32:13Z

Adds an alternative parser to tree-sitter-swift, using the Swift-native swift-syntax package instead.

The current implementation exposes this parser through a Rust wrapper, using the C FFI to communicate with Swift. The Rust component passes the source file contents to the Swift component, which in turn responds with a JSON string containing the AST.

Note that at present, the AST is quite verbose (though some effort has been made to make it less so). In particular, it contains detailed location information about every single token in the input, which is almost certainly overkill for our purposes.

This PR does not attempt to replace the existing yeast rules to be based on the swift-syntax AST. That is left as follow-up work.

Can be reviewed commit-by-commit.

Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.

These were taking up roughly 20% of the JSON payload.

Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.

Introduces new yeast types for `Point`s and `Range`s (corresponding to the tree-sitter equivalents), since it seemed silly to have `swift-syntax-rs` pull in `tree-sitter` just to have those types available. This _does_ mean there is a slight overhead in the shared extractor when converting between these types, but I think this is negligible.

Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.

At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.

Copilot

Pull request overview

Adds a Swift-native swift-syntax parser exposed through Rust and prepares unified extraction to consume its JSON AST.

Changes:

Adds Swift/Rust FFI parser and Bazel/Cargo builds.
Adds JSON-to-yeast AST adaptation and minimal Swift rules.
Extends yeast for externally constructed ASTs and owned locations.

Show a summary per file

File	Description
`MODULE.bazel`	Registers Swift dependencies and toolchains.
`Cargo.toml`	Adds the parser crate to the workspace.
`Cargo.lock`	Locks the new crate.
`shared/yeast-schema/src/schema.rs`	Supports merging schema names.
`shared/yeast-macros/src/parse.rs`	Uses yeast-owned ranges.
`shared/yeast/src/build.rs`	Migrates build contexts to owned ranges.
`shared/yeast/src/lib.rs`	Adds external AST construction/desugaring APIs.
`shared/yeast/src/range.rs`	Defines owned point and range types.
`shared/yeast/tests/test.rs`	Tests desugaring hand-built ASTs.
`shared/tree-sitter-extractor/src/extractor/mod.rs`	Converts yeast points for extraction.
`unified/extractor/src/languages/mod.rs`	Exposes the Swift JSON adapter.
`unified/extractor/src/languages/swift/adapter.rs`	Converts JSON into a yeast AST.
`unified/extractor/src/languages/swift/swift.rs`	Adds minimal swift-syntax rules.
`unified/extractor/tests/fixtures/let_x.swiftsyntax.json`	Provides parser output fixture.
`unified/extractor/tests/swift_syntax_pipeline.rs`	Tests the adapter/desugarer pipeline.
`unified/swift-syntax-rs/.gitignore`	Ignores generated outputs.
`unified/swift-syntax-rs/.swift-version`	Pins Swift 6.3.2.
`unified/swift-syntax-rs/BUILD.bazel`	Defines Bazel Swift/Rust targets.
`unified/swift-syntax-rs/Cargo.toml`	Defines the Rust crate and CLI.
`unified/swift-syntax-rs/README.md`	Documents building and usage.
`unified/swift-syntax-rs/build.rs`	Builds and links the Swift shim for Cargo.
`unified/swift-syntax-rs/src/lib.rs`	Implements safe Rust FFI bindings.
`unified/swift-syntax-rs/src/main.rs`	Adds the parser CLI.
`unified/swift-syntax-rs/swift/Package.resolved`	Locks swift-syntax.
`unified/swift-syntax-rs/swift/Package.swift`	Defines the Swift FFI package.
`unified/swift-syntax-rs/swift/Sources/SwiftSyntaxFFI/SwiftSyntaxFFI.swift`	Serializes SwiftSyntax ASTs as JSON.
`unified/swift-syntax-rs/xcode_transition.bzl`	Isolates macOS Xcode configuration.

Review details

Files reviewed: 26/27 changed files
Comments generated: 9
Review effort level: Medium

+    println!("cargo:rustc-link-search=native={}", build_dir.display());
+    println!("cargo:rustc-link-lib=dylib=SwiftSyntaxFFI");
+    println!("cargo:rustc-link-arg=-Wl,-rpath,{}", build_dir.display());


- BUILD.bazel: require x86_64 on the Linux branch of the compatibility select. The only registered standalone Linux Swift toolchain is x86_64-only, so without the CPU constraint Linux/aarch64 targets stayed "compatible" and then failed toolchain resolution instead of being skipped cleanly. - build.rs: emitting `rerun-if-changed` disables Cargo's default whole-package scan, so list every input to `swift build` — the package manifests, `Package.resolved`, `.swift-version`, and the whole Sources tree — so stale shim/toolchain output can't linger. (`.build/` is left unwatched, as it is this build's own output.) - src/lib.rs: a null result from the shim is not a parse failure — SwiftParser recovers from invalid syntax and always yields a tree — so it means the shim failed to serialize/allocate the JSON. Fix the error message and the variant doc accordingly. - README.md: `.swift-version` is not honored by the macOS Bazel build (which uses the host Xcode toolchain); document that it pins the Linux and local builds only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The doc claimed the schema passed to `Ast::with_schema` must already have every node kind and field name registered, contradicting the registration methods just below and the swift-syntax adapter, which starts from `Schema::new()` and registers names on demand during construction. Document that up-front registration is optional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.

Registers the `unified/swift-syntax-rs` workspace member (added in "unified: Add swift-syntax-rs") in the tree-sitter extractors crate universe. It has no external Rust dependencies, so its dependency entries are empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jketema

I mostly skimmed the Rust changes.

For the Swift code I wonder whether we really need to write the complete serializer ourselves, or whether we could leverage things that are already there in swift-syntax. This is not blocking though.

Do we need someone with more Bazel knowledge to review those parts?

jketema · 2026-07-16T12:45:23Z

+If your `swift`/`swiftc` are not on `PATH`, point the build at them explicitly:
+
+```sh
+SWIFT=/path/to/swift SWIFTC=/path/to/swiftc cargo build
+```


Is it common to have something like this in rust builds that also compile code in other languages? Note that I'm not suggesting to change this.

I don't think we do it anywhere else, so it's a valid concern. Originally I had swift-syntax-rs much more tightly integrated with the extractor, in which case it made sense to build both at the same time. However, now that we have (for the time being) moved to a model where we call an external binary for the parsing, we could also decouple the builds themselves.

jketema · 2026-07-16T13:27:16Z

+//!   swift-syntax JSON -> `swift_adapter::json_to_ast` -> yeast `Ast`
+//!   -> `Desugarer::run_from_ast` (the real Swift translation rules) -> dump.
+//!
+//! This exercises the whole chain *without* the Swift toolchain: the JSON


This seems potentially fragile if swift-syntax changes?

Agreed. This was a "proof of concept" test that was useful at the time, but it is brittle. We can probably just get rid of it once the whole extraction is using swift-syntax-rs.

jketema · 2026-07-16T13:33:04Z

+# The Linux branch additionally requires x86_64: the only registered standalone
+# Linux Swift toolchain (`swift_toolchain_exec_ubuntu24.04`, see //:MODULE.bazel)
+# is x86_64-only. Without the CPU constraint, Linux/aarch64 targets would stay
+# "compatible" and then fail toolchain resolution instead of being skipped.


Why is this change needed? And why is this the right approach to this change (instead of registering an Arm64 tool chain)?

Honestly, I have no idea why Copilot chose that solution. A better solution is probably to just register an Arm64 tool chain.

This goes back to your other comment as well -- we probably ought to have someone more intimate with the Bazel setup take a look at this. (Or at the very least once we get close to being production-ready.)

Honestly, I have no idea why Copilot chose that solution. A better solution is probably to just register an Arm64 tool chain.

I can have a look at that.

This goes back to your other comment as well -- we probably ought to have someone more intimate with the Bazel setup take a look at this. (Or at the very least once we get close to being production-ready.)

This we should probably do as soon as possible, because I am concerned that the changes here might affect the rest of the build in ways that I do not understand.

tausbn and others added 8 commits July 10, 2026 11:35

unified: Add swift-syntax-rs

89e11e5

Adds an initial prototype of an interface from Rust to Swift, which enables us to use the `swift-syntax` package for parsing. At present, the parsed AST is passed between Swift and Rust as a JSON string.

unified: Don't emit empty trivia

87c8173

These were taking up roughly 20% of the JSON payload.

unified: Add mapping from swift-syntax JSON to yeast AST

446ff7c

Adds a preliminary mapping that decodes the JSON into the yeast AST. The JSON AST itself is still very verbose, but it would be useful to see if it actually contains all of the things we want it to contain.

unified: Emit locations for comments

ddab6cc

Also gathers these into a separate side-channel during the initial AST construction. That way, we don't encounter these as weird extra nodes while running yeast.

unified: Hook up swift-syntax AST

58ddeac

At this point it's only a proof-of-concept translation -- it translates `sourceFile` nodes, but everything else gets mapped to 'unsupported_node`.

Support building swift-syntax-rs on macOS

3e2daf2

Upgrade swift-syntax-rs to swift-syntax 603.0.2 / Swift 6.3.2

1577d82

github-actions Bot added the documentation label Jul 15, 2026

tausbn requested a review from Copilot July 15, 2026 11:32

Copilot started reviewing on behalf of tausbn July 15, 2026 11:33 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

tausbn and others added 5 commits July 15, 2026 12:14

swift-syntax-rs: Fix bazel formatting

205c9a9

aCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

unified: Hardcode swift_version

7474a33

It seems that using swift_version_file has some issues when `codeql` is consumed as a dependency module. I'm hoping hardcoding the version instead will fix this, but ideally we should find a more robust solution.

tausbn force-pushed the tausbn/swift-syntax-rs branch from 4ba73bd to da1bbb7 Compare July 15, 2026 13:15

tausbn marked this pull request as ready for review July 15, 2026 13:58

tausbn requested review from a team as code owners July 15, 2026 13:58

jketema approved these changes Jul 16, 2026

View reviewed changes

Uh oh!

Conversation

tausbn commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jketema left a comment

Choose a reason for hiding this comment

Uh oh!

jketema Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

tausbn Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

jketema Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

tausbn Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

jketema Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

tausbn Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

jketema Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tausbn commented Jul 15, 2026 •

edited

Loading