Repository navigation
Conversation
unsafe-libyaml is unmaintained (RUSTSEC-2024-0449) and reaches us only through serde_yaml, which is itself deprecated. serde-saphyr is a maintained serde deserializer written in safe Rust, and it is the serde option the advisory suggests. All YAML parsing now goes through src/yaml.rs. Callers read the whole file and pass the bytes to crate::yaml::from_slice, which calls from_slice_with_options with options that keep serde_yaml's semantics where serde-saphyr allows it: a repeated key keeps its last value instead of failing the file, `<<` stays an ordinary key, non-finite floats and `!!binary` are not interpreted, only true and false are booleans, and errors stay on one line. Resource budgets stay at serde-saphyr's defaults. Two deserialize_with helpers keep empty values in string fields and list items working as before. The config tests now go through Config::load_from_path, so they cover the same path as production. New tests pin each option, the helpers, empty package.yml files and the depth limit. Fixes #141
With DuplicateKeyPolicy::LastWins, serde-saphyr buffers mappings before serde sees them, and nested mappings once per level. Under the default budget a crafted 480 KB package.yml made `generate` peak at 2.4 GB and take 15 s, against 35 MB and 0.5 s on main. Cap depth at 16 and nodes at 50,000, and lower the scalar-byte and anchor-copy caps. The worst file that fits now peaks at about 195 MB. The 1,015 files read in two large private monorepos nest at most 5 deep with at most 2,162 nodes, and `generate` output for both is unchanged. strings_or_nulls now reads an empty, `~` or `null` list as empty itself, rather than relying on serde-saphyr to do so (serde-saphyr#233 asks it to stop). The empty-value tests now cover every deserialize_with field, so removing any of them fails a test; before, 11 of the 15 could be removed with the suite still passing. The depth test now nests 20 deep, between the new limit and the default, and asserts a budget error; a new test does the same for the node limit. Dependabot's cargo group now excludes serde-saphyr, so its bumps get their own pull request.
A team file or package.yml over the node limit aborts the whole run, and until now raising the limit took a new release. The new `yaml_max_nodes` key in config/code_ownership.yml (default 50,000) sets it for every path that reads those files: generate and validate, for-file, and for-file --from-codeowners. The team cache is keyed on it, so teams loaded under one limit are not reused under another. A file over the limit now also gets a hint naming the key. The config file itself is still read with the default, since it can't raise its own limit and has at most a few hundred nodes.
50,000 left 23x headroom over the largest real file because the limit could not be raised without a release. Now that yaml_max_nodes can raise it, 20,000 still leaves 9x headroom over the 2,162 nodes of the largest of the 1,015 files read in two large monorepos, and cuts the worst-case peak for a file that fits from about 195 MB to about 40 MB. `generate` output for both monorepos is unchanged.
Raising yaml_max_nodes did not help a file that then hit the fixed 4 MiB scalar-byte cap or an anchor cap, and those breaches got no hint. The scalar-byte, parser-event and anchor caps now grow with the node limit: 256 bytes, 50 events and 5 anchor copies (64 bytes) per node. At the 20,000 default that is about 5 MB, the 1,000,000 events serde-saphyr defaults to, 100,000 anchor copies and about 1.3 MB. A breach of any of them, including one reached through an alias, gets the yaml_max_nodes hint. Also update the team cache comment, which is keyed on the node limit too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #141.
unsafe-libyamlis unmaintained (RUSTSEC-2024-0449) and reaches us only throughserde_yaml, which is itself deprecated. This replacesserde_yamlwith serde-saphyr 2.0.0, the serde option the advisory suggests. serde-saphyr and its parser, granit-parser, are pure Rust with#![forbid(unsafe_code)], though granit-parser depends on arraydeque, which usesunsafe. serde-saphyr is actively maintained: 32 releases in the past year, outside issues closed in a median of under a day, and kube-rs among its 400+ dependents. Both crates have a single maintainer, though, and their security policy covers only the latest release. 2.0.0 shipped about ten weeks after 1.0.0, so we should expect to take major versions promptly, which is why Dependabot now opens serde-saphyr bumps on their own.Alternatives considered
unowned_globs:that four fixtures use. It is still 0.0.x, so every release is semver-incompatible with the last.unsafe-libyamlor a fork of it.What changed
All YAML parsing goes through the new
src/yaml.rs:fs::readand parse it withfrom_slice_with_options, which accepts the same encodings serde_yaml did (UTF-8, with or without a BOM).from_readerwould also accept UTF-16 and silently replace invalid UTF-8 after a BOM, and it parses about 2.3x slower than serde_yaml.duplicate_keys: LastWins. The default rejects a repeated key anywhere in a file, including keys we never read, and one suchpackage.ymlwould abortgenerateandvalidatefor the whole repo. These exist in the wild: this public packwerk app repeatsenforce_privacy. The other options keep serde_yaml's behavior for<<, for!!binary, for.infand.nanin values we ignore, for YAML 1.1 booleans (yesandnostay errors), and for single-line error messages.LastWinsbuffers mappings before serde sees them, nested ones once per level, so under the defaults a crafted 480 KBpackage.ymlmadegeneratepeak at 2.4 GB and take 15 s, against 35 MB and 0.5 s onmain. Under these budgets that file fails with a budget error, and the worst file that fits peaks at about 40 MB. The depth limit also keeps nested input from overflowing the stack. The 1,015 files read in two large private monorepos nest at most 5 deep, with at most 2,162 nodes and no anchors, so the node limit leaves 9x headroom. It is the newyaml_max_nodesconfig key, so a repo whose files outgrow it can raise it, and the caps that grow with it, without waiting for a release. A file over any of those limits fails with a hint naming the key. A 50,000-node default would give 23x headroom, but raises the worst case to about 195 MB.~ornullvalue in aStringfield or list item, which serde_yaml accepted. Two smalldeserialize_withhelpers keep accepting them, as"". The list helper also reads an empty,~ornulllist as empty itself, rather than relying on serde-saphyr to, since serde-saphyr#233 asks it to stop.src/common_test.rsnow go throughConfig::load_from_path, so they exercise the production path.src/yaml.rs, next to the options they cover. Every option, the depth and node limits, everydeserialize_withattribute, and each of the four places that passyaml_max_nodeson fail at least one test when removed. Eleven tests assert what serde_yaml already did (checked by running the same assertions onmain). The other ten pin intended changes: a repeatedownertakes the last value, a BOM before several keys now parses, a~ornulllist reads as empty, a value nested 20 deep is rejected, and so is a list of 100,000 items, unlessyaml_max_nodesraises the limit. That last part is checked throughgenerate,for-fileandfor-file --from-codeowners, and for the team cache, which is keyed on the limit. The tenth checks that the scalar-byte cap grows withyaml_max_nodesand that its breach, even through an alias, gets the hint.cargogroup now excludes serde-saphyr, so its bumps arrive as their own pull requests and never wait on unrelated breakage. granit-parser is a transitive dependency, which Dependabot version updates skip, so its fixes arrive through serde-saphyr bumps or RustSec advisories.Behavior changes
Verified unchanged:
package.ymlfiles, the empty-file cases in the last bullet, and all 13 fixtures (includingdocs/CODEOWNERSfrom the custom-path fixture).package.yml, team and config files from GitHub, and about 1,200 files from nine private repositories. The one exception is two public GitHub Actions workflows namedpackage.ymlthat start with a UTF-8 BOM, which serde_yaml rejected and serde-saphyr accepts. For the seven private repositories with a config,generategives byte-identical CODEOWNERS, stdout, stderr and exit codes.generateon two large private monorepos, which reads 1,015 YAML files, gives byte-identical CODEOWNERS, empty stdout and stderr and exit code 0, the same asmain. None of a wider local corpus of 1,615package.yml, team and config files, or the 42 fixtures, hits a budget.----onlypackage.ymlstill means "no owner".Changed: error text on every failure path, and a handful of rare inputs that are now rejected, accepted, or read differently. Apart from the error text of files that fail in both, and the two BOM files above, none of these occur in the 2,356 files.
Every change, with examples
Error text changes on every failure path. Messages no longer start with the key path, type errors no longer say what was found, and locations read
line L, column C, including theline 1, column 1that serde_yaml left out:githubmissing field `github`missing field `github` at line 1, column 1owned_globs: 5owned_globs: invalid type: integer `5`, expected a sequence at line 1 column 14expected sequence start at line 1, column 14do_not_add_to_codeowners_file: yesgithub.do_not_add_to_codeowners_file: invalid type: string "yes", expected a boolean at line 4 column 34invalid boolean (strict mode expects true/false) at line 4, column 34missing field `name`unexpected end of input at line 1, column 1found unexpected end of stream at line 4 column 1, while scanning a quoted scalar at line 3 column 9invalid indentation in multiline quoted scalar at line 4, column 1deserializing from YAML containing more than one document is not supportedmultiple YAML documents detected; use from_str_multiple or from_str_multiple_with_options at line 3, column 1~: xat the top levelcannot deserialize null into string; use Option<String> at line 1, column 1The last two messages give advice meant for Rust callers of serde-saphyr. A budget breach reads like
budget breached: Depth { depth: 17 } at line 2, column 70, under the path of the file. A breach of the node limit or a cap that grows with it adds a hint namingyaml_max_nodes.A missing field is also reported at a different line. serde_yaml gave the line where the mapping starts. serde-saphyr gives the line of the last key it read in that mapping, so a longer team file without
githubcan point at an unrelated key such asruby:.Newly rejected. serde_yaml accepts these, and so does Psych except where noted:
name: "Payroll"# note.~: xdirectly in a mapping we read: the top level,metadata,githuborruby. Nested deeper inside a value we ignore, null keys still parse.!!timestampwith any value (Psych'sload_filerejects it too),!!intwith a value outside 64 bits or in YAML 1.1 form such as017or1:20,!!float 1_000.5, or a value that does not fit, such as!!int abc. In a field we read, a non-string core tag such asname: !!int 123is rejected too, and so is!!str truefor a boolean.yaml_max_nodes), anywhere in the file. serde_yaml had no limits for values we ignore.*a: 1. Psych 4'sload_filerejects aliases by default.---.Newly accepted:
duplicate field `owner`.---or before the first of several keys. serde_yaml failed on these, usually with "deserializing from YAML containing more than one document is not supported".~for a list field, read as an empty list, and a quoted"true"or an odd-casetRuEfor a boolean.package.ymlthat is only~,nullor..., read as no owner, as in Psych.metadata:; anchor names containing/or non-ASCII characters; unknown%directives; and surrogate-pair escapes such as"😀".Different values:
~,nullin any capitalization such asnUlL, or a!!null-tagged value in a string field or list item reads as"", and inowneras no owner, instead of the literal text. Psych also reads these as nil.|or>block scalar that ends the file without a trailing newline now ends in\n, soowner: |followed byTeamAat end of file reads as"TeamA\n".name: TeamAfollowed by U+0085 reads as"TeamA\u{85}".In an
owneror a teamname, either of the last two means the name no longer matches its team, the way a typo would. In code_ownership, which compares the names Rust reads against the ones Psych reads,for_filethen returns nil for that team's files.Cost
strip -x, from 3.71 MB to 4.50 MB on aarch64-apple-darwin. Unstripped, each architecture's binary grows by 0.9 to 1.0 MB, and the universal mac binary the release job uploads by 1.9 MB. Most of it is serde-saphyr's deserializer, plus annotate-snippets, which stays linked even with snippets off.generateon a large monorepo is unchanged within noise, because walking the tree dominates.Verification
cargo audit: no warnings, down from RUSTSEC-2024-0449.cargo tree -i unsafe-libyamlandcargo tree -i serde_yamlmatch nothing.cargo fmt --all -- --check,RUSTFLAGS=-Dwarnings cargo clippy --all-targets --all-features, andcargo test: 199 passed, 0 failed, 3 ignored (178 before, plus the 21 new tests).lipostep succeed. Itscrossbuild succeeds for x86_64-unknown-linux-gnu, and for aarch64-unknown-linux-gnu it builds and linkscodeowners.Cargo.lockis exactly whatcargo buildresolves frommain's lockfile, with nocargo update: 6 packages removed, 11 added.grep "\+version"check finds nothing in this diff with BSD grep or GNU grep, so merging does not cut a release.main.IgnoredAnywith this PR's options, caused no panics.Notes
yaml_max_nodeskey is documented in the README's configuration section.serde_yaml::from_reader. The two PRs merge without conflicts, and an earlier green CI run on whichever lands second won't catch it, so rebase or re-run CI before merging that one. The test needs switching toConfig::load_from_path, and the newallow_unowned_globsfield needsdeserialize_with = "crate::yaml::strings_or_nulls"to match the other list fields. Merging this first means Add allow_unowned_globs to permit files with no owner #135's 0.6.0 release ships the fix.