diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 75a835e..b597e0e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -92,7 +92,7 @@ The records under `docs/adr/` stay as history and are cited as they are. No new one is added: the `adr-freeze` rule in `policy/principles.toml` refuses any other file under that directory, and its message names the remedy. [ADR 0012](docs/adr/0012-a-rule-may-reach-the-content-its-repository-pins.md) -stands at Proposed until the owner rules it. +is Accepted and implemented as `files.reach`. ## Working on the engine diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md index ca4c31b..5144460 100644 --- a/docs/REFERENCE.md +++ b/docs/REFERENCE.md @@ -60,7 +60,7 @@ does not run.** | keys | vocabulary | runs it | |---|---|---| -| `files.*` | ripgrep scoping — `glob`, `multiline`, `fixed_strings` — and `min_selected`, the floor under what the scoping leaves | `uphold scan` | +| `files.*` | ripgrep scoping — `glob`, `multiline`, `fixed_strings` — `min_selected`, the floor under what the scoping leaves, and `reach`, `"repository"` (the default) or `"pinned"` for the content the repository's submodules pin | `uphold scan` | | `git.hooks` | githooks(5) names — `pre-commit`, `commit-msg`, `pre-merge-commit`, `pre-push`, `manual` | `uphold guard --stage ` | | `command.before` | the command line as typed — `"gh pr create"`, `"git push"` | `uphold shim ` | @@ -346,6 +346,62 @@ counts links and `require_any_anchor` counts anchors. A `links-resolve` rule may hold both; they fail differently, and a glob typo is caught by the one that counts files. +### A rule may reach the content its repository pins + +`files.reach` says whose tracked files a rule selects from. `"repository"` is +this repository's own, and is what an absent key means. `"pinned"` adds the +content of every submodule the index pins, selected under its mount path: + +```toml +[rule.member-readmes-name-an-owner] +require_regexp = '(?m)^Owner: ' +message = "every member says who owns it" +files.glob = ["/*/README.md"] +files.reach = "pinned" +files.min_selected = 1 +``` + +The pins are the gitlinks in the index (mode `160000` in `git ls-files -s`), +and each member is asked for its own `git ls-files` inside it. A member that +pins members of its own is followed the same way, at every depth. Not +`git ls-files --recurse-submodules`: that follows git's active-submodule +filter, and a rule that claims the pinned content must not pass over part of it +without a word. + +- **A pinned mount that cannot be read is exit `2`**: not checked out, or + checked out and marked inactive by `submodule..active` or + `submodule.active`. The message names the mount and + `git submodule update --init `, which checks it out and marks it active. + A CI checkout without `--recurse-submodules` goes red on a pinned rule, on + purpose. A repository that pins nothing reads the same as at + `"repository"`. +- **Paths are mount-prefixed everywhere**: findings, path baselines and size + baselines key on `member/sub/file`, and `files.include` may name a directory + inside a mount. `files.min_selected` counts the prefixed files. +- **The globs are the superproject's.** `include`, `exclude` and `glob` keep + their gitignore meaning, rooted at the superproject: `/vendor.txt` is the + superproject's own file, and `vendor.txt` matches at any depth, inside mounts + too. +- **A member's `.gitattributes` is asked inside the member**, so a file it + declares `-text` is skipped and listed like the superproject's own. A member + whose attributes cannot be asked is exit `2`, as the superproject's is. +- **A link in a member's Markdown is the member's.** A leading `/` resolves + against the member's root, and a link leaving the member is outside the + repository, for `links-resolve` and `anchors-resolve` alike. +- **Nothing the member declares about policy is read**: not its policy file, + not its excludes. The rule is the superproject's, and so is the verdict. + +The direction is one-way. A member never borrows upward: run in a member with +no policy of its own, uphold stops at the member's root rather than loading the +superproject's. A repository may judge, downward, the content it pins, and +reports it under the mount path. + +`files.reach` is refused on a guard built-in's `[rule.files]`, for the reason +`min_selected` is: there it scopes the bytes a hook is about to record, and no +path it is handed lies inside a mount. `uphold rules --effective --json` carries +`"reach": "pinned"` on a rule that declares it, and nothing on one that does +not. + ### A rule may not be about its own declaration A policy file is a tracked file, so a rule's `regexp` and `require_regexp` are @@ -407,6 +463,11 @@ it. In a directory git has no index for, the tree is walked instead with **no** ignore file consulted, which selects a superset of what would be tracked. Over-reporting is the direction a checker may fail in; hiding a file is not. +**A submodule is its own repository**, so its content is not among a rule's +files: the gitlink is a pointer, and the scan passes over it. A rule that +declares `files.reach = "pinned"` claims that content too. See +[A rule may reach the content its repository pins](#a-rule-may-reach-the-content-its-repository-pins). + A path a rule selected and could not open — an unstaged deletion, a sparse checkout, a directory this process may not enter — is **named on stderr and is exit `2`**, after every other rule has reported. It is not dropped from the diff --git a/docs/adr/0012-a-rule-may-reach-the-content-its-repository-pins.md b/docs/adr/0012-a-rule-may-reach-the-content-its-repository-pins.md index 9027646..a1ed967 100644 --- a/docs/adr/0012-a-rule-may-reach-the-content-its-repository-pins.md +++ b/docs/adr/0012-a-rule-may-reach-the-content-its-repository-pins.md @@ -1,6 +1,6 @@ # ADR 0012: a rule may reach the content its repository pins -Status: Proposed +Status: Accepted `uphold scan` reads what `git ls-files -z` lists (`index_bytes`, `src/selection.rs:150`) and drops every gitlink on purpose diff --git a/src/config.rs b/src/config.rs index a4b9f3c..f50724e 100644 --- a/src/config.rs +++ b/src/config.rs @@ -239,6 +239,35 @@ pub(crate) struct Files { /// shape `trivial_comments = false` is refused for. #[serde(default)] pub min_selected: Option, + /// Whose tracked files the selection is drawn from: this repository's own + /// (`"repository"`, and what an absent key means), or those plus the + /// content of every repository its index pins (`"pinned"`). + /// + /// An `Option` rather than a defaulted value so that a rule written before + /// the field existed serializes exactly as it did, into the bundled-set + /// lock and everywhere else a rule is printed. + #[serde(default)] + pub reach: Option, +} + +/// How far down a rule's selection reaches. +/// +/// `Repository` is what every rule did before the field existed: the files +/// this repository's index lists, a gitlink skipped as another repository's +/// content. `Pinned` is a claim about the content the repository pins as well: +/// each gitlink in the index is asked for its own tracked files, which are +/// selected under the mount path. The pin is what makes the content this +/// repository's business -- a member never borrows upward, and a repository +/// may judge, downward, what it pins. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)] +#[serde(rename_all = "lowercase")] +pub(crate) enum Reach { + /// This repository's own tracked files. + #[default] + Repository, + /// This repository's own tracked files and those of every repository it + /// pins, at every depth, under their mount paths. + Pinned, } /// The five stage names a rule may run at: four of git's own, spelled as diff --git a/src/config/rule.rs b/src/config/rule.rs index 74df7f1..900b4e8 100644 --- a/src/config/rule.rs +++ b/src/config/rule.rs @@ -17,7 +17,7 @@ use std::sync::OnceLock; use serde::{Deserialize, Serialize}; -use super::{CommandWhere, Files, Git, Origin}; +use super::{CommandWhere, Files, Git, Origin, Reach}; use crate::error::{Fatal, Result}; use crate::text::{Judged, Seam}; @@ -1322,6 +1322,12 @@ impl Rule { .unwrap_or_else(|| DEFAULTS.get_or_init(Files::default)) } + /// How far this rule's selection reaches, `Repository` where the rule + /// does not say. + pub(crate) fn reach(&self) -> Reach { + self.files().reach.unwrap_or_default() + } + /// Whether this rule searches files at all. Absent `files.*` keys are the /// answer, not a default to fill in. pub(crate) const fn reads_files(&self) -> bool { @@ -1537,6 +1543,7 @@ impl Rule { self.validate_prose(check)?; self.validate_selection_floor()?; + self.validate_reach()?; // The label is resolved at load, so a typo is a refusal here and not a // rule that fails every file it selects. @@ -1859,6 +1866,30 @@ impl Rule { ) } + /// The `files.reach` half of [`Rule::validate`]: a reach needs a selection + /// for it to widen. + /// + /// Refused where `min_selected` is refused and for the same reason: a + /// guard built-in's `[rule.files]` scopes the bytes git is about to record + /// one path at a time, and nothing it is handed lies inside a mount. A + /// `"pinned"` there would read as a claim on the pinned content that no + /// seam ever makes. + fn validate_reach(&self) -> Result<()> { + if self.reach() == Reach::Repository || self.selection_is_counted() { + return Ok(()); + } + Err(Fatal::new(format!( + "rule {:?}: `files.reach = \"pinned\"` widens the selection `uphold scan` builds \ + to the repositories this one pins, and built-in {:?} runs at a git hook over the \ + bytes git is about to record -- its `files.*` keys scope that guard one path at a \ + time, and no path it is handed lies inside a mount. On this rule the field would \ + be read by nothing. The built-ins the scan selects for are {}", + self.id, + self.builtin().unwrap_or_default(), + crate::guard::SCAN_BUILTINS.join(", ") + ))) + } + /// The `seams` half of [`Rule::validate`]: the list names seams this /// rule's kind can run at, beside the table that makes it published-text /// rule at all. diff --git a/src/git.rs b/src/git.rs index 3205537..1da781b 100644 --- a/src/git.rs +++ b/src/git.rs @@ -60,10 +60,30 @@ pub(crate) fn try_run(root: &Path, args: &[&str]) -> Result> { pub(crate) fn try_run_elsewhere(directory: &Path, args: &[&str]) -> Result> { let mut command = crate::shim::inner_tool("git"); command.current_dir(directory); + answer(elsewhere(&mut command), args) +} + +/// A git command aimed at a repository other than the hooked one, with the +/// hooked repository's environment taken away. +/// +/// The one place that decides what "elsewhere" strips, so a caller that has to +/// build its own `Command` -- one that speaks to git over stdin, which +/// [`try_run_elsewhere`] does not -- strips the same list rather than a copy. +pub(crate) fn elsewhere(command: &mut std::process::Command) -> &mut std::process::Command { for name in REPOSITORY_ENVIRONMENT { command.env_remove(name); } - answer(&mut command, args) + command +} + +/// Whether a gitlink's mount holds a checkout: a `.git` file or directory is +/// what `git submodule update --init` leaves there, and an uninitialised mount +/// is an empty directory, or no directory at all. +/// +/// Asked by every reader that follows a pin into the member, so "not checked +/// out" means one thing across them. +pub(crate) fn is_checked_out(mount: &Path) -> bool { + mount.join(".git").symlink_metadata().is_ok() } fn answer(command: &mut std::process::Command, args: &[&str]) -> Result> { diff --git a/src/main.rs b/src/main.rs index a0af48b..6846a39 100644 --- a/src/main.rs +++ b/src/main.rs @@ -637,7 +637,7 @@ fn scan_command(arguments: &[OsString]) -> Result { }; let policy = config::load(&root, &policy_path)?; - let scanner = scan::Scan::new(&root, &policy); + let scanner = scan::Scan::new(&root, &policy)?; let failures = scanner.run()?; for failure in &failures { failure.print(); @@ -967,6 +967,12 @@ struct EffectiveRule<'rule> { /// whose only place is `command.before` reconciled green in a repository /// where nothing runs it. The loader knows; it says so here. seams: Vec<&'static str>, + /// `"pinned"` where the rule's selection reaches the content this + /// repository pins. Only there, for the reason `overridden` gives: the + /// entry of every rule that reads only its own repository is what it was + /// before the field existed. + #[serde(skip_serializing_if = "Option::is_none")] + reach: Option, /// Which fields of an inherited rule an override changed. Only where one /// did, so the entry of every rule no override touches is what it was /// before the field existed. @@ -1009,11 +1015,16 @@ fn effective_rules_command(as_json: bool) -> Result { // Which parts of an inherited rule this policy changed, so a // reworded message or a widened selection reads as local rather // than as the set's. + let reach = if rule.reach() == config::Reach::Pinned { + " [reach: pinned]" + } else { + "" + }; if rule.overridden.is_empty() { - println!(" {} ({at})", rule.id); + println!(" {} ({at}){reach}", rule.id); } else { println!( - " {} ({at}) [override: {}]", + " {} ({at}){reach} [override: {}]", rule.id, rule.overridden.join(", ") ); @@ -1029,6 +1040,7 @@ fn effective_rules_command(as_json: bool) -> Result { id: &rule.id, git_hooks: rule.hooks(), seams: rule.seams(), + reach: (rule.reach() == config::Reach::Pinned).then_some(config::Reach::Pinned), overridden: &rule.overridden, }) .collect(); diff --git a/src/scan.rs b/src/scan.rs index a572c46..b360a63 100644 --- a/src/scan.rs +++ b/src/scan.rs @@ -9,11 +9,11 @@ use regex::Regex; use tree_sitter::Parser; use unicode_script::{Script, UnicodeScript}; -use crate::config::{Check, CheckKind, Files, Policy, Rule}; +use crate::config::{Check, CheckKind, Files, Policy, Reach, Rule}; use crate::engine::{self, Hit, Query}; use crate::error::{Fatal, Result}; use crate::report::{Failure, body_for}; -use crate::selection::{Selection, normalize_rel, not_text_paths}; +use crate::selection::{Pinned, Selection, normalize_rel, not_text_paths}; /// The command name a `command_sources` pattern captures out of a path. /// @@ -137,6 +137,10 @@ pub(crate) struct Scan<'a> { root: &'a Path, policy: &'a Policy, not_text: Vec, + /// The pinned content, read once where a rule declares + /// `files.reach = "pinned"` and not at all otherwise, so a policy with no + /// such rule runs no git command it did not run before the field existed. + pinned: Option, /// Every path any rule's selection knew about and could not open. /// /// Interior mutability because `run` takes `&self` and every check arm @@ -177,7 +181,7 @@ pub(crate) struct Scan<'a> { } impl<'a> Scan<'a> { - pub(crate) fn new(root: &'a Path, policy: &'a Policy) -> Self { + pub(crate) fn new(root: &'a Path, policy: &'a Policy) -> Result { // A `.gitattributes` question that could not be answered is seeded into // the unreadable list rather than dropped, because the scan continues // either way and the reader has to be told which of the two answers they @@ -187,18 +191,64 @@ impl<'a> Scan<'a> { if let Some(reason) = unmeasured { unreadable.insert(reason); } - Self { + // Read before any rule runs, so a mount that cannot be read stops the + // run as a whole: the pinned rules would each have to report it, and + // the repository rules' findings would arrive beside a claim on the + // pinned content that nothing made. + let pinned = if policy + .rules + .iter() + .any(|rule| rule.reads_files() && rule.reach() == Reach::Pinned) + { + Pinned::read(root)? + } else { + None + }; + Ok(Self { root, policy, not_text, + pinned, unreadable: RefCell::new(unreadable), below_floor: RefCell::new(BTreeMap::new()), declared_encodings: RefCell::new(None), + }) + } + + /// Every path the scan skipped as declared not text: this repository's, + /// and the pinned members' under their mount paths where a pinned rule + /// read them. + pub(crate) fn not_text(&self) -> Vec { + let mut skipped = self.not_text.clone(); + if let Some(pinned) = &self.pinned { + skipped.extend(pinned.not_text().iter().cloned()); } + skipped } - pub(crate) fn not_text(&self) -> &[String] { - &self.not_text + /// One rule's selection: over the pins read once for this run where the + /// rule reaches them, and over this repository's own index otherwise. + fn selection(&self, rule: &Rule) -> Result { + if rule.reach() == Reach::Pinned { + return Selection::build_over(self.root, rule, &self.not_text, self.pinned.as_ref()); + } + Selection::build(self.root, rule, &self.not_text) + } + + /// The root of the repository that tracks `relative`, and the path + /// relative to that root: a pinned member's own root for a path inside a + /// mount, where a link written `/docs/a.md` means the member's `docs/` and + /// not the superproject's. This repository's root, and `relative` itself, + /// for every other path. + fn repository_of<'path>(&self, relative: &'path str) -> (PathBuf, &'path str) { + match self + .pinned + .as_ref() + .and_then(|pinned| pinned.mount_of(relative)) + { + Some((mount, within)) => (self.root.join(mount), within), + None => (self.root.to_path_buf(), relative), + } } /// The paths this scan could not read, each with its reason. @@ -313,7 +363,7 @@ impl<'a> Scan<'a> { // declaration, so the file it selects stays undeclared rather // than being decoded as something nobody wrote. if let Some(encoding) = encoding_rs::Encoding::for_label(label.as_bytes()) { - let selection = Selection::build(self.root, rule, &self.not_text)?; + let selection = self.selection(rule)?; self.unreadable .borrow_mut() .extend(selection.unreadable().iter().cloned()); @@ -407,7 +457,7 @@ impl<'a> Scan<'a> { } fn select(&self, rule: &Rule) -> Result> { - let selection = Selection::build(self.root, rule, &self.not_text)?; + let selection = self.selection(rule)?; // Gathered here, at the one place every rule's selection passes // through, so no future check kind can acquire its own way of dropping // a path it could not open. @@ -898,13 +948,22 @@ impl<'a> Scan<'a> { Err(error) if error.kind() == std::io::ErrorKind::InvalidData => continue, Err(error) => return Err(Fatal::at(&self.root.join(relative), error)), }; + // A member's links are the member's: a leading `/` is its root, and + // leaving it is leaving the repository the document belongs to. + let (repository, within) = self.repository_of(relative); + let member_root = (repository != self.root).then(|| { + repository + .canonicalize() + .unwrap_or_else(|_| repository.clone()) + }); + let boundary = member_root.as_ref().unwrap_or(&canonical_root); for (line, target) in link_targets(&text) { checked += 1; - let resolved = resolve_link(self.root, &target, relative); + let resolved = resolve_link(&repository, &target, within); let canonical = resolved .canonicalize() .unwrap_or_else(|_| lexically_normalize(&resolved)); - let inside = canonical.starts_with(&canonical_root); + let inside = canonical.starts_with(boundary); if !inside { if rule.allow_outside_repo() { continue; @@ -1146,7 +1205,8 @@ impl<'a> Scan<'a> { }; for anchor in crate::anchors::parse(&text) { checked += 1; - let Some(finding) = crate::anchors::resolve(&anchor, self.root) else { + let (repository, _) = self.repository_of(relative); + let Some(finding) = crate::anchors::resolve(&anchor, &repository) else { continue; }; hits.push(Hit { diff --git a/src/selection.rs b/src/selection.rs index 5701453..ad62d27 100644 --- a/src/selection.rs +++ b/src/selection.rs @@ -33,7 +33,7 @@ use std::sync::Once; use ignore::WalkBuilder; use ignore::overrides::{Override, OverrideBuilder}; -use crate::config::Rule; +use crate::config::{Reach, Rule}; use crate::error::{Fatal, Result}; /// Paths this repository declares are NOT TEXT in `.gitattributes`. @@ -63,30 +63,47 @@ pub(crate) fn not_text_paths(root: &Path) -> (Vec, Option) { if listed.is_empty() { return (Vec::new(), None); } - - let unmeasured = |reason: &str| { - ( + match declared_not_text(root, &listed, false) { + Ok(found) => (found, None), + Err(reason) => ( Vec::new(), Some(format!( ".gitattributes: {reason}, so which paths this repository declares are not \ text is unknown. Every tracked path was treated as text, which means a \ declared binary file was searched by the content rules rather than skipped." )), - ) - }; + ), + } +} - let Ok(mut child) = crate::shim::inner_tool("git") +/// Which of `listed` (NUL-separated paths) git reads as `-text` in `directory`, +/// or why it could not say. +/// +/// `elsewhere` is a repository other than the hooked one -- a pinned member -- +/// which is asked with the hooked repository's environment taken away, for the +/// reason [`crate::git::try_run_elsewhere`] gives. That function cannot be used +/// itself because this question travels over stdin. +fn declared_not_text( + directory: &Path, + listed: &[u8], + elsewhere: bool, +) -> std::result::Result, String> { + let mut command = crate::shim::inner_tool("git"); + if elsewhere { + crate::git::elsewhere(&mut command); + } + let Ok(mut child) = command .args(["check-attr", "--stdin", "-z", "text"]) - .current_dir(root) + .current_dir(directory) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::null()) .spawn() else { - return unmeasured("git check-attr could not be started"); + return Err(String::from("git check-attr could not be started")); }; let (Some(mut sink), Some(mut source)) = (child.stdin.take(), child.stdout.take()) else { - return unmeasured("git check-attr gave no pipe to speak to"); + return Err(String::from("git check-attr gave no pipe to speak to")); }; // The two pipes move at the same time, on two threads, and that is not a @@ -103,7 +120,7 @@ pub(crate) fn not_text_paths(root: &Path) -> (Vec, Option) { // Whatever git makes of the list, the handle is dropped when this // closure ends, and closing stdin is what tells `--stdin` the list // is finished. - sink.write_all(&listed).ok(); + sink.write_all(listed).ok(); }); source.read_to_end(&mut answered) }); @@ -112,18 +129,18 @@ pub(crate) fn not_text_paths(root: &Path) -> (Vec, Option) { // complete answer from a truncated one. let finished = child.wait(); if drained.is_err() { - return unmeasured("its answer could not be read to the end"); + return Err(String::from("its answer could not be read to the end")); } match finished { Ok(status) if status.success() => {} Ok(status) => { - return unmeasured(&format!( + return Err(format!( "git check-attr exited {}", status.code().unwrap_or(-1) )); } Err(error) => { - return unmeasured(&format!("git check-attr could not be waited for: {error}")); + return Err(format!("git check-attr could not be waited for: {error}")); } } @@ -138,7 +155,7 @@ pub(crate) fn not_text_paths(root: &Path) -> (Vec, Option) { found.push(String::from_utf8_lossy(path).into_owned()); } } - (found, None) + Ok(found) } /// Every path in git's index, NUL separated, exactly as git wrote them. @@ -170,6 +187,281 @@ fn index_paths(root: &Path) -> Option> { ) } +/// Every file a `files.reach = "pinned"` rule may select, read once per run. +/// +/// The pins are the gitlinks the index records -- mode `160000` in +/// `git ls-files -s`, the same entries [`from_index`] passes over as another +/// repository's content -- and each is asked for its own index by running +/// `git ls-files` inside it, with the hooked repository's environment taken +/// away. Not `git ls-files --recurse-submodules`: that follows git's +/// active-submodule filter, so a mount marked `submodule..active = false` +/// would be left out without a word, and a rule claiming the pinned content +/// would pass over content it never read. The pin is the claim; git's +/// activity setting is not. +/// +/// A member that pins members of its own is followed the same way, because a +/// pin is a claim at every depth: the superproject pins the member's commit, +/// and that commit pins its own members. +/// +/// Nothing a member declares about policy is read -- not its principles file, +/// not its excludes. The member's index and its `.gitattributes` are read +/// because they are git's answer to what the member tracks and what it +/// declares not text, and the selection is judged by the superproject's rule. +#[derive(Debug, Default)] +pub(crate) struct Pinned { + /// The superproject's own tracked paths and every pinned member's, the + /// member's under its mount path. + tracked: Vec, + /// Every mount, at every depth, as a superproject-relative path. + mounts: Vec, + /// What the members declare not text, under their mount paths. The + /// superproject's own declarations are `not_text_paths`'. + not_text: Vec, + /// A member whose `.gitattributes` could not be asked, with the reason -- + /// carried to every selection a pinned rule builds, and so to exit 2. + unmeasured: Vec, +} + +impl Pinned { + /// Read the pins under `root`. + /// + /// `Ok(None)` where there is no index to read, which is the answer + /// [`index_bytes`] gives: a pinned rule there walks the tree like any + /// other. A pin whose mount cannot be read -- not checked out, marked + /// inactive, or an index git will not list -- is a `Fatal`, because a rule + /// that claims the pinned content and could not read part of it has not + /// looked. + pub(crate) fn read(root: &Path) -> Result> { + let Some(listed) = staged_index(root, false) else { + return Ok(None); + }; + let mut pinned = Self::default(); + pinned.gather(root, "", &listed)?; + Ok(Some(pinned)) + } + + /// What the members declare not text, under their mount paths. + pub(crate) fn not_text(&self) -> &[String] { + &self.not_text + } + + /// The mount that holds `relative`, the deepest one where mounts nest, and + /// the path relative to that member's own root. `None` for a path of the + /// superproject's own. + pub(crate) fn mount_of<'path>(&self, relative: &'path str) -> Option<(&str, &'path str)> { + self.mounts + .iter() + .filter_map(|mount| { + relative + .strip_prefix(mount.as_str()) + .and_then(|rest| rest.strip_prefix('/')) + .map(|rest| (mount.as_str(), rest)) + }) + .max_by_key(|(mount, _)| mount.len()) + } + + /// One repository's index: its files under `prefix`, and each pin followed. + fn gather(&mut self, root: &Path, prefix: &str, listed: &[u8]) -> Result<()> { + let (files, pins) = split_index(listed); + if !prefix.is_empty() { + self.ask_not_text(root, prefix, &files); + } + self.tracked + .extend(files.iter().map(|file| under_mount(prefix, file))); + let inactive = inactive_pins(&root.join(prefix), !prefix.is_empty(), &pins)?; + for pin in &pins { + let mount = under_mount(prefix, pin); + let member = open_mount(root, prefix, pin, &mount, inactive.contains(pin))?; + self.mounts.push(mount.clone()); + self.gather(root, &mount, &member)?; + } + Ok(()) + } + + /// A member's own `-text` declarations, asked inside the member. + fn ask_not_text(&mut self, root: &Path, mount: &str, files: &[String]) { + if files.is_empty() { + return; + } + let mut listed: Vec = Vec::new(); + for file in files { + listed.extend_from_slice(file.as_bytes()); + listed.push(0); + } + match declared_not_text(&root.join(mount), &listed, true) { + Ok(found) => self + .not_text + .extend(found.iter().map(|path| under_mount(mount, path))), + Err(reason) => self.unmeasured.push(format!( + "{mount}/.gitattributes: {reason}, so which paths the repository pinned at \ + {mount} declares are not text is unknown. Every path it tracks was treated \ + as text, which means a declared binary file was searched by the content \ + rules rather than skipped." + )), + } + } +} + +/// `path` under `prefix`, with git's separator whatever the platform's is. +fn under_mount(prefix: &str, path: &str) -> String { + if prefix.is_empty() { + path.to_owned() + } else { + format!("{prefix}/{path}") + } +} + +/// `git ls-files -z -s` in `directory`: the index with each entry's mode, which +/// is where a gitlink is told apart from a file. `None` where git said no. +fn staged_index(directory: &Path, elsewhere: bool) -> Option> { + let mut command = crate::shim::inner_tool("git"); + if elsewhere { + crate::git::elsewhere(&mut command); + } + let listed = command + .args(["ls-files", "-z", "-s"]) + .current_dir(directory) + .stderr(Stdio::null()) + .output() + .ok()?; + listed.status.success().then_some(listed.stdout) +} + +/// One `git ls-files -z -s` listing, split into the paths it tracks as files +/// and the paths it pins, each sorted and once. A path in conflict is listed +/// once per stage by git, and is one path here. +fn split_index(listed: &[u8]) -> (Vec, Vec) { + let mut files: BTreeSet = BTreeSet::new(); + let mut pins: BTreeSet = BTreeSet::new(); + for entry in listed.split(|byte| *byte == 0) { + let entry = String::from_utf8_lossy(entry); + // ` \t`: the tab is the one separator a + // path cannot be confused with, since git quotes nothing under `-z`. + let Some((meta, path)) = entry.split_once('\t') else { + continue; + }; + if meta.split(' ').next() == Some("160000") { + pins.insert(path.to_owned()); + } else { + files.insert(path.to_owned()); + } + } + (files.into_iter().collect(), pins.into_iter().collect()) +} + +/// The index of the member pinned at `pin` in the repository at `parent`, or +/// why it cannot be read. +/// +/// Refused in the shape `uphold supply-chain` refuses a moved submodule that is +/// not checked out: the mount by name and the command that fixes it. A mount +/// git marks inactive is refused even where a checkout is present, because +/// that is the mount `--recurse-submodules` would have skipped without a word, +/// and `git submodule update --init ` is what marks it active again. +fn open_mount( + root: &Path, + parent: &str, + pin: &str, + mount: &str, + inactive: bool, +) -> Result> { + let remedy = if parent.is_empty() { + format!("git submodule update --init {pin}") + } else { + format!("git -C {parent} submodule update --init {pin}") + }; + let member = root.join(mount); + if !crate::git::is_checked_out(&member) { + return Err(Fatal::new(format!( + "the repository pinned at {mount} is not checked out, so the files a \ + `files.reach = \"pinned\"` rule claims cannot be read. Run `{remedy}`. A rule \ + that claims the pinned content and cannot read part of it has not looked" + ))); + } + if inactive { + return Err(Fatal::new(format!( + "the repository pinned at {mount} is checked out and git marks it inactive \ + (`submodule..active`, or a `submodule.active` that does not match it), so \ + the files a `files.reach = \"pinned\"` rule claims are not ones this checkout \ + keeps current. Run `{remedy}`, which marks it active. A rule that claims the \ + pinned content and cannot read part of it has not looked" + ))); + } + staged_index(&member, true).ok_or_else(|| { + Fatal::new(format!( + "git ls-files failed inside the repository pinned at {mount}, so the files a \ + `files.reach = \"pinned\"` rule claims cannot be listed. Run `{remedy}`. A rule \ + that claims the pinned content and cannot read part of it has not looked" + )) + }) +} + +/// Which of `pins` git counts inactive in the repository at `container`. +/// +/// `git submodule status` marks an inactive submodule `-`, which is git's own +/// reading of `submodule..active`, `submodule.active` and the URL +/// fallback together, rather than a second reading of the three here. Asked +/// once for the repository, because each call costs a process that walks +/// every submodule's state, and asked per pin only where the whole question +/// was refused -- git refuses it outright when any one gitlink has no +/// `.gitmodules` entry -- or a pin's line could not be found in the answer. +fn inactive_pins(container: &Path, elsewhere: bool, pins: &[String]) -> Result> { + let mut inactive = BTreeSet::new(); + if pins.is_empty() { + return Ok(inactive); + } + let listing = submodule_status(container, elsewhere, None)?; + for pin in pins { + let flag = match listing + .as_deref() + .and_then(|listing| status_flag(listing, pin)) + { + Some(flag) => Some(flag), + None => submodule_status(container, elsewhere, Some(pin))? + .as_deref() + .and_then(|alone| status_flag(alone, pin)), + }; + if flag == Some('-') { + inactive.insert(pin.clone()); + } + } + Ok(inactive) +} + +/// `git submodule status`, for the repository or for one pin in it. `None` +/// where git refused, which for a gitlink with no `.gitmodules` entry is the +/// answer: it has no activity setting to read, and is not inactive -- its +/// checkout is read or refused on whether it is there. +fn submodule_status( + container: &Path, + elsewhere: bool, + pin: Option<&str>, +) -> Result> { + let mut args = vec!["submodule", "status"]; + if let Some(pin) = pin { + args.extend(["--", pin]); + } + if elsewhere { + crate::git::try_run_elsewhere(container, &args) + } else { + crate::git::try_run(container, &args) + } +} + +/// The flag `git submodule status` gave `pin`: ` `, `-`, `+` or `U`. +/// +/// A line is ` `, followed by ` ()` for a +/// checkout. The path is matched whole against the pin, so `sub` does not +/// answer for `sub2`, nor `sub (x` for `sub`. +fn status_flag(listing: &str, pin: &str) -> Option { + let described = format!("{pin} ("); + listing.lines().find_map(|line| { + let mut characters = line.chars(); + let flag = characters.next()?; + let (_, path) = characters.as_str().split_once(' ')?; + (path == pin || path.starts_with(&described)).then_some(flag) + }) +} + /// The files one rule searches, chosen once, at build time. /// /// Chosen at build time because every way choosing them can fail -- an @@ -195,6 +487,10 @@ pub(crate) struct Selection { impl Selection { pub(crate) fn build(root: &Path, rule: &Rule, not_text: &[String]) -> Result { + if rule.reach() == Reach::Pinned { + let pinned = Pinned::read(root)?; + return Self::build_over(root, rule, not_text, pinned.as_ref()); + } let overrides = overrides_for(root, rule, not_text)?; let roots = search_roots(root, rule)?; // An index if there is one, and a walk only where there is not. @@ -205,6 +501,35 @@ impl Selection { Ok(Self { files, unreadable }) } + /// The selection of a `files.reach = "pinned"` rule, over pins the caller + /// read once for the whole run rather than once per rule. + /// + /// The globs are the superproject's and are applied to superproject-relative + /// paths, so gitignore's rooting holds across the mounts: a leading-slash + /// exclude is anchored at the superproject's root, and a bare name matches + /// at any depth, inside a mount as well. `None` is a tree with no index, + /// which is walked as a repository rule's is. + pub(crate) fn build_over( + root: &Path, + rule: &Rule, + not_text: &[String], + pinned: Option<&Pinned>, + ) -> Result { + let Some(pinned) = pinned else { + let overrides = overrides_for(root, rule, not_text)?; + let roots = search_roots(root, rule)?; + let (files, unreadable) = by_walking(root, &roots, &overrides); + return Ok(Self { files, unreadable }); + }; + let mut declared = not_text.to_vec(); + declared.extend(pinned.not_text.iter().cloned()); + let overrides = overrides_for(root, rule, &declared)?; + let roots = search_roots(root, rule)?; + let (files, mut unreadable) = from_index(root, &roots, &overrides, &pinned.tracked); + unreadable.extend(pinned.unmeasured.iter().cloned()); + Ok(Self { files, unreadable }) + } + /// Repository-relative paths, sorted, deduplicated. /// /// Sorted because a report whose order depends on directory iteration is a @@ -806,4 +1131,306 @@ mod tests { "{declared:?}" ); } + + // -- files.reach = "pinned" ---------------------------------------------- + + /// A superproject pinning `sub`, whose own tracked files are `files`. + fn superproject(label: &str, files: &[(&str, &str)]) -> PathBuf { + let member = repository(&format!("{label}-member")); + for (relative, contents) in files { + write(&member, relative, contents); + } + crate::fixture::git(&member, &["add", "-f", "-A", "."]); + crate::fixture::git(&member, &["commit", "-q", "-m", "member"]); + let root = repository(label); + write(&root, "README.md", "root\n"); + crate::fixture::git( + &root, + &[ + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "-q", + &member.display().to_string(), + "sub", + ], + ); + crate::fixture::git(&root, &["add", "-f", "-A", "."]); + root + } + + fn pinned(files: Files) -> Rule { + rule(Files { + reach: Some(Reach::Pinned), + ..files + }) + } + + #[test] + fn the_index_splits_into_files_and_pins_and_a_conflict_is_one_path() { + let listed = b"100644 aaaa 0\ta file.txt\0\ + 160000 bbbb 0\tsub\0\ + 100644 cccc 1\tboth.txt\0\ + 100644 dddd 2\tboth.txt\0\ + 120000 eeee 0\tlink\0\ + no tab here\0"; + let (files, pins) = split_index(listed); + assert_eq!(files, ["a file.txt", "both.txt", "link"]); + assert_eq!(pins, ["sub"]); + } + + #[test] + fn a_path_under_an_include_root_is_selected_and_one_outside_it_is_not() { + let root = repository("selects"); + let scoped = rule(Files { + include: Some(vec!["src".to_owned()]), + ..Files::default() + }); + assert!(selects(&root, &scoped, Path::new("src/a.rs")).unwrap()); + assert!(!selects(&root, &scoped, Path::new("docs/a.md")).unwrap()); + assert!(selects(&root, &rule(Files::default()), Path::new("a.md")).unwrap()); + } + + #[test] + fn a_status_line_answers_for_its_own_pin_and_no_other() { + let listing = + "-1111 sub\n 2222 sub2 (heads/main)\n+3333 a b (v1.0-2-g3333)\nU4444 odd (x\n"; + assert_eq!(status_flag(listing, "sub"), Some('-')); + assert_eq!(status_flag(listing, "sub2"), Some(' ')); + assert_eq!(status_flag(listing, "a b"), Some('+')); + assert_eq!(status_flag(listing, "odd (x"), Some('U')); + assert_eq!(status_flag(listing, "su"), None); + assert_eq!(status_flag("", "sub"), None); + } + + #[test] + fn a_path_under_a_mount_is_joined_with_gits_separator() { + assert_eq!(under_mount("", "a.txt"), "a.txt"); + assert_eq!(under_mount("sub", "docs/a.txt"), "sub/docs/a.txt"); + } + + #[test] + fn the_mount_that_holds_a_path_is_the_deepest_and_not_a_name_prefix() { + let pinned = Pinned { + mounts: vec!["a".to_owned(), "a/b".to_owned(), "ab".to_owned()], + ..Pinned::default() + }; + assert_eq!(pinned.mount_of("a/b/c.md"), Some(("a/b", "c.md"))); + assert_eq!(pinned.mount_of("a/c.md"), Some(("a", "c.md"))); + assert_eq!(pinned.mount_of("ab/c.md"), Some(("ab", "c.md"))); + assert_eq!(pinned.mount_of("abc/c.md"), None); + assert_eq!(pinned.mount_of("a"), None); + assert_eq!(pinned.mount_of("README.md"), None); + } + + #[test] + fn the_pins_are_read_from_the_index_and_each_member_asked_for_its_own() { + let root = superproject("pins", &[("a.txt", "a\n"), ("docs/b.md", "b\n")]); + let pinned = Pinned::read(&root).unwrap().unwrap(); + assert_eq!(pinned.mounts, ["sub"]); + for path in ["README.md", ".gitmodules", "sub/a.txt", "sub/docs/b.md"] { + assert!( + pinned.tracked.iter().any(|tracked| tracked == path), + "{path} in {:?}", + pinned.tracked + ); + } + // The gitlink itself is a pointer and not a file of anybody's. + assert!(!pinned.tracked.iter().any(|tracked| tracked == "sub")); + assert!(pinned.unmeasured.is_empty(), "{:?}", pinned.unmeasured); + } + + #[test] + fn a_directory_with_no_index_has_no_pins_to_read() { + let root = workspace("pins-no-index"); + assert!(Pinned::read(&root).unwrap().is_none()); + } + + #[test] + fn a_pinned_rule_selects_the_members_files_and_a_repository_rule_does_not() { + let root = superproject("reach", &[("a.txt", "a\n")]); + let reaching = Selection::build(&root, &pinned(Files::default()), &[]) + .unwrap() + .files(); + assert!(reaching.contains(&"sub/a.txt".to_owned()), "{reaching:?}"); + assert!(reaching.contains(&"README.md".to_owned()), "{reaching:?}"); + let staying = selected(&root, Files::default()); + assert!( + !staying.iter().any(|path| path.starts_with("sub/")), + "{staying:?}" + ); + assert!(staying.contains(&"README.md".to_owned()), "{staying:?}"); + } + + #[test] + fn an_anchored_exclude_stays_at_the_superproject_root_and_a_bare_one_reaches_into_mounts() { + let root = superproject("reach-globs", &[("vendor.txt", "m\n"), ("keep.txt", "k\n")]); + write(&root, "vendor.txt", "r\n"); + crate::fixture::git(&root, &["add", "-f", "vendor.txt"]); + + let anchored = Selection::build( + &root, + &pinned(Files { + exclude: vec!["/vendor.txt".to_owned()], + ..Files::default() + }), + &[], + ) + .unwrap() + .files(); + assert!(!anchored.contains(&"vendor.txt".to_owned()), "{anchored:?}"); + assert!( + anchored.contains(&"sub/vendor.txt".to_owned()), + "{anchored:?}" + ); + + let bare = Selection::build( + &root, + &pinned(Files { + exclude: vec!["vendor.txt".to_owned()], + ..Files::default() + }), + &[], + ) + .unwrap() + .files(); + assert!( + !bare.iter().any(|path| path.ends_with("vendor.txt")), + "{bare:?}" + ); + assert!(bare.contains(&"sub/keep.txt".to_owned()), "{bare:?}"); + + // And a glob is the same: rooted where it has a slash, at any depth + // where it has none. + let globbed = Selection::build( + &root, + &pinned(Files { + glob: vec!["vendor.txt".to_owned()], + ..Files::default() + }), + &[], + ) + .unwrap() + .files(); + assert_eq!(globbed, ["sub/vendor.txt", "vendor.txt"]); + } + + #[test] + fn an_include_inside_a_mount_selects_only_under_it() { + let root = superproject("reach-include", &[("docs/a.md", "a\n"), ("b.md", "b\n")]); + let files = Selection::build( + &root, + &pinned(Files { + include: Some(vec!["sub/docs".to_owned()]), + ..Files::default() + }), + &[], + ) + .unwrap() + .files(); + assert_eq!(files, ["sub/docs/a.md"]); + } + + #[test] + fn a_members_not_text_declaration_is_asked_inside_the_member() { + let root = superproject( + "reach-attributes", + &[ + (".gitattributes", "*.bin -text\n"), + ("capture.bin", "c\n"), + ("a.txt", "a\n"), + ], + ); + let pinned_content = Pinned::read(&root).unwrap().unwrap(); + assert_eq!(pinned_content.not_text(), ["sub/capture.bin"]); + // The superproject's own question does not see it. + let (own, unmeasured) = not_text_paths(&root); + assert!(own.is_empty(), "{own:?}"); + assert!(unmeasured.is_none()); + + let files = Selection::build(&root, &pinned(Files::default()), &[]) + .unwrap() + .files(); + assert!(!files.contains(&"sub/capture.bin".to_owned()), "{files:?}"); + assert!(files.contains(&"sub/a.txt".to_owned()), "{files:?}"); + } + + #[test] + fn a_member_that_cannot_be_asked_for_its_attributes_is_unmeasured_not_clean() { + let mut pinned_content = Pinned::default(); + let root = workspace("reach-unmeasured"); + // Not a repository, so `check-attr` exits non-zero inside it. + std::fs::create_dir_all(root.join("sub")).unwrap(); + pinned_content.ask_not_text(&root, "sub", &["a.txt".to_owned()]); + assert!(pinned_content.not_text.is_empty()); + assert_eq!( + pinned_content.unmeasured.len(), + 1, + "{:?}", + pinned_content.unmeasured + ); + assert!( + pinned_content.unmeasured[0].starts_with("sub/.gitattributes: "), + "{:?}", + pinned_content.unmeasured + ); + // And carried to exit 2 through every pinned selection. + let selection = + Selection::build_over(&root, &pinned(Files::default()), &[], Some(&pinned_content)) + .unwrap(); + assert_eq!(selection.unreadable(), pinned_content.unmeasured.as_slice()); + } + + #[test] + fn an_uninitialised_mount_is_refused_naming_it_and_the_remedy() { + let root = superproject("reach-uninitialised", &[("a.txt", "a\n")]); + crate::fixture::git(&root, &["commit", "-q", "-m", "pin"]); + crate::fixture::git(&root, &["submodule", "deinit", "-q", "-f", "sub"]); + let error = Pinned::read(&root).unwrap_err().to_string(); + assert!( + error.contains("pinned at sub is not checked out"), + "{error}" + ); + assert!( + error.contains("`git submodule update --init sub`"), + "{error}" + ); + } + + #[test] + fn a_checked_out_mount_git_marks_inactive_is_refused() { + let root = superproject("reach-inactive", &[("a.txt", "a\n")]); + crate::fixture::git(&root, &["config", "submodule.sub.active", "false"]); + let error = Pinned::read(&root).unwrap_err().to_string(); + assert!(error.contains("marks it inactive"), "{error}"); + assert!( + error.contains("`git submodule update --init sub`"), + "{error}" + ); + assert!( + inactive_pins(&root, false, &["absent".to_owned()]) + .unwrap() + .is_empty() + ); + } + + #[test] + fn a_gitlink_with_no_gitmodules_entry_is_read_where_it_is_checked_out() { + // `git add` of a nested repository records a gitlink with no URL, so + // git keeps no activity setting for it and refuses to be asked. The + // pin is still the claim, and its checkout is right there. + let root = repository("reach-embedded"); + let embedded = root.join("embedded"); + std::fs::create_dir_all(&embedded).unwrap(); + crate::fixture::git(&embedded, &["init", "-q", "-b", "main"]); + crate::fixture::git(&embedded, &["config", "user.name", "Test"]); + crate::fixture::git(&embedded, &["config", "user.email", "test@example.test"]); + write(&embedded, "inside.txt", "i\n"); + crate::fixture::git(&embedded, &["add", "inside.txt"]); + crate::fixture::git(&embedded, &["commit", "-q", "-m", "embedded"]); + crate::fixture::git(&root, &["add", "embedded"]); + let pinned_content = Pinned::read(&root).unwrap().unwrap(); + assert_eq!(pinned_content.tracked, ["embedded/inside.txt"]); + } } diff --git a/src/supply.rs b/src/supply.rs index 2a8f9c3..6881eb9 100644 --- a/src/supply.rs +++ b/src/supply.rs @@ -749,7 +749,7 @@ fn expand_gitlink( out: &mut BTreeSet, ) -> Result<()> { let directory = root.join(submodule); - if directory.join(".git").symlink_metadata().is_err() { + if !crate::git::is_checked_out(&directory) { return Err(Fatal::new(format!( "the submodule {} moved in this range and is not checked out, so its manifests \ cannot be read. Run `git submodule update --init {}`, or scan the whole tree \ diff --git a/tests/reach_cli.rs b/tests/reach_cli.rs new file mode 100644 index 0000000..3459be7 --- /dev/null +++ b/tests/reach_cli.rs @@ -0,0 +1,571 @@ +//! CLI tests for `files.reach`: a rule may reach the content its repository +//! pins. +//! +//! Every fixture is a real superproject with a real submodule, because what is +//! under test is what git reports about the pins -- a mode-160000 entry, a +//! member's own index, a mount git marks inactive -- and a stand-in for any of +//! those would pass a test git itself would fail. + +#![expect( + clippy::let_underscore_must_use, + clippy::tests_outside_test_module, + clippy::unwrap_used, + reason = "A CLI test asserts on the outcome; a panic in the harness that builds the fixture IS the failure report, and there is no caller to hand a Result to" +)] + +mod support; + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +/// A pattern that does not match its own spelling, so the policy file that +/// declares it is not a finding of its own. +const CANARY: &str = "CANAR[Y]"; + +fn repository(kind: &str) -> PathBuf { + let root = support::scratch(kind); + std::fs::create_dir_all(&root).unwrap(); + support::git(&root, &["init", "-q", "-b", "main"]); + support::git(&root, &["config", "user.name", "Test"]); + support::git(&root, &["config", "user.email", "test@example.test"]); + root +} + +fn write(root: &Path, relative: &str, contents: &str) { + let path = root.join(relative); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent).unwrap(); + } + std::fs::write(path, contents).unwrap(); +} + +fn commit(root: &Path, message: &str) { + support::git(root, &["add", "-A"]); + support::git(root, &["commit", "-q", "--allow-empty", "-m", message]); +} + +/// A rule refusing the canary, at the reach given, with `extra` lines added to +/// its `files` table. +fn policy(root: &Path, reach: Option<&str>, extra: &str) { + let reach = reach.map_or_else(String::new, |reach| format!("reach = \"{reach}\"\n")); + write( + root, + "policy/principles.toml", + &format!( + "[rule.no-canary]\nmessage = \"no canary\"\nregexp = '{CANARY}'\n\n\ + [rule.no-canary.files]\n{reach}{extra}\n" + ), + ); +} + +/// A superproject pinning one member, `canary`, whose `docs/note.md` carries +/// the canary, and whose own policy would exclude every file it has -- which a +/// pinned rule of the superproject's must not read. +fn superproject(kind: &str) -> PathBuf { + let root = repository(kind); + write(&root, "README.md", "the superproject's own, and clean\n"); + support::submodule( + &root, + "canary", + &[ + ("docs/note.md", "fine\nCANARY here\n"), + ( + "policy/principles.toml", + "[rule.no-canary]\nmessage = \"the member's own\"\nregexp = 'CANAR[Y]'\n\ + [rule.no-canary.files]\nexclude = [\"**\"]\n", + ), + ], + ); + commit(&root, "pin the member"); + root +} + +fn uphold(root: &Path, args: &[&str]) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_uphold")); + support::without_git_environment(&mut command); + command.args(args).current_dir(root).output().unwrap() +} + +fn scan(root: &Path) -> Output { + uphold(root, &["scan"]) +} + +fn code(output: &Output) -> i32 { + output.status.code().unwrap() +} + +fn text(output: &Output) -> String { + let mut all = String::from_utf8_lossy(&output.stdout).into_owned(); + all.push_str(&String::from_utf8_lossy(&output.stderr)); + all +} + +#[test] +fn a_pinned_rule_finds_the_canary_inside_the_member_under_the_mount_path() { + let root = superproject("reach-pinned"); + policy(&root, Some("pinned"), ""); + let output = scan(&root); + assert_eq!(code(&output), 1, "{}", text(&output)); + assert!( + text(&output).contains("canary/docs/note.md:2:"), + "{}", + text(&output) + ); +} + +#[test] +fn a_repository_rule_on_the_same_tree_does_not_look_inside_the_member() { + let root = superproject("reach-repository"); + for reach in [None, Some("repository")] { + policy(&root, reach, ""); + let output = scan(&root); + assert_eq!(code(&output), 0, "{reach:?}: {}", text(&output)); + assert!(!text(&output).contains("canary/"), "{}", text(&output)); + } +} + +#[test] +fn a_pinned_rule_in_a_repository_that_pins_nothing_is_a_repository_rule() { + let root = repository("reach-no-pins"); + write(&root, "a.txt", "CANARY\n"); + commit(&root, "no pins"); + for reach in [Some("pinned"), None] { + policy(&root, reach, ""); + let output = scan(&root); + assert_eq!(code(&output), 1, "{reach:?}: {}", text(&output)); + assert!(text(&output).contains("a.txt:1:"), "{}", text(&output)); + } +} + +#[test] +fn an_uninitialised_member_is_exit_2_naming_the_mount_and_the_remedy() { + let source = superproject("reach-uninitialised-source"); + policy(&source, Some("pinned"), "exclude = [\"/policy/**\"]"); + commit(&source, "the policy"); + let clone = support::scratch("reach-uninitialised"); + support::git( + source.parent().unwrap(), + &[ + "clone", + "-q", + &source.display().to_string(), + &clone.display().to_string(), + ], + ); + assert!( + std::fs::read_dir(clone.join("canary")) + .unwrap() + .next() + .is_none() + ); + + let output = scan(&clone); + assert_eq!(code(&output), 2, "{}", text(&output)); + assert!(text(&output).contains("canary"), "{}", text(&output)); + assert!( + text(&output).contains("git submodule update --init canary"), + "{}", + text(&output) + ); + assert!(!text(&output).contains("policy checks passed")); + + // The same tree read by a rule of its own repository is not a claim on + // the member, and the absent checkout is nothing to it. + policy(&clone, None, "exclude = [\"/policy/**\"]"); + assert_eq!(code(&scan(&clone)), 0, "{}", text(&scan(&clone))); +} + +#[test] +fn a_checked_out_member_git_marks_inactive_is_exit_2_until_the_remedy_is_run() { + // The case `git ls-files --recurse-submodules` would have skipped without + // a word: the files are on disk, and git's activity filter leaves them out. + let root = superproject("reach-inactive"); + policy(&root, Some("pinned"), ""); + support::git(&root, &["config", "submodule.canary.active", "false"]); + assert!(root.join("canary/docs/note.md").is_file()); + + let output = scan(&root); + assert_eq!(code(&output), 2, "{}", text(&output)); + assert!(text(&output).contains("inactive"), "{}", text(&output)); + assert!( + text(&output).contains("git submodule update --init canary"), + "{}", + text(&output) + ); + + // The remedy it names is the one that works. + support::git( + &root, + &[ + "-c", + "protocol.file.allow=always", + "submodule", + "update", + "--init", + "canary", + ], + ); + let remedied = scan(&root); + assert_eq!(code(&remedied), 1, "{}", text(&remedied)); + assert!( + text(&remedied).contains("canary/docs/note.md:2:"), + "{}", + text(&remedied) + ); +} + +#[test] +fn a_leading_slash_exclude_is_anchored_at_the_superproject_and_a_bare_name_is_not() { + let root = superproject("reach-globs"); + write(&root, "vendor.txt", "CANARY at the root\n"); + write(&root.join("canary"), "vendor.txt", "CANARY in the member\n"); + commit(&root.join("canary"), "the member's vendor file"); + commit(&root, "the root's vendor file, and the bumped pin"); + + policy( + &root, + Some("pinned"), + "exclude = [\"/vendor.txt\", \"note.md\"]", + ); + let output = scan(&root); + assert_eq!(code(&output), 1, "{}", text(&output)); + assert!( + text(&output).contains("canary/vendor.txt:1:"), + "{}", + text(&output) + ); + assert!( + !text(&output).contains("\nvendor.txt:1:") && !text(&output).starts_with("vendor.txt"), + "the anchored exclude did not exempt the root's own file: {}", + text(&output) + ); + + policy( + &root, + Some("pinned"), + "exclude = [\"vendor.txt\", \"note.md\"]", + ); + let bare = scan(&root); + assert_eq!(code(&bare), 0, "{}", text(&bare)); +} + +#[test] +fn an_include_may_name_a_path_inside_a_mount_and_the_floor_counts_the_member_files() { + let root = superproject("reach-include"); + policy( + &root, + Some("pinned"), + "include = [\"canary/docs\"]\nmin_selected = 1", + ); + let output = scan(&root); + assert_eq!(code(&output), 1, "{}", text(&output)); + assert!( + text(&output).contains("canary/docs/note.md:2:"), + "{}", + text(&output) + ); + assert!( + !text(&output).contains("selection floor"), + "{}", + text(&output) + ); + + // The same include at repository reach selects nothing under the mount, + // and the floor says so. + policy(&root, None, "include = [\"canary/docs\"]\nmin_selected = 1"); + let unpinned = scan(&root); + assert!( + text(&unpinned).contains("selected 0 file(s)"), + "{}", + text(&unpinned) + ); +} + +#[test] +fn a_members_own_not_text_declaration_is_honoured_for_its_files() { + let root = superproject("reach-attributes"); + let member = root.join("canary"); + write(&member, ".gitattributes", "*.bin -text\n"); + write(&member, "capture.bin", "CANARY in a declared capture\n"); + commit(&member, "a declared capture"); + commit(&root, "bump the pin"); + + policy(&root, Some("pinned"), "exclude = [\"note.md\"]"); + let output = scan(&root); + assert_eq!(code(&output), 0, "{}", text(&output)); + assert!( + text(&output).contains("declared not text in .gitattributes"), + "{}", + text(&output) + ); + assert!( + text(&output).contains(" canary/capture.bin"), + "{}", + text(&output) + ); +} + +#[test] +fn a_path_baseline_keyed_under_the_mount_suppresses_exactly_that_finding() { + let root = superproject("reach-baseline"); + let member = root.join("canary"); + write(&member, "other.md", "CANARY too\n"); + commit(&member, "a second canary"); + commit(&root, "bump the pin"); + write(&root, "policy/canary.baseline", "canary/docs/note.md\n"); + + policy( + &root, + Some("pinned"), + "baseline = \"policy/canary.baseline\"", + ); + let output = scan(&root); + assert_eq!(code(&output), 1, "{}", text(&output)); + assert!( + text(&output).contains("canary/other.md:1:"), + "{}", + text(&output) + ); + assert!( + !text(&output).contains("canary/docs/note.md"), + "{}", + text(&output) + ); + assert!(!text(&output).contains("stale"), "{}", text(&output)); +} + +#[test] +fn a_size_baseline_keyed_under_the_mount_holds_that_file_and_no_other() { + let root = superproject("reach-size"); + let member = root.join("canary"); + write(&member, "held.txt", "1\n2\n3\n"); + write(&member, "free.txt", "1\n2\n3\n"); + commit(&member, "two long files"); + commit(&root, "bump the pin"); + write(&root, "policy/size.baseline", "canary/held.txt 3\n"); + write( + &root, + "policy/principles.toml", + "[rule.short]\nmessage = \"short files\"\nmax_lines = 2\n\n[rule.short.files]\n\ + reach = \"pinned\"\nglob = [\"*.txt\"]\nbaseline = \"policy/size.baseline\"\n", + ); + let output = scan(&root); + assert_eq!(code(&output), 1, "{}", text(&output)); + assert!( + text(&output).contains("canary/free.txt: 3 lines"), + "{}", + text(&output) + ); + assert!( + !text(&output).contains("canary/held.txt"), + "{}", + text(&output) + ); +} + +#[test] +fn a_leading_slash_link_in_a_member_resolves_against_the_members_root() { + let root = superproject("reach-links"); + let member = root.join("canary"); + write( + &member, + "guide.md", + "[ok](/docs/note.md)\n[gone](/docs/absent.md)\n", + ); + commit(&member, "a guide"); + commit(&root, "bump the pin"); + write( + &root, + "policy/principles.toml", + "[rule.links]\nbuiltin = \"links-resolve\"\nmessage = \"links resolve\"\n\n\ + [rule.links.files]\nreach = \"pinned\"\nglob = [\"*.md\"]\n", + ); + let output = scan(&root); + assert_eq!(code(&output), 1, "{}", text(&output)); + assert!( + text(&output).contains("canary/guide.md:2: /docs/absent.md -> no such file"), + "{}", + text(&output) + ); + // `/docs/note.md` is the member's, and the superproject has no `docs/`. + assert!(!root.join("docs").exists()); + assert!( + !text(&output).contains("/docs/note.md ->"), + "{}", + text(&output) + ); +} + +/// A superproject pinning `outer`, which pins `inner`, whose `deep.md` carries +/// the canary. `inner` is not checked out: the clone `submodule add` made of +/// the outer member carries the pin and not the content. +fn nested(kind: &str) -> PathBuf { + let outer = repository(&format!("{kind}-outer-source")); + write(&outer, "outer.md", "clean\n"); + support::submodule(&outer, "inner", &[("deep.md", "CANARY at depth\n")]); + commit(&outer, "pin the inner member"); + + let root = repository(kind); + write(&root, "README.md", "clean\n"); + support::git( + &root, + &[ + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "-q", + &outer.display().to_string(), + "outer", + ], + ); + commit(&root, "pin the outer member"); + policy(&root, Some("pinned"), "exclude = [\"/policy/**\"]"); + root +} + +/// Check out the inner member of a [`nested`] fixture. +fn initialise_inner(root: &Path) { + support::git( + &root.join("outer"), + &[ + "-c", + "protocol.file.allow=always", + "submodule", + "update", + "--init", + "inner", + ], + ); +} + +#[test] +fn a_member_that_pins_a_member_is_followed_at_every_depth() { + // The ADR leaves the depth open. A pin is a claim at every depth: the + // superproject pins the outer member's commit, and that commit pins its + // own member, so a rule claiming the pinned content reads both. + let root = nested("reach-nested"); + let output = scan(&root); + assert_eq!(code(&output), 2, "{}", text(&output)); + assert!(text(&output).contains("outer/inner"), "{}", text(&output)); + assert!( + text(&output).contains("git -C outer submodule update --init inner"), + "{}", + text(&output) + ); + + initialise_inner(&root); + let initialised = scan(&root); + assert_eq!(code(&initialised), 1, "{}", text(&initialised)); + assert!( + text(&initialised).contains("outer/inner/deep.md:1:"), + "{}", + text(&initialised) + ); +} + +#[test] +fn a_scan_run_from_a_hook_asks_each_member_about_itself() { + // A hook runner exports `GIT_DIR` and `GIT_INDEX_FILE` for the repository + // the hook fired in, and each outranks `current_dir`. A git asked about a + // member with them still set answers about the superproject: its + // submodules, its index. So the member is asked with them taken away, at + // every depth -- here the inner member, which only the outer member's own + // configuration marks inactive. + let root = nested("reach-hooked"); + initialise_inner(&root); + support::git( + &root.join("outer"), + &["config", "submodule.inner.active", "false"], + ); + let hooked = |superproject: &Path| { + let mut command = Command::new(env!("CARGO_BIN_EXE_uphold")); + support::without_git_environment(&mut command); + command + .arg("scan") + .env("GIT_DIR", superproject.join(".git")) + .env("GIT_INDEX_FILE", superproject.join(".git/index")) + .current_dir(superproject) + .output() + .unwrap() + }; + + let output = hooked(&root); + assert_eq!(code(&output), 2, "{}", text(&output)); + assert!( + text(&output).contains("pinned at outer/inner is checked out and git marks it inactive"), + "{}", + text(&output) + ); + + support::git( + &root.join("outer"), + &["config", "submodule.inner.active", "true"], + ); + let active = hooked(&root); + assert_eq!(code(&active), 1, "{}", text(&active)); + assert!( + text(&active).contains("outer/inner/deep.md:1:"), + "{}", + text(&active) + ); +} + +#[test] +fn the_effective_rules_show_a_pinned_reach_and_only_that() { + let root = superproject("reach-effective"); + write( + &root, + "policy/principles.toml", + "[rule.reaches]\nmessage = \"m\"\nregexp = 'CANAR[Y]'\nfiles.reach = \"pinned\"\n\n\ + [rule.stays]\nmessage = \"m\"\nregexp = 'CANAR[Y]'\nfiles.include = [\".\"]\n", + ); + let output = uphold(&root, &["rules", "--effective", "--json"]); + assert_eq!(code(&output), 0, "{}", text(&output)); + let rules: Vec = serde_json::from_slice(&output.stdout).unwrap(); + let entry = |id: &str| rules.iter().find(|rule| rule["id"] == id).unwrap().clone(); + assert_eq!( + entry("reaches"), + serde_json::json!({"id": "reaches", "git_hooks": [], "seams": ["scan"], "reach": "pinned"}) + ); + assert_eq!( + entry("stays"), + serde_json::json!({"id": "stays", "git_hooks": [], "seams": ["scan"]}) + ); + + let listed = uphold(&root, &["rules", "--effective"]); + assert!( + text(&listed).contains("reaches (scan) [reach: pinned]"), + "{}", + text(&listed) + ); +} + +#[test] +fn a_reach_that_is_neither_value_is_refused_naming_both() { + let root = superproject("reach-refused"); + policy(&root, Some("everywhere"), ""); + let output = scan(&root); + assert_eq!(code(&output), 2, "{}", text(&output)); + assert!(text(&output).contains("everywhere"), "{}", text(&output)); + assert!( + text(&output).contains("repository") && text(&output).contains("pinned"), + "{}", + text(&output) + ); +} + +#[test] +fn a_pinned_reach_on_a_guard_scope_is_refused() { + let root = superproject("reach-guard"); + write( + &root, + "policy/principles.toml", + "[rule.unicode]\nbuiltin = \"prevent-unusual-unicode-in-files\"\n\ + message = \"m\"\ngit.hooks = [\"pre-commit\"]\n\n\ + [rule.unicode.files]\nreach = \"pinned\"\n", + ); + let output = scan(&root); + assert_eq!(code(&output), 2, "{}", text(&output)); + assert!(text(&output).contains("files.reach"), "{}", text(&output)); +} diff --git a/tests/supply_chain_cli.rs b/tests/supply_chain_cli.rs index 46181a0..bfe4910 100644 --- a/tests/supply_chain_cli.rs +++ b/tests/supply_chain_cli.rs @@ -886,36 +886,7 @@ fn a_whole_tree_sweep_declares_the_ci_configuration_it_did_not_scan() { /// A member repository, cloned into the fixture as a real submodule. fn with_a_submodule(root: &Path) { - let member = support::scratch("supply-chain-member"); - std::fs::create_dir_all(&member).unwrap(); - support::git(&member, &["init", "-q", "-b", "main"]); - support::git(&member, &["config", "user.name", "Test"]); - support::git(&member, &["config", "user.email", "test@example.test"]); - write(&member, "uv.lock", "version = 1\n"); - commit(&member, "the member's own lock"); - // `protocol.file.allow` because git refuses a local-path submodule by - // default since CVE-2022-39253, and the fixture is exactly a local path. - support::git( - root, - &[ - "-c", - "protocol.file.allow=always", - "submodule", - "add", - "-q", - &member.display().to_string(), - "sub", - ], - ); - // The submodule in the working tree is a CLONE, and a clone carries none of - // the source repository's local config. Every other repository this fixture - // builds is handed an identity at `init`; this one is handed one here, - // because the tests that commit into it commit into the clone and not into - // the source. Without it the fixture borrows whoever is configured globally, - // which is a machine that has somebody -- and CI is a machine that does not. - let checkout = root.join("sub"); - support::git(&checkout, &["config", "user.name", "Test"]); - support::git(&checkout, &["config", "user.email", "test@example.test"]); + support::submodule(root, "sub", &[("uv.lock", "version = 1\n")]); commit(root, "track the member"); } diff --git a/tests/support/mod.rs b/tests/support/mod.rs index 4ee303b..ed3740a 100644 --- a/tests/support/mod.rs +++ b/tests/support/mod.rs @@ -217,3 +217,53 @@ pub fn git(root: &Path, args: &[&str]) { String::from_utf8_lossy(&output.stderr) ); } + +/// A member repository holding `files`, committed, and added under `root` at +/// `mount` as a real submodule. Returns the checkout inside `root`. +/// +/// The superproject is not committed: a caller that wants the pin recorded +/// commits it, and one building a member of a member adds more first. +pub fn submodule(root: &Path, mount: &str, files: &[(&str, &str)]) -> PathBuf { + let member = scratch("member"); + std::fs::create_dir_all(&member).expect("the member directory"); + git(&member, &["init", "-q", "-b", "main"]); + git(&member, &["config", "user.name", "Test"]); + git(&member, &["config", "user.email", "test@example.test"]); + for (relative, contents) in files { + let path = member.join(relative); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent).expect("a member subdirectory"); + } + std::fs::write(path, contents).expect("a member file"); + } + git(&member, &["add", "-A"]); + git( + &member, + &["commit", "-q", "--allow-empty", "-m", "the member's own"], + ); + // `protocol.file.allow` because git refuses a local-path submodule by + // default since CVE-2022-39253, and the fixture is exactly a local path. + git( + root, + &[ + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "-q", + &member.display().to_string(), + mount, + ], + ); + // The submodule in the working tree is a CLONE, and a clone carries none of + // the source repository's local config. Every other repository a fixture + // builds is handed an identity at `init`; this one is handed one here, + // because a test that commits into it commits into the clone and not into + // the source. Without it the fixture borrows whoever is configured + // globally, which is a machine that has somebody -- and CI is a machine + // that does not. + let checkout = root.join(mount); + git(&checkout, &["config", "user.name", "Test"]); + git(&checkout, &["config", "user.email", "test@example.test"]); + checkout +}