From cf7786e22f8ba44f87aede24b0b135b1091cf644 Mon Sep 17 00:00:00 2001 From: IceDBorn <51162078+IceDBorn@users.noreply.github.com> Date: Tue, 8 Sep 2026 04:29:30 +0300 Subject: [PATCH 1/2] feat: add random tips message as bottom bar on icedos rebuild (#84) --- .gitignore | 1 + AGENTS.md | 53 ++++++++- README.md | 32 +++++- lib/bash.nix | 210 ++++++++++++++++++++++++++++++++++- lib/prelude.sh | 5 + modules/build-vm.nix | 4 + modules/cache.nix | 6 +- modules/config-diff.nix | 4 + modules/config-history.nix | 4 + modules/config-manage.nix | 4 + modules/config-rollback.nix | 4 + modules/configuration.nix | 11 ++ modules/git.nix | 21 +++- modules/nh.nix | 6 +- modules/nix.nix | 31 +++++- modules/options.nix | 53 +++++++++ modules/rebuild.nix | 13 ++- modules/repl.nix | 5 + modules/ssh.nix | 6 +- modules/status.nix | 4 + modules/sudo.nix | 6 +- modules/toolset.nix | 26 ++++- modules/users.nix | 4 + tests/tests.nix | 214 ++++++++++++++++++++++++++++++++++++ 24 files changed, 699 insertions(+), 28 deletions(-) diff --git a/.gitignore b/.gitignore index 727b6f21..3f05df4c 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,4 @@ __pycache__/ # Generated by `nix flake check` / any nix command in this repo — # deliberately NOT pinned (library flake; see AGENTS.md "Core lib tests"). flake.lock +/result diff --git a/AGENTS.md b/AGENTS.md index f74abd10..fca753d9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ Exposed to every module as **`icedosLib`**. |---|---| | `lib/options/helpers.nix` | The `mk*Option` family: `mkBoolOption`, `mkStrOption`, `mkStrListOption`, `mkNumberOption`, `mkEnumOption`, `mkIntBetweenOption`, `mkFloatBetweenOption`, `mkNullableOption`, `mkListOption`, `mkAttrsOfOption`, `mkSubmodule{,List,Attrs}Option`, `mkRecordOption`, `mkUsersOption`. | | `lib/options/validate.nix` | `validate.{int,float,enum,str,nonEmpty,list,requires,abort}` — rich, path-aware error messages. | -| `lib/bash.nix` | `bash.{prelude,exportSystemPath,genHelpFlags,mkFlags,blueString,dimBlueString,greenString,dimGreenString,purpleString,dimPurpleString,redString,dimRedString,yellowString,dimYellowString,configSet,gcTimerCheckSnippet,requireConfigOwner}` — runtime shell helpers shared between Nix-embedded scripts and `prelude.sh` (color vars + the `*String` builders that emit `$(...)`-interpolated escape sequences — the **only** way the dispatcher/completions add color to command help text; `log_*`/`die`/`is_help_flag`; `bash.requireConfigOwner` is the permission guard for executing the baked `configurationLocation` — capture `ORIG_ARGS=("$@")` before arg parsing and only use where `$0` is the leaf command script). Also `injectIfExists` (emits `()` when a path exists — used by genflake for `/etc/nixos/extras.nix`). | +| `lib/bash.nix` | `bash.{prelude,exportSystemPath,genHelpFlags,mkFlags,blueString,dimBlueString,greenString,dimGreenString,purpleString,dimPurpleString,redString,dimRedString,yellowString,dimYellowString,configSet,gcTimerCheckSnippet,requireConfigOwner,printTip}` — runtime shell helpers shared between Nix-embedded scripts and `prelude.sh` (color vars + the `*String` builders that emit `$(...)`-interpolated escape sequences — the **only** way the dispatcher/completions add color to command help text; `log_*`/`die`/`is_help_flag`; `CLR_LINE` — line-erase prefix for printing over a progress line a child left behind, auto-stripped with the colours when stdout isn't a TTY; `bash.requireConfigOwner` is the permission guard for executing the baked `configurationLocation` — capture `ORIG_ARGS=("$@")` before arg parsing and only use where `$0` is the leaf command script; `bash.printTip` takes `config.icedos.system.tips` and returns `{ head, foot }` shell fragments for the tips bottom bar, both `""` when disabled or the list is empty — see §10). Also `injectIfExists` (emits `()` when a path exists — used by genflake for `/etc/nixos/extras.nix`). | | `lib/toolset.nix` | `toolset.mk{Dispatcher,BashCompletion,ZshCompletion,FishCompletion}` — the CLI dispatcher generator (used to build `icedos` itself and every subcommand attrset that has children) + the per-shell completion generators. | | `lib/users.nix` | `users.{getNormal,genDefaults,mkGroupInjector}`. | | `lib/color.nix` | `color.hexToRgbInts`. | @@ -567,7 +567,7 @@ command is a `toolsetCommandType` submodule (`modules/options.nix`): |---|---|---| | `command` | string (required) | subcommand name; must match `[a-zA-Z0-9_-]+`. | | `help` | string (required) | one-line help, shown in the parent listing and `icedos --tree`. | -| `script` | lines | inline bash. **Auto-prefixed with `bash.prelude`** (`modules/toolset.nix`), so `log_ok`/`log_warn`/`log_fail`/`log_info`/`log_step`/`die`/`is_help_flag` + colour vars are available. | +| `script` | lines | inline bash. **Auto-prefixed with `bash.prelude`** (`modules/toolset.nix`), so `log_ok`/`log_warn`/`log_fail`/`log_info`/`log_step`/`die`/`is_help_flag` + colour vars and `CLR_LINE` are available. Top-level leaves named in `tipsCommands` (`modules/toolset.nix`, currently just `rebuild`) are additionally wrapped in `bash.printTip`'s `head`/`foot` — see "Tips bottom bar" below. | | `bin` | string | absolute path to an executable instead of `script` (e.g. a `pkgs.writeShellScript`). | | `commands` | list | nested subcommands — arbitrarily deep. | | `completion.files` | bool | offer file-path completion for this leaf's arguments. | @@ -610,14 +610,59 @@ icedos.system.toolset.commands = [{ installed as `xdg.desktopEntries`. Modules adding session actions gate their own entries on the same flag. +### Tips bottom bar + +`icedos.system.tips.list` is a module-facing contribution list, like +`sessionCommands`: every loaded module appends its own tips and NixOS list merging +concatenates them, so a module advertises its own features and the user's +`config.toml` adds machine-local ones. Entries are bare message strings (rendered +with the lamp default, `💡: `), or `{ title, message }` records — an +explicit `title` renders `title: message`, and `title = ""` renders the message +alone. **Gate a tip on the option it advertises** (`optionals [ … ]`, as +`modules/{ssh,sudo,cache}.nix` do) or it will claim a feature the machine has +turned off. + +`icedosLib.bash.printTip` turns that list into `{ head, foot }`, which +`modules/toolset.nix` wraps around the top-level leaves listed in `tipsCommands` (currently +just `rebuild`). `head` pins one random tip to the terminal's last row and confines +output to a scroll region above it; `foot` restores the region while preserving the +leaf's exit status. Both are `""` when `tips.enable` is false or the list is empty. + +Constraints on a wrapped leaf: + +- **Don't install your own `EXIT` trap** — it would replace `head`'s + `trap _icedos_tip EXIT` and leave the scroll region set. Compose instead: + `command -v _icedos_tip >/dev/null 2>&1 && _icedos_tip` (see `modules/git.nix`, + `modules/nh.nix`). +- **Finalize before any `exec`** with the same guarded call — `exec` replaces the + shell, so the `EXIT` trap never runs (`modules/nix.nix`, `modules/repl.nix`). + `tests/tests.nix` asserts that every `exec` across `rebuild`, `nix`, `repl`, + `git` and `nh` is preceded by it — the guarded count must equal the total. +- **Don't erase to end of screen** (`\033[J`) in a redraw loop; it takes the pinned + bar with it. Erase only the rows you own, saving and restoring the cursor around + the walk with **DECSC/DECRC** — written `\e7`/`\e8`, since `printf %b` reads + `\0337` as a single octal byte. Not `\033[s`/`\033[u`: that pair is an ANSI.SYS + extension xterm honours only while DECLRMM is off, and a terminal that ignores it + leaves each later frame drawn below the last instead of over it. +- `ICEDOS_TIP_ACTIVE` marks "a bar is pinned in this process tree", so nested + `icedos` calls and owner re-runs keep the outer bar. `bash.requireConfigOwner` + forwards it across the re-exec; `_icedos_tip` unsets it on restore. + +The bar is TTY-only. It needs the cursor row, so it asks the terminal with a DSR +query; a terminal that does not answer within 200 ms gets the tip as a plain +trailing line instead, and the reply is drained so a late answer cannot surface as +stray keystrokes in a later `read`. + ## 11. Hook authoring contract `icedos.system.toolset.rebuild.hooks.{preRebuild,postRebuild,preUpdate,postUpdate}` and `icedos.system.gc.hooks.{preGc,postGc}` are lists of shell snippets. Each snippet is compiled to its **own** `pkgs.writeShellScript` with `bash.prelude` prepended (`modules/rebuild.nix`, `modules/nh.nix`), so it runs in a fresh shell with the same -helpers a command gets (`log_*`, `die`, `is_help_flag`, colour vars; colours auto-strip -when stdout isn't a TTY). +helpers a command gets (`log_*`, `die`, `is_help_flag`, colour vars, `CLR_LINE`; +colours and `CLR_LINE` auto-strip when stdout isn't a TTY). A hook runs in its own +shell, so it never owns the tips bar — it inherits `ICEDOS_TIP_ACTIVE` from the +rebuild that spawned it and must not pin one of its own. ### Execution identity — hooks don't run as root by default diff --git a/README.md b/README.md index a0f5bfc2..8728215b 100644 --- a/README.md +++ b/README.md @@ -252,7 +252,7 @@ Everything under `icedos` is IceDOS's own, checked settings. The top-level group | Key | What it controls | | --- | --- | | `icedos.repositories` | Which module repos to load and which modules to enable (see above). | -| `icedos.system` | System-wide settings: `arch`, `version` (stateVersion), `nixpkgsChannel`, `allowUnfree`, `generations`, `packages`, `permittedInsecurePackages`, `loadHardwareConfiguration`, the binary `cache`, `gc` (auto-cleanup), the `toolset` (CLI + hooks), extra `channels`/`overlays`, and `build-vm`. | +| `icedos.system` | System-wide settings: `arch`, `version` (stateVersion), `nixpkgsChannel`, `allowUnfree`, `generations`, `packages`, `permittedInsecurePackages`, `loadHardwareConfiguration`, the binary `cache`, `gc` (auto-cleanup), the `toolset` (CLI + hooks), the rebuild `tips` line, extra `channels`/`overlays`, and `build-vm`. | | `icedos.users` | User accounts (home-manager integrated): password, groups, sudo, packages, … | | `icedos..*` | Options exposed by the module repos you load, grouped by category — e.g. `icedos.applications.*` (apps like `btop`, `steam`), `icedos.hardware.*`, `icedos.desktop.*`, `icedos.tweaks.*`. Which categories exist depends on which repos you enable. | @@ -430,6 +430,36 @@ New here? These cover almost everything: | `icedos gc [--dry] [--days ] [--gens ]` | Free up disk space (--dry to preview). | | `icedos repl` | Open a Nix REPL preloaded with your evaluated config, packages, and lib. | +### The tips line + +Once `icedos.system.tips.list` is populated, `icedos rebuild` runs with a +persistent bottom bar: output is confined above a blank line and a one-line tip +pinned to the terminal's last row, visible from the first frame to the last. +The bar stays on the last row for the whole run, so short output leaves it a +screenful below the final line; restoring the terminal on exit scrolls it up so +the shell prompt lands beneath it: + +``` +$ icedos rebuild +> Caching config set + +💡: icedos rebuild --dry prepares a rebuild without building anything. +``` + +`list` merges tips contributed by every loaded icedos module, so each +module advertises its own features; add machine-local tips from config if you +like. Set `enable = false` under `[icedos.system.tips]` to turn the line off. The +bar is TTY-only, so piped output stays clean. It makes room below your typed +command, restores the terminal on exit and on TERM/HUP/QUIT (SIGKILL excepted), +and leaves helpers and non-script dispatchers untouched. + +Pinning the bar needs the cursor position, so the terminal is asked for it with +a DSR query. Terminals that do not answer within 200 ms — and multiplexers or +recorders that swallow the reply — get the tip as a plain trailing line instead; +nothing is scrolled blind. A resize re-pins the bar, but bash only runs the +`WINCH` handler once the current foreground command returns, so resizing mid-build +leaves the bar at its old coordinates until that build step finishes. + ### Full command reference | Command | Description | diff --git a/lib/bash.nix b/lib/bash.nix index c4e4b714..0e6dfb02 100644 --- a/lib/bash.nix +++ b/lib/bash.nix @@ -1,7 +1,5 @@ { - icedosLib, lib, - self, ... }: @@ -17,9 +15,11 @@ let escapeShellArg fileContents max + replaceStrings ; + in -rec { +{ # Runtime helpers shared by Nix-embedded scripts (prelude auto-prepended by # toolset.nix) and standalone .sh files that source lib/prelude.sh. bash = { @@ -394,11 +394,11 @@ rec { owner_path="/run/wrappers/bin:$owner_home/.nix-profile/bin:$owner_home/.local/state/nix/profile/bin:/etc/profiles/per-user/$owner/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin" /run/current-system/sw/bin/runuser -u "$owner" -- \ /run/current-system/sw/bin/env -i "HOME=$owner_home" "USER=$owner" "LOGNAME=$owner" "PATH=$owner_path" \ - ICEDOS_OWNER_RERUN=1 "$0" "$@" + ICEDOS_OWNER_RERUN=1 "ICEDOS_TIP_ACTIVE=''${ICEDOS_TIP_ACTIVE:-}" "$0" "$@" else # sudo resets the env, so pass the re-entry marker explicitly; only # the setuid wrapper path works here. - /run/wrappers/bin/sudo -u "$owner" -- /run/current-system/sw/bin/env ICEDOS_OWNER_RERUN=1 "$0" "$@" + /run/wrappers/bin/sudo -u "$owner" -- /run/current-system/sw/bin/env ICEDOS_OWNER_RERUN=1 "ICEDOS_TIP_ACTIVE=''${ICEDOS_TIP_ACTIVE:-}" "$0" "$@" fi rc=$? [ "$rc" -eq 0 ] && exit 0 @@ -417,6 +417,206 @@ rec { return 0 } ''; + + # Bottom bar for toolset leaves: one random tip pinned to the last line and + # command output confined to the scroll region above it; TTY-only. + printTip = + tips: + let + enabled = tips.enable && tips.list != [ ]; + # The tip is printed onto one pinned row, so a newline from a `"""…"""` + # tip in config.toml would scroll the screen and tear the region open. + oneLine = replaceStrings [ "\n" "\r" "\t" ] [ " " " " " " ]; + tipLine = + t: + if builtins.isString t then + "💡: ${t}" + else if !(t ? title) then + "💡: ${t.message}" + else if t.title == "" then + t.message + else + "${t.title}: ${t.message}"; + head = '' + _icedos_tips=( + ${concatStringsSep " +" (map (t: escapeShellArg (oneLine (tipLine t))) tips.list)} + ) + _icedos_tip_shown=0 + _icedos_tip_i=0 + _icedos_bar=0 + _icedos_tip_plain=0 + _icedos_rows= + _icedos_max= + _icedos_cpr_row= + _icedos_cpr_col= + # A cursor-position reply that misses the timeout would otherwise + # surface as stray keystrokes in the next `read` a leaf runs. + _icedos_drain_tty() { + local junk + read -rs -t 0.05 -n 4096 junk /dev/null + read -rs -t 0.05 -n 4096 junk /dev/null + return 0 + } + # Row and col come back empty when the terminal does not answer, or + # answers with something that is not a CPR. + _icedos_cpr() { + local p c + _icedos_cpr_row= + _icedos_cpr_col= + printf '\033[6n' >/dev/tty 2>/dev/null + if IFS=';' read -rsdR -t 0.2 p c /dev/null; then + p=''${p##*[} + p=''${p%%;*} + c=''${c%%R*} + case "$p" in + "" | *[!0-9]*) p= ;; + esac + case "$c" in + "" | *[!0-9]*) c= ;; + esac + else + p= + c= + fi + [ -n "$p" ] || _icedos_drain_tty + _icedos_cpr_row=$p + _icedos_cpr_col=$c + return 0 + } + _icedos_tip_init() { + [ -t 1 ] || return 0 + # A parent leaf already pinned the bar (nested icedos call, or an + # owner re-run that carried the marker): keep the outer bar. + [ -n "''${ICEDOS_TIP_ACTIVE:-}" ] && return 0 + local size rows cols max p n i + size=$(stty size /dev/null) || size="" + rows=''${size%% *} + cols=''${size##* } + [ "$rows" -ge 4 ] 2>/dev/null || rows=$(tput lines /dev/null || echo 24) + [ "$rows" -ge 4 ] 2>/dev/null || rows=24 + # `cols - 2` is a substring length: 0 blanks the bar and a negative + # value means "all but the last N chars", so keep it at least 1. + [ "$cols" -ge 3 ] 2>/dev/null || cols=80 + _icedos_rows=$rows + _icedos_max=$(( cols - 2 )) + _icedos_tip_i=$(( RANDOM % ''${#_icedos_tips[@]} )) + export ICEDOS_TIP_ACTIVE=1 + _icedos_cpr + p=$_icedos_cpr_row + if [ -z "$p" ]; then + # Scrolling blind would wipe the screen, so drop the bar and + # degrade to a plain tip line printed on exit. + _icedos_tip_plain=1 + return 0 + fi + max=$_icedos_max + _icedos_bar=1 + n=0 + if [ "$p" -ge "$((rows - 2))" ] 2>/dev/null; then + n=$(( p - rows + 3 )) + fi + if [ "$n" -gt 0 ]; then + # The prompt sits where the bar and its blank row go, so scroll it + # up by that overlap instead of drawing over the command line. + printf '\033[%d;1H' "$rows" + i=$n + while [ "$i" -gt 0 ]; do printf '\n'; i=$((i - 1)); done + fi + # Autowrap off around the tip: a tip wider than the terminal would + # otherwise wrap into the scroll region and push output up a row. + printf '\033[?7l\033[%d;1H\033[K%b%s%b\033[?7h' "$rows" "$DIM_GREEN" "''${_icedos_tips[$_icedos_tip_i]:0:$max}" "$NC" + printf '\033[%d;1H\033[K' "$((rows - 1))" + printf '\033[1;%dr' "$((rows - 2))" + # One line below the prompt, corrected for the rows just scrolled: + # setting the region homed the cursor, so it has to be placed back. + printf '\033[%d;1H' "$(( p + 1 - n ))" + } + # A resize leaves the region and bar at stale coordinates. Bash defers + # this until the foreground command returns, so a build holds it stale. + _icedos_tip_winch() { + [ "$_icedos_tip_shown" -eq 1 ] && return 0 + [ -t 1 ] || return 0 + [ "$_icedos_bar" = 1 ] || return 0 + local size rows cols max p c + size=$(stty size /dev/null) || size="" + rows=''${size%% *} + cols=''${size##* } + [ "$rows" -ge 4 ] 2>/dev/null || return 0 + [ "$cols" -ge 3 ] 2>/dev/null || return 0 + max=$(( cols - 2 )) + [ "$rows" = "$_icedos_rows" ] && [ "$max" = "$_icedos_max" ] && return 0 + local old_rows=$_icedos_rows + _icedos_rows=$rows + _icedos_max=$max + _icedos_cpr + p=$_icedos_cpr_row + c=$_icedos_cpr_col + printf '\033[r' + printf '\033[?7l\033[%d;1H\033[K%b%s%b\033[?7h' "$rows" "$DIM_GREEN" "''${_icedos_tips[$_icedos_tip_i]:0:$max}" "$NC" + printf '\033[%d;1H\033[K' "$((rows - 1))" + printf '\033[1;%dr' "$((rows - 2))" + # Growing puts the old bar rows inside the region; clear that band so + # resumed output cannot fuse with the text left there. + if [ "$rows" -gt "$old_rows" ]; then + local r + r=$((old_rows - 1)) + while [ "$r" -le "$old_rows" ]; do + printf '\033[%d;1H\033[K' "$r" + r=$((r + 1)) + done + fi + if [ -n "$p" ] && [ "$p" -le "$((rows - 2))" ] 2>/dev/null && [ -n "$c" ]; then + # Back to where output was, clamped: a narrower terminal leaves the + # pre-resize column past the new right edge. + [ "$c" -gt "$cols" ] 2>/dev/null && c=$cols + printf '\033[%d;%dH' "$p" "$c" + else + printf '\033[1;1H' + fi + } + _icedos_tip() { + [ "$_icedos_tip_shown" -eq 1 ] && return 0 + _icedos_tip_shown=1 + trap - WINCH + # Children spawned after the bar is gone (an exec'd program that + # calls icedos again) must be free to pin a bar of their own. + unset ICEDOS_TIP_ACTIVE + [ -t 1 ] || return 0 + if [ "$_icedos_tip_plain" = 1 ]; then + printf '%b%s%b\n' "$DIM_GREEN" "''${_icedos_tips[$_icedos_tip_i]}" "$NC" + return 0 + fi + [ "$_icedos_bar" = 1 ] || return 0 + local size rows + size=$(stty size /dev/null) || size="" + rows=''${size%% *} + [ "$rows" -ge 4 ] 2>/dev/null || rows=$_icedos_rows + # Newline on the bar's own row scrolls it up one, so the shell prompt + # lands below the tip instead of overwriting it. + printf '\033[r\033[%d;1H\n' "$rows" + } + _icedos_tip_init + trap _icedos_tip EXIT + # Signal death must also restore the region (SIGKILL cannot); the + # explicit exits give leaves the usual 128+SIG status. + trap '_icedos_tip; exit $((128 + 15))' TERM + trap '_icedos_tip; exit $((128 + 1))' HUP + trap '_icedos_tip; exit $((128 + 3))' QUIT + trap _icedos_tip_winch WINCH + + ''; + in + if !enabled then + { + head = ""; + foot = ""; + } + else + { + inherit head; + foot = "_icedos_tip_rc=$?\n_icedos_tip\nexit \"$_icedos_tip_rc\"\n"; + }; }; injectIfExists = diff --git a/lib/prelude.sh b/lib/prelude.sh index 25a2d423..673ea533 100644 --- a/lib/prelude.sh +++ b/lib/prelude.sh @@ -13,11 +13,16 @@ DIM_PURPLE='\033[0;35m' DIM_RED='\033[0;31m' DIM_YELLOW='\033[0;33m' +# Prefix for the first message after a child that ends mid-line: erases the +# progress line it left behind instead of appending to it. +CLR_LINE='\033[2K\r' + # Strip escape codes when stdout isn't a terminal so piped/redirected # output stays clean (e.g. `icedos download | cat`). if [ ! -t 1 ]; then NC='' BLUE='' GREEN='' PURPLE='' RED='' YELLOW='' DIM_BLUE='' DIM_GREEN='' DIM_PURPLE='' DIM_RED='' DIM_YELLOW='' + CLR_LINE='' fi log_info() { printf '%b>%b %s\n' "$DIM_BLUE" "$NC" "$*"; } diff --git a/modules/build-vm.nix b/modules/build-vm.nix index 84cd0a1b..ec5893df 100644 --- a/modules/build-vm.nix +++ b/modules/build-vm.nix @@ -40,4 +40,8 @@ in } ]; }; + + icedos.system.tips.list = [ + "icedos rebuild --run-vm tries your config in a throwaway VM first." + ]; } diff --git a/modules/cache.nix b/modules/cache.nix index c0c3ed91..2d630c9c 100644 --- a/modules/cache.nix +++ b/modules/cache.nix @@ -1,7 +1,7 @@ { config, lib, ... }: let - inherit (lib) mkIf; + inherit (lib) mkIf optionals; inherit (config.icedos.system.cache) enable @@ -15,4 +15,8 @@ in substituters = [ "${url}?priority=${toString priority}" ]; trusted-public-keys = [ key ]; }; + + icedos.system.tips.list = optionals enable [ + "The binary cache downloads prebuilt custom and unfree packages, so rebuilds compile less." + ]; } diff --git a/modules/config-diff.nix b/modules/config-diff.nix index 83664128..aac2c03c 100644 --- a/modules/config-diff.nix +++ b/modules/config-diff.nix @@ -60,4 +60,8 @@ in ''; } ]; + + icedos.system.tips.list = [ + "icedos configuration diff shows what you changed since the last rebuild." + ]; } diff --git a/modules/config-history.nix b/modules/config-history.nix index a52a7a53..b2646be1 100644 --- a/modules/config-history.nix +++ b/modules/config-history.nix @@ -167,4 +167,8 @@ in ''; } ]; + + icedos.system.tips.list = [ + "icedos configuration history browses your past configs, one per rebuild." + ]; } diff --git a/modules/config-manage.nix b/modules/config-manage.nix index 16449ab6..8a03ce17 100644 --- a/modules/config-manage.nix +++ b/modules/config-manage.nix @@ -302,4 +302,8 @@ in action = "disable"; }) ]; + + icedos.system.tips.list = [ + "icedos configuration enable/disable picks modules; rebuild applies them." + ]; } diff --git a/modules/config-rollback.nix b/modules/config-rollback.nix index fd6e5c49..348b7aaf 100644 --- a/modules/config-rollback.nix +++ b/modules/config-rollback.nix @@ -170,4 +170,8 @@ in ''; } ]; + + icedos.system.tips.list = [ + "icedos configuration rollback puts your system back to how it was." + ]; } diff --git a/modules/configuration.nix b/modules/configuration.nix index 31a3d3b3..06cb014a 100644 --- a/modules/configuration.nix +++ b/modules/configuration.nix @@ -368,4 +368,15 @@ in ++ configurationCommands; } ]; + + icedos.system.tips.list = [ + "icedos configuration search finds any setting or module by name." + "icedos configuration get shows what a setting is set to." + "icedos configuration validate checks your config before you rebuild." + "Split your settings across configs/*.toml; they all merge together." + "Put custom Nix modules under modules/; files and folders load automatically." + "Set [icedos.system] extraConfigs/extraModules to scan more folders." + "Add enable = false at the top of any extra config *.toml to skip that file." + "/.name.toml stays out of git, but it is not a place for secrets." + ]; } diff --git a/modules/git.nix b/modules/git.nix index c07019b1..221f2ae6 100644 --- a/modules/git.nix +++ b/modules/git.nix @@ -263,6 +263,7 @@ in printf "\033[?25h" fi ${coreutils}/bin/rm -rf "$tmpdir" + command -v _icedos_tip >/dev/null 2>&1 && _icedos_tip ' EXIT function showRepos() { @@ -520,7 +521,7 @@ in frame+=("$footer") fi - local out="" n="''${#frame[@]}" idx2=0 + local out="" n="''${#frame[@]}" idx2=0 t if [ "$prev_lines" -gt 1 ]; then out="\r\033[$((prev_lines - 1))A" elif [ "$prev_lines" -eq 1 ]; then @@ -531,7 +532,19 @@ in out="''${out}\033[2K''${i}" [ "$idx2" -lt "$n" ] && out="''${out}\n" done - out="''${out}\033[J" + # Erase only the rows a longer previous frame left; erase-to-end would + # take the pinned tip. Cursor-down clamps, so save/restore with DECSC — + # SCOSC is not universal, and `printf %b` folds `\0337` to one byte. + t=$(( prev_lines - n )) + if [ "$t" -gt 0 ]; then + idx2=0 + out="''${out}\e7" + while [ "$idx2" -lt "$t" ]; do + out="''${out}\033[B\r\033[2K" + idx2=$((idx2 + 1)) + done + out="''${out}\e8" + fi printf '%b' "$out" prev_lines=$n spin_i=$(( (spin_i + 1) % ''${#spin[@]} )) @@ -634,4 +647,8 @@ in ]; } ]; + + icedos.system.tips.list = [ + "icedos git rpull recursively updates every git repo under a folder at once." + ]; } diff --git a/modules/nh.nix b/modules/nh.nix index fe0be126..adecf104 100644 --- a/modules/nh.nix +++ b/modules/nh.nix @@ -432,7 +432,7 @@ in ${cacheCleanScript} $CACHE_ARGS --preview _summary_clean=$(mktemp) || die "mktemp failed" _summary_cache=$(mktemp) || die "mktemp failed" - trap "rm -f '$_summary_clean' '$_summary_cache'" EXIT + trap "rm -f '$_summary_clean' '$_summary_cache'; command -v _icedos_tip >/dev/null 2>&1 && _icedos_tip" EXIT ${cleanExtra} --remove --summary-file "$_summary_clean" ${cacheCleanScript} $CACHE_ARGS --remove --summary-file "$_summary_cache" cat "$_summary_clean" @@ -470,4 +470,8 @@ in ++ runHooksAsUsers "postGc" postGc; }; }; + + icedos.system.tips.list = [ + "icedos gc frees disk space by removing old system versions and junk." + ]; } diff --git a/modules/nix.nix b/modules/nix.nix index c770fc8f..e90a8716 100644 --- a/modules/nix.nix +++ b/modules/nix.nix @@ -110,6 +110,7 @@ in if [ -z "$RUN_ARG" ]; then echo "$OUT" else + command -v _icedos_tip >/dev/null 2>&1 && _icedos_tip exec "$OUT/bin/$RUN_ARG" "''${BIN_ARGS[@]}" fi ''; @@ -185,6 +186,7 @@ in setsid -f "$@" /dev/null 2>&1 exit 0 fi + command -v _icedos_tip >/dev/null 2>&1 && _icedos_tip exec "$@" } @@ -234,8 +236,27 @@ in { command = "repair"; - script = "nix-store --verify --check-contents --repair"; - help = "repair nix store"; + help = "verify the nix store and re-fetch damaged paths (elevates with sudo)"; + + script = '' + if [[ ${genHelpFlags { excludeNoArgs = true; }} ]]; then + echo "Usage: icedos repair" + echo "Verifies every path in the nix store and re-fetches damaged ones." + echo "Repairing is root-only, so this elevates with sudo when needed." + exit 0 + fi + + [ "$#" -gt 0 ] && die "unknown arg: $1" + + # The daemon rejects repairPath for untrusted users with a bare + # "operation is not supported by store 'daemon'", so elevate first. + if [ "$(id -u)" -eq 0 ]; then + nix-store --verify --check-contents --repair + else + log_step "repairing the nix store (needs root)..." + /run/wrappers/bin/sudo nix-store --verify --check-contents --repair + fi + ''; } { @@ -281,4 +302,10 @@ in }; nixpkgs.config = icedosLib.pkgs.mkConfig config.icedos; + + icedos.system.tips.list = [ + "icedos pkgs list shows everything in your system, dependencies included." + "icedos pkgs run runs an app once without installing it." + "icedos repair re-downloads damaged files in the package store." + ]; } diff --git a/modules/options.nix b/modules/options.nix index 0266c409..ed130a02 100644 --- a/modules/options.nix +++ b/modules/options.nix @@ -292,6 +292,59 @@ in rs = mkBoolOption { default = true; }; }; + tips = { + enable = mkBoolOption { + default = true; + + description = '' + Pin a tip from `list` to the terminal's bottom row while + `icedos rebuild` runs, keeping command output above it in a + scroll region. Skipped when stdout is not a terminal; when the + terminal does not answer a cursor-position query the tip is + printed as a plain trailing line instead of a pinned bar. + ''; + }; + + list = + mkListOption + { + default = [ ]; + + description = '' + Tips shown under `icedos rebuild` output, one per run. Every + loaded module contributes its own tips here; extend the list + with machine-local hints. Entries are bare message strings + (rendered with the default lamp title) or `{ title, message }` + records — the bottom bar shows "title: message". An empty list + hides the tip line even when `enable` is true. + ''; + } + ( + types.either types.str ( + types.submodule { + options = { + title = mkStrOption { + default = "💡"; + + description = '' + Label shown before `message` in the bottom bar. Defaults to + a lamp emoji; set it to the empty string for a plain + message-only tip with no title and no separator. + ''; + }; + + message = mkStrOption { + description = '' + The tip text itself. Keep it short — the bar clips anything + wider than the terminal. + ''; + }; + }; + } + ) + ); + }; + version = mkStrOption { }; # Set according to docs at https://search.nixos.org/options?show=system.stateVersion zsh = { diff --git a/modules/rebuild.nix b/modules/rebuild.nix index d0f9364d..52b8ee7f 100644 --- a/modules/rebuild.nix +++ b/modules/rebuild.nix @@ -344,7 +344,7 @@ in exit_if_interrupted "$BUILD_STATUS" if [ "$BUILD_STATUS" -ne 0 ]; then - echo -e "${redString "error"}: build failed with exit code $BUILD_STATUS" + echo -e "''${CLR_LINE}${redString "error"}: build failed with exit code $BUILD_STATUS" exit "$BUILD_STATUS" fi @@ -365,7 +365,7 @@ in if [ ''${#CACHED_NAMES[@]} -gt 0 ]; then printf -v JOINED '%s, ' "''${CACHED_NAMES[@]}" - echo -e "${dimGreenString ">"} Caching ''${JOINED%, }" + echo -e "''${CLR_LINE}${dimGreenString ">"} Caching ''${JOINED%, }" fi # Only switch/boot mint a generation, so only they record which snapshot @@ -416,8 +416,8 @@ in if [ ''${#REBOOT_REASONS[@]} -gt 0 ]; then printf -v REASONS_JOINED '%s, ' "''${REBOOT_REASONS[@]}" - echo -e "${purpleString "warning"}: reboot recommended for ''${REASONS_JOINED%, } changes to apply" - printf -v PROMPT '%b' "${dimGreenString ">"} Reboot now? [y/N] " + echo -e "''${CLR_LINE}${purpleString "warning"}: reboot recommended for ''${REASONS_JOINED%, } changes to apply" + printf -v PROMPT '%b' "''${CLR_LINE}${dimGreenString ">"} Reboot now? [y/N] " read -r -p "$PROMPT" ANSWER case "$ANSWER" in [yY]|[yY][eE][sS]) systemctl reboot -i || sudo systemctl reboot -i ;; @@ -427,4 +427,9 @@ in ''; } ]; + + icedos.system.tips.list = [ + "icedos rebuild --dry prepares a rebuild without building anything." + "icedos rebuild --update gets the newest packages, repos and inputs before rebuilding." + ]; } diff --git a/modules/repl.nix b/modules/repl.nix index 6a381b73..db708b71 100644 --- a/modules/repl.nix +++ b/modules/repl.nix @@ -47,8 +47,13 @@ in die "no generated flake at '${configurationLocation}'; run 'icedos rebuild' once." fi + command -v _icedos_tip >/dev/null 2>&1 && _icedos_tip exec nix repl --file ${replExpr} ''; } ]; + + icedos.system.tips.list = [ + "icedos repl opens a nix repl with your whole config loaded." + ]; } diff --git a/modules/ssh.nix b/modules/ssh.nix index f99793f1..815eb322 100644 --- a/modules/ssh.nix +++ b/modules/ssh.nix @@ -1,10 +1,14 @@ { config, lib, ... }: let - inherit (lib) mkIf; + inherit (lib) mkIf optionals; inherit (config.icedos.system) ssh; in { services.openssh.enable = mkIf ssh true; programs.zsh.shellAliases.ssh = mkIf ssh "TERM=xterm-256color ssh"; + + icedos.system.tips.list = optionals ssh [ + "SSH is on, so you can reach this machine from another computer." + ]; } diff --git a/modules/status.nix b/modules/status.nix index a1bea462..c934e27e 100644 --- a/modules/status.nix +++ b/modules/status.nix @@ -232,4 +232,8 @@ in ''; } ]; + + icedos.system.tips.list = [ + "icedos status shows your system info, disk use, and health checks." + ]; } diff --git a/modules/sudo.nix b/modules/sudo.nix index 71d684c4..aa493676 100644 --- a/modules/sudo.nix +++ b/modules/sudo.nix @@ -1,7 +1,7 @@ { config, lib, ... }: let - inherit (lib) mkIf; + inherit (lib) mkIf optionals; inherit (config.icedos.system.sudo) passwordFeedback rs; pwfeedback = mkIf passwordFeedback "Defaults pwfeedback"; # Show asterisks when typing sudo password @@ -14,4 +14,8 @@ in execWheelOnly = true; extraConfig = pwfeedback; }; + + icedos.system.tips.list = optionals passwordFeedback [ + "Most password prompts show asterisks as you type." + ]; } diff --git a/modules/toolset.nix b/modules/toolset.nix index 9f672e9b..9b7a7b10 100644 --- a/modules/toolset.nix +++ b/modules/toolset.nix @@ -9,6 +9,7 @@ let inherit (config.icedos.system.toolset) commands desktopEntries sessionCommands; inherit (icedosLib.bash) prelude; + printTip = icedosLib.bash.printTip config.icedos.system.tips; inherit (icedosLib.toolset) mkBashCompletion @@ -68,13 +69,13 @@ let leafCount = builtins.foldl' (acc: x: if x.hasScriptOrBin then acc + 1 else acc) 0 entries; in if n > 1 && anyBranch && anyLeaf then - builtins.abort '' + abort '' icedos toolset: command "${name}" is registered as BOTH a leaf (script/bin) and a branch (subcommands) by different modules. This is ambiguous — NixOS list ordering decides which definition wins. Use unique command names or consolidate the definitions into a single module.'' else if n > 1 && leafCount > 1 then - builtins.abort '' + abort '' icedos toolset: command "${name}" has multiple leaf definitions (script/bin) from different modules (${toString leafCount} registrations). Only the first survives — rename or consolidate.'' @@ -87,7 +88,7 @@ let builtins.seq validated ( map ( cmd: - builtins.removeAttrs cmd [ "_entries" ] + removeAttrs cmd [ "_entries" ] // { commands = mergeCommands cmd.commands; } @@ -116,12 +117,17 @@ let exec ${pkgs.systemd}/bin/run0 ${pkgs.systemd}/bin/systemctl suspend -i ''; + tipsCommands = [ "rebuild" ]; + + # `tipsCommands` names top-level leaves, so `top` keeps a nested subcommand + # (or a repo module's leaf) that shares the name from inheriting a bar. resolve = - cmd: + top: cmd: let - resolvedChildren = map resolve cmd.commands; + resolvedChildren = map (resolve false) cmd.commands; hasChildren = cmd.commands != [ ]; hasScript = cmd.script != ""; + wrapTips = top && hasScript && builtins.elem cmd.command tipsCommands; in cmd // { @@ -133,13 +139,17 @@ let commands = resolvedChildren; }) ) + else if wrapTips then + toString ( + pkgs.writeShellScript cmd.command "${prelude}\n${printTip.head}\n${cmd.script}\n${printTip.foot}" + ) else if hasScript then toString (pkgs.writeShellScript cmd.command "${prelude}\n${cmd.script}") else cmd.bin; }; - resolvedCommands = map resolve mergedCommands; + resolvedCommands = map (resolve true) mergedCommands; flatten = cmd: [ cmd ] ++ concatMap flatten cmd.commands; allCommands = concatMap flatten mergedCommands; @@ -329,4 +339,8 @@ in settings.Keywords = "suspend;sleep;"; }; }; + + icedos.system.tips.list = [ + "Hide this bar with [icedos.system.tips] enable = false in config.toml." + ]; } diff --git a/modules/users.nix b/modules/users.nix index fadbb869..93915fdb 100644 --- a/modules/users.nix +++ b/modules/users.nix @@ -93,4 +93,8 @@ in ] ) users ); + + icedos.system.tips.list = [ + "Each user gets their own packages and settings in toml config files." + ]; } diff --git a/tests/tests.nix b/tests/tests.nix index a8d36bc1..b626d8b1 100644 --- a/tests/tests.nix +++ b/tests/tests.nix @@ -494,6 +494,22 @@ let optional = i.repoFetchOptional; deps = i.repoFetchDeps; }; + + # --- icedosLib.bash.printTip (icedos.system.tips bottom bar) ------------- + tipsLib = + (import ../lib/bash.nix { + inherit lib; + self = "tests"; + icedosLib = { }; + }).bash; + tipsFor = opts: tipsLib.printTip opts; + tipsHead = + list: + (tipsFor { + enable = true; + inherit list; + }).head; + tipsSampleHead = tipsHead [ "a tip" ]; in { intHappy = expectOk ( @@ -3211,4 +3227,202 @@ in }; } ); + + # --- tips bottom bar (icedos.system.tips) -------------------------------- + tipsDisabled = + expectEq + { + head = ""; + foot = ""; + } + (tipsFor { + enable = false; + list = [ "a tip" ]; + }); + + tipsEmptyList = + expectEq + { + head = ""; + foot = ""; + } + (tipsFor { + enable = true; + list = [ ]; + }); + + tipsEnabledHeadHasTrap = expectOk (lib.strings.hasInfix "trap _icedos_tip EXIT" tipsSampleHead); + + tipsBarPinsTipAndRegion = expectOk ( + lib.strings.hasInfix "printf '\\033[?7l\\033[%d;1H\\033[K%b%s%b\\033[?7h'" tipsSampleHead + && lib.strings.hasInfix "printf '\\033[1;%dr'" tipsSampleHead + && lib.strings.hasInfix "stty size /dev/tty" tipsSampleHead + && lib.strings.hasInfix "read -rsdR -t 0.2" tipsSampleHead + ); + + # A reply that misses the 0.2s window would be swallowed by the next `read` + # a leaf runs, so every failed query drains the terminal first. + tipsCprDrainsOnTimeout = expectOk ( + lib.strings.hasInfix "_icedos_drain_tty() {" tipsSampleHead + && lib.strings.hasInfix "read -rs -t 0.05 -n 4096 junk 0 && guarded == total); + + # The file list above is hand-maintained, so pin `tipsCommands`: growing it + # trips here until the new leaf's module joins `tipsExecLeavesFinalize`. + tipsCommandsPinned = expectOk ( + lib.strings.hasInfix ''tipsCommands = [ "rebuild" ];'' (builtins.readFile ../modules/toolset.nix) + ); + + # Leaves with their own EXIT trap compose the finalize in, and _icedos_tip + # no-ops once shown, so signal traps cannot double-scroll. + tipsTrapComposition = expectOk ( + lib.strings.hasInfix "command -v _icedos_tip" (builtins.readFile ../modules/git.nix) + && lib.strings.hasInfix "command -v _icedos_tip" (builtins.readFile ../modules/nh.nix) + && lib.strings.hasInfix "_icedos_tip_shown" tipsSampleHead + && lib.strings.hasInfix "[ \"$_icedos_tip_shown\" -eq 1 ] && return 0" tipsSampleHead + ); + + # rpull's redraw must not erase to end of screen, and must save/restore with + # DECSC — SCOSC is not universal, and `\0337` is one octal byte to `printf %b`. + tipsRendererPreservesBar = + let + git = builtins.readFile ../modules/git.nix; + in + expectOk ( + !lib.strings.hasInfix "\\033[J" git + && lib.strings.hasInfix "\\e7" git + && lib.strings.hasInfix "\\e8" git + && !lib.strings.hasInfix "\\033[s" git + && !lib.strings.hasInfix "\\033[u" git + ); + + # The list is embedded via escapeShellArg, so a quote in a tip must not + # survive verbatim into the generated script. + tipsEscapesQuotes = expectOk (!lib.strings.hasInfix "it's" (tipsHead [ "it's broken" ])); + + # The bar prints onto one pinned row, so a newline reaching it would scroll + # the screen and leave the region without its bar. + tipsCollapsesNewlines = expectOk (lib.strings.hasInfix "'💡: a b c'" (tipsHead [ "a\nb\tc" ])); + + # Every tip must be embedded; rotation happens at runtime. `one` etc. appear + # nowhere else in the snippet, so bare presence is a safe check here. + tipsAllEmbedded = + let + head = tipsHead [ + "one" + "two" + "three" + ]; + in + expectOk ( + lib.strings.hasInfix "one" head + && lib.strings.hasInfix "two" head + && lib.strings.hasInfix "three" head + ); + + # Bare strings and title-less records take the lamp default, an explicit + # title renders "title: message", and an empty title drops both. + tipsRenderVariants = expectOk ( + lib.strings.hasInfix "'💡: bare'" (tipsHead [ "bare" ]) + && lib.strings.hasInfix "'💡: implicit'" (tipsHead [ { message = "implicit"; } ]) + && lib.strings.hasInfix "'note: titled'" (tipsHead [ + { + title = "note"; + message = "titled"; + } + ]) + # escapeShellArg leaves a shell-safe word unquoted, so the empty-title tip + # has to be matched bare rather than inside the quotes the others get. + && lib.strings.hasInfix "plain\n)" (tipsHead [ + { + title = ""; + message = "plain"; + } + ]) + && !lib.strings.hasInfix "💡: plain" (tipsHead [ + { + title = ""; + message = "plain"; + } + ]) + ); + + # The prefix must come from CLR_LINE, blanked for a non-terminal stdout, or + # `icedos rebuild | cat` grows literal escape codes. + preludeClearLineStripped = + let + prelude = builtins.readFile ../lib/prelude.sh; + in + expectOk ( + lib.strings.hasInfix "CLR_LINE='\\033[2K\\r'" prelude + && lib.strings.hasInfix "CLR_LINE=''" prelude + && !lib.strings.hasInfix "\\033[2K" (builtins.readFile ../modules/rebuild.nix) + ); + + tipsResizeTrap = expectOk ( + lib.strings.hasInfix "trap _icedos_tip_winch WINCH" (tipsHead [ "one" ]) + ); } From 08733bb8231c37ccba5baf90339eb832c70c3298 Mon Sep 17 00:00:00 2001 From: IceDBorn Date: Wed, 2 Sep 2026 19:34:04 +0300 Subject: [PATCH 2/2] feat(lock): unpin inputs from cache server ones --- AGENTS.md | 34 +- README.md | 2 + build/genflake.py | 6 +- build/main.py | 18 + build/options.py | 17 + build/pins.py | 634 +++++++++++++++++++++++++++++++ build/tests/test_lockfile.py | 28 +- build/tests/test_options.py | 35 +- build/tests/test_pins.py | 710 +++++++++++++++++++++++++++++++++++ build/tests/test_token.py | 24 +- lib/genflake.nix | 10 + lib/icedos.nix | 66 +++- lib/inputs.nix | 98 +++-- modules/options.nix | 3 +- modules/rebuild.nix | 8 + tests/tests.nix | 62 +++ 16 files changed, 1690 insertions(+), 65 deletions(-) create mode 100644 build/pins.py create mode 100644 build/tests/test_pins.py diff --git a/AGENTS.md b/AGENTS.md index fca753d9..01347147 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -467,7 +467,7 @@ at your checkout, and enable/configure the module you touched) → run `icedos r | Check | What it does | |---|---| | `lib-tests` | Evaluates `tests/tests.nix`; fails if any result is not "ok" (or the eval throws). | -| `python-tests` | `unittest` over `build/tests/` — the orchestrator's arg parsing, `flake.lock` reading, and GitHub-token precedence, all pure functions needing no build. | +| `python-tests` | `unittest` over `build/tests/` — the orchestrator's arg parsing, `flake.lock` reading, and GitHub-token precedence, all pure functions needing no build. | `nixfmt-check` | `nixfmt --check` over every `*.nix`; without it a commit lands unformatted and the next one absorbs the reformat. | Run it **without `--no-build`**: `lib-tests` reaches `builtins.path`/`readDir` on a @@ -514,8 +514,36 @@ consumed via flake inputs, and a committed lock would pin core's own inputs — `icedos rebuild` flags (full list in `README.md`): `--boot`, `--build`, `--build-vm`, `--dry`/`-n`/`--dry-run`, `--run-vm`, `--update`, `--update-core`, `--update-core-only`, `--update-state-inputs "..."`, `--update-repos`, -`--update-repos-only`, `--update-repos-select "..."`, `--update-repo-inputs-only`, `--update-hooks`, `--ask`, -`--builder `, `--logs`, `--nh-args …`, `--build-args …` (must be last). +`--update-repos-only`, `--update-repos-select "..."`, `--update-repo-inputs-only`, `--unpin-inputs "..."`, +`--unpin-inputs-all`, `--update-hooks`, `--ask`, `--builder `, `--logs`, `--nh-args …`, +`--build-args …` (must be last). + +**Unpin path (`--unpin-inputs` / `--unpin-inputs-all`, needs `system.cache.enable` + +`system.cache.pinInputs`).** The orchestrator (`build/pins.py`) diffs each named +cache-pinned input's `tracked-inputs.json` rev (exposed by genflake as `pinRevs`) +against the input's upstream rev (`git ls-remote`): the declared url ref +(genflake exports it as `pinRefs`, since the state lock loses it once a rev is +baked in; both the `` path segment and the `?ref=` query spelling count), +else the remote HEAD. It then asks (TTY only) whether to re-pin. +Each accepted re-pin is remembered in +`.state/unpinned-inputs.json`; genflake bakes that rev into the sub-flake url in +place of the cache pin (`_unpinnedRevs` + the saved-pin branch of `_cachePin` in +`lib/icedos.nix` — still a pin, just a custom one). The whole pin machinery, +cache revs included, is gated on `_cachePinsActive` = `enable` && `pinInputs`; +`--unpin-inputs-all` covers every cache-tracked module-leaf input the config +locks (state-lock membership via `_lock_leaf`; root-declared repo-level tracked +names are skipped, and `--unpin-inputs` rejects them). A remembered pin is +dropped automatically once the cache-server's rev reaches or passes it +(`git merge-base --is-ancestor` in a throwaway blobless clone; unknown ancestry +keeps the pin), so steady state always returns to the cache-server's pins — plain +rebuilds run that cleanup themselves (`expire_unpinned`, only when the state file +exists, at the cost of one extra genflake evaluation per such rebuild). +`--genflake-only` / +`--dry` runs skip both unpin paths entirely. Patched module inputs reach the +custom rev only via their lock bypass +(`--update-repo-inputs-only`/`--update`), matching the cache-pin path. The +rebuild help only lists the flags when the gate is on. Deleting +the state file reverts to cache pins immediately. ## 8. Hard rules (do not violate) diff --git a/README.md b/README.md index 8728215b..f51ef17e 100644 --- a/README.md +++ b/README.md @@ -519,6 +519,8 @@ Rebuilds can pass a GitHub token to nix as a `github.com` access token (higher A | `--update-repos-only` | Pull new revisions of the IceDOS module repos only. Does **not** re-lock inputs declared *inside* those modules — the sub-flake texts are generated before the repo bump in the same run, so if the bumped rev changes a module's declared inputs, those land on the **next** build (one-build lag, self-healing). | Get latest modules without bumping their dependencies. | | `--update-repos-select "..."` | Pull new revisions of only the named repos (space-separated `[[icedos.repositories]].url` values), without re-locking inputs declared *inside* those modules. Same one-build lag as `--update-repos-only` for the selected repos. | Get latest modules for one or a few repos without bumping everything. | | `--update-repo-inputs-only` | Re-lock every module-declared dependency without pulling new repo revisions. Each module's inputs live in its own input-namespace sub-flake. | Bump module dependencies without updating repos. | +| `--unpin-inputs "..."` | With cache `enable` + `pinInputs` set, offer to re-pin cache-pinned module inputs (space-separated tracked names) to their upstream revs — the input's declared url ref when it declares one, else the default branch. Repo-level tracked names are rejected. Each accepted re-pin is remembered in `.state/unpinned-inputs.json` and keeps overriding the cache pin until the cache-server's rev reaches or passes it — the memory is then dropped automatically on the next rebuild (or `--unpin-inputs` run). Non-interactive sessions keep the current pin instead of prompting. Module inputs declared with `patches` only pick the custom rev once their lock entry is gone — pair the run with `--update-repo-inputs-only` for those. | Track one input at master while the rest stay cache-pinned. | +| `--unpin-inputs-all` | Same as `--unpin-inputs` (requires cache `enable` + `pinInputs`), but offers the re-pin for every cache-tracked module-leaf input this config locks (repo-level tracked names are skipped). | Catch the whole config up to its default branches in one run. | | `--update-hooks` | Run only the `preUpdate`/`postUpdate` hooks and exit — no build, no activation. | Refresh non-Nix things (e.g. `flatpak update`). | #### Behavior flags diff --git a/build/genflake.py b/build/genflake.py index ccbc789b..87aa0da5 100644 --- a/build/genflake.py +++ b/build/genflake.py @@ -8,7 +8,7 @@ from .util import JSON, capture, run, write_json, write_text -def _nix_eval_json(env: BuildEnv, apply: str, trace: list[str]) -> str: +def nix_eval_json(env: BuildEnv, apply: str, trace: list[str]) -> str: cmd = [ "nix", "eval", @@ -27,7 +27,7 @@ def export_search_index(env: BuildEnv, trace: list[str]) -> None: cache = env.state_dir / ".cache" _ = cache.mkdir(parents=True, exist_ok=True) - search_docs = _nix_eval_json( + search_docs = nix_eval_json( env, "g: { inherit (g) optionsDoc modulesDoc; }", trace ) docs = cast("dict[str, JSON]", json.loads(search_docs)) @@ -42,7 +42,7 @@ def export_search_index(env: BuildEnv, trace: list[str]) -> None: write_json(cache / filename, value) run(["jsonfmt", str(cache / filename), "-w"], check=True) - user_config = _nix_eval_json(env, "g: g.userConfigRaw", trace) + user_config = nix_eval_json(env, "g: g.userConfigRaw", trace) write_json(cache / "config.json", cast(JSON, json.loads(user_config))) run(["jsonfmt", str(cache / "config.json"), "-w"], check=True) diff --git a/build/main.py b/build/main.py index f8032a62..ff3afc8f 100644 --- a/build/main.py +++ b/build/main.py @@ -8,6 +8,7 @@ from .context import from_environment from .genflake import export_search_index, generate_flake from .options import Options, parse_args +from .pins import UNPINNED_FILE, expire_unpinned, unpin_inputs from .runner import build from .update import maybe_re_exec_update_core, prepare_lock, refresh_config_root_paths from .util import warn @@ -112,6 +113,23 @@ def main(argv: list[str] | None = None) -> int: refresh_config_root_paths(env, opts) maybe_re_exec_update_core(env, opts, previous_arguments) + # --dry/--genflake-only must not mutate state: no expiry, no prompts. + mutating = not opts.genflake_only + if ( + mutating + and (env.state_dir / UNPINNED_FILE).exists() + and not (opts.unpin_inputs or opts.unpin_all) + ): + # Best-effort case-4 expiry; runs only while a remembered pin exists. + expire_unpinned(env, trace) + + # Runs before genflake, which bakes the pins into the sub-flake urls. + if opts.unpin_inputs or opts.unpin_all: + if mutating: + unpin_inputs(env, opts, trace) + else: + warn("warning: --unpin-input* have no effect together with --genflake-only/--dry") + generate_flake( env, trace, diff --git a/build/options.py b/build/options.py index 875df070..d804ae09 100644 --- a/build/options.py +++ b/build/options.py @@ -18,6 +18,8 @@ class Options: update_repos_inputs: bool = False state_inputs: list[str] = field(default_factory=list) repos_select: list[str] = field(default_factory=list) + unpin_inputs: list[str] = field(default_factory=list) + unpin_all: bool = False github_token: str | None = None github_token_path: str | None = None nh_build_args: list[str] = field(default_factory=list) @@ -38,6 +40,10 @@ def trace(self) -> list[str]: error: --update-state-inputs requires a space-separated list of input names usage: --update-state-inputs "nixpkgs home-manager\"""" +_ERR_UNPIN_INPUTS = """\ +error: --unpin-inputs requires a space-separated list of tracked input names + usage: --unpin-inputs "plasmazones jovian\"""" + def _die(message: str) -> NoReturn: print(message, file=sys.stderr) @@ -110,6 +116,14 @@ def parse_args(argv: list[str]) -> tuple[Options, list[str]]: _die(_ERR_STATE_INPUTS) opts.state_inputs.extend(_take_list(arg, argv[i + 1])) i += 2 + elif arg == "--unpin-inputs": + if i + 1 >= len(argv) or argv[i + 1].startswith("--"): + _die(_ERR_UNPIN_INPUTS) + opts.unpin_inputs.extend(_take_list(arg, argv[i + 1])) + i += 2 + elif arg == "--unpin-inputs-all": + opts.unpin_all = True + i += 1 elif arg == "--ask": opts.nh_build_args.append("-a") i += 1 @@ -147,4 +161,7 @@ def parse_args(argv: list[str]) -> tuple[Options, list[str]]: else: _die(f"Unknown arg: {arg}") + if opts.unpin_all and opts.unpin_inputs: + _die("error: --unpin-inputs-all cannot be combined with --unpin-inputs") + return opts, previous_arguments diff --git a/build/pins.py b/build/pins.py new file mode 100644 index 00000000..aef56e3a --- /dev/null +++ b/build/pins.py @@ -0,0 +1,634 @@ +from __future__ import annotations + +import json +import os +import re +import shutil +import subprocess +import sys +import tempfile +from collections.abc import Callable +from typing import Protocol +from pathlib import Path +from typing import TypeAlias, cast + +from .context import BuildEnv +from .genflake import nix_eval_json +from .lockfile import load_lock +from .options import Options +from .util import JSON, warn + +# .state file holding remembered custom pins: name -> { rev; repo; }. +UNPINNED_FILE = "unpinned-inputs.json" + +# ls-remote/fetch base urls for the forge schemes the pin machinery tracks. +_FORGE_URLS = { + "github": "https://github.com/{}/{}", + "gitlab": "https://gitlab.com/{}/{}", + "sourcehut": "https://git.sr.ht/{}/{}", +} + +# Full or abbreviated git sha; revs reach git argv positions unquoted. +_SHA_RE = re.compile(r"^[0-9a-f]{7,40}$") + + +def _is_sha(rev: str) -> bool: + return _SHA_RE.match(rev) is not None + + +def _git_env() -> dict[str, str]: + # Unauthenticated remotes must fail fast, not prompt on an invisible tty. + return { + **os.environ, + "GIT_TERMINAL_PROMPT": "0", + "GIT_ASKPASS": "", + "GIT_SSH_COMMAND": "ssh -oBatchMode=yes", + } + + +AskFn: TypeAlias = Callable[[str], bool] +# is x an ancestor-or-equal of y? True/False when decidable, None when git +# could not fetch the commits (offline, forge refusing fetch-by-sha, ...). +AncestryFn: TypeAlias = Callable[[str, str], bool | None] + + +class _GitLike(Protocol): + """Structural interface _prune_pass needs, so tests can fake it.""" + + def head(self, url: str) -> str: ... + + def ref_head(self, url: str, ref: str) -> str: ... + + def is_ancestor(self, url: str, a: str, b: str) -> bool | None: ... + + +def _never(_message: str) -> bool: + return False + + +class _Git: + """Cached upstream lookups; git failures degrade to ""/None (unknown).""" + + def __init__(self) -> None: + self._heads: dict[str, str] = {} + self._ref_heads: dict[tuple[str, str], str] = {} + self._ancestry: dict[tuple[str, str, str], bool | None] = {} + + def head(self, url: str) -> str: + if url not in self._heads: + try: + proc = subprocess.run( + ["git", "ls-remote", "--", url, "HEAD"], + capture_output=True, + text=True, + timeout=30, + env=_git_env(), + ) + except (OSError, subprocess.TimeoutExpired): + proc = None + lines = proc.stdout.split() if proc is not None and proc.returncode == 0 else [] + # Revs reach git argvs and sub-flake urls: never accept non-shas. + sha = lines[0] if lines else "" + self._heads[url] = sha if _is_sha(sha) else "" + return self._heads[url] + + def ref_head(self, url: str, ref: str) -> str: + # Branch or tag lookup for inputs declared `github:o/r/`; HEAD + # would offer a commit from the default branch of a different line. + if not re.fullmatch(r"[A-Za-z0-9._/-]+", ref): + return "" + key = (url, ref) + if key not in self._ref_heads: + try: + proc = subprocess.run( + ["git", "ls-remote", "--", url, f"refs/heads/{ref}", f"refs/tags/{ref}"], + capture_output=True, + text=True, + timeout=30, + env=_git_env(), + ) + except (OSError, subprocess.TimeoutExpired): + proc = None + resolved: dict[str, str] = {} + if proc is not None and proc.returncode == 0: + for line in proc.stdout.splitlines(): + sha, _, name = line.partition("\t") + if name: + resolved[name.strip()] = sha.strip() + # Prefer the branch; for tags take the peeled commit, not the tag object. + sha = ( + resolved.get(f"refs/heads/{ref}") + or resolved.get(f"refs/tags/{ref}^{{}}") + or resolved.get(f"refs/tags/{ref}") + or "" + ) + self._ref_heads[key] = sha if _is_sha(sha) else "" + return self._ref_heads[key] + + def is_ancestor(self, url: str, a: str, b: str) -> bool | None: + # Revs come from remote JSON / the state file: never let non-shas + # reach a git argv position. + if not (_is_sha(a) and _is_sha(b)): + return None + if a == b: + return True + key = (url, a, b) + if key not in self._ancestry: + self._ancestry[key] = self._probe_ancestor(url, a, b) + return self._ancestry[key] + + @staticmethod + def _probe_ancestor(url: str, a: str, b: str) -> bool | None: + tmp = Path(tempfile.mkdtemp(prefix="icedos-ancestry-")) + try: + + def git(*args: str) -> int: + try: + return subprocess.run( + ["git", *args], + cwd=tmp, + capture_output=True, + timeout=120, + env=_git_env(), + ).returncode + except subprocess.TimeoutExpired: + return -1 + + if git("init", "-q") != 0 or git("remote", "add", "origin", "--", url) != 0: + return None + # Blobless fetch: only history metadata, never full trees. + rc = git("fetch", "-q", "--no-tags", "--filter=blob:none", "origin", a, b) + if rc != 0: + # Some forges refuse fetch-by-sha; fall back to all branches. + rc = git("fetch", "-q", "--no-tags", "--filter=blob:none", "origin") + if rc != 0: + # ...and partial clones (uploadpack.allowFilter off). + rc = git("fetch", "-q", "--no-tags", "origin") + if rc != 0: + return None + try: + probe = subprocess.run( + ["git", "merge-base", "--is-ancestor", a, b], + cwd=tmp, + capture_output=True, + timeout=30, + ) + except subprocess.TimeoutExpired: + return None + # 0 = ancestor, 1 = not; anything else (128: bad object, fetch + # missed the rev, ...) must stay unknown rather than read as False. + return {0: True, 1: False}.get(probe.returncode) + finally: + shutil.rmtree(tmp, ignore_errors=True) + + +def _normalise_entry(value: JSON) -> dict[str, str]: + # tracked-inputs.json entries are a bare rev or { rev; repo; }; remembered + # pins additionally carry `checked` (the cache rev last probed against). + if isinstance(value, str): + return {"rev": value, "repo": "", "checked": ""} + if isinstance(value, dict): + return { + "rev": str(value.get("rev") or ""), + "repo": str(value.get("repo") or ""), + "checked": str(value.get("checked") or ""), + } + return {"rev": "", "repo": "", "checked": ""} + + +def _lock_leaf(lock: dict[str, JSON], name: str) -> dict[str, JSON]: + # Same node-key lookup as cache-server's tracked-revs.py: exact match, else + # a "-" suffixed one. + nodes = lock.get("nodes") + if not isinstance(nodes, dict): + return {} + # Exact key wins; refuse ambiguous suffixes like the nix-side lookup does. + matches = [name] if name in nodes else [ + k for k in nodes if k.endswith("-" + name) + ] + if len(matches) != 1: + return {} + key = matches[0] + node = nodes.get(key) + if not isinstance(node, dict): + return {} + locked = node.get("locked") + if isinstance(locked, dict) and locked.get("rev"): + return node # already the leaf node + # A rev-less match is the node's sub-flake ROOT (a path node); its inputs + # map the bare input name, so try that when the full name misses. + inputs = node.get("inputs") + if not isinstance(inputs, dict): + return node + candidates = [name] if "-" not in name else [name, name.rsplit("-", 1)[-1]] + for candidate in candidates: + ref = inputs.get(candidate) + if isinstance(ref, str): + nested = nodes.get(ref) + if isinstance(nested, dict): + return nested + return node + + +def _root_input_names(lock: dict[str, JSON]) -> set[str]: + # Repo/channel-level names declared on the lock root. Sub-flake ROOTS are + # root inputs too, but they are legit tracked keys (their leaf hangs off + # their inputs map), so they are excluded here. + nodes = lock.get("nodes") + root_key = lock.get("root") + node = ( + nodes.get(root_key if isinstance(root_key, str) else "root") + if isinstance(nodes, dict) + else None + ) + inputs = node.get("inputs") if isinstance(node, dict) else None + if not isinstance(inputs, dict): + return set() + + def is_subflake_root(key: JSON) -> bool: + entry = nodes.get(key) if isinstance(key, str) and isinstance(nodes, dict) else None + original = entry.get("original") if isinstance(entry, dict) else None + path = original.get("path") if isinstance(original, dict) else None + return isinstance(path, str) and path.endswith("-subflake") + + return {k for k in inputs if isinstance(k, str) and not is_subflake_root(inputs[k])} + + +def _leaf_ident(lock: dict[str, JSON], name: str) -> str: + # Bare "scheme:owner/repo" of the leaf the config actually locks under + # `name` — the same shape the nix-side repo guard compares against. + orig = _lock_leaf(lock, name).get("original") + if not isinstance(orig, dict): + return "" + kind = orig.get("type") + if kind in _FORGE_URLS: + return f"{kind}:{orig.get('owner')}/{orig.get('repo')}" + if kind == "git": + url = orig.get("url", "") + return url if isinstance(url, str) else "" + return "" + + +def _leaf_url(lock: dict[str, JSON], name: str) -> str: + # https url of that leaf, honoring the self-hosted `host` attribute + # (mirrors the nix-side fetchTree shape). + orig = _lock_leaf(lock, name).get("original") + if not isinstance(orig, dict): + return "" + kind = orig.get("type") + if kind == "git": + url = str(orig.get("url") or "").removeprefix("git+") + return url if url.startswith(("https://", "http://", "ssh://", "git://")) else "" + if kind not in _FORGE_URLS: + return "" + owner = str(orig.get("owner") or "").replace("%2F", "/") + repo = str(orig.get("repo") or "") + if not owner or not repo: + return "" + host = orig.get("host") + if isinstance(host, str) and host: + return f"https://{host}/{owner}/{repo}" + return _FORGE_URLS[kind].format(owner, repo) + + +def _ident_to_url(ident: str) -> str: + ident = ident.removeprefix("git+") + # Scheme allowlist: blocks git's ext:: command-execution transport and any + # other non-forge scheme smuggled through remote-supplied repo strings. + if ident.startswith(("https://", "http://", "ssh://", "git://")): + return ident + scheme, _, rest = ident.partition(":") + host_override = "" + if "?" in rest: + rest, query = rest.split("?", 1) + for param in query.split("&"): + name, _, value = param.partition("=") + if name == "host": + host_override = value + template = _FORGE_URLS.get(scheme) + if template and "/" in rest: + owner, repo = rest.split("/", 1) + # Gitlab subgroups arrive percent-encoded in the owner segment. + owner = owner.replace("%2F", "/") + if host_override: + return f"https://{host_override}/{owner}/{repo}" + return template.format(owner, repo) + return "" + + +def _git_url(entry: dict[str, str], lock: dict[str, JSON], name: str) -> str: + # The config's own locked leaf is authoritative; the tracked entry's repo is + # a fallback that must agree when both exist (mirrors the nix repo guard). + leaf_url = _leaf_url(lock, name) + if leaf_url: + if entry["repo"] and entry["repo"] != _leaf_ident(lock, name): + return "" + return leaf_url + return _ident_to_url(entry["repo"]) + + +def decide_pin( + name: str, + cache_rev: str, + saved_rev: str, + master_rev: str, + ancestry: AncestryFn, + ask: AskFn, +) -> tuple[str, str]: + """Run the four unpin cases for one tracked input. + + Args are the cache-server rev, remembered pin, and master ("" = unknown). + Returns (action, rev): save/clear/keep/noop/skip. + """ + if not saved_rev: + if not cache_rev: + return "skip", "" + if not master_rev or cache_rev == master_rev: + return "noop", "" + # Cache already at-or-ahead of master: re-pinning would move backward. + # Diverged/unknown still asks, naming both revs. + if ancestry(master_rev, cache_rev) is True: + return "noop", "" + if ask(f"{name}: pinned to {cache_rev[:12]}, master is at {master_rev[:12]} — re-pin to master?"): + return "save", master_rev + return "keep", "" + + # Remembered pin: drop it once the cache-server's rev reaches or passes it. + if not cache_rev or cache_rev == saved_rev: + return "clear", "" + if ancestry(saved_rev, cache_rev) is True: + return "clear", "" + # Cache is still behind the custom pin (or ancestry unknown): stay on it + # unless master moved past the pin too. + if not master_rev or master_rev == saved_rev: + return "keep", "" + if ancestry(saved_rev, master_rev) is False: + return "keep", "" + if ask(f"{name}: custom pin {saved_rev[:12]}, master moved to {master_rev[:12]} — re-pin to master?"): + return "save", master_rev + return "keep", "" + + +def _pin_revs(env: BuildEnv, trace: list[str]) -> tuple[dict[str, JSON], dict[str, str]]: + # One eval for revs and declared refs: genflake's output asserts already + # force full module discovery, so a revs-only eval would cost the same. + data = cast( + "dict[str, JSON]", + json.loads(nix_eval_json(env, "g: { revs = g.pinRevs; refs = g.pinRefs; }", trace)), + ) + revs = cast("dict[str, JSON]", data.get("revs") or {}) + refs = cast("dict[str, str]", data.get("refs") or {}) + return revs, refs + + +def _read_unpinned(path: Path) -> dict[str, JSON]: + if not path.exists(): + return {} + try: + raw = path.read_text() + except OSError: + warn(f"warning: unreadable {path}; ignoring remembered custom pins") + # Unlink so nix's readFile path degrades to "no pins" too, if it can. + try: + _ = path.unlink() + except OSError: + pass + return {} + except UnicodeDecodeError: + # Non-UTF-8 bytes are corrupt for the nix-side read as well; heal. + warn(f"warning: unreadable {path}; ignoring remembered custom pins") + _write_json_atomic(path, {}) + return {} + try: + value: JSON = cast(JSON, json.loads(raw)) + except ValueError: + # Corrupt JSON must heal here: the nix-side read has no tolerance. + warn(f"warning: unreadable {path}; ignoring remembered custom pins") + _write_json_atomic(path, {}) + return {} + if not isinstance(value, dict): + # Valid JSON but not an object — nix needs an attrset; heal it too. + _write_json_atomic(path, {}) + return {} + return value + + +def _write_json_atomic(path: Path, value: JSON) -> None: + # write_json truncates in place; a crash mid-write would leave a file that + # kills every genflake eval, so publish via rename instead. + _ = path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=path.name, suffix=".tmp") + try: + with os.fdopen(fd, "w") as fh: + json.dump(value, fh, ensure_ascii=False) + os.replace(tmp, path) + except BaseException: + os.unlink(tmp) + raise + + +def _drop(saved: dict[str, JSON], name: str, reason: str) -> bool: + print(f"unpin: {reason}") + _ = saved.pop(name, None) + return True + + +def _prune_pass( + saved: dict[str, JSON], + pin_revs: dict[str, JSON], + lock: dict[str, JSON], + git: _GitLike, + skip: set[str], +) -> bool: + """Case-4 cleanup over remembered pins; returns True when anything changed. + + No ask here: only case 4 is handled, the rest belongs to the main loop. + """ + changed = False + for name in sorted(saved): + if name in skip: + continue + entry = _normalise_entry(saved[name]) + if name not in pin_revs: + changed = _drop(saved, name, f"{name} is no longer cache-tracked; dropping the custom pin") or changed + continue + if not entry["rev"]: + changed = _drop(saved, name, f"{name} has an empty custom pin; dropping it") or changed + continue + cache_rev = _normalise_entry(pin_revs[name])["rev"] + if entry["checked"] == cache_rev: + continue # already probed against this cache rev; nothing new to clear + url = _git_url(entry, lock, name) + + def ancestry(a: str, b: str) -> bool | None: + return git.is_ancestor(url, a, b) if url else None + + action, _ = decide_pin(name, cache_rev, entry["rev"], "", ancestry, _never) + if action == "clear": + reason = ( + f"cache-server reached the custom pin for {name}; dropping it" + if cache_rev + else f"{name} is no longer cache-pinned; dropping the custom pin" + ) + changed = _drop(saved, name, reason) or changed + elif ancestry(entry["rev"], cache_rev) is False: + # Definitive "cache behind": stamp the probed cache rev so plain + # rebuilds skip the git probe until the cache-server moves. An + # inconclusive probe (offline, no url) must re-probe next time. + saved[name] = {"rev": entry["rev"], "repo": entry["repo"], "checked": cache_rev} + changed = True + return changed + + +def expire_unpinned(env: BuildEnv, trace: list[str]) -> None: + """Best-effort case-4 cleanup for plain rebuilds (no --unpin-inputs).""" + file_path = env.state_dir / UNPINNED_FILE + saved = _read_unpinned(file_path) + if not saved: + return + try: + pin_revs, _ = _pin_revs(env, trace) + except SystemExit: + warn("warning: could not evaluate the cache pin revs; keeping remembered custom pins") + return + if not pin_revs: + return + saved_copy = dict(saved) + if _prune_pass(saved_copy, pin_revs, load_lock(env.state_dir / "flake.lock"), _Git(), set()): + _write_json_atomic(file_path, cast(JSON, saved_copy)) + + +def unpin_inputs(env: BuildEnv, opts: Options, trace: list[str]) -> None: + pin_revs, refs = _pin_revs(env, trace) + if not pin_revs: + print( + "error: --unpin-inputs found no cache-pinned inputs. Either\n" + + "icedos.system.cache.enable or icedos.system.cache.pinInputs is not set,\n" + + "or there is nothing to diff against yet (first build, or a channel\n" + + "publish without tracked-inputs.json — both self-heal on the next run).", + file=sys.stderr, + ) + raise SystemExit(1) + lock = load_lock(env.state_dir / "flake.lock") + root_inputs = _root_input_names(lock) + if opts.unpin_all: + # Every cache-tracked module leaf input this config actually locks; the + # rest is not part of this build, so re-pinning it would change nothing. + names = [ + n + for n in sorted(pin_revs) + # Lock membership, not url derivability: a newer { rev; repo; } + # entry carries its own repo even for inputs this config never locks. + if _lock_leaf(lock, n).get("original") and n not in root_inputs + ] + if not names: + warn("warning: no cache-tracked input is locked by this config; nothing to unpin") + else: + skipped = [n for n in sorted(pin_revs) if n not in names] + if skipped: + warn( + "warning: cache-tracked inputs not locked by this config are skipped: " + + " ".join(skipped) + ) + else: + for name in opts.unpin_inputs: + if name not in pin_revs: + tracked = ", ".join(sorted(pin_revs)) or "none" + print( + f"error: '{name}' is not tracked by the cache server (tracked: {tracked})", + file=sys.stderr, + ) + raise SystemExit(1) + if name in root_inputs: + print( + f"error: '{name}' is a repo-level input, not a module-declared " + + "leaf input; the pin machinery does not apply to it", + file=sys.stderr, + ) + raise SystemExit(1) + names = opts.unpin_inputs + + file_path = env.state_dir / UNPINNED_FILE + saved = _read_unpinned(file_path) + + git = _Git() + interactive = sys.stdin.isatty() + + def ask(message: str) -> bool: + if not interactive: + warn(f"warning: {message} [y/N] — not a terminal, keeping the current pin") + return False + try: + answer = input(f"{message} [y/N] ") + except EOFError: + return False + return answer.strip().lower() in ("y", "yes") + + changed = _prune_pass(saved, pin_revs, lock, git, set(names)) + + for name in names: + cache = _normalise_entry(pin_revs[name]) + url = _git_url(cache, lock, name) + saved_rev = _normalise_entry(saved.get(name, ""))["rev"] + # Case 4 is decidable without the network; run it before the master + # lookup so an ls-remote failure cannot leave an expired pin in place. + if saved_rev and (not cache["rev"] or cache["rev"] == saved_rev): + reason = ( + "custom pin dropped; back on the cache-server's rev" + if cache["rev"] + else "is no longer cache-pinned; dropping the custom pin" + ) + changed = _drop(saved, name, f"{name} {reason}") or changed + saved_rev = "" + # The locked leaf's repo is authoritative, so the nix-side repo guard + # always agrees. Non-forge inputs (git urls) never reach the pin + # machinery (_cachePin's scheme gate), so skip before any network call. + leaf = _leaf_ident(lock, name) + repo = leaf if leaf.partition(":")[0] in _FORGE_URLS else cache["repo"] + if repo.partition(":")[0] not in _FORGE_URLS: + warn( + f"warning: {name}: not a github/gitlab/sourcehut input; " + + "the pin machinery cannot apply it; skipping" + ) + continue + ref = refs.get(name, "") + master = git.ref_head(url, ref) if (url and ref) else (git.head(url) if url else "") + + def ancestry(a: str, b: str) -> bool | None: + return git.is_ancestor(url, a, b) if url else None + + if not master: + ref_note = f" for refs '{ref}'" if ref else "" + warn( + f"warning: {name}: no resolvable git url or upstream rev{ref_note} " + + "(no lock entry, or ls-remote failed); skipping" + ) + continue + action, rev = decide_pin( + name, cache["rev"], saved_rev, master, ancestry, ask, + ) + if action == "clear": + changed = _drop(saved, name, f"{name} custom pin dropped; back on the cache-server's rev") or changed + # The memory is gone — cases 1-2 may still offer a fresh re-pin. + action, rev = decide_pin( + name, cache["rev"], "", master, ancestry, ask, + ) + saved_rev = "" + if action == "save": + saved[name] = {"rev": rev, "repo": repo} + target = f"'{ref}'" if ref else "master" + print(f"unpin: {name} re-pinned to {target} ({rev[:12]})") + changed = True + elif action == "keep": + if saved_rev: + print(f"unpin: {name} keeps its custom pin ({saved_rev[:12]})") + else: + print(f"unpin: {name} keeps the cache-server's pin ({cache['rev'][:12]})") + elif action == "skip": + print(f"unpin: {name} is not cache-pinned; nothing to unpin") + elif action == "noop": + print(f"unpin: {name} is already at master; nothing to do") + + if changed: + _write_json_atomic(file_path, cast(JSON, saved)) + print(f"unpin: wrote {file_path}") diff --git a/build/tests/test_lockfile.py b/build/tests/test_lockfile.py index 9c0bf0d8..28ddae3c 100644 --- a/build/tests/test_lockfile.py +++ b/build/tests/test_lockfile.py @@ -4,12 +4,14 @@ import tempfile import unittest from pathlib import Path +from typing import cast from build import lockfile +from build.util import JSON # A state lock with one sub-flake root, a nested local path input, a nested store # path input, a plain github root input, and follows entries (lists, not keys). -LOCK = { +LOCK: dict[str, JSON] = { "nodes": { "root": { "inputs": { @@ -37,18 +39,25 @@ } +def _lock_with_sub_node_path(path: str) -> dict[str, JSON]: + lock = cast("dict[str, JSON]", json.loads(json.dumps(LOCK))) + node = cast("dict[str, JSON]", lock["nodes"]) + sub = cast("dict[str, JSON]", node["sub-node"]) + locked = cast("dict[str, JSON]", sub["locked"]) + locked["path"] = path + return lock + + class SubflakeTest(unittest.TestCase): def test_requires_store_prefix_and_name_suffix(self): self.assertEqual(lockfile.subflakes_from_lock(LOCK), ["sub"]) def test_store_path_without_the_suffix_is_not_a_subflake(self): - lock = json.loads(json.dumps(LOCK)) - lock["nodes"]["sub-node"]["locked"]["path"] = "/nix/store/hash-sub" + lock = _lock_with_sub_node_path("/nix/store/hash-sub") self.assertEqual(lockfile.subflakes_from_lock(lock), []) def test_local_path_is_not_a_subflake(self): - lock = json.loads(json.dumps(LOCK)) - lock["nodes"]["sub-node"]["locked"]["path"] = "/home/u/sub-subflake" + lock = _lock_with_sub_node_path("/home/u/sub-subflake") self.assertEqual(lockfile.subflakes_from_lock(lock), []) @@ -89,7 +98,12 @@ def test_input_is_string_rejects_follows_entries(self): class MalformedTest(unittest.TestCase): def test_every_helper_tolerates_an_empty_lock(self): - for lock in ({}, {"nodes": {}}, {"nodes": {"root": {}}}): + empty_locks: list[dict[str, JSON]] = [ + {}, + {"nodes": {}}, + {"nodes": {"root": {}}}, + ] + for lock in empty_locks: self.assertEqual(lockfile.path_nodes(lock), []) self.assertEqual(lockfile.path_inputs(lock), []) self.assertEqual(lockfile.subflakes_from_lock(lock), []) @@ -124,4 +138,4 @@ def test_round_trips_a_real_lock(self): if __name__ == "__main__": - unittest.main() + _ = unittest.main() diff --git a/build/tests/test_options.py b/build/tests/test_options.py index 84698a20..03ab0623 100644 --- a/build/tests/test_options.py +++ b/build/tests/test_options.py @@ -64,15 +64,36 @@ def test_repos_select_splits_on_whitespace(self): def test_repos_select_rejects_a_following_flag(self): with self.assertRaises(SystemExit): - _parse(["--update-repos-select", "--logs"]) + _ = _parse(["--update-repos-select", "--logs"]) def test_repos_select_rejects_an_empty_list(self): with self.assertRaises(SystemExit): - _parse(["--update-repos-select", " "]) + _ = _parse(["--update-repos-select", " "]) + + def test_unpin_inputs_splits_on_whitespace(self): + opts, _ = _parse(["--unpin-inputs", "plasmazones jovian"]) + self.assertEqual(opts.unpin_inputs, ["plasmazones", "jovian"]) + + def test_unpin_inputs_rejects_a_following_flag(self): + with self.assertRaises(SystemExit): + _ = _parse(["--unpin-inputs", "--logs"]) + + def test_unpin_inputs_rejects_an_empty_list(self): + with self.assertRaises(SystemExit): + _ = _parse(["--unpin-inputs", " "]) + + def test_unpin_inputs_all_sets_only_the_flag(self): + opts, _ = _parse(["--unpin-inputs-all"]) + self.assertTrue(opts.unpin_all) + self.assertEqual(opts.unpin_inputs, []) + + def test_unpin_inputs_all_rejects_combination_with_list(self): + with self.assertRaises(SystemExit): + _ = _parse(["--unpin-inputs-all", "--unpin-inputs", "x"]) def test_state_inputs_rejects_a_following_flag(self): with self.assertRaises(SystemExit): - _parse(["--update-state-inputs", "--logs"]) + _ = _parse(["--update-state-inputs", "--logs"]) def test_builder_and_target_map_to_nh_flags(self): opts, _ = _parse(["--builder", "b.example", "--target", "t.example"]) @@ -83,11 +104,11 @@ def test_builder_and_target_map_to_nh_flags(self): def test_github_token_requires_a_value(self): with self.assertRaises(SystemExit): - _parse(["--github-token"]) + _ = _parse(["--github-token"]) def test_github_token_path_requires_a_value(self): with self.assertRaises(SystemExit): - _parse(["--github-token-path"]) + _ = _parse(["--github-token-path"]) def test_github_token_flags_are_captured(self): opts, _ = _parse(["--github-token", "tok", "--github-token-path", "/p"]) @@ -96,8 +117,8 @@ def test_github_token_flags_are_captured(self): def test_unknown_arg_exits(self): with self.assertRaises(SystemExit): - _parse(["--nope"]) + _ = _parse(["--nope"]) if __name__ == "__main__": - unittest.main() + _ = unittest.main() diff --git a/build/tests/test_pins.py b/build/tests/test_pins.py new file mode 100644 index 00000000..505908a3 --- /dev/null +++ b/build/tests/test_pins.py @@ -0,0 +1,710 @@ +from __future__ import annotations + +import io +import json +import tempfile +import unittest +from collections.abc import Generator +from contextlib import contextmanager, redirect_stderr, redirect_stdout +from pathlib import Path +from types import ModuleType +from typing import cast +from unittest import mock + +import subprocess + +from build import pins as pins_mod +from build.context import BuildEnv +from build.pins import _Git +from build.options import Options +from build.pins import ( + UNPINNED_FILE, + AncestryFn, + _git_url, + _ident_to_url, + _lock_leaf, + _normalise_entry, + _read_unpinned, + decide_pin, + expire_unpinned, + unpin_inputs, +) +from build.util import JSON + + +def _no_ask(_message: str) -> bool: + return False + + +def _yes_ask(_message: str) -> bool: + return True + + +def _ancestry(map: dict[tuple[str, str], bool]) -> AncestryFn: + return lambda a, b: map.get((a, b)) + + +class DecidePinTest(unittest.TestCase): + def test_case_1_cache_behind_master_asks_to_repin(self): + action, rev = decide_pin( + "x", "aaa", "", "bbb", _ancestry({("aaa", "bbb"): True}), _yes_ask + ) + self.assertEqual((action, rev), ("save", "bbb")) + + def test_case_1_declined_keeps_the_cache_pin(self): + action, rev = decide_pin( + "x", "aaa", "", "bbb", _ancestry({("aaa", "bbb"): True}), _no_ask + ) + self.assertEqual((action, rev), ("keep", "")) + + def test_case_2_cache_equals_master_is_a_noop(self): + action, rev = decide_pin("x", "aaa", "", "aaa", _ancestry({}), _yes_ask) + self.assertEqual((action, rev), ("noop", "")) + + def test_cache_already_ahead_of_master_does_not_repin(self): + action, rev = decide_pin( + "x", "bbb", "", "aaa", _ancestry({("aaa", "bbb"): True}), _yes_ask + ) + self.assertEqual((action, rev), ("noop", "")) + + def test_case_3_master_moved_past_the_custom_pin_asks(self): + # Cache (ccc) is behind the custom pin (bbb); master (ddd) moved past it. + ancestry = _ancestry({("bbb", "ddd"): True}) + action, rev = decide_pin("x", "ccc", "bbb", "ddd", ancestry, _yes_ask) + self.assertEqual((action, rev), ("save", "ddd")) + action, rev = decide_pin("x", "ccc", "bbb", "ddd", ancestry, _no_ask) + self.assertEqual((action, rev), ("keep", "")) + + def test_custom_pin_already_at_master_keeps(self): + action, rev = decide_pin("x", "ccc", "bbb", "bbb", _ancestry({}), _yes_ask) + self.assertEqual((action, rev), ("keep", "")) + + def test_case_4_cache_reached_the_custom_pin_clears(self): + self.assertEqual(decide_pin("x", "bbb", "bbb", "bbb", _ancestry({}), _yes_ask)[0], "clear") + self.assertEqual( + decide_pin("x", "ddd", "bbb", "ddd", _ancestry({("bbb", "ddd"): True}), _yes_ask)[0], + "clear", + ) + + def test_untracked_cache_clears_the_remembered_pin(self): + action, rev = decide_pin("x", "", "bbb", "", _ancestry({}), _yes_ask) + self.assertEqual((action, rev), ("clear", "")) + + def test_no_saved_pin_and_untracked_cache_skips(self): + action, rev = decide_pin("x", "", "", "aaa", _ancestry({}), _yes_ask) + self.assertEqual((action, rev), ("skip", "")) + + def test_unknown_master_keeps_the_current_pin(self): + self.assertEqual(decide_pin("x", "aaa", "", "", _ancestry({}), _yes_ask)[0], "noop") + self.assertEqual(decide_pin("x", "ccc", "bbb", "", _ancestry({}), _yes_ask)[0], "keep") + + def test_master_moved_backward_keeps_the_custom_pin(self): + action, rev = decide_pin( + "x", "ccc", "bbb", "aaa", _ancestry({("bbb", "aaa"): False}), _yes_ask + ) + self.assertEqual((action, rev), ("keep", "")) + + +class EntryHelpersTest(unittest.TestCase): + def test_normalise_entry_string_and_attrs(self): + self.assertEqual(_normalise_entry("abc"), {"rev": "abc", "repo": "", "checked": ""}) + self.assertEqual( + _normalise_entry({"rev": "abc", "repo": "github:o/r"}), + {"rev": "abc", "repo": "github:o/r", "checked": ""}, + ) + self.assertEqual(_normalise_entry(None), {"rev": "", "repo": "", "checked": ""}) + + def test_lock_leaf_exact_and_suffixed_keys(self): + lock: JSON = { + "nodes": { + "root": {"inputs": {}}, + "github:o/r": {"original": {"type": "github", "owner": "o", "repo": "r"}}, + "icedos-sub-r": {"original": {"type": "github", "owner": "o2", "repo": "r2"}}, + } + } + first = cast("dict[str, JSON]", _lock_leaf(lock, "github:o/r")["original"]) + second = cast("dict[str, JSON]", _lock_leaf(lock, "r")["original"]) + self.assertEqual(first["owner"], "o") + self.assertEqual(second["repo"], "r2") + + def test_lock_leaf_follows_subflake_root_to_the_leaf(self): + # The cache-server publishes keys shaped like the sub-flake root + # (`icedos--`); the leaf hangs off its inputs map. + lock: JSON = { + "nodes": { + "icedos-github_icedos_kde-plasmazones": { + "inputs": {"plasmazones": "github:polito/plasmazones"}, + "original": {"type": "path", "path": "/nix/store/x-subflake"}, + }, + "github:polito/plasmazones": { + "original": {"type": "github", "owner": "polito", "repo": "PlasmaZones"} + }, + } + } + leaf = _lock_leaf(lock, "icedos-github_icedos_kde-plasmazones") + original = cast("dict[str, JSON]", leaf["original"]) + self.assertEqual(original["owner"], "polito") + + def test_lock_leaf_refuses_ambiguous_suffixes(self): + lock: JSON = { + "nodes": { + "icedos-a-r": {"original": {"type": "github", "owner": "o", "repo": "r"}}, + "icedos-b-r": {"original": {"type": "github", "owner": "o2", "repo": "r2"}}, + } + } + self.assertEqual(_lock_leaf(lock, "r"), {}) + + def test_git_url_from_entry_and_lock_fallback(self): + self.assertEqual( + _git_url({"rev": "a", "repo": "github:o/r"}, {}, "x"), + "https://github.com/o/r", + ) + lock: JSON = {"nodes": {"x": {"original": {"type": "github", "owner": "o", "repo": "r"}}}} + self.assertEqual(_git_url({"rev": "a", "repo": ""}, lock, "x"), "https://github.com/o/r") + + def test_ident_to_url_scheme_allowlist_and_host_override(self): + self.assertEqual(_ident_to_url("ext::sh -c evil"), "") + self.assertEqual(_ident_to_url("file:///etc/passwd"), "") + self.assertEqual( + _git_url({"rev": "a", "repo": "ext::sh -c evil://x"}, {}, "x"), "" + ) + lock: JSON = { + "nodes": { + "x": { + "original": { + "type": "gitlab", + "owner": "o", + "repo": "r", + "host": "ghe.example", + } + } + } + } + self.assertEqual(_git_url({"rev": "a", "repo": ""}, lock, "x"), "https://ghe.example/o/r") + + def test_git_url_refuses_tracked_repo_that_disagrees_with_the_lock(self): + lock: JSON = {"nodes": {"x": {"original": {"type": "github", "owner": "o", "repo": "r"}}}} + self.assertEqual(_git_url({"rev": "a", "repo": "github:o2/r2"}, lock, "x"), "") + # The lock leaf is authoritative even when the entry carries a repo. + self.assertEqual(_git_url({"rev": "a", "repo": "github:o/r"}, lock, "x"), "https://github.com/o/r") + + def test_git_url_sourcehut_keeps_owner_tilde_and_gitlab_subgroups(self): + self.assertEqual( + _git_url({"rev": "a", "repo": "sourcehut:~ice/r"}, {}, "x"), + "https://git.sr.ht/~ice/r", + ) + self.assertEqual( + _git_url({"rev": "a", "repo": "gitlab:g%2Fsub/r"}, {}, "x"), + "https://gitlab.com/g/sub/r", + ) + + +class PrunePassTest(unittest.TestCase): + def test_drops_pins_the_cache_no_longer_tracks(self): + saved: dict[str, JSON] = { + "x": {"rev": "bbb", "repo": ""}, + "y": {"rev": "ccc", "repo": ""}, + } + changed = pins_mod._prune_pass(saved, {"y": "ddd"}, {}, _FakeGit({}, {}), set()) + self.assertTrue(changed) + self.assertEqual(saved, {"y": {"rev": "ccc", "repo": ""}}) + + +class GitGuardTest(unittest.TestCase): + def test_is_ancestor_refuses_non_sha_revs_without_spawning_git(self): + git = _Git() + with mock.patch.object( + subprocess, "run", side_effect=AssertionError("git called") + ): + self.assertIsNone(git.is_ancestor("https://github.com/o/r", "--evil", "bbb")) + self.assertIsNone(git.is_ancestor("https://github.com/o/r", "aaa", "not a sha")) + + +class UnpinnedFileTest(unittest.TestCase): + def test_read_unpinned_roundtrip_and_tolerates_garbage(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / UNPINNED_FILE + self.assertEqual(_read_unpinned(path), {}) + _ = path.write_text(json.dumps({"x": {"rev": "abc", "repo": "github:o/r"}})) + self.assertEqual(_read_unpinned(path), {"x": {"rev": "abc", "repo": "github:o/r"}}) + _ = path.write_text("not json") + self.assertEqual(_read_unpinned(path), {}) + self.assertEqual(path.read_text(), "{}") # garbage self-heals + _ = path.write_text("null") # valid JSON, but not an object + self.assertEqual(_read_unpinned(path), {}) + self.assertEqual(path.read_text(), "{}") + + +class _FakeGit: + def __init__( + self, + heads: dict[str, str], + ancestry: dict[tuple[str, str], bool | None], + ref_heads: dict[tuple[str, str], str] | None = None, + ) -> None: + self._heads: dict[str, str] = heads + self._ref_heads: dict[tuple[str, str], str] = ref_heads or {} + self._ancestry: dict[tuple[str, str], bool | None] = ancestry + self.head_calls: list[str] = [] + self.ref_calls: list[tuple[str, str]] = [] + self.probes: int = 0 + self._counted: set[tuple[str, str, str]] = set() + + def head(self, url: str) -> str: + self.head_calls.append(url) + return self._heads.get(url, "") + + def ref_head(self, url: str, ref: str) -> str: + self.ref_calls.append((url, ref)) + return self._ref_heads.get((url, ref), "") + + def is_ancestor(self, url: str, a: str, b: str) -> bool | None: + # Count unique triples only, mirroring the real _Git's per-key memoization. + if (url, a, b) not in self._counted: + self._counted.add((url, a, b)) + self.probes += 1 + return self._ancestry.get((a, b)) + + +@contextmanager +def _patched( + pins: ModuleType, + pin_revs: dict[str, JSON], + git: object, + refs: dict[str, str] | None = None, +) -> Generator[None]: + def fake_revs(_env: BuildEnv, _trace: list[str]) -> tuple[dict[str, JSON], dict[str, str]]: + return pin_revs, refs or {} + + with ( + mock.patch.object(pins, "_pin_revs", fake_revs), + mock.patch.object(pins, "_Git", lambda: git), + # io.StringIO().isatty() is False, forcing the non-interactive path. + mock.patch("sys.stdin", io.StringIO()), + ): + yield + + +class OrchestrationTest(unittest.TestCase): + # A state lock whose leaf node resolves "x" to github:o/x, so _git_url's + # lock fallback produces a url for the fake git. + LOCK: str = json.dumps( + {"nodes": {"x": {"original": {"type": "github", "owner": "o", "repo": "x"}}}} + ) + + def _env(self, tmp: str) -> BuildEnv: + return BuildEnv( + root=Path("/x"), + config_root=None, + state_dir=Path(tmp), + inputs_prefix="icedos", + ) + + def test_expired_pin_is_cleared_then_freshly_evaluated(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "bbb", "repo": ""}})) + # Cache moved past the remembered pin; master equals the cache rev. + git = _FakeGit({"https://github.com/o/x": "ddd"}, {("bbb", "ddd"): True}) + opts = Options() + opts.unpin_inputs = ["x"] + with _patched(pins_mod, {"x": "ddd"}, git): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + self.assertEqual(json.loads(path.read_text()), {}) + self.assertIn("x custom pin dropped; back on the cache-server's rev", out.getvalue()) + self.assertIn("x is already at master; nothing to do", out.getvalue()) + + def test_stale_pin_with_master_behind_is_kept(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "bbb", "repo": ""}})) + # Cache behind the pin, master behind it too. + git = _FakeGit({"https://github.com/o/x": "aaa"}, {("bbb", "aaa"): False}) + opts = Options() + opts.unpin_inputs = ["x"] + with _patched(pins_mod, {"x": "ccc"}, git): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + self.assertEqual( + json.loads(path.read_text()), {"x": {"rev": "bbb", "repo": ""}} + ) + self.assertIn("keeps its custom pin", out.getvalue()) + + def test_unresolvable_master_warns_and_skips(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + # A forge leaf must pass the scheme gate so the failure under test + # (no upstream rev) is actually reached. + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "bbb", "repo": "github:o/x"}})) + git = _FakeGit({}, {}) # no lock, no heads -> no master rev + opts = Options() + opts.unpin_inputs = ["x"] + with _patched(pins_mod, {"x": "ccc"}, git): + err = io.StringIO() + with redirect_stdout(io.StringIO()), mock.patch("sys.stderr", err): + unpin_inputs(self._env(tmp), opts, []) + self.assertEqual( + json.loads(path.read_text()), {"x": {"rev": "bbb", "repo": "github:o/x"}} + ) + self.assertIn("no resolvable git url or upstream rev", err.getvalue()) + + def test_expire_unpinned_prunes_without_prompting(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "bbb", "repo": ""}})) + git = _FakeGit({}, {("bbb", "ddd"): True}) + with _patched(pins_mod, {"x": "ddd"}, git): + expire_unpinned(self._env(tmp), []) + self.assertEqual(json.loads(path.read_text()), {}) + + def test_unpin_all_covers_every_lock_resolvable_tracked_input(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + git = _FakeGit( + {"https://github.com/o/x": "ddd", "https://github.com/o/y": "eee"}, + {("aaa", "ddd"): True}, + ) + opts = Options() + opts.unpin_all = True + # x is behind master and accepts the re-pin; y carries its own + # repo but no lock node, so it is still skipped; z is bare string. + pin_revs: dict[str, JSON] = { + "x": "aaa", + "y": {"rev": "ccc", "repo": "github:o/y"}, + "z": "bbb", + } + + def _answer(_msg: str) -> str: + return "y" + + # MagicMock().isatty() is truthy, forcing the interactive path. + with ( + _patched(pins_mod, pin_revs, git), + mock.patch("sys.stdin", mock.MagicMock()), + mock.patch("builtins.input", _answer), + ): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + self.assertEqual( + json.loads(path.read_text()), + {"x": {"rev": "ddd", "repo": "github:o/x"}}, + ) + self.assertIn("x re-pinned to master", out.getvalue()) + + def test_expire_unpinned_memoizes_the_ancestry_probe(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "bbb", "repo": ""}})) + # Cache behind the pin, ancestry definitively False: stamp the + # probed cache rev so the second run skips the git probe entirely. + # One fresh _FakeGit per run (real _Git-per-run behavior), so the + # second run's zero probes can only come from the checked stamp. + pin_revs: dict[str, JSON] = {"x": "ccc"} + + def fake_pin_revs( + _env: BuildEnv, _trace: list[str] + ) -> tuple[dict[str, JSON], dict[str, str]]: + return pin_revs, {} + + fakes = [ + _FakeGit({"https://github.com/o/x": ""}, {("bbb", "ccc"): False}), + _FakeGit({"https://github.com/o/x": ""}, {("bbb", "ccc"): False}), + ] + git_iter = iter(fakes) + + def fake_git() -> _FakeGit: + return next(git_iter) + + with ( + mock.patch.object(pins_mod, "_pin_revs", fake_pin_revs), + mock.patch.object(pins_mod, "_Git", fake_git), + mock.patch("sys.stdin", io.StringIO()), + ): + env = self._env(tmp) + expire_unpinned(env, []) + expire_unpinned(env, []) + self.assertEqual([f.probes for f in fakes], [1, 0]) + self.assertEqual( + json.loads(path.read_text()), + {"x": {"rev": "bbb", "repo": "", "checked": "ccc"}}, + ) + + def test_expire_unpinned_reprobes_when_ancestry_is_inconclusive(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "bbb", "repo": ""}})) + # A fresh _Git per run (real behavior); ancestry unknown, so every + # run must re-probe instead of trusting a stamp. + fakes = [_FakeGit({"https://github.com/o/x": ""}, {}) for _ in range(2)] + pin_revs: dict[str, JSON] = {"x": "ccc"} + + def fake_pin_revs( + _env: BuildEnv, _trace: list[str] + ) -> tuple[dict[str, JSON], dict[str, str]]: + return pin_revs, {} + + git_iter = iter(fakes) + + def fake_git() -> _FakeGit: + return next(git_iter) + + with ( + mock.patch.object(pins_mod, "_pin_revs", fake_pin_revs), + mock.patch.object(pins_mod, "_Git", fake_git), + mock.patch("sys.stdin", io.StringIO()), + ): + env = self._env(tmp) + expire_unpinned(env, []) + expire_unpinned(env, []) + self.assertEqual(sum(f.probes for f in fakes), 2) + self.assertEqual(json.loads(path.read_text()), {"x": {"rev": "bbb", "repo": ""}}) + + def test_prompt_survives_eof(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + git = _FakeGit({"https://github.com/o/x": "bbb"}, {("aaa", "bbb"): True}) + opts = Options() + opts.unpin_inputs = ["x"] + + def _eof(_msg: str) -> str: + raise EOFError + + with _patched(pins_mod, {"x": "aaa"}, git), mock.patch( + "sys.stdin", mock.MagicMock() + ), mock.patch("builtins.input", _eof): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + # A declined prompt writes nothing: the state file is not created. + self.assertFalse(path.exists()) + self.assertIn("keeps the cache-server's pin", out.getvalue()) + + def test_expired_pin_clears_before_the_master_lookup(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + _ = path.write_text(json.dumps({"x": {"rev": "ddd", "repo": ""}})) + # Case 4 (cache_rev == saved_rev) is decidable without the network: + # no ancestry probe, and the clear happens even though ls-remote + # would fail (no heads). + git = _FakeGit({}, {}) + opts = Options() + opts.unpin_inputs = ["x"] + with _patched(pins_mod, {"x": "ddd"}, git): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + self.assertEqual(json.loads(path.read_text()), {}) + self.assertEqual(git.probes, 0) + self.assertIn("x custom pin dropped", out.getvalue()) + + def test_unpin_skips_non_forge_inputs_before_prompting(self): + # A git-type leaf (no forge repo): skipped before any upstream lookup, + # so the user is never asked for a pin that cannot be applied. + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(json.dumps({ + "nodes": { + "x": {"original": {"type": "git", "url": "https://example.com/x.git"}} + } + })) + path = state / UNPINNED_FILE + git = _FakeGit({"https://example.com/x.git": "ddd"}, {}) + opts = Options() + opts.unpin_inputs = ["x"] + + def _no_ask(_msg: str) -> str: + raise AssertionError("must not prompt for non-forge inputs") + + with _patched(pins_mod, {"x": "aaa"}, git, refs={}), mock.patch( + "sys.stdin", mock.MagicMock() + ), mock.patch("builtins.input", _no_ask): + with redirect_stderr(io.StringIO()) as err: + unpin_inputs(self._env(tmp), opts, []) + self.assertFalse(path.exists()) + self.assertIn("not a github/gitlab/sourcehut input", err.getvalue()) + self.assertEqual(git.head_calls, []) + + def test_root_input_names_exclude_subflake_roots(self): + # A sub-flake ROOT is a root input but a legit tracked key (its leaf + # hangs off its inputs map); only repo/channel-level names count here. + lock: dict[str, JSON] = { + "nodes": { + "root": { + "inputs": { + "icedos-github_icedos_apps": "repo", + "icedos-github_icedos_apps-celluloid": "sub", + } + }, + "repo": {"original": {"type": "github", "owner": "IceDOS", "repo": "apps"}}, + "sub": { + "original": { + "type": "path", + "path": "/nix/store/x-icedos-github_icedos_apps-celluloid-subflake", + } + }, + } + } + self.assertEqual( + pins_mod._root_input_names(lock), + {"icedos-github_icedos_apps"}, + ) + + def test_ref_head_prefers_branch_then_peeled_tag_commit(self): + branch_sha, tag_obj, tag_commit = "a" * 40, "b" * 40, "c" * 40 + git = _Git() + out = ( + f"{branch_sha}\trefs/heads/release\n" + + f"{tag_obj}\trefs/tags/v1\n" + + f"{tag_commit}\trefs/tags/v1^{{}}\n" + ) + with mock.patch.object(subprocess, "run", return_value=mock.MagicMock(returncode=0, stdout=out)): + self.assertEqual(git.ref_head("https://github.com/o/x", "release"), branch_sha) + # An annotated tag resolves to the peeled commit, not the tag object. + self.assertEqual(git.ref_head("https://github.com/o/x", "v1"), tag_commit) + + def test_unpin_targets_the_declared_ref_not_head(self): + # A module declared `github:o/x/release`; the re-pin must offer that + # branch's rev, never the remote's HEAD. + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + git = _FakeGit( + {"https://github.com/o/x": "ddd"}, + {("aaa", "bbb"): True, ("bbb", "ccc"): False}, + {("https://github.com/o/x", "release"): "ccc"}, + ) + opts = Options() + opts.unpin_inputs = ["x"] + + def _answer(_msg: str) -> str: + return "y" + + with _patched(pins_mod, {"x": "aaa"}, git, refs={"x": "release"}), mock.patch( + "sys.stdin", mock.MagicMock() + ), mock.patch("builtins.input", _answer): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + self.assertEqual(git.ref_calls, [("https://github.com/o/x", "release")]) + self.assertEqual(json.loads(path.read_text()), {"x": {"rev": "ccc", "repo": "github:o/x"}}) + self.assertIn("re-pinned to 'release'", out.getvalue()) + + def test_unpin_skips_when_the_declared_ref_has_no_remote_rev(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(self.LOCK) + path = state / UNPINNED_FILE + git = _FakeGit({"https://github.com/o/x": "ddd"}, {}, {("https://github.com/o/x", "gone"): ""}) + opts = Options() + opts.unpin_inputs = ["x"] + with _patched(pins_mod, {"x": "aaa"}, git, refs={"x": "gone"}), mock.patch( + "sys.stdin", mock.MagicMock() + ): + with redirect_stderr(io.StringIO()) as err: + unpin_inputs(self._env(tmp), opts, []) + self.assertFalse(path.exists()) + self.assertIn("upstream rev for refs 'gone'", err.getvalue()) + + def test_unpin_all_skips_repo_level_inputs(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(json.dumps({ + "nodes": { + "root": {"inputs": {"icedos-github_icedos_kde": "icedos-github_icedos_kde"}}, + "icedos-github_icedos_kde": { + "original": {"type": "github", "owner": "IceDOS", "repo": "kde"} + }, + "x": {"original": {"type": "github", "owner": "o", "repo": "x"}}, + } + })) + path = state / UNPINNED_FILE + git = _FakeGit({"https://github.com/o/x": "bbb"}, {("aaa", "bbb"): True}) + opts = Options() + opts.unpin_all = True + + def _answer(_msg: str) -> str: + return "y" + + with _patched(pins_mod, {"icedos-github_icedos_kde": "aaa", "x": "aaa"}, git), mock.patch( + "sys.stdin", mock.MagicMock() + ), mock.patch("builtins.input", _answer): + with redirect_stdout(io.StringIO()) as out: + unpin_inputs(self._env(tmp), opts, []) + # The repo-level tracked input is skipped; the leaf re-pins. + self.assertEqual( + json.loads(path.read_text()), + {"x": {"rev": "bbb", "repo": "github:o/x"}}, + ) + self.assertIn("re-pinned to master", out.getvalue()) + + def test_unpin_inputs_rejects_repo_level_inputs(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + _ = (state / "flake.lock").write_text(json.dumps({ + "nodes": { + "root": {"inputs": {"icedos-github_icedos_kde": "icedos-github_icedos_kde"}}, + "icedos-github_icedos_kde": { + "original": {"type": "github", "owner": "IceDOS", "repo": "kde"} + }, + } + })) + opts = Options() + opts.unpin_inputs = ["icedos-github_icedos_kde"] + with _patched(pins_mod, {"icedos-github_icedos_kde": "aaa"}, _FakeGit({}, {})): + err = io.StringIO() + with redirect_stderr(err): + with self.assertRaises(SystemExit): + unpin_inputs(self._env(tmp), opts, []) + self.assertIn("repo-level input", err.getvalue()) + + def test_save_skips_non_forge_inputs(self): + with tempfile.TemporaryDirectory() as tmp: + state = Path(tmp) + git_leaf = json.dumps( + {"nodes": {"x": {"original": {"type": "git", "url": "https://github.com/o/x"}}}} + ) + _ = (state / "flake.lock").write_text(git_leaf) + path = state / UNPINNED_FILE + git = _FakeGit({"https://github.com/o/x": "bbb"}, {("aaa", "bbb"): True}) + opts = Options() + opts.unpin_inputs = ["x"] + + def _answer(_msg: str) -> str: + return "y" + + with _patched(pins_mod, {"x": "aaa"}, git), mock.patch( + "sys.stdin", mock.MagicMock() + ), mock.patch("builtins.input", _answer): + with redirect_stdout(io.StringIO()) as out, redirect_stderr(io.StringIO()) as err: + unpin_inputs(self._env(tmp), opts, []) + self.assertFalse(path.exists()) + self.assertIn("not a github/gitlab/sourcehut input", err.getvalue()) + + def test_expire_unpinned_skips_the_eval_without_the_file(self): + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object( + pins_mod, "_pin_revs" + ) as pin_revs, mock.patch("sys.stdin", io.StringIO()): + expire_unpinned(self._env(tmp), []) + pin_revs.assert_not_called() + + +if __name__ == "__main__": + _ = unittest.main() diff --git a/build/tests/test_token.py b/build/tests/test_token.py index 7553fbb3..71d3f05a 100644 --- a/build/tests/test_token.py +++ b/build/tests/test_token.py @@ -12,10 +12,6 @@ from build.options import Options -def _opts(**kwargs) -> Options: - return Options(**kwargs) - - @contextlib.contextmanager def _env(**values: str | None): # None removes the variable; everything else is set for the block. @@ -31,15 +27,15 @@ class TokenFilePathTest(unittest.TestCase): def test_flag_beats_env_beats_default(self): with _env(ICEDOS_GITHUB_TOKEN_PATH="/from/env"): self.assertEqual( - build_main._token_file_path(_opts(github_token_path="/from/flag")), + build_main._token_file_path(Options(github_token_path="/from/flag")), Path("/from/flag"), ) self.assertEqual( - build_main._token_file_path(_opts()), Path("/from/env") + build_main._token_file_path(Options()), Path("/from/env") ) with _env(ICEDOS_GITHUB_TOKEN_PATH=None): self.assertEqual( - build_main._token_file_path(_opts()), + build_main._token_file_path(Options()), Path(build_main.DEFAULT_TOKEN_PATH), ) @@ -53,7 +49,7 @@ def test_literal_flag_wins_over_env_and_file(self): ICEDOS_GITHUB_TOKEN="from-env", ICEDOS_GITHUB_TOKEN_PATH=str(path) ): self.assertEqual( - build_main._resolve_token(_opts(github_token="from-flag")), + build_main._resolve_token(Options(github_token="from-flag")), "from-flag", ) @@ -64,14 +60,14 @@ def test_env_literal_wins_over_file(self): with _env( ICEDOS_GITHUB_TOKEN="from-env", ICEDOS_GITHUB_TOKEN_PATH=str(path) ): - self.assertEqual(build_main._resolve_token(_opts()), "from-env") + self.assertEqual(build_main._resolve_token(Options()), "from-env") def test_falls_through_to_the_file(self): with tempfile.TemporaryDirectory() as d: path = Path(d) / "token" _ = path.write_text("from-file\n") with _env(ICEDOS_GITHUB_TOKEN=None, ICEDOS_GITHUB_TOKEN_PATH=str(path)): - self.assertEqual(build_main._resolve_token(_opts()), "from-file") + self.assertEqual(build_main._resolve_token(Options()), "from-file") def test_blank_literals_do_not_shadow_the_file(self): with tempfile.TemporaryDirectory() as d: @@ -79,7 +75,7 @@ def test_blank_literals_do_not_shadow_the_file(self): _ = path.write_text("from-file\n") with _env(ICEDOS_GITHUB_TOKEN=" ", ICEDOS_GITHUB_TOKEN_PATH=str(path)): self.assertEqual( - build_main._resolve_token(_opts(github_token=" ")), "from-file" + build_main._resolve_token(Options(github_token=" ")), "from-file" ) @@ -142,11 +138,11 @@ def test_non_utf8_file_does_not_crash(self): class NixConfigTest(unittest.TestCase): def test_base_config_only_without_a_token(self): with _env(ICEDOS_GITHUB_TOKEN=None, ICEDOS_GITHUB_TOKEN_PATH="/nonexistent"): - self.assertEqual(build_main._nix_config(_opts()), build_main.BASE_NIX_CONFIG) + self.assertEqual(build_main._nix_config(Options()), build_main.BASE_NIX_CONFIG) # NIX_CONFIG is nix.conf content, so the token has to land on its own line. def test_token_is_appended_on_its_own_line(self): - config = build_main._nix_config(_opts(github_token="tok")) + config = build_main._nix_config(Options(github_token="tok")) self.assertEqual( config.splitlines(), [build_main.BASE_NIX_CONFIG, "access-tokens = github.com=tok"], @@ -154,4 +150,4 @@ def test_token_is_appended_on_its_own_line(self): if __name__ == "__main__": - unittest.main() + _ = unittest.main() diff --git a/lib/genflake.nix b/lib/genflake.nix index f39e48eb..b697da6f 100644 --- a/lib/genflake.nix +++ b/lib/genflake.nix @@ -420,6 +420,14 @@ let # The complete merged config set for the webui, not just config.toml. userConfigRaw = toJSON userConfig; + # Cache-server's tracked revs (empty unless `pinInputs` is set), for + # `--unpin-inputs` to diff against upstream master revs. + pinRevs = icedosLib._cacheTrackedRevs; + + # Tracked key -> author-declared url ref, so a re-pin targets the declared + # branch; the state lock's `original` has lost it once a rev is baked in. + pinRefs = modulesFromConfig.pinRefs; + # Sub-flake texts leave this stage only as the root inputs' store paths; nothing # else is exported (the build orchestrator reads the resulting flake.lock). @@ -441,6 +449,8 @@ assert githubTokenStoreWarning; flakeInputsNix optionsDoc modulesDoc + pinRevs + pinRefs userConfigRaw ; diff --git a/lib/icedos.nix b/lib/icedos.nix index 3513708d..64acddbe 100644 --- a/lib/icedos.nix +++ b/lib/icedos.nix @@ -54,6 +54,11 @@ let default = true; }; + # The whole pin machinery (cache revs + remembered custom pins) requires + # the cache itself to be enabled, not just the pinInputs opt-in. + _cachePinsActive = + (config.system.cache.enable or false) && (config.system.cache.pinInputs or false); + # cache-server's published tracked-inputs.json (name -> rev | { rev; repo; }). # Resolved through the state lock chain root -> icedos-config -> icedos -> # cache-server, so it matches the tree the build evaluates. Genflake-only: the @@ -88,7 +93,7 @@ let // lib.optionalAttrs (locked ? host) { inherit (locked) host; } ); in - if (icedosLib.ICEDOS_STAGE != "genflake") || !(config.system.cache.pinInputs or false) then + if (icedosLib.ICEDOS_STAGE != "genflake") || !_cachePinsActive then { } # First build: no state lock to resolve the channel through. Documented to # self-heal on the next run, so it stays quiet. @@ -111,6 +116,19 @@ let else { }; + # Remembered custom pins from --unpin-inputs (.state/unpinned-inputs.json, + # name -> { rev; repo; }); genflake-only, baked in place of the cache pin. + _unpinnedRevs = + let + file = "${icedosLib.ICEDOS_STATE_DIR}/unpinned-inputs.json"; + in + if (icedosLib.ICEDOS_STAGE != "genflake") || !_cachePinsActive then + { } + else if !(pathExists file) then + { } + else + builtins.fromJSON (builtins.readFile file); + # Patch a flake source into a realised, context-free store path usable as a # locked `path:` input (readDir realises it; --raw genflake forbids context). _mkPatchedSource = @@ -420,20 +438,27 @@ let inputName = "${i}_source"; }; - # Bare rev cache-server last built for this leaf input, "" when - # untracked or not a forge url (git-scheme revs are out of scope). + # Pin for this leaf input, "" when untracked/non-forge; a + # remembered custom pin wins while the cache is tracking it and + # differs from its rev. + _trackedUrl = inputs.${i}.url or ""; + _cachePin = let - url = inputs.${i}.url or ""; + url = _trackedUrl; rev = icedosLib._cacheRevLookup { inherit url; name = i; revs = _cacheTrackedRevs; }; + savedRev = icedosLib._cacheRevLookup { + inherit url; + name = i; + revs = _unpinnedRevs; + }; in if - rev == "" - || !(builtins.elem (head (lib.splitString ":" url)) [ + !(builtins.elem (head (lib.splitString ":" url)) [ "github" "gitlab" "sourcehut" @@ -441,7 +466,7 @@ let then "" else - rev; + icedosLib._cachePinRev { inherit rev savedRev; }; # Lock rev wins, then the cache pin, then the author's ref; only the # lock hands back a pre-formed suffix, the other two are bare revs. @@ -518,12 +543,31 @@ let else [ i ] ); + # The input's tracked key (same matching _cacheRevLookup uses) + # mapped to the author's declared url ref, so --unpin-inputs can + # re-pin the declared branch instead of the remote's HEAD. Both + # spellings count: `scheme:o/r/` and `scheme:o/r?ref=`. + pinRefEntries = + let + key = icedosLib._cacheTrackedKey { + name = i; + url = _trackedUrl; + revs = _cacheTrackedRevs; + }; + # Parse `_trackedUrl`, not `_patchSrcParsed`: a url-less + # follows-only input is legal here (key "" makes it moot), + # and _parseFlakeUrl folds both ref spellings into `ref`. + parsed = icedosLib._parseFlakeUrl _trackedUrl; + ref = if parsed.ref != null then parsed.ref else ""; + in + if key == "" then { } else { "${key}" = ref; }; in { inherit decls hasPatches maskedEntries + pinRefEntries ; }; in @@ -586,6 +630,9 @@ let }; masked = flatten (map (i: (perInput i).maskedEntries) inputNames); + + # Tracked key -> declared url ref, for the --unpin-inputs flow. + pinRefs = foldl' (acc: e: acc // e) { } (map (i: (perInput i).pinRefEntries) inputNames); } ) modulesWithInputs; @@ -1068,6 +1115,9 @@ let value = r.text; }) moduleSubFlakes ); + + # Tracked key -> author-declared url ref, for the --unpin-inputs flow. + pinRefs = foldl' (acc: r: acc // r.pinRefs) { } moduleSubFlakes; in { inherit @@ -1075,6 +1125,7 @@ let nixosModules nixosModulesText options + pinRefs subFlakes ; }; @@ -1606,6 +1657,7 @@ let # Names are unique per declaring module, so a plain merge is exact. subFlakes = externalOutputs.subFlakes // extraOutputs.subFlakes; + pinRefs = externalOutputs.pinRefs // extraOutputs.pinRefs; options = externalOutputs.options ++ extraOutputs.options; nixosModulesText = externalOutputs.nixosModulesText ++ extraOutputs.nixosModulesText; }; diff --git a/lib/inputs.nix b/lib/inputs.nix index 77268790..79c24d4e 100644 --- a/lib/inputs.nix +++ b/lib/inputs.nix @@ -237,7 +237,24 @@ rec { # "-" suffixed one. The optional `repo` guard ("scheme:owner/repo") only # exists in the newer { rev; repo; } format and must equal the url's host repo; # older string entries pin on the name alone. - _cacheRevLookup = + # `{ rev; repo; }` view of a tracked-inputs.json entry (string or attrset). + _cacheEntryOf = + value: + if builtins.isString value then + { + rev = value; + repo = ""; + } + else + { + rev = value.rev or ""; + repo = value.repo or ""; + }; + + # The tracked key this input matches: exact, else a lone "-" suffix + # guarded by the url's repo; "" when none or ambiguous. _cacheRevLookup and + # the declared-ref export (_cachePin's tracked-ref map) share this matching. + _cacheTrackedKey = { name, url, @@ -253,19 +270,6 @@ rec { match ; - _entryOf = - value: - if builtins.isString value then - { - rev = value; - repo = ""; - } - else - { - rev = value.rev or ""; - repo = value.repo or ""; - }; - keys = filter (k: k == name || lib.hasSuffix "-${name}" k) (attrNames revs); # "scheme:owner/repo" of `url`, "" when it is not a github/gitlab/sourcehut url. @@ -277,11 +281,11 @@ rec { # `{ rev; repo; }` entries name their own repo, so one naming this url is # unambiguous even when several keys share the "-" suffix. - repoMatched = filter (k: urlRepo != "" && (_entryOf revs.${k}).repo == urlRepo) keys; + repoMatched = filter (k: urlRepo != "" && (_cacheEntryOf revs.${k}).repo == urlRepo) keys; # Several entries may claim the same repo (a publish-side mistake). That is # only unambiguous while they agree on the rev. - repoMatchedRevs = lib.unique (map (k: (_entryOf revs.${k}).rev) repoMatched); + repoMatchedRevs = lib.unique (map (k: (_cacheEntryOf revs.${k}).rev) repoMatched); # Exact key wins, then a repo-guarded match, then a lone string suffix; # the rest is ambiguous (same leaf in different repos), so refuse. @@ -296,15 +300,42 @@ rec { head keys else null; + in + if key == null then "" else key; + + # A remembered custom pin wins while the cache still tracks the input and + # its rev differs; otherwise the cache rev (or no pin) applies. + _cachePinRev = + { rev, savedRev }: + if rev != "" && savedRev != "" && savedRev != rev then savedRev else rev; + + _cacheRevLookup = + { + name, + url, + revs, + }: + let + inherit (builtins) elemAt match; + + urlRepo = + let + m = match "(github|gitlab|sourcehut):([^/?]+)/([^/?]+)(.*)" url; + in + if m == null then "" else "${elemAt m 0}:${elemAt m 1}/${elemAt m 2}"; + + key = _cacheTrackedKey { + inherit name url revs; + }; entry = - if key == null then + if key == "" then { rev = ""; repo = ""; } else - _entryOf revs.${key}; + _cacheEntryOf revs.${key}; in if entry.rev == "" || (entry.repo != "" && entry.repo != urlRepo) then "" else entry.rev; @@ -346,16 +377,37 @@ rec { url: let match = builtins.match "(github|gitlab|sourcehut):([^/?]+)/([^/?]+)/([^?]+)(.*)" url; + # `?ref=` spelling, possibly among other query params. Folding it into + # `ref` (and out of the query) means a later baked rev REPLACES the branch + # instead of stacking with it — nix rejects both at once. + qref = builtins.match "(github|gitlab|sourcehut):([^/?]+)/([^/?]+)[?]([^#]*)" url; in - if match == null then + if match != null then { - baseUrl = url; - ref = null; + baseUrl = "${builtins.elemAt match 0}:${builtins.elemAt match 1}/${builtins.elemAt match 2}${builtins.elemAt match 4}"; + ref = builtins.elemAt match 3; + } + else if qref != null then + let + params = builtins.filter builtins.isString (builtins.split "&" (builtins.elemAt qref 3)); + refParams = builtins.filter (p: builtins.match "ref=.+" p != null) params; + rest = builtins.filter (p: builtins.match "ref=.+" p == null) params; + kept = concatStringsSep "&" rest; + in + { + baseUrl = "${builtins.elemAt qref 0}:${builtins.elemAt qref 1}/${builtins.elemAt qref 2}${ + lib.optionalString (kept != "") "?${kept}" + }"; + ref = + if refParams == [ ] then + null + else + builtins.head (builtins.match "ref=(.*)" (builtins.head refParams)); } else { - baseUrl = "${builtins.elemAt match 0}:${builtins.elemAt match 1}/${builtins.elemAt match 2}${builtins.elemAt match 4}"; - ref = builtins.elemAt match 3; + baseUrl = url; + ref = null; }; # Generate a unique key for a module (url/name combination). diff --git a/modules/options.nix b/modules/options.nix index ed130a02..df040e37 100644 --- a/modules/options.nix +++ b/modules/options.nix @@ -199,7 +199,8 @@ in # Pin tracked leaf inputs to the revs cache-server last built (its # `tracked-inputs.json`), so rebuilds consume cached closures instead of - # resolving fresh upstream revs. Needs one built cycle before it can pin. + # resolving fresh upstream revs. Needs one built cycle before it can + # pin; the pin machinery also requires `enable`. pinInputs = mkBoolOption { default = false; }; priority = mkNumberOption { default = 100; }; diff --git a/modules/rebuild.nix b/modules/rebuild.nix index 52b8ee7f..5039bd4d 100644 --- a/modules/rebuild.nix +++ b/modules/rebuild.nix @@ -24,6 +24,10 @@ let inherit (config) icedos; inherit (icedos) configurationLocation; + + # The unpin flags only work when the pin machinery is active; the help block + # is baked at build time, so it can reflect the config. + cachePinsActive = icedos.system.cache.enable && icedos.system.cache.pinInputs; inherit (icedos.system.toolset.rebuild) hooks; inherit (hooks) @@ -81,6 +85,10 @@ in echo " --update-repos-only update repos only before rebuild" echo " --update-repos-select update specific repos (space-separated urls) before rebuild" echo " --update-repo-inputs-only update repo inputs only before rebuild" + ${optionalString cachePinsActive '' + echo " --unpin-inputs offer to re-pin cache-pinned module inputs to master (space-separated names)" + echo " --unpin-inputs-all offer to re-pin every cache-tracked module input to master" + ''} echo " --build-vm build a VM test image" echo " --run-vm build and run a VM test image" echo " --genflake-only generate .state/flake.nix and exit (--dry's underlying mechanism)" diff --git a/tests/tests.nix b/tests/tests.nix index b626d8b1..625d3efe 100644 --- a/tests/tests.nix +++ b/tests/tests.nix @@ -3019,6 +3019,28 @@ in } ); + # The key lookup behind _cacheRevLookup, also feeding the declared-ref export. + cacheTrackedKeyExact = expectEq "plasmazones" ( + helpers._cacheTrackedKey { + name = "plasmazones"; + url = "github:fuddlesworth/PlasmaZones"; + revs = { + plasmazones = "abc"; + }; + } + ); + + cacheTrackedKeyAmbiguous = expectEq "" ( + helpers._cacheTrackedKey { + name = "plasmazones"; + url = ""; + revs = { + "a-plasmazones" = "x"; + "b-plasmazones" = "y"; + }; + } + ); + # { rev; repo; } entries guard the host repo; string entries pin on name alone. cacheRevRepoMatch = expectEq "abc" ( helpers._cacheRevLookup { @@ -3101,6 +3123,46 @@ in } ); + # The remembered-pin preference behind _cachePin: a custom pin wins only + # while it differs from the cache rev, and an untracked input pins nothing. + cachePinRevSavedWins = expectEq "saved" ( + helpers._cachePinRev { + rev = "cache"; + savedRev = "saved"; + } + ); + cachePinRevAgrees = expectEq "cache" ( + helpers._cachePinRev { + rev = "cache"; + savedRev = "cache"; + } + ); + cachePinRevNoSaved = expectEq "cache" ( + helpers._cachePinRev { + rev = "cache"; + savedRev = ""; + } + ); + cachePinRevUntracked = expectEq "" ( + helpers._cachePinRev { + rev = ""; + savedRev = "saved"; + } + ); + + # `?ref=` folds into `ref` and out of the query, so a baked rev replaces it. + parseFlakeUrlQueryRef = expectEq "main" ((helpers._parseFlakeUrl "github:o/r?ref=main").ref); + parseFlakeUrlQueryRefBaseUrl = expectEq "github:o/r" ( + (helpers._parseFlakeUrl "github:o/r?ref=main").baseUrl + ); + parseFlakeUrlQueryRefMixedRef = expectEq "main" ( + (helpers._parseFlakeUrl "github:o/r?dir=x&ref=main").ref + ); + parseFlakeUrlQueryRefMixedBaseUrl = expectEq "github:o/r?dir=x" ( + (helpers._parseFlakeUrl "github:o/r?dir=x&ref=main").baseUrl + ); + parseFlakeUrlRevQueryNoRef = expectEq null ((helpers._parseFlakeUrl "github:o/r?rev=abc").ref); + # _appendRev (inputs.nix): `_parseFlakeUrl`'s baseUrl keeps the query string, so a # rev must be spliced in front of it — concatenating would bury it inside `?dir=`. appendRevPlain = expectEq "github:o/r/deadbeef" (