Repository navigation
Conversation
Add standalone Zig packages for the Nix formats a binary cache needs: nix-base32, nix-store-path, nix-archive, nix-narinfo, nix-derivation and nix-flake-lock. Each builds and tests on its own with zig build test, and scripts/test-zig.sh runs them all. nix-archive serialises store paths to NAR straight from the kernel with openat, getdents64 and statx, mmapping large files, and hashes and zstd-compresses the stream in one pass. Its SHA-256 keeps the SHA-NI state in registers across blocks, which matches OpenSSL where the std implementation, reloading state for every block, is 12% slower. The same package verifies uploads by decompressing and hashing them as they stream in. nix-narinfo validates every field before rendering, so no input can inject narinfo lines, and signs with a fixed-base comb that needs a quarter of the point operations of std's Ed25519 base multiplication. The scanning and validation loops in base32, store paths, narinfo, derivations and flake.lock use @vector, each with a scalar reference and a test that the two agree. nix-flake-lock produces output byte-identical to the nix-flake-lock crate.
libhelios exposes the pkg packages through a C ABI, declared in core/include/helios.h, and builds as a static library. crates/helios-core runs zig build from its build script, links the result together with the system libzstd and wraps it in a safe Rust API. Calls across the boundary are coarse: a whole NAR dump, upload or narinfo per call, with output chunks handed back through a callback. A test checks that NAR hashes match the Nix database for a real closure.
helios-server replaces the Cloudflare worker. It keeps metadata in SQLite in WAL mode and NARs as content-addressed files on local disk, and serves the substituter protocol and a push API over HTTP. Narinfo misses are answered from an in-memory index of every published path, so the common case of Nix probing a substituter never touches SQLite; hits come from an LRU of narinfo rendered and signed once at publish time. Uploads stream to disk while libhelios decompresses and hashes them, and paths are only published against a NAR hash the server computed itself. This fixes several problems the worker had. Garbage collection could delete a content-addressed NAR that another path still referenced, uploads were never checked against their hash, unvalidated fields could inject narinfo lines, and private caches and pull tokens were not enforced. Token revocation now takes effect immediately instead of after a cache expiry.
The CLI pushes store paths and closures and manages caches and tokens. Each NAR is serialised, compressed and hashed by libhelios and streamed straight into the upload request; the TypeScript CLI spawned nix store dump-path and zstd for every path and wrote a temporary file. Path metadata comes from a single nix path-info call, every NAR hash is checked against the Nix database, and paths are published dependencies first. A 1,350-path system closure pushes in 3.3 seconds. scripts/e2e.sh starts a server, pushes a closure, and has Nix substitute it back into a fresh store with signatures required.
bench/run.sh times the pkg packages against the Haskell Nix libraries (nix-narinfo, nix-derivation, hnix-store-core and hnix-store-nar) and against Nix's C++ NAR serialiser, on a corpus built from the local store. bench/flake-lock.sh first checks that pkg/nix-flake-lock produces the same canonical output and validation results as the nix-flake-lock crate on fixtures, real lock files and mutated documents, then times both. All comparisons run single-threaded.
Remove the Cloudflare worker, its D1 migrations, the TypeScript CLI and the shared domain package, along with the pnpm workspace and Wrangler configuration. The flake now builds the Rust server and CLI, and its dev shell provides Zig, Rust, zstd and pkg-config. The package builds libhelios for the baseline CPU so it runs on any machine, which gives up SHA-NI hashing; building on the target host with HELIOS_ZIG_CPU=native keeps it.
Rewrite the README, the reference documentation and the agent instructions for the Rust and Zig stack: the layout, server configuration, private caches, the push API, nginx with X-Accel-Redirect, and how to run the tests and benchmarks.
Export SHA-256 (the multi-block SHA-NI implementation), HMAC-SHA256 and the OS random number generator from libhelios, with Rust wrappers, a constant-time comparison and a version 4 UUID helper. This lets the server drop the hmac, sha2, subtle and uuid crates. Zig 0.16 moved randomness behind std.Io, so the export calls getrandom directly on Linux. The helios-core tests use a small scratch directory helper instead of the tempfile crate.
The server is switching from a compiled-in SQLite to the system library, so the dev shell and the package now provide it.
Serve HTTP/1.1 only, since TLS and HTTP/2 end at the reverse proxy, and parse the upload query by hand, dropping axum's http2 and query features. Link the system SQLite by default, with a bundled-sqlite feature to compile it in, and turn off default features on rusqlite, quick_cache, clap and futures-util. tracing is replaced by a small logger that emits journald priorities under systemd and reads its level from HELIOS_LOG, parking_lot by std locks, tokio-util's ReaderStream by an unfold stream, and hmac, sha2, subtle and uuid by the libhelios exports. mimalloc is gone too; without it narinfo misses still run at 1.18M requests per second. The server now also shuts down gracefully on SIGTERM, which is how systemd stops services.
reqwest brought in aws-lc-rs, URL and IDNA handling, tower and proxy support that a CLI talking to one configured server does not need. The CLI now uses a small HTTP/1.1 client over hyper, with rustls on the ring provider and the system CA roots, honouring NIX_SSL_CERT_FILE. This takes the CLI from 115 crates to 56. tokio-stream and mimalloc are dropped as well; a full closure push takes the same 3.3 seconds without mimalloc.
Replace RUST_LOG with HELIOS_LOG in the reference, note the bundled-sqlite feature, and record in AGENTS.md which dependencies are deliberately absent and why.
The flake now exports nixosModules.default, which runs helios-server on any NixOS machine as the services.helios systemd unit, and overlays.default for pkgs.helios. Secrets are file paths passed to the unit as systemd credentials, so they can be root-only and come from any secret manager. The unit runs as its own user under a strict sandbox: no capabilities, a read-only system, and write access only to its data directory, which defaults to the StateDirectory and can point anywhere else. When a domain is set, nginx fronts the server with ACME TLS and HTTP/2. NAR downloads are served by nginx itself: the server checks access and answers with X-Accel-Redirect to an internal location aliased to the data directory. A VM test in the flake checks deploys the module behind nginx, pushes a closure from a client, substitutes it back through Nix with signatures required, and checks the sandbox, NARs on disk under a custom data directory, restarts, and clean shutdown on SIGTERM. The package source is now limited to what the build reads, so module and documentation edits no longer rebuild it.
Add a NixOS section to the reference with a flake and configuration example and the module options, point the README at it, and list the VM test among the checks in AGENTS.md.
helios-daemon needs to evict paths, scrub blobs and maintain the database, but the server's in-memory path index and narinfo cache must stay in step with SQLite, so the server remains the only process that changes it. With --admin-socket it serves a small internal API on a Unix socket, created 0660 so that directory and group permissions decide who may use it: stats, least-recently-used paths, eviction, blob listing, quarantine of a bad blob, GC, WAL checkpoints, optimize, and rotated online backups via VACUUM INTO. Eviction deletes the rows under the writer lock, drops the paths from the index and narinfo cache, and only then collects the blobs no path references any more, so nothing is served for a NAR whose file is gone. Quarantine unpublishes every path a blob backs and moves its file to quarantine/ for inspection. Least-recently-used order comes from a new paths.accessed_at column. Narinfo hits are recorded in sharded in-memory sets and flushed once a minute, before every LRU query and at shutdown; request counters are striped across cache lines. Neither shows up in narinfo throughput, which stays at about 1.2M requests per second. The upload grace period that protects in-flight pushes from GC is now configurable with --upload-grace-seconds.
Evicting a path only unpublishes it; its NAR is deleted by GC once no path uses it and its upload grace period has passed. A quota policy that watched deleted bytes alone would see nothing freed for recent NARs and keep evicting. The stats endpoint now reports unreferencedBytes, the NARs no path uses, so callers can budget against what is actually in use. Blob GC also compared creation time strictly against the cutoff, so with a zero grace period a NAR created in the current second was never collected; the comparison is now inclusive.
helios-daemon runs beside helios-server and drives it over the maintenance socket; it never writes the database itself. Auto-GC keeps the NARs in use under a byte quota, and optionally the data filesystem above a free-space floor, by evicting least-recently-used paths between a high and a low watermark. Each round re-reads the server's stats, so NARs still shared with other paths or inside the upload grace period cannot drive it into evicting everything. An integrity scrub re-reads every NAR at a rate limit, decompresses and re-hashes it through libhelios, and compares file hash, NAR hash and NAR size with the database. A missing or corrupt NAR is quarantined, which unpublishes the paths it backs. The daemon also schedules WAL checkpoints, PRAGMA optimize and rotated database backups, and serves Prometheus metrics that combine the server's request counters, its own task counters and disk space. Failed runs are logged and counted and retried on the next tick, since the server may be restarting. The crate adds no dependencies the workspace does not already build.
helios queue-paths is meant for Nix's post-build-hook. It writes the paths in $OUT_PATHS to a spool directory under a temporary name and renames them into place, then returns, so builds never wait on the network or on the cache being up. helios watch-store drains the spool and pushes each batch's closure, deleting entries only after a successful push and backing off up to a minute while the server is unreachable. Services and CI can now point the CLI at a server with --url and --token-file (or HELIOS_URL, HELIOS_TOKEN_FILE and HELIOS_TOKEN) instead of a saved login.
services.helios.daemon runs helios-daemon beside the server by default, with WAL checkpoints, daily backups and a weekly scrub; auto-GC starts once a quota or a free-space floor is set. It runs as its own user at idle I/O priority, without network access unless metrics are enabled. The server's maintenance socket lives in a setgid directory owned by a dedicated helios-admin group, so the daemon can reach it and nginx, which shares the helios group to read NARs, cannot. services.helios.watchStore is for build machines and works without the server. It installs a Nix post-build hook that spools each build's outputs and a unit that pushes them with helios watch-store. The hook never fails, because a failing post-build hook fails the build. The VM test now also checks socket permissions, metrics, rotated backups, a scrub that quarantines a corrupted NAR, a build pushed by watch-store without anyone running push, eviction under a quota, and a clean SIGTERM shutdown of the daemon.
Describe helios-daemon's auto-GC, integrity scrub, SQLite upkeep and metrics, the maintenance socket they run over, the watch-store flow for build machines, and the NixOS options for both. The server reference gains --admin-socket and --upload-grace-seconds, and the storage table the backups and quarantine directories.
Signer.generate derives an Ed25519 key pair from a seed and writes it in the name:base64 form that nix key generate-secret produces, so a fresh deployment can create its own signing key without Nix installed. libhelios exposes it as hl_signer_generate, seeded from hl_random, and helios-core as Signer::generate.
helios-server generate-secrets --dir creates a signing key, a token secret and an admin secret, each 0600 and only when missing, and writes and prints the public key. Rerunning it is safe, which lets a container entrypoint call it on every start.
packages.docker is a layered image that runs helios-server and helios-daemon as uid 10000 with all state in /var/lib/helios. The entrypoint generates secrets on first start unless HELIOS_*_FILE provides them, forwards SIGTERM to both processes, and exits non-zero when either dies so the restart policy applies. helios-admin and helios-public-key are wrappers for docker exec, since the image has no shell utilities. /etc/passwd and /etc/group are copied in rather than linked into the store: docker exec rejects absolute symlinks when resolving the user. checks.docker runs the image under Docker in a VM, pushes to it, and substitutes back with signatures required.
The code was written to long lines by hand and never run through rustfmt. max_width 150 with the Max heuristics matches that style, so applying it changes little, and CI can now enforce it.
A restarted container keeps /run/helios, so the entrypoint saw the last run's socket, started the daemon before the server was listening, and its first backup failed with connection refused. Remove the socket before starting the server. The entrypoint now has coreutils, which also replaces the read -t workaround for sleep.
Each push and pull request runs rustfmt, clippy, the Rust and Zig tests and the end-to-end test on x86_64 and aarch64 runners, the NixOS and Docker VM tests on x86_64 (the only runners with KVM), and builds the package and image on aarch64. Pushes to master and v* tags then push each architecture's image with skopeo and join them into one multi-arch tag on ghcr.io/polymath-as/helios: master, or the version and latest for a tag, which must match the package version. Actions are pinned by commit.
Every auth failure answered a bare "unauthorized" or "forbidden", which leaves the token holder guessing whether it is expired, revoked, scoped to another cache or missing a permission. Name the reason instead: the holder already knows the claims, so this leaks nothing to them.
helios token create defaulted to push on every cache. Default to pull and require --caches, so a token only gets what it is given. The end-to-end test covers the default, that push does not imply pull, and the refusal messages.
The server and daemon each had a hand-rolled logger that formatted a line, and the CLI wrote to stderr directly. Replace all three with tracing, set up once in the new helios-log crate, and log events with fields (blob, cache, bytes) rather than prose. HELIOS_LOG takes a level or per-target directives, parsed with Targets instead of env-filter and its regex engine. HELIOS_LOG_FORMAT picks journald (structured entries, the default when stderr is the journal), json for container log collectors, or text. JOURNAL_STREAM only counts when it matches stderr: a shell started from a service inherits it, and the CLI would otherwise log into the journal instead of the terminal. The CLI keeps its plain output, with warning: and error: prefixes. Requests log at debug; a 5xx also logs at warn with its method and path. Fatal errors at startup now go through the logger too. This adds 12 crates to the server (64 to 76) and the daemon. Narinfo hits measure about 1% slower (726k to 717k req/s), with or without the request middleware, which points at code layout rather than per-request work.
A NAR was always one PUT, so pushes failed behind proxies that cap request bodies: 100 MB on Cloudflare's free plan, 32 MiB on Cloud Run. Attic users hit this repeatedly (zhaofengli/attic#141, #220, #287). The server gains a chunked upload: open a session, append chunks at a stated offset, complete. A chunk is verified as it arrives and appended whole or not at all, so after a failed request the client reads the offset and resends only if it did not land; a stale offset gets 409 with the current one. Sessions belong to the token that opened them, expire after an hour idle and do not survive a restart. helios push and watch-store send NARs larger than --chunk-size (32 MiB) that way and retry each chunk with backoff; smaller NARs still go in one request, and --chunk-size 0 streams every NAR whole.
Setting up a client meant writing nix.conf and, for a private cache, a netrc entry by hand, and getting either wrong only shows up as "does not appear to be a binary cache" (zhaofengli/attic#140). helios use <cache> adds the substituter, the signing key and, for a private cache, a netrc entry with the login's token to the user's Nix config; rerunning it changes nothing, and a new token replaces the old entry. The URL is the one given to helios login rather than one the server reports, which behind a proxy is wrong (attic#325). --print gives system-wide settings, and it warns when the Nix daemon would ignore a user's substituters. The server gains GET /_api/v2/caches/<cache> for the signing key.
Caches could only be created by hand with helios cache create, so a NixOS host was not fully described by its configuration (zhaofengli/attic#169). helios-server --caches (HELIOS_CACHES) takes names such as main,team:private, creates missing ones at startup and gives existing ones the listed visibility. Unlisted caches are left alone: removing a cache from the list must not delete its paths. The module exposes it as services.helios.caches.<name>.public.
Auto-GC evicts the least recently used paths, so a release or system fetched rarely could vanish from the cache with nothing to hold it (zhaofengli/attic#83). A pin names a store path in a cache. It protects the path and its closure, followed through the published narinfo References when auto-GC runs, so dependencies pushed after the pin count too. The candidate list skips protected paths, and evict re-checks under the writer lock since a pin may be newer than the list. Pins do not stop the scrub from unpublishing a corrupt NAR. The API is GET, POST and DELETE on /caches/<cache>/pins (pull to list, push to change), with helios pin, unpin, pins and push --pin. The daemon logs when what is left over quota is pinned.
hl_signer_sign signs an arbitrary message with the key, as <name>:<base64>, for signatures that are not a narinfo's. A test checks it gives the narinfo's Sig for the same fingerprint.
A CA derivation's output path is only known once built, so Nix looks it up in the cache's build traces (realisations) before it can substitute. The server served none, so CA derivations were always rebuilt (zhaofengli/attic#188). helios push now gathers the build traces of the CA outputs it pushes from nix store build-trace info (nix realisation info on older Nix) and publishes them. The server accepts an entry only when its output path is published in the cache, signs it with the cache key over the fingerprint Nix defines, and serves it from memory in the format the pusher produced: build-trace-v2/<drv>/<output>.doi from Nix 2.35 on, realisations/sha256:<hash>!<output>.doi before. The VM test builds a two-level CA derivation with Nix 2.34, pushes it, and substitutes it into a fresh store with --max-jobs 0. Nix 2.34 and Determinate Nix 2.35 accepted entries with a deliberately wrong signature too, so the signatures follow the upstream definition but no available Nix checks them.
Describe the comparisons by the libraries they run (the Haskell Nix libraries, the nix-flake-lock crate, Nix C++), and take the nix-flake-lock crate from crates.io instead of a git checkout. CI installs upstream Nix with the nix-installer-action.
A secret key embeds its public half. Ed25519.KeyPair.fromSecretKey only checks it against the seed under runtime safety, so the ReleaseFast build libhelios ships accepted a corrupt key, advertised a public key no signature verified under, and signed under two public halves, which leaks the secret scalar. Derive the key pair from the seed and refuse a mismatch. render sorted references but kept duplicates, so References and the signed fingerprint listed one twice while Nix, which reads them into a set, fingerprints it once and rejects the signature. Drop adjacent duplicates. test-zig.sh now runs every package in ReleaseFast as well as Debug, so behaviour that depends on runtime safety is covered.
The compressor takes window_log, which turns on zstd long-distance matching with a window of up to 2^27 (128 MiB, the largest a stock Nix decoder accepts), and pledges the exact NAR size instead of hinting it. With the size pledged zstd fits the window to the NAR and records the size in the frame, so a decoder allocates no more than each NAR needs. hl_dump_options gains window_log and size_hint becomes nar_size. Also: the CCtx leaked when allocating the output buffer failed, and Signer::public_key and sign panicked on key names longer than their fixed buffers; they now grow the buffer to the size libhelios reports.
The serious one: a push token for any cache could publish a path in its own public cache over a NAR that only another, private cache held, given just the NAR hash, then fetch it without auth. nars/known even confirmed the hash existed. A new blob_caches table records which caches a NAR was uploaded to or published in, and a cache may only reuse a NAR it holds or that a public cache serves; anything else must be uploaded again. The narSize mismatch error no longer reveals the real size. Also fixed: - New paths had accessed_at 0, so auto-GC evicted them first. - Publish and evict updated the in-memory index after releasing the writer lock, so the index could disagree with the database; both now update it under the lock. A narinfo read before an eviction or re-push could be cached after it; an epoch check prevents that. - Publish re-checked a blob by rowid, which SQLite reuses; it now checks the file hash too, and a blob collected meanwhile becomes a nar_required the client recovers from instead of a dead-end 409. - GC swept the file of a chunked upload idle past an hour while its session lived on; idle sessions now expire with their files, GC skips live ones, sessions are capped per token, and offsets are read without blocking on a chunk in progress. - Upload I/O failures were reported as 400; they are 500 now. A blob file whose row failed to insert is removed, and the directory is synced after the rename. - Quarantine moved the file outside the writer lock, racing a re-upload of the same NAR. - Private caches sent cacheable 404s; every private response is now private, no-cache. narinfo and build traces lost immutable, since eviction removes them. - A build trace whose output was evicted could never be replaced; trace signing ran under the index lock. - Pins may only name published paths, at most 10000 per cache. - Two backups in one second failed, and a failed one could count towards --backup-keep. - PathKey hashing used client-chosen bytes directly; it is now keyed per process, so pushes cannot flood the index with collisions. - The audit log trusted the leftmost X-Forwarded-For entry. The server advertises compression defaults (--compression-level 3, --compression-window-log 27) in GET /_api/v2/caches/<cache>, which push tokens may now read. scripts/e2e.sh covers the leak, uncompressed uploads and the defaults.
helios push and watch-store take --level (0-19, 0 uploads raw NARs) and --window-log (10-27, or 0), and use the cache's advertised defaults for whichever is not given: zstd level 3 with a 2^27 window, about 9% smaller than plain level 2 on a NixOS closure. Fixes from review: - One NAR that kept failing stopped push from publishing anything, and stopped watch-store for good. Push now publishes everything that uploaded, leaving out failed paths and their dependents. watch-store pushes in batches, falls back to one spool entry at a time, and after five failures, counted only while the server is reachable, moves an entry to failed/. - A large spool overflowed nix path-info's argument list; long path lists now go through --stdin. - The transport had no timeouts or keepalive, so a half-open connection hung push and watch-store forever. Connect, TLS and the handshake are bounded, keepalive is on, and a connection that moves no bytes for five minutes fails. - A local serialisation failure was reported as the server's "invalid NAR stream". - helios use appended its netrc entry after a default entry, where curl never reaches it, and rewrote the file in ways that broke macdefs and quoted values. It now edits only its own entry. It also shadowed /etc/nix/netrc by pointing netrc-file at a new file; it writes to the effective netrc where it can. - Token files kept loose permissions and were rewritten in place; they are now written to a 0600 temp file and renamed. - A trailing slash in --url gave substituter URLs with //. - Unreadable spool entries were deleted; they move to failed/.
A read error ended the whole scrub, so the blob at a bad sector, the case scrub exists for, stopped every later pass at the same place, and any restart began a multi-day pass again a full interval later. Read errors are now counted and skipped, the position is saved in the state directory (--state-dir, or systemd's StateDirectory) so a restart resumes, and admin-socket calls are retried. The rate limit applies per buffer rather than between blobs, and shutdown stops a scrub in progress. With --min-free, a disk filled by something else made auto-GC evict the whole cache. It now evicts for free space only when that can help, and at most a quarter of the cache per run. Also: one accept() error ended the metrics endpoint, and requests over the admin socket had no timeout.
- The secret options were types.path, so a path literal copied a signing key into the world-readable store. They are absolute path strings now, and a store path is refused. - Without nginx the module created an empty nginx user and failed evaluation. - The image ignored docker stop while starting: bash runs as PID 1 and set its trap only after starting the server. The VM test now stops the container during startup. - CI ran pull request code with permission to publish images. The test job is read-only with credentials not persisted, and a separate job builds and pushes images on push events only. - The daemon gets a state directory for its scrub position, and the module exposes the compression defaults as settings.compressionLevel and settings.compressionWindowLog.
bench/compression.sh compresses every NAR of a closure separately, as a cache serves them, with each codec a stock Nix can decode, and reports size, speed and round-trip correctness. The numbers behind the default come from it, and docs.md gains a compression section.
nixpkgs moves from 2026-04-18 to 2026-10-01: rustc 1.98.1, Nix 2.34.8,
zstd 1.5.7, sqlite 3.53.3, GCC 16 and binutils 2.46; zig stays 0.16.0,
still the newest.
binutils 2.46 refuses to link libhelios into the Rust binaries
(".eh_frame_hdr refers to overlapping FDEs"): the compiler-rt Zig
bundled into the archive duplicates functions Rust's compiler_builtins
already provides. Stop bundling it; those symbols come from
compiler_builtins and libc as for the rest of the binary.
Raise every requirement to the newest stable release and refresh the lock file: hyper 1.11, http 1.5, http-body-util 0.1.5, libc 0.2.190, rustls 0.23.45, rustls-native-certs 0.8.4, tokio 1.53.2 and tokio-rustls 0.26.6; cc, lazy_static and mio move with them. The rest were already current. libc 1.0 and rustls 0.24 are pre-releases and stay out.
bench/flake-lock.sh had a stray "| head -1)" from an earlier edit and failed to find the nix-flake-lock fixtures. bench/run.sh created its push token with the default permissions, which became read-only.
nixpkgs now packages zig 0.17.0 (zig_0_17; plain zig is still 0.16), so the flake builds and develops with it, and nixpkgs moves to 2026-10-03 for it. The port is mechanical: @intFromEnum to @backingInt, array repetition to @Splat, dupeZ to dupeSentinel, std.meta.Int to @int, stackFallback to BufferFirstAllocator, std.fmt.allocPrint to Allocator.print, and minimum_zig_version 0.17.0 in every package. Zig 0.17 keeps frame pointers in ReleaseFast, which 0.16 omitted. In the hot paths here that cost up to 10% (drv-parse, narinfo-render), so every module omits them again in ReleaseFast builds.
Results were passed to doNotOptimizeAway by value. For an array of 64 bytes or less that is one register barrier per byte, 20 or 32 asm statements around a 5 ns routine, so the harness timed its own barriers: under Zig 0.17 base32-encode-20 seemed twice as slow while its machine code was identical. A pointer barrier makes the output escape once.
Measured against zig 0.16 (median of five runs, same corpus): - nix32 decode packed each 8-digit group by extracting digits from a vector register one byte at a time, which LLVM 22 schedules worse: base32-decode-20 went from 7.2 to 9.1 ns. One big-endian u64 load and three mask-and-shift steps now pack a group without extracts: 6.9 ns, and base32-decode-32 16.3 to 13.1 ns. - Zig 0.17 inlines the vector name check into baseName, which made store path parsing 7% slower than calling it. It stays out of line. - flake.lock validation on 8000-input documents was 13-17% slower in the follows resolver's hash maps. Edges are now numbered (node, input) through a prefix sum, so cycle detection and the target cache are plain arrays: validation is 0-8% faster than under 0.16, and output is still identical to the nix-flake-lock crate on 20302 documents.
The 0.17 port compiled, but still used what 0.17 deprecates. Checked against every declaration the 0.17 std documents as deprecated and the release notes: - builtin.cpu and builtin.os become builtin.target.cpu and .os - std.mem.indexOf and indexOfScalar become find and findScalar, and our own indexOfNewline and indexOfQuoteOrEscape follow as findNewline and findQuoteOrEscape - std.fmt.bufPrint becomes std.mem.print - std.fs.path becomes std.Io.Dir.path - std.unicode.utf8Decode, deprecated as easy to misuse, becomes the length-specific utf8Decode2/3/4; flake.lock output is still identical to the nix-flake-lock crate on 20310 documents Benchmarks are unchanged.
bench/poop.sh runs three comparisons through poop, which counts wall time, peak RSS, cycles, instructions and cache and branch misses across child processes: NAR serialise and SHA-256 against nix hash path, flake.lock canonicalisation against the nix-flake-lock crate, and helios push against nix copy to a zstd file cache, both at level 3 and signing. Nix reads the store read-only itself there, so none of its work hides in the nix-daemon. The bench binary gains a single-pass nar-once mode for it. Results print to the terminal; .gitignore now keeps benchmark output (txt, tsv, perf.data) out of the tree.
Files of 1 MiB or more were mapped whole with MAP_POPULATE, so peak RSS grew with the largest file in a closure: hashing python3's closure peaked at 70.7 MB for its 64 MiB libpython3.14.a, against 47.8 MB for nix hash path. They are now mapped 16 MiB at a time, falling back to read() from where the mappings stopped if one fails: 20.4 MB peak, at the same speed (poop, bench/poop.sh). A test serialises a 40 MiB file that ends mid-page against an independent NAR encoding.
helios push hashed every NAR twice, raw and compressed, and the server hashed both again while verifying the upload: four SHA-256 passes where nix copy makes two, 29% of a push's cycles. The server's hashes are the authoritative ones, so the compressor gains a hash option (on by default) that the CLI turns off, and the client checks the NAR hash the server verified against the Nix database instead. A corrupt local store is still caught, after the upload rather than before, and its NAR is never published. Pushing python3's closure at zstd 3 (poop, bench/poop.sh): 416 to 360 ms and 4.40 to 3.74 G cycles, against 336 ms and 3.42 G for nix copy to a file cache, which verifies nothing on the other side. The default long window: 503 to 473 ms.
nix path-info, nix store info, nix config show and the nix-store --dump fallback inherited helios's stdin. Nix does not read it, but a test that stood in for nix with `cat` hung for half an hour whenever the test runner left stdin open as a pipe, and a subprocess has no business waiting on whatever the caller left open. They now get /dev/null; only the --stdin path list still gets a pipe.
Rust: - helios-core: the three Signer binding tests become one, and the nix32 smoke test joins the primitives test; signing logic is tested in pkg/nix-narinfo. - auth: the header constant and scope checks join the token test. - traces: drop a test that serde picks an enum variant; e2e publishes and serves both formats. - daemon: the duration and size parsers share a test, and so do the two free-space floor tests. Zig: - nix-base32: the encodedLen check folds into the known-vector test. - nix-archive: the two "refused" tests merge; a test now covers the hash = false option, which had none. e2e drops seven checks a later check already proves: the token being non-empty, nix-cache-info, a narinfo hit and a NAR download (all needed by the signed substitution that follows), the chunk-size log scan (the chunked closure substitutes), the signing key in nix.conf (Nix verifies signatures through it) and an unknown trace 404.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the Cloudflare Workers/R2/D1 cache with a self-hosted one: a Rust
server, CLI and maintenance daemon over a Zig core (libhelios), storing
NARs on the local filesystem and metadata in SQLite (WAL). Cloudflare
support and the pnpm workspace are removed.
What's in it
pkg/*: Zig packages for the Nix formats (narinfo, derivation, NAR,flake.lock, store paths), with SIMD routines checked against scalar
references.
core/exposes them as a C library;crates/helios-corebinds it.
helios-server: axum; narinfo misses never reach SQLite and hits aren'tsigned per request. Tokens, caches, audit log, and a maintenance API on a
Unix socket.
heliosCLI: push (with closure), login, admin, andwatch-storeforpushing as paths are built.
helios-daemon: LRU auto-GC under a quota, integrity scrub withquarantine, SQLite checkpoint/optimize, backups, Prometheus metrics.
nix/: the NixOS moduleservices.helios(hardened systemd, nginx withACME and X-Accel-Redirect, daemon, watch-store) and an OCI image that
generates its keys on first start.
VM tests; publishes
ghcr.io/polymath-as/helios(amd64 + arm64) onpushes to master and
v*tags.Not in it