Rewrite as a self-hosted cache: Rust server over a Zig core

.github/workflows/ci.yml

@@ -0,0 +1,132 @@
+# Tests on x86_64 and aarch64, then publishes the OCI image to
+# ghcr.io/<owner>/helios on pushes to master and on v* tags.
+name: ci
+
+on:
+  push:
+    branches: [master]
+    tags: ["v*"]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ubuntu-24.04
+            system: x86_64-linux
+            arch: amd64
+          - runner: ubuntu-24.04-arm
+            system: aarch64-linux
+            arch: arm64
+    runs-on: ${{ matrix.runner }}
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: DeterminateSystems/nix-installer-action@3138316df39ed29be04236d7ffc686fa525866aa # v23
+        with:
+          # Upstream Nix, as users of the module and image run.
+          determinate: false
+          extra-conf: |
+            system-features = nixos-test benchmark big-parallel kvm
+
+      - name: Format
+        run: nix develop -c cargo fmt --check
+      - name: Clippy
+        run: nix develop -c cargo clippy --release --all-targets -- -D warnings
+      - name: Rust tests
+        run: nix develop -c cargo test --release
+      - name: Zig tests
+        run: nix develop -c ./scripts/test-zig.sh
+      - name: End to end
+        run: nix develop -c sh -c 'cargo build --release && ./scripts/e2e.sh'
+
+      # The NixOS and Docker VM tests need KVM, which only the x86_64
+      # runners have.
+      - name: Enable KVM
+        if: matrix.system == 'x86_64-linux'
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
+      - name: Nix checks
+        if: matrix.system == 'x86_64-linux'
+        run: nix flake check -L
+      - name: Nix build
+        if: matrix.system != 'x86_64-linux'
+        run: nix build -L .#checks.${{ matrix.system }}.package .#docker
+
+  # Builds and pushes each architecture's image. Kept apart from the tests,
+  # which run pull request code, so only pushes ever hold a write token.
+  push:
+    if: github.event_name == 'push'
+    needs: test
+    strategy:
+      matrix:
+        include:
+          - runner: ubuntu-24.04
+            arch: amd64
+          - runner: ubuntu-24.04-arm
+            arch: arm64
+    runs-on: ${{ matrix.runner }}
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: DeterminateSystems/nix-installer-action@3138316df39ed29be04236d7ffc686fa525866aa # v23
+        with:
+          determinate: false
+
+      - name: Push image
+        env:
+          ARCH: ${{ matrix.arch }}
+          TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          version=$(nix eval --raw .#helios.version)
+          if [[ "$GITHUB_REF_TYPE" == tag && "$GITHUB_REF_NAME" != "v$version" ]]; then
+            echo "tag $GITHUB_REF_NAME does not match package version $version" >&2
+            exit 1
+          fi
+          image="ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/helios"
+          nix run --inputs-from . nixpkgs#skopeo -- copy \
+            --dest-creds "$GITHUB_ACTOR:$TOKEN" \
+            "docker-archive:$(nix build --print-out-paths .#docker)" \
+            "docker://$image:sha-${GITHUB_SHA::12}-$ARCH"
+
+  # Joins the per-architecture images into one multi-arch tag.
+  image:
+    if: github.event_name == 'push'
+    needs: push
+    runs-on: ubuntu-24.04
+    permissions:
+      packages: write
+    steps:
+      - name: Publish manifest
+        env:
+          TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          echo "$TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin
+          image="ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/helios"
+          sha="sha-${GITHUB_SHA::12}"
+          tags=(-t "$image:$sha")
+          if [[ "$GITHUB_REF_TYPE" == tag ]]; then
+            tags+=(-t "$image:${GITHUB_REF_NAME#v}" -t "$image:latest")
+          else
+            tags+=(-t "$image:$GITHUB_REF_NAME")
+          fi
+          docker buildx imagetools create "${tags[@]}" "$image:$sha-amd64" "$image:$sha-arm64"

.gitignore

@@ -1,8 +1,17 @@
-node_modules/
-.pnpm-store/
-.wrangler/
-coverage/
-dist/
-*.log
 .DS_Store
+*.log
 result
+result-*
+
+# Rust
+/target
+/bench/flake-lock-rs/target
+
+# Zig
+.zig-cache/
+zig-out/
+
+# Benchmark output (bench/*.sh, perf, poop): keep results out of the tree
+/bench/**/*.txt
+*.tsv
+perf.data*

AGENTS.md

@@ -1,31 +1,72 @@
 # Helios
 
-This repository is a `pnpm` workspace for Cloudflare-native services.
+A self-hosted Nix binary cache: Rust server and CLI over a Zig core.
 
 ## Workspace Layout
 
-- `workers/*`: deployable Cloudflare Workers
-- `apps/*`: operator or end-user applications that sit beside Workers
-- `packages/*`: shared code used by workers or apps
+- `pkg/*`: Zig packages for Nix formats and the NAR pipeline, one concern
+  each. Each builds and tests on its own (`zig build test`).
+- `core/`: libhelios, the C ABI over `pkg/*`, compiled to a static library
+  by `crates/helios-core/build.rs`. `core/include/helios.h` is the contract.
+- `crates/helios-core`: safe Rust bindings to libhelios.
+- `crates/helios-server`, `crates/helios-cli`, `crates/helios-daemon`: the
+  binaries. `crates/helios-log` sets up their logging. The daemon drives
+  the server over its maintenance socket and never writes the database
+  itself.
+- `nix/`: the NixOS module (`services.helios`), the OCI image, and their VM
+  tests.
+- `bench/`: comparisons against the Haskell Nix libraries, the Rust
+  nix-flake-lock crate and Nix C++. Not part of the Cargo workspace.
 
-Keep deployable runtime code inside `workers/`. Move shared protocol, parsing, and storage helpers into `packages/` only after they are reused.
+Put Nix format logic in a `pkg/*` package and expose it through
+`core/src/root.zig` only when Rust needs it. Keep calls across the C ABI
+coarse, such as a whole NAR or narinfo per call.
 
 ## Tooling
 
-- Use `pnpm`, not `npm`, for dependency management and scripts.
-- Do not commit `package-lock.json` files.
-- If `pnpm` or `wrangler` are missing locally, use `nix shell nixpkgs#nodejs nixpkgs#pnpm nixpkgs#wrangler`.
-- Run workspace commands from the repository root unless a package-specific command is clearer.
+- Use `nix develop` for zig 0.17, cargo, zstd and pkg-config.
+- Zig 0.17 APIs changed a lot; check `zig env` for the std source rather
+  than relying on older examples.
+- System libraries: zstd and sqlite (pkg-config).
+- `HELIOS_ZIG_CPU` sets libhelios's `-Dcpu` (default `native`; the Nix
+  package uses `baseline`).
 
-## Cloudflare Rules
+## Dependencies
 
-- Before changing Workers, R2, D1, Durable Objects, Queues, or platform limits, fetch current Cloudflare docs.
-- Prefer platform-native APIs such as `fetch`, Web Streams, and Web Crypto.
-- Do not enable `nodejs_compat` unless a concrete dependency requires it.
-- After changing a worker's bindings in `wrangler.jsonc`, run that package's `cf-typegen` script.
+Keep the dependency tree small; every crate is build time, audit surface and
+supply-chain risk. Before adding one, check whether libhelios, std or a few
+dozen lines cover it, and disable default features. Deliberately absent:
+reqwest (and `url`/`idna`), aws-lc-rs, parking_lot, uuid,
+hmac/sha2 (libhelios provides SHA-256, HMAC and randomness) and mimalloc
+(measured: no gain here).
+
+Logging is `tracing` with `tracing-subscriber` (fmt, ansi and json only; no
+`env-filter`, whose regex engine `Targets` makes unnecessary) and
+`tracing-journald`, set up once in `crates/helios-log`.
+
+## Performance Rules
+
+- Measure before and after; `bench/run.sh` and `bench/flake-lock.sh`
+  compare against the reference implementations on the same corpus.
+  `bench/poop.sh` compares whole processes with poop (wall time, peak RSS,
+  cycles, cache and branch misses), including `helios push` against
+  `nix copy`. Benchmark output stays out of the tree.
+- Keep the narinfo read path free of SQLite on misses and of per-request
+  signing.
+- Keep a scalar reference for every SIMD routine, with a test asserting the
+  two agree.
 
 ## Verification
 
-- Install dependencies with `pnpm install` from the repo root.
-- Run `pnpm check` before finishing substantial changes.
-- For worker-only iteration, use `pnpm --filter <package> dev`.
+CI (`.github/workflows/ci.yml`) runs all of this on x86_64 and aarch64
+(the VM tests on x86_64 only) and publishes the image on pushes.
+
+- `cargo fmt --check` and `cargo clippy --release --all-targets -- -D warnings`
+- `cargo test --release`
+- `./scripts/test-zig.sh`
+- `nix build .#checks.x86_64-linux.nixos` after changing `nix/`: a VM test of
+  the NixOS module. `.#checks.x86_64-linux.docker` likewise for the OCI
+  image.
+- `./scripts/e2e.sh` before finishing changes to the server, the CLI or wire
+  formats; it substitutes a real closure through Nix with signatures
+  required.