A small Linux proof of concept that captures process execution events with eBPF, chains them with SHA-256, ships them to a remote collector, and lets anyone independently verify whether the stored audit history has been modified, deleted, or reordered — and, given one hash recorded outside the log, whether it has been rewritten wholesale.
Three binaries, one file format, no infrastructure.
audit-agent capture, chain, send
audit-collector validate, append
audit-verify re-verify from scratch
Audit logs are usually trusted because of where they live: a directory only root can write to, a bucket with a retention policy, a SIEM someone else operates. That trust is administrative, and it disappears the moment the administrator is the adversary.
A hash chain replaces most of that administrative trust with arithmetic. Each event carries the hash of the one before it, so every stored event commits to the entire history that preceded it. Change one byte of one event and every hash from that point on stops matching — and anyone holding the file can check it, without access to the machine that produced it and without a key.
What the chain cannot do on its own is tell a rewritten history from a real one:
whoever can edit the file can also recompute every hash in it. Closing that gap
needs one number kept outside the log — the final hash from a previous check.
audit-verify -expect-hash H -expect-events N makes that check explicit rather
than aspirational, and everything below is careful to say which of the two
properties it is talking about.
This PoC demonstrates the whole path end to end on the noisiest signal a Linux host produces: process execution.
┌───────────────┐
│ Linux Kernel │
│ eBPF execve │ sched_process_exec tracepoint -> ring buffer
└───────┬───────┘
│
▼
┌───────────────┐
│ audit-agent │ normalize, assign sequence, link SHA-256 chain
│ seq + SHA256 │
└───────┬───────┘
│ HTTP POST /events
▼
┌───────────────┐
│audit-collector│ revalidate hash, sequence and link, then append
│ append-only │ data/<hostname>.audit.jsonl
└───────┬───────┘
│
▼
┌───────────────┐
│ audit-verify │ recompute every hash, exit 0 or 1
│ integrity CLI │
└───────────────┘
Each event is one line of JSON:
{
"seq": 3,
"timestamp": "2026-08-18T11:22:49.653239539Z",
"hostname": "devbox-01",
"pid": 8123,
"ppid": 8099,
"uid": 1000,
"gid": 1000,
"command": "git",
"args": ["git", "push", "origin", "main"],
"prev_hash": "ae710a75e816a67e49ded8943d5337eefd3cd7e6ff044d57e94cb4a70c8e2dbb",
"hash": "4efa4e4a73e9633f4a9747a81e0e46caa52c58c80c6300775d6b637e750b6d79"
}hash is SHA-256 over a canonical serialization of every other field,
prev_hash included, in a fixed order with every variable-length value
length-prefixed.
Requirements: Linux 5.8+ with BTF (/sys/kernel/btf/vmlinux) for the eBPF
agent. Everything except the agent's kernel probe runs unprivileged.
go.mod requires Go 1.26.6. The standard library is linked into these
binaries, so the toolchain is a runtime dependency, not just build tooling:
releases before 1.26.6 carry reachable net/http vulnerabilities, including one
that lets a cleartext HTTP/2 request bypass the collector's header timeout.
make vulncheck reports what the toolchain you have would actually ship — it is
clean on 1.26.6.
make build # -> bin/audit-agent, bin/audit-collector, bin/audit-verify
make test # unit + end-to-end tests
make probe-test # the eBPF probe test, which needs rootCI runs the same checks on every push: build, gofmt, vet, tests with and
without the race detector, cross-builds for the platforms the eBPF build tags
claim to cover, govulncheck (weekly as well, since advisories land without
anyone touching the code), the demo end to end, and — on a real kernel — the
committed eBPF artifacts against their source plus the privileged probe test.
See .github/workflows/ci.yml.
Terminal 1 — the collector:
./bin/audit-collector -config configs/collector.yamlPass -config explicitly: the binary does not look for that file on its own, so
running without it means the settings you edited there — including the
allowlist — are not in force. (-listen and -data override the file; with no
-config at all you get the built-in defaults, which accept any hostname up to
max_hosts.)
This version has no agent authentication, so name the hosts you expect:
# configs/collector.yaml
allowed_hosts: [devbox-01, devbox-02]
max_hosts: 64
max_concurrent_requests: 64An unknown key in that file is a startup error rather than a silently ignored
setting — a misspelled allowd_hosts would otherwise leave admission control
off while looking configured.
A valid genesis event is unauthenticated by construction — a fresh chain always
starts at sequence 1 with the zero hash, and SHA-256 is public — so without a
list, anyone who can reach the collector can invent host identities, each
costing a log file and an inode. max_hosts is the backstop when you cannot
enumerate them; hosts that already have a log are always admitted.
Terminal 2 — the agent. Only this part needs root, and only to load the eBPF program:
sudo ./bin/audit-agent -collector http://127.0.0.1:8080/eventsTerminal 3 — run some commands, then verify:
whoami
uname -a
git status
./bin/audit-verify ./data/$(hostname).audit.jsonlAudit Verification
File: ./data/devbox-01.audit.jsonl
Events: 6
First sequence: 1
Last sequence: 6
Sequence: PASS
Hash chain: PASS
Event hashes: PASS
Trusted anchor: NOT CHECKED
Result: VERIFIED
Final hash:
ec078396b7b5ad11f6e65c44a90a3a4d15e02c74fbd8a6303f4620d4d1abf740
Checkpoint for the next run:
-expect-hash ec078396b7b5ad11f6e65c44a90a3a4d15e02c74fbd8a6303f4620d4d1abf740 -expect-events 6
This log was not checked against a trusted anchor, so the result means
the file is internally consistent - not that it is unaltered. Store the
checkpoint above somewhere the log writer cannot reach, and pass it back
next time.
No root? The agent has a synthetic event source that exercises the identical chain, transport, storage and verification path:
./bin/audit-agent -demo -demo-count 7make demoscripts/demo.sh builds everything, starts a collector, runs the agent,
executes a few commands, and verifies the resulting log.
Run it as your normal user — not under sudo. Only the agent needs
privileges, and the script elevates just that part; running the whole thing as
root would fail at the build step (sudo's secure_path has no Go toolchain) and
leave root-owned binaries and audit logs behind. Without usable sudo it falls
back to the synthetic event source.
make tamper-demoscripts/tamper-demo.sh takes the log the demo produced and performs the four
edits an attacker would make — rewrite an event, delete an event, swap two
events, drop the tail — then runs the verifier on each. The original log is left
untouched; each attempt is reported:
== 1. Modified event 3 (command rewritten to "echo")
Events verified: 2 (before the failure below)
Verified range: seq 1..2
Result: FAILED
Tampering detected at event 3 (line 3).
Stored hash:
3aff4d40ac2c502e52710da50e5eb654ad0067fc58e1f893d001ae005b4d344f
Calculated hash:
8e22997e9ae2e0b4246e34991f553804300897339eaaa49f5de8ea398caa838a
Every hash in this README is sample output from a devbox-01 demo run. The
worked example under docs/ is the one with values you can reproduce exactly;
its canonical byte string and digest are pinned in internal/event/golden_test.go.
The fourth case is the interesting one: the truncated log passes an ordinary verification, because a shorter chain is still a consistent chain, and fails the moment the final hash recorded before the truncation is supplied.
audit-verify exits 0 when verified, 1 on an integrity failure (including a
mismatch against the anchor), 2 on bad input or usage, 3 on an internal
error — so it drops straight into CI.
The on-disk format is byte-exact: each line must be the one canonical encoding
of the event it contains. Without that rule a record can be rewritten into
different bytes that decode to the same values — a shadowing duplicate key, say
— leaving every hash and the anchor intact while grep and any first-match log
reader see the attacker's version.
What this project provides:
A tamper-evident audit trail for Linux process execution events.
Precisely, and the precision matters:
If a stored event is modified, deleted, or reordered and the chain is not rebuilt, the verifier detects that the cryptographic chain is no longer valid. Detecting a wholesale rewrite, in which every hash after the edit is recomputed, requires comparing the log against a final hash recorded earlier and kept out of the writer's reach.
Three checks run on every record, in the collector at ingest time and again in the verifier at rest — plus a fourth that only the caller can supply.
| Attack | Detected by | Needs an anchor? |
|---|---|---|
| Edit an event's contents | its stored hash no longer matches the recomputed one | no |
| Rewrite a record's bytes without changing its values (duplicate keys, reordering, escapes) | the stored line is not the one canonical encoding of the event | no |
| Edit an event and re-hash just it | the next event's prev_hash no longer matches |
no |
| Delete an event, leaving a later record | sequence numbers are no longer consecutive | no |
| Reorder events | sequence numbers are out of order, and the links break | no |
| Truncate the head of the file | the log no longer starts at sequence 1 with the zero hash | no |
| Add a field to a record | records with unknown fields are rejected outright | no |
| Replay an older event into the collector | the collector refuses a sequence number it has moved past | no |
| Reset the chain to erase history | the collector refuses a restart onto an existing log | no |
| Relabel a log by renaming it to another host | its records name a different host than the file does | no |
| Splice another host's records into a log | the records disagree about which host they came from | no |
| Truncate the tail of the file | the log is shorter than the checkpoint | yes |
| Delete every record (an empty log verifies) | the log is shorter than the checkpoint | yes |
| Rewrite history before the checkpoint and re-chain all of it | the event at the checkpoint no longer has its recorded hash | yes |
| Invent a host identity | the collector refuses hostnames outside allowed_hosts, and a new host must arrive at its genesis event |
no |
The last two rows are why the anchor exists. A hash chain proves internal consistency; an external anchor is what turns that into evidence:
# First run: it prints a checkpoint. Store it somewhere the log writer
# cannot reach - a ticket, a CI artifact, another host.
./bin/audit-verify ./data/devbox-01.audit.jsonl
# Checkpoint for the next run:
# -expect-hash 7ed3b4a8768c... -expect-events 318
# Later runs, after the log has grown: event 318 must still hash to that
# value, and the log must still contain at least 318 events.
./bin/audit-verify \
-expect-hash 7ed3b4a8768c... \
-expect-events 318 \
./data/devbox-01.audit.jsonl
# Trusted anchor: PASS (through event 318)The pair matters. An audit log grows, so a hash on its own can only describe a frozen snapshot — checked against a log that has since gained an event, it would report tampering. Paired with the count it was taken at, the checkpoint keeps working as the log grows, and the report says how far the attestation reaches: events after the checkpoint are chained to the anchored history but are not themselves attested until you store a new one.
Passing -expect-hash alone keeps the strict reading — the log must still end
on that hash — which is what you want for an archived snapshot.
Without a checkpoint, the report says Trusted anchor: NOT CHECKED and explains
what the result does and does not mean. Signed checkpoints would make the anchor
self-describing instead of an operator's responsibility; they are not in this
version.
What this project does not claim:
- that root cannot stop or bypass the agent;
- that every action on the host is captured;
- that the local machine is trustworthy;
- that the collector's storage is production-grade WORM;
- that an unanchored verification proves a log is unaltered.
An audit trail that stops recording is as useless as one that lies, so two failure modes are handled explicitly rather than documented away:
- A failed delivery costs one event, not the host. The agent chains an event, offers it to the collector, retries transient failures with backoff, and only keeps the new chain position once the event is accepted. If it is given up on, the chain rewinds so the next event is still the collector's next expected link. Dropped events are counted in the shutdown summary. (There is no durable spool, so a dropped event is gone and leaves no trace in the log.)
- A host that stops being audited says so. The rewind above is wrong in one case: the collector committed the event and the agent never found out. The two are then permanently out of step, and resynchronizing would mean handing an unauthenticated caller the host's exact chain position. So the agent gives up after three consecutive chain disagreements and exits non-zero, as it does when its eBPF probe dies. Run it under a supervisor that restarts on failure — silently dropping every event looks exactly like a quiet host.
- One damaged log costs one host. A log whose tail cannot be read
quarantines that host: its events are refused,
/healthzreports it, and every other host keeps being collected. - A misconfigured agent fails at startup, not silently forever. The hostname goes into every event, so the agent checks it against the collector's rule before loading the probe. If the collector nonetheless refuses event after event, the agent gives up rather than observing everything and delivering nothing.
- A re-delivered event is not a conflict. If the agent never saw the response, it retries; the collector recognizes its own accepted tail and acknowledges without storing a second copy.
- Arguments are byte strings, not text. Non-UTF-8 arguments are encoded losslessly before hashing, so an ordinary process cannot halt collection by passing one bad byte.
Read docs/limitations.md before drawing conclusions from
a verified log. The short version: it audits execve only, speaks plain HTTP,
drops events it cannot deliver, restarts its sequence when the agent restarts,
and stores the log in a file that root can still overwrite.
What it guarantees is narrower than "overwriting is detectable", and the
distinction is the whole point of the checkpoint: an edit that does not
rebuild the chain is detected by the file alone, while a wholesale re-chain, a
truncated tail, or deleting every record is detected only by comparison with
a checkpoint recorded beforehand and covering the changed position. An unanchored
VERIFIED means internally consistent, not unaltered.
The obvious next steps, roughly in the order they matter:
- HTTPS and mTLS, agent authentication
- a durable local spool with retry, and persistent sequence state across restarts
- Ed25519-signed checkpoints, so the anchor is not an operator's responsibility
- immutable remote storage (S3 Object Lock or equivalent)
- more than
execve: network connections, file mutations, privilege transitions
cmd/ three binaries
internal/event/ the audit event and its canonical serialization
internal/chain/ the SHA-256 hash chain
internal/agent/ normalize -> chain -> send
internal/collector/ HTTP ingest, validation, append-only storage
internal/verifier/ independent re-verification
bpf/ the eBPF program and its Go loader
test/ the end-to-end test: agent -> collector -> file -> verifier
scripts/ demo.sh, tamper-demo.sh
docs/ architecture, limitations
MIT, except the eBPF program in bpf/exec.bpf.c, which is GPL-2.0 because the
kernel helpers it calls are GPL-only, and the libbpf headers in bpf/headers/,
which keep their upstream dual LGPL-2.1/BSD-2-Clause license. See
LICENSE.