Skip to content

fix(dax): stop publishing failed installs and truncated disk sizes - #387

Open
tode-rl wants to merge 1 commit into
computesdk:masterfrom
tode-rl:fix/dax-install-masking-and-du-overflow
Open

fix(dax): stop publishing failed installs and truncated disk sizes#387
tode-rl wants to merge 1 commit into
computesdk:masterfrom
tode-rl:fix/dax-install-masking-and-du-overflow

Conversation

@tode-rl

@tode-rl tode-rl commented Sep 2, 2026

Copy link
Copy Markdown

Two defects in benchmarks/scripts/dax-benchmark.sh that put wrong numbers into the published results rather than failing loudly. Both are currently visible on the live benchmark pages.

1. A failed bun install is recorded as a successful phase

install_dependencies() {
  cd "$ROOT/repo"
  bun install
  git diff --exit-code -- bun.lock package.json   # <- exits 0, and it is the LAST command
}

A function returns its last command's status, so git diff --exit-code — which exits 0 whenever the lockfile is untouched — masks whatever bun install did. set -e does not catch it either, because phase() runs its callee under set +e.

Observed on a 1 GiB sandbox:

/tmp/dax-benchmark.sh: line 206: 2099 Killed   bun install
BENCH_PHASE  install  4248                      <- recorded as a 4.2s success

node_modules never exists, yet the run reports a fast successful install and proceeds into typecheck, which then fails with a confusing Cannot run "…/turbo.json" — because the toolchain was never installed. Any provider whose install dies without dirtying the lockfile publishes the same silently-wrong timing.

Fixed with bun install || return.

2. Workspace size above 2 GiB is truncated to 3 bytes

disk() { du -skx "$ROOT" | awk '{print $1 * 1024}'; }

mawk (the default awk on Debian/Ubuntu images) switches to %.6g once $1 * 1024 exceeds INT_MAX, emitting 3.07106e+09. The consumer's parseInt() then reads that as 3.

This is in the results today. For the identical workload:

  • diskAfterInstall: 3 — e2b, miosa, mosaic, runloop, sail, superserve, upstash (mawk images), rendered as 0.0 MB on the benchmark pages
  • diskAfterInstall: ~3.1e9 — arker, daytona, tenki, tensorlake, vercel (gawk/busybox images)

It splits by awk implementation, not by anything about the provider. printf "%.0f" formats the value exactly on every awk.

Verification

Ground truth from a complete run of the pinned workload (opencode @ 08fb473): 38,182 directories, 276,525 files, du -skx = 3,239,731,200 bytes — now reported verbatim instead of as 3.

Not included

While investigating I found a third, separate issue worth raising on its own: on some providers du reports only directory inodes because their filesystem leaves st_blocks unpopulated for regular files, so diskAfter* understates real usage by 20-150x (isorun/blaxel report directory-count x 512; archil/beam/createos/declaw/namespace report directory-count x 4096). That is not fixable with a formatting change — it would need du --apparent-size, which changes what the metric means — so I have left it out of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VXBre1j9JaGUwwzXta1B3E

Two defects in dax-benchmark.sh that put wrong numbers in the published
results rather than failing loudly.

**A failed `bun install` is recorded as a successful phase.**
`install_dependencies` ends with `git diff --exit-code`, which exits 0 whenever
the lockfile is untouched — masking the status of the `bun install` before it.
`set -e` does not catch this either, because `phase()` runs its callee under
`set +e`. Observed on a 1 GiB sandbox: `bun install` is SIGKILLed by the OOM
killer at t=27s, node_modules never exists, and the run still emits
`BENCH_PHASE install 4248` and proceeds into typecheck, which then fails with a
confusing "Cannot run turbo.json" because the toolchain was never installed.
Any provider whose install dies without dirtying the lockfile publishes the
same silently-wrong timing. `bun install || return` propagates it.

**Workspace size above 2 GiB is truncated to 3 bytes.**
`disk()` used awk's `print`, and mawk switches to `%.6g` once `$1 * 1024`
exceeds INT_MAX, emitting `3.07106e+09`; the consumer's `parseInt()` then reads
that as `3`. This is in the current results: every provider on a mawk image
(e2b, miosa, mosaic, runloop, sail, superserve, upstash) reports
`diskAfterInstall: 3`, rendered as "0.0 MB" on the benchmark pages, while
gawk/busybox images report the true ~3.1 GB for the identical workload.
`printf "%.0f"` formats it exactly on every awk.

Verified against a full run: 38,182 directories, 276,525 files,
`du -skx` = 3,239,731,200 bytes, now reported verbatim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VXBre1j9JaGUwwzXta1B3E
@tode-rl
tode-rl force-pushed the fix/dax-install-masking-and-du-overflow branch from 096c806 to 01bdbb5 Compare September 2, 2026 20:04
@tode-rl
tode-rl marked this pull request as ready for review September 2, 2026 20:41
@open-cla

open-cla Bot commented Sep 2, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant