Skip to content

Repin to Solid 2.0.0-rc.21; eval installs Solid from npm; grade script #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ryansolid merged 2 commits into from
Nov 4, 2026
Merged

Repin to Solid 2.0.0-rc.21; eval installs Solid from npm; grade script #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
41a6b1f
Repin to Solid 2.0.0-rc.21 (next 2105ffb8); eval installs Solid from …
claude Nov 4, 2026
ad57d42
runtime: regenerate package-lock for rc.21
claude Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion compiler/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion compiler/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ regex = "1"
serde_json = { version = "1", features = ["preserve_order"] }
# CP-1b: Solid's compiler, at the commit ref/ runs on (ref/SOLID_NEXT), and the Oxc release it is built against (CP-1c:
# the program crosses the seam, so the versions must match)
solidjs-compiler = { git = "https://github.com/solidjs/solid", rev = "5e68925cc2fd216a9fb06345019e66d5da5c523c", default-features = false }
solidjs-compiler = { git = "https://github.com/solidjs/solid", rev = "2105ffb8ca2fced55f5fac0763177fc93f97c2c2", default-features = false }
oxc_allocator = "0.144"
oxc_ast = "0.144"
oxc_span = "0.144"
Expand Down
11 changes: 10 additions & 1 deletion eval/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,15 @@ mutants/ naive React drafts the hidden tests must catch
results/ scored runs
```

Setup: `npm install` here; `cargo build --release --bin rxc` in `../compiler` (its Solid pin is d2323b0, rc.19).
Setup: `npm install` here (Solid 2.0.0-rc.21 from npm); `cargo build --release --bin rxc` in `../compiler` (its Solid pin is d2323b0, rc.19).
`RXC` and `RT_WEB` override the paths to the binary and `ref/compile/rt-web.ts`. (The pilot itself ran on PR #6's
crate before it merged, built with the same pin.)

## Running another agent harness (Cursor, Grok, Codex, …)

A desk is a self-contained folder: `node harness/scaffold.mjs desk9 rx <dir> --brief plain --inline-docs` writes
`.prompt` (the whole instructions), `BRIEF.md`, `src/`, `./check` and `./test`. Give the agent `.prompt` as its task
and the folder as its workspace, then grade with `sh harness/grade.sh <dir>`. Use the `react` arm for the control.
Defects are comparable across harnesses; cost needs that harness's own token count (`cost.mjs` reads Claude
transcripts). The hidden tests and references are in this public repo: an outside run relies on the prompt's
"don't look elsewhere" rule, so check its transcript.
10 changes: 10 additions & 0 deletions eval/harness/grade.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/sh
# Grade one desk (scaffolded with `node harness/scaffold.mjs desk9 <arm> <dir> --brief plain [--inline-docs]`), however
# the agent that wrote it was run: hidden tests, hidden overlap tests, and 20 timing-fuzzer seeds.
# Usage: sh harness/grade.sh <desk dir> [out.txt] (run from eval/; defects = ✗ lines + fuzz seeds with a hole)
set -e
d=$(cd "$1" && pwd); out=${2:-/dev/stdout}
cd "$(dirname "$0")/.."
{ echo "== hidden"; node harness/test.mjs "$d" tasks/desk9/hidden.mjs 2>&1 | grep "^✗\|passed\|build failed" || true
echo "== overlap"; node harness/test.mjs "$d" tasks/desk9/hidden-overlap.mjs 2>&1 | grep "^✗\|passed\|build failed" || true
echo "== fuzz"; node harness/fuzz.mjs "$d" --seeds 20 --r9 2>&1 | cut -c1-400; } > "$out"
Loading
Loading