Skip to content

Find why trail sync crashes a laptop, and fix it - #736

Open
anirudhkumar-nanonets wants to merge 1 commit into
mainfrom
claude/sync-crash-9a5d00
Open

anirudhkumar-nanonets wants to merge 1 commit into
mainfrom
claude/sync-crash-9a5d00

Conversation

@anirudhkumar-nanonets

@anirudhkumar-nanonets anirudhkumar-nanonets commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Issue: On a big repo like assign, trail could run two or three full code-map builds at once in the background, each peaking above 2 GB of memory and using a CPU core or more.
Why: The build lock counted as abandoned after 5 minutes, even with its build still running, so on a slow laptop the next trail query started another build beside it.
Fix: The lock now names the process doing the build and is taken over only once that process is gone. Background builds run at the lowest CPU priority, and a build now reports when another was already running.

Moves: active users, people with session_summary in the last 7 days (PostHog db 11, posthog_event_full_view): 7,214 on 2026-10-29 (996 of them from the trail CLI).

What changed

  • src/util/state.ts: a lock is reclaimed only when its owner pid is gone and it's past LOCK_STALE_MS (5 min), or when it's past the new LOCK_MAX_MS (15 min) whoever holds it. 15 min is longer than any build's own limit (a first build gets 10), and it covers a reused pid. lockHeldIn uses the same rule.
  • src/claude/sync-run.ts: writes its own pid into the lock before building (claimLock). Whoever took the lock (a query, the Stop hook) exits at once, so its pid said nothing about whether a build was running. It also sets PRIORITY_LOW; the build child inherits it.
  • build_completed gets overlap (src/cli.ts, src/telemetry/contract.ts, TELEMETRY.md): true when another live process held this map's lock as the build started. Nothing recorded overlaps before. It can't see two worktrees building at once, since each worktree has its own lock.
  • A learning for the next session: .trail/learnings/2026-10-29-on-a-big-repo-every-trail-anirudhkumar.md.

Measured (copy of assign, 12,281 files, 12-core Mac, 24 GB)

Builds at once Processes Peak memory Peak CPU Busy
Before (main 19865ff4) 3 6 4.78 GB 3.8 cores 32 s
After 1 3 2.26 GB 1.5 cores 22 s

Three trail ask calls 4 s apart on a repo with no map, run the way the prompt hook runs them (GRAFT_FIRST_BUILD_WAIT_MS=0). A build here takes 23 s, so the lock's mtime was set 6 minutes back before the 2nd and 3rd query, which is what a laptop's lock looks like once its build passes 5 minutes. At nice 19 an idle-machine build took 22 s vs 23 s before.

What this does and doesn't explain

The report (Slack DM, 29 Oct 12:43 IST) came from an install on 0.26.0, working in assign: a first build at trail init, 14 end-of-turn builds of 5–30 s on 27–28 Oct, no failures, last event 28 Oct 13:53 IST. 0.26.0 predates the first-query build (#704), and on it every background build is killed at 2 minutes, so its lock never reaches 5 minutes. This bug can't be what hung that laptop. It protects 0.27+.

What more likely hung it is memory. assign's map is a 98.6 MB wiring.json, and every process that loads it holds about 1.2 GB:

  • trail's MCP server, one per Claude session: 1.24 GB, held for the whole session
  • trail ask from the prompt hook after an edit: 1.29 GB peak, 2.3 s
  • the end-of-turn build: 2.15 GB peak, 23 s

Estimate: three sessions open on assign hold ~3.7 GB, and each turn end adds ~2.2 GB. This PR makes that build yield the CPU but doesn't lower memory (follow-up 1).

Follow-ups

  1. Cut memory per process on big repos (a compact map, or load only what a query needs).
  2. The MCP server holds the map for the whole session. Dropping a parsed 98.6 MB map freed only ~200 MB of ~590 MB after 30 s, so this needs a design, not an idle timer.
  3. One build at a time per repo across worktrees.
  4. A peak-memory bucket on build_completed and query.
  5. Ask the reporter for laptop RAM, how many Claude sessions were open, and whether Docker was running, then move them to 0.28 with this fix.

Tests

npm test with a scratch HOME: 4125 pass, 9 skipped, 4 fail. Clean main the same way: 4123 pass, 3 fail. None of the failures come from this change: the upkeep-hooks "machine without trail" test fails on main too, and the 3 trail-scenario-hooks tests pass 9/9 in a fresh HOME but fail on a second run in the same HOME (they leave state there), which is how they ran here. Main's other 2 failures pass on this branch. New tests cover a live owner keeping an old lock, a gone owner's lock being reclaimed, the 15-minute cap, lockHeldByOther, and sync-run owning the lock during its build.

trail check flags 3 conflicts with the 27 Oct learning's "Reusing sync-run's 120s limit for a first build". That doesn't happen here: the first build keeps its 10-minute limit. The check matches on the words "sync-run", "first build" and "limit".

🤖 Generated with Claude Code

@trailhq-graft

trailhq-graft Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🌱 graft blast radius

4 areas changed → 9 areas can be affected. 29 dependent symbols, depth 2.
Tests: 2 areas updated their tests.
Tag: @shhdwi — 12 of 13 areas · s4kura — File Locking · @Rakly3 — Sync Run

flowchart TB
  A0(("Hook System<br/>13 symbols"))
  A1(("Graph Freshness<br/>4 symbols"))
  A2(("Cloud Pairing<br/>3 symbols"))
  A3(("Note Storage<br/>2 symbols"))
  A4(("Initialization<br/>2 symbols"))
  AX(("4 smaller areas<br/>5 symbols"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3,A4 reached;
  classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
  class AX tail;
Loading
Can be affected Symbols Nearest hop Reached from
Hook System 13 src/claude/hooks.ts:L1-L580 hooks.ts — imports, depth 1 Telemetry Contract, File Locking
Graph Freshness 4 src/graph/refresh.ts:L134-L159 buildMissingMap — calls, depth 1 File Locking
Cloud Pairing 3 src/cloud/nudge.ts:L1-L503 nudge.ts — imports, depth 1 Telemetry Contract
Note Storage 2 src/notes/memory-notes.ts:L1-L280 memory-notes.ts — imports, depth 1 Telemetry Contract
Initialization 2 src/init/run.ts:L1-L447 run.ts — imports, depth 1 Telemetry Contract
Session Telemetry 2 src/telemetry/sessions.ts:L1-L113 sessions.ts — imports, depth 1 Telemetry Contract
3 smaller areas 3 CLI Skills, Learning Sessions, MCP Tools see below
Who knows this code — 5 people across 13 areas
Area Who knows it
Telemetry Contract · changed @shhdwi — 1 commit, last 6d ago
File Locking · changed @shhdwi — 1 commit, last 6d ago · s4kura — 1 commit, last 7d ago
CLI Entry Point · changed @shhdwi — 51 commits, last yesterday
Sync Run · changed @shhdwi — 3 commits, last 3mo ago · @Rakly3 — 1 commit, last 7d ago
Hook System · affected @shhdwi — 27 commits, last 2d ago
Graph Freshness · affected 贾旭卿 — 1 commit, last 2mo ago
Cloud Pairing · affected @shhdwi — 22 commits, last 2d ago
Note Storage · affected @shhdwi — 4 commits, last 2d ago
…5 further areas

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 29 dependent symbols, grouped by area

Hook System — 13 symbols in 12 files

  • src/claude/hooks.ts:L1-L580 — hooks.ts (imports, depth 1)
    21: import { main as preBashMain } from './pre-bash.js';
  • src/claude/session-metrics.ts:L1-L215 — session-metrics.ts (imports, depth 1)
    36: import { readSession, writeSession, sessionDir, listSessionIds, type SessionState } from './state.js';
  • src/claude/skills-wiring.ts:L1-L116 — skills-wiring.ts (imports, depth 1)
    16: import { hookMsBucket } from '../telemetry/contract.js';
  • src/claude/state.ts:L1-L165 — state.ts (imports, depth 1)
    15: acquireLockIn,
  • src/claude/format.ts:L1-L416 — format.ts (imports, depth 2)
    2: import type { Stats, SessionState } from './state.js';
  • src/claude/hook-io.ts:L1-L140 — hook-io.ts (imports, depth 2)
    9: import { resolveContextDir } from './state.js';
  • src/claude/prompt-hook.ts:L1-L307 — prompt-hook.ts (imports, depth 2)
    42: * ask took 13 s at p95 before this (the lab, on main).
  • src/claude/prompt-skills.ts:L1-L327 — prompt-skills.ts (imports, depth 2)
    34: import { lastUsedIn } from '../skills/state.js';
  • src/claude/skill-use.ts:L1-L80 — skill-use.ts (imports, depth 2)
    14: import { readSession, writeSession } from "./state.js";
  • src/claude/stats.ts:L1-L27 — stats.ts (imports, depth 2)
    4: import type { Stats } from './state.js';
  • src/claude/statusline.ts:L1-L39 — statusline.ts (imports, depth 2)
    26: export function main(): void {
  • src/claude/stop.ts:L1-L216 — stop.ts (imports, depth 2)
    21: import { acquireLock, patchStats, readSession, readStats, writeSession, type SessionState } from './state.js';
  • src/claude/stop.ts:L159-L177 — startSync (calls, depth 2)
    166: const syncRun = process.env.GRAFT_TEST_SYNC_RUN ?? claudeScriptPath('sync-run.js');

Graph Freshness — 4 symbols in 1 file

  • src/graph/refresh.ts:L134-L159 — buildMissingMap (calls, depth 1)
    145: if (!lockHeldIn(lockCache) && acquireLockIn(lockCache) && !startFirstBuild(dir)) releaseLockIn(lockCache);
  • src/graph/refresh.ts:L190-L197 — waitForLock (calls, depth 1)
    193: if (acquireLockIn(cache)) return true;
  • src/graph/refresh.ts:L233-L309 — ensureFreshGraph (calls, depth 2)
    256: const seedNote = seededFrom ? `copied the graph from the main checkout (${seededFrom})` : builtNote;
  • src/graph/refresh.ts:L207-L224 — seedUnderLock (calls, depth 2)

Cloud Pairing — 3 symbols in 3 files

  • src/cloud/nudge.ts:L1-L503 — nudge.ts (imports, depth 1)
    28: import { durationBucket } from "../telemetry/contract.js";
  • src/cloud/lookup-activity.ts:L1-L60 — lookup-activity.ts (imports, depth 2)
  • src/cloud/pair.ts:L1-L368 — pair.ts (imports, depth 2)

Note Storage — 2 symbols in 2 files

  • src/notes/memory-notes.ts:L1-L280 — memory-notes.ts (imports, depth 1)
    18: import { countBucket } from "../telemetry/contract.js";
  • src/notes/pr-reviews.ts:L1-L577 — pr-reviews.ts (imports, depth 1)
    469: if (name.length > 2 && !/^(index|main|mod|init|utils?|types?|__init__)$/i.test(name)) words.add(name.toLowerCase());

Initialization — 2 symbols in 2 files

  • src/init/run.ts:L1-L447 — run.ts (imports, depth 1)
    37: import { countBucket, durationBucket } from '../telemetry/contract.js';
  • src/init/keep.ts:L1-L386 — keep.ts (imports, depth 2)

Session Telemetry — 2 symbols in 2 files

  • src/telemetry/sessions.ts:L1-L113 — sessions.ts (imports, depth 1)
    26: import { readSession, writeSession, sessionDir, listSessionIds } from '../claude/state.js';
  • src/telemetry/track.ts:L1-L162 — track.ts (imports, depth 1)
    21: import { EVENTS, allowedValue } from './contract.js';

CLI Skills — 1 symbol in 1 file

  • src/skills/cli-skills.ts:L1-L376 — cli-skills.ts (imports, depth 2)
    224: if (c.getOptionValueSource(o.attributeName()) === 'cli') cmd.error(`error: unknown option '${o.long ?? o.flags}'`, { exitCode: 1, code: 'commander.unknownOption' });

Learning Sessions — 1 symbol in 1 file

  • src/learn/sessions.ts:L1-L310 — sessions.ts (imports, depth 1)
    27: import { countBucket, durationBucket } from "../telemetry/contract.js";

MCP Tools — 1 symbol in 1 file

  • src/mcp/tools.ts:L1-L337 — tools.ts (imports, depth 2)
    13: import { callersSavings, headerOf, hitLine, looseNoteFor } from '../graph/traverse-cli.js';
Test signal per changed area — 2 ✓ · 2 –

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

  • – Telemetry Contract — 8 test files reach it, none changed here
  • ✓ File Locking — 2 of 7 reached · 1 test file changed here: test/claude-state.test.ts
    • not reached: acquireLockIn, claimLock, lockPid, lockStale, pidAlive
  • – CLI Entry Point — no function, method or class changed here
  • ✓ Sync Run — 1 of 2 reached · 1 test file changed here: test/claude-hooks.test.ts
    • not reached: main
50 test suites also reference this code

50 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

  • test/claude-copy-dashes.test.ts
  • test/claude-format.test.ts
  • test/claude-hooks-1b.test.ts
  • test/claude-hooks-perf.test.ts
  • test/claude-hooks-skills.test.ts
  • test/claude-host-standdown.test.ts
  • test/claude-post-bash-walls.test.ts
  • test/claude-prompt-learnings.test.ts
  • test/claude-prompt-private.test.ts
  • test/claude-session-start-learnings.test.ts
  • test/claude-skills-journey-fixtures.ts
  • test/claude-start-skills-1b.test.ts
  • test/claude-statusline.test.ts
  • test/claude-stop-trace.test.ts
  • test/claude-stop.test.ts
  • test/claude-tally.test.ts
  • test/graph-refresh.test.ts
  • test/hosts-codex-hooks.test.ts
  • test/init-keep-cli-review.test.ts
  • test/init-keep-cli.test.ts
  • …30 more

⚠️ 2 changed files not in the graph (.trail/learnings/2026-10-29-on-a-big-repo-every-trail-anirudhkumar.md, TELEMETRY.md) — no parser claims the extension, or the index predates the file.

graft blast · origin/main...HEAD · depth 2 · 8 changed files

Open the interactive graph → — click an area to see its dependent symbols at file:line.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant