fix(backfill): dedicated embed-backfill arm and a time budget for the query path

[DamieMoon]

Tracking issue, with the full server and llama.cpp logs: https://force-push.me/damienmoon/ctx/issues/1 (my Forgejo; damienmoon/ctx there is a read-only mirror of this fork).

Summary

On a fresh v5.21.1 install with the documented CPU embed backend (Qwen3-Embedding-8B on llama.cpp, about 23 tokens per second), storing one block of about 10k tokens made every following ctx query fail with HTTP 500 after 120 seconds. Two things combine to produce that:

1. The background backfill arm is gated on Dream. backfillOneEmbedding is called only at the top of runDreamLoop. With dream.enabled=false, the registry default, or with Dream switched to DreamModeOff at runtime, no goroutine embeds pending blocks in the background. The query path's backfillPending is the only embedder, and it runs inside the caller's deadline.
2. backfillPending has a count cap but no time budget. embed_backfill.sync_cap bounds how many blocks a request embeds inline, not how long. One block that needs seven minutes consumes the CLI client's entire 120-second deadline; the question's own embed is never admitted (embedcache: admission: context canceled) and the query answers 500 instead of degrading. The failure is memoized as class wire, which means "backend down", although the backend was healthy and only slow for this caller. Every query that lands after the backoff lapses repeats the burn.

This PR adds a dedicated backfill goroutine that runs regardless of Dream, a wall-clock budget for the query path's inline backfill, and a memo class that hands an overrunning block to the background arm instead of parking it on the wire backoff curve.

Observed

Server log of the failing request (POST /api/query through the CLI client, default timeout 120 s):

query pipeline start
query translated                               (1452 ms)
WARN  query backfill: embed failed — memoized  block_id=… class=wire
      error: Post "http://ctx-embed:8081/v1/embeddings": context canceled
INFO  query: backfilled embeddings before search  count=1
ERROR embedding failed  error="embedcache: admission: context canceled"
http request POST /api/query status=500 duration_ms=119997

llama.cpp on the same block: prompt processing, n_tokens = 3072, progress = 0.30, t = 110.30 s / 27.85 tokens per second, then srv stop: cancel task. The same query over /mcp, which carries no deadline, ran 317 seconds, embedded the block and answered 200. Turning Dream on made the scheduler embed the next pending block within 27 seconds, which is what pointed at the gating.

Change

Area	What
internal/events/scheduler.go	runEmbedBackfillLoop, started unconditionally in Run. It does the same work as the call at the top of the dream loop, backfillOneEmbedding under the iterated tenant's snapshot and router, on its own cadence: a successful pick loops at once, an empty pick waits embed_backfill.interval, an error waits 10 s. Under !enforcing() it yields to interactive demand exactly like the dream loop; under Enforcing the wire call waits at its target in its background lease. interval <= 0 turns the arm off; it then polls the config every 15 s so a hot enable needs no restart. Each pick runs under its own guardPanic. The dream loop keeps its inline call; the tx-wrapped FOR UPDATE SKIP LOCKED pick makes two pickers safe (Welle-49).
internal/handler/query.go	backfillPending gets a wall-clock budget. Each wire call runs under context.WithDeadline(ctx, budgetDeadline), and the loop head checks the budget before the next pick. A call cut by the budget or by the request context is classified caller_timeout; the embed context's error is read before its cancel, so a genuine wire error on a live context still classifies wire or oversize. The pick uses the query-path predicate below.
internal/store/embed_failures.go	New class EmbedFailureCallerTimeout = "caller_timeout". RecordEmbedFailure writes it with next_attempt_at = now(), so the block is eligible for the background arm at once while attempts still counts. New EmbedFailureQueryPathExcludedPredicate: the shared backoff exclusion plus last_class = 'caller_timeout', so the query path never re-picks such a block; the background arm keeps the plain predicate. A later wire failure in the arm reclassifies the block as wire with the ordinary curve. last_class is unconstrained TEXT (migration 113, line 11286), so there is no schema change.
internal/config/config.go, descriptions.go	embed_backfill.sync_budget (seconds, default 30, 0 = unbounded, hot, global-only) and embed_backfill.interval (seconds, default 15, 0 = arm off, hot, global-only). Both follow the explicit opt-out convention of sync_cap.
docker-compose.yml	The two env declarations in the embed_backfill.* block, so TestComposeDeclaresEveryRegistryKey holds.
docs/operations.md	One paragraph after the backfill-memo paragraph describing the arm, the budget and the new class.
internal/handler/query.go, one log line	german detected, translating becomes non-english heuristic matched, translating. The heuristic fires on umlauts and stopwords, so Swedish, Danish and Finnish questions land there as well; the translation was right, the label was not.

The branch has three commits: the fix (8e11ca79), a comments-only follow-up that names the tracking issue by URL (83a0aca3), and the bump of the duration-key drift guard in validate_test.go that CI asked for (218d5f84).

Behavior before and after

Situation	Before	After
Dream off, block pending	Never embedded in the background	The arm embeds it within interval
Dream switched off at runtime (DreamModeOff)	Backfill stops with it	The arm is unaffected
A query lands on a block slower than the caller's deadline	500 after the full deadline, memo wire, repeated after the backoff	Loop ends at sync_budget, memo caller_timeout, the question embed runs, the query answers; the arm embeds the block
Same block, next query	Re-picked after the backoff lapses	Never re-picked by the query path
Backend really down	memo wire, exponential backoff	unchanged
Oversize block	memo oversize, infinity	unchanged
sync_budget=0, interval=0	n/a	Pre-PR behavior for each path, individually

The defaults change observable behavior in two ways: a background goroutine now exists in every deployment, and a query spends at most 30 seconds on inline backfill. A deployment that relies on the inline loop draining a backlog in one request can set sync_budget=0.

Tests

Rot/Grün through the production functions, in the style of backfill_synccap_integration_test.go and backfill_headofline_integration_test.go:

• internal/handler/backfill_syncbudget_integration_test.go, TestBackfillPending_SyncBudget_Integration
  • red_unbudgeted_waits_for_the_slow_backend: SyncBudget=0 holds the request for the backend's full 1.5 s delay.
  • budget_cuts_the_slow_block_and_hands_it_to_the_arm: SyncBudget=300ms against a 5 s backend returns inside the budget with the parent context intact; the backend sees the cancel; the memo is caller_timeout with next_attempt_at <= now() and attempts = 1; a second backfillPending with a 10 s budget makes no wire call; the shared predicate still returns the block for the arm.
  • request_cancel_is_caller_timeout_not_wire: a request cancelled mid-embed classifies caller_timeout.
• internal/events/backfill_arm_integration_test.go, TestEmbedBackfillArm_RunsWithoutDream_Integration
  • arm_embeds_a_pending_block_with_dream_off: with StartupConfig{DreamEnabled: false} the arm alone embeds a pending block within 5 s at Interval=100ms.
  • arm_picks_a_caller_timeout_block_at_once: a caller_timeout memo does not hold the arm back.
  • interval_zero_turns_the_arm_off: a pending block stays pending.
• Existing suites re-run green: TestBackfillPending_SyncCap_Integration, TestBackfillPending_MemoSurvivesRequestCancel, TestBackfillPending_ExcludedType, TestBackfillHeadOfLine_*, TestBackfillQI3_Integration, TestBackfillExcludedType_Integration, the store EmbedFailure and ClearEmbedding suites, and the config gates TestComposeDeclaresEveryRegistryKey and TestKeyDescriptionsComplete.

Run in golang:1.27-alpine with CTX_TEST_PG_IMAGE pointing at a locally built PostgreSQL 18 + pgvector + TimescaleDB image: go build ./..., go vet -tags=integration ./internal/{config,store,handler,events}/..., go test -short on the same packages, and the integration suites above. One pre-existing -short failure in that container, TestRebuildViaWorker_TimeoutKillsHangingChild, fails identically on unpatched 38fdb74f (the helper process reports a Go runtime deadlock instead of being killed) and is unrelated to this change.

Live confirmation

Deployed on the host that hit the issue at the fix commit as first pushed, a677b2d7, whose tree is identical to 8e11ca79 on this branch (the branch was re-signed before opening this PR) (48 vCPU VPS, Qwen3-Embedding-8B Q8_0 on CPU, --threads 16, Dream on) and exercised on the path that failed originally, the CLI client with its 120 s timeout:

Time (UTC)	Event
02:07:36	ctx save of a 29,163-character block (7,117 tokens)
02:07:37	ctx query through the same client
02:08:08	inline embed cut at the 30 s budget, memo caller_timeout (embed-backfill llmlog row: 29,998 ms, no tokens); question embed served; search ran
02:08:13	query answered, exit 0, 36 s end to end; the arm picked the block (scheduler: backfilling embedding) 6 s after the memo
02:12:50	scheduler: embedding backfilled, 7,117 tokens in 276,893 ms (25.9 tokens per second)

The same sequence on 38fdb74f answered 500 after 120 s. llama.cpp shows the inline attempt cancelled at 1,024 tokens, so a cut attempt no longer occupies the backend for the full deadline.

Out of scope, noted on the way

• MaxBlockChars = 1500 in internal/llm/synthesize.go truncates every source in the synthesis prompt. A block of the kind docs/using-ctx-effectively.md recommends (a dedicated RLHF-warnings block, loaded with ctx query) is retrieved but not readable through query once it passes 1,500 characters; get by id is the working path. Worth a sentence in that doc or a key. I can file it separately if you prefer.
• The embed client has no per-call timeout by design (embed.go, "No artificial deadline"), so the new arm, like the dream loop's inline call today, has no bound on one embed call. A background_timeout would need a memo path that survives a dead context, since the memo is written in the same transaction as the pick. Left out of this PR on purpose.

I am glad to adjust naming, defaults or the test shape to the project's conventions.