fix(watchdog): checkpoint-operator-diagnostics - #33
Conversation
stephenctw
changed the title
stephenctw
changed the title
|
86b99f3 — diagnose watchdog dirs as recovery dumps |
endersonmaia
left a comment
endersonmaia
left a comment
There was a problem hiding this comment.
Minor comment about the logs. Not blocking, feel free to tackle this on following versions.
I tested the recovery process and the logs are much more detailed helping me to look at the right place.
While at it, I found that some directories had different casing, but unfortunately I coulnd't reproduce it.
Those are the same address:
0x8cD07E9089bbB734f667F5AdaD318299c0A904370x8cd07e9089bbb734f667f5adad318299c0a90437
I think we could agree to normalize to lowercase everywhere, this could be done at another PR.
| " 1. ls " | ||
| .. state_dir | ||
| .. " — expect config.json, head.json, and checkpoints/<safe_block>/", | ||
| " 2. if head.json is missing: run `sequencer-watchdog init` with CARTESI_WATCHDOG_CM_SNAPSHOT_DIR and CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK set to a CM snapshot whose safe block equals the sequencer's current finalized inclusion_block, then run `tick`", | ||
| " 3. schedule `sequencer-watchdog init && sequencer-watchdog tick` (init is a no-op when state is already complete)", | ||
| " 4. if head.json exists but config.json or the selected snapshot is missing/corrupt: wipe state_dir and re-init", | ||
| " 5. ensure CARTESI_WATCHDOG_STATE_DIR is a persistent volume (empty/ephemeral dirs lose head.json across restarts)", | ||
| }, "\n")) |
There was a problem hiding this comment.
I'd avoid multiline error logs and this kind of detailed step by step should be on the docs
so I think we shoud give a detailed error with paths and stuff, like walredy done before and point to a documentation with this step by step, also like you did before like See docs/snapshots/lifecycle.md and docs/recovery/cockroach.md.
|
@endersonmaia I've made all changes you requested, please take a look! |
04ef167 to
757882d
Compare
Do not burn the tick retry budget on an operator/state error that cannot self-heal; surface an init hint and write status.prom promptly.
Include state_dir, expected layout, init&&tick, wipe, and volume hints so operators can repair from stderr alone.
Pointing setup --recovery at a watchdog CM checkpoint produced a bare ENOENT on info.toml. Detect that layout and tell operators to use a sequencer dumps/<id>/ directory instead.
Keep the load-failure error short (paths + init hint) and move the step-by-step repair into operator-deployment.md. Normalize app and input-box addresses to lowercase so EIP-55 vs lower forms do not diverge in config.json or status.prom labels.
757882d to
174b7bc
Compare
2 participants
Summary
head.jsonas a non-retryable operator/state error sotickexits once instead of burning the default 3×5s retry budget.state_dir, expected layout,init && tick, wipe/re-init, and persistent-volume troubleshooting so operators can repair from stderr alone.status.promis rewritten promptly tostate="warning"on this path.Context
Production ticks were failing with
failed to load watchdog head: missing head.jsonafter loadingconfig.json. That means incomplete state (no checkpoint pointer).ticknever bootstraps a head from env; retrying cannot create one. Retries delayed exit and could leave metrics stale if the process was killed mid-retry.Test plan
lua watchdog/tests/run.lua(includes new cases: no retry on missing head; tick writesstatus.promwarning once; enriched error assertions)