Skip to content

veche: unvalidated --option names make seats fail silently (COULD_NOT_DECIDE) unless they are short tokens #2136

Closed

Description

@remigiusz-antczak

What happened

--option accepts any non-empty text: the only checks are distinct names and at most 8 including do-nothing (.claude/skills/idea-critic/scripts/veche.py:170-172). The names are sent to every seat as one line, OPTIONS: a; b; c (:193), and each seat must echo every name verbatim as a key of its scores object (:195); scores_ok then looks the names up one by one (:133-137). A name containing ; makes that line ambiguous, and a sentence-length name (the docs suggest --option "<the claim>", references/council.md:22,34) is hard for a model to echo exactly. In our runs with claude -p seats, sentence-length option names came back keyed a and b; every seat was dropped as "unparseable or malformed answer" and the council ended COULD_NOT_DECIDE with only N of M seat answers parseable. --dry-run accepts the same arguments, so the mistake shows up only after the seats have run. Neither the --option help (:359) nor the docs state a naming rule. (Punctuation in --claim and --steelman is not affected; they are inserted as plain text.)

What you expected

A clear error at start-up, also under --dry-run, for example --option names must match [A-Za-z0-9][A-Za-z0-9_-]{0,39}; got 'ship it now; then measure', and help text that states the rule.

How to reproduce

Save as seat.py, a stand-in for a model that splits the OPTIONS: line on ; and scores every piece:

import json, sys
prompt = sys.stdin.read()
line = next(l for l in prompt.splitlines() if l.startswith("OPTIONS: "))[len("OPTIONS: "):]
scores = {name: {"impact": 3, "cost": 3} for name in line.split("; ")}
print(json.dumps({"verdict": "REVISE", "claims": [], "would_change_it": "x", "premortem": "x",
                  "kill_criterion": "x", "leading_metric": "x", "scores": scores}))
V=/path/to/perun/.claude/skills/idea-critic/scripts/veche.py
python3 $V --cmd "python3 seat.py" --claim c --steelman s --option "ship it now; then measure" --criteria impact:1,cost:1 --dry-run
python3 $V --cmd "python3 seat.py" --claim c --steelman s --option "ship it now; then measure" --criteria impact:1,cost:1

Observed: the dry run exits 0 with "verdict": "DRY_RUN". The real run exits 3 with "reason": "only 0 of 3 seat answers parseable", because the seat returned the keys ship it now and then measure, not ship it now; then measure.

Proposed fix (smallest)

Validate in the option handling at :170-172: reject names that do not match a short token pattern (the exact rule is the maintainers' call) with a Stop that names the offending option; this also covers --dry-run. Change the --option help (:359) and the examples in references/council.md (:22, :34) to short tokens, and say that the claim itself goes in --claim ("include the claim" currently invites a sentence). If free-text option names are wanted instead, send labelled options (option-1: <text>) and key the scores by label, as the critique round already does for answers (:281-285).

Environment

  • deep-code-review version (from .claude/skills/deep-code-review/VERSION): 1.620.0
  • Host agent (Claude Code / Cursor / Codex / other): Claude Code
  • OS: Linux

Confidentiality

This report contains no secrets, real names, internal IDs, or private hostnames.

Activity

  1. remigiusz-antczak commented on Oct 30, 2026

    @remigiusz-antczak
    CollaboratorAuthor

    Fixed across #2145 (release 1.622.0) and #2147 (release 1.623.0).

    --option names must now match [A-Za-z0-9][A-Za-z0-9_-]{0,31}. Any other name stops the run at start-up with an error that quotes the rule and the bad name, and --dry-run refuses it the same way. The --option help states the rule, and references/council.md states it next to the short-token examples (--option ship --option wait). Covered by scripts/test_veche_input.py and scripts/test_veche_surface.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:decisionsDecision quality and roadmapbugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions