Skip to content

[Bug]: Chained comparisons such as 0 <= x <= 10 in workflow expressions silently return the wrong result #5071

Open

Description

@hnipps

Bug Description

The workflow expression evaluator accepts a chained comparison such as {{ 0 <= inputs.n <= 10 }} but does not evaluate it as a range check. It splits the expression at the first comparison operator it finds and evaluates the whole remainder as the right operand. As a result, 0 <= inputs.n <= 10 is evaluated as 0 <= (inputs.n <= 10), which compares a number with a boolean. Validation accepts the condition, and nothing is raised at run time, so an if, while or do-while step can silently take the wrong branch.

The splitting happens in _evaluate_simple_expression (src/specify_cli/workflows/expressions.py:908-912). The operators are tried in the fixed order of _COMPARISON_OPERATORS (expressions.py:715), not left to right.

Steps to Reproduce

  1. From a fresh clone of spec-kit, run uv sync.
  2. Save the script below as repro.py in the repository root.
  3. Run uv run python -I repro.py.
import pathlib
import tempfile

from specify_cli.workflows.base import StepContext
from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine, validate_workflow
from specify_cli.workflows.expressions import evaluate_condition

for n in (5, 50):
    ctx = StepContext(inputs={"n": n})
    for cond in ("{{ 1 < inputs.n < 10 }}", "{{ 0 <= inputs.n <= 10 }}"):
        print(f"n={n}: {cond} -> {evaluate_condition(cond, ctx)}")

wf = WorkflowDefinition.from_string("""
schema_version: "1.0"
workflow: {id: chained, name: chained, version: "1.0.0"}
inputs:
  n: {type: number, default: 50}
steps:
  - id: range
    type: if
    condition: "{{ 0 <= inputs.n <= 10 }}"
    then:
      - {id: inside, type: shell, run: "echo inside"}
    else:
      - {id: outside, type: shell, run: "echo outside"}
""")
print("validate_workflow errors:", validate_workflow(wf))
project = pathlib.Path(tempfile.mkdtemp())
(project / ".specify").mkdir()
state = WorkflowEngine(project).execute(wf, {"n": "50"})
print("status:", state.status.value, "| steps run:", sorted(state.step_results))

Expected Behavior

The module docstring describes the evaluator as "a safe Jinja2 subset" (src/specify_cli/workflows/expressions.py:3), and the docstring of _evaluate_simple_expression lists the comparisons as supported (expressions.py:787). Jinja2 and Python both chain comparisons, so 1 < 5 < 10 is true and 0 <= 50 <= 10 is false. With n = 50, the if step should run the outside branch. If chaining is not meant to be supported, it should be rejected with an error rather than evaluated with a different meaning.

Actual Behavior

n=5: {{ 1 < inputs.n < 10 }} -> False
n=5: {{ 0 <= inputs.n <= 10 }} -> True
n=50: {{ 1 < inputs.n < 10 }} -> False
n=50: {{ 0 <= inputs.n <= 10 }} -> True
validate_workflow errors: []
status: completed | steps run: ['inside', 'range']

1 < 5 < 10 evaluates to false. 0 <= 50 <= 10 evaluates to true, so the workflow run with n = 50 takes the then branch (inside). validate_workflow reports no errors.

Specify CLI Version

1.2.4.dev0 (main at 27d14e2)

AI Agent

Not applicable

Operating System

Linux 6.18 (Ubuntu container)

Python Version

Python 3.12.3

Additional Context

The evaluator raises an error for other ambiguous forms, such as a comparison after a filter (docs/reference/workflows.md, "Trailing comparisons after a filter are rejected"). A chained comparison could be treated the same way, or evaluated pairwise.

Related: #5060 proposes replacing the hand-written expression parser with a grammar so that invalid input raises. It does not mention this case.

Also reproduced on macOS (Darwin 25.6, arm64) with Python 3.14.4 and uv run python -I repro.py against 27d14e2; the output matches the above.

AI Disclosure

Found and reproduced with Claude Code (model: Claude Opus 4.8), run autonomously under human supervision. The human reporter re-ran every reproduction above before filing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage-must-haveVerdict: high-value, important work for Spec Kit — do first

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions