Skip to content

Make the text DSL a first-class, runnable graph description - #8

Open
psyinf wants to merge 2 commits into
mainfrom
feature/dsl-first-class
Open

psyinf wants to merge 2 commits into
mainfrom
feature/dsl-first-class

Conversation

@psyinf

@psyinf psyinf commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Summary

Until now the text DSL could only be parsed, validated and drawn; running a runtime graph still required JSON. This PR makes the DSL executable and the primary way to describe runtime graphs. The JSON format stays fully supported.

Changes

Running DSL graphs

  • DslFilterGraph<In, Out> (DslFilterGraph.hpp) builds, type-checks and runs named-edge DSL graphs, and is itself a MessageFilter, so it can be nested or registered as a stage.
    • Stages run in source order, waiting until their inputs have been produced.
    • Every reader of an edge gets its own copy (fan-out).
    • A dropped message (std::nullopt) skips everything downstream.
    • Merges receive an empty slot ("hole") for each dropped edge.
    • Values routed to end are discarded.
  • The output type sets the graph's output shape: a concrete type (exactly one out), GraphOutputs (several, optionally keyed outputs of different types) or Void (sink-only).
  • MergeFilter / registerMergeFilter provide fan-in stages for (a, b) -> Merge, using the same combiner signature as JoinFilter.
  • Construction checks the whole graph up front: unknown stages, bad arguments, type mismatches along every edge, fan-in and Void misuse, and the output shape. All problems are thrown together in one GraphError with line:column; validateDslGraph<In, Out>(text) returns them without throwing.

Parser

  • dsl::parseGraphProgram now uses the lexy-based parser, which moved into GraphLang.hpp.
  • Syntax errors are located and descriptive, e.g. 1:17: expected '->' but found 'Print'. Previously the lexy path reported every problem as "syntax error" at column 1.
  • A syntax error ends its statement, so there are no follow-on errors.
  • New checks: malformed or out-of-range numbers, empty groups (), and unterminated strings, which no longer swallow the following lines.
  • Added support for decimal numbers, string escapes and # inside strings.
  • Deprecated: the hand-written parser moved to GraphLangHandwritten.hpp as [[deprecated]] dsl::parseGraphProgramHandwritten. Tests check it against the lexy parser for identical output and diagnostics. parseGraphProgramLexy remains as an alias.
  • A stage written without arguments now receives {} instead of null, matching JSON stages without "config".
  • toMermaid uses generated node ids (no clashes with end or internal $out edges), labels outputs and lists stage arguments.

Docs and example

  • README, EXAMPLE.md and apps/textPipeline lead with the DSL: tap, merge, named outputs, diagnostics, and the same pipeline in JSON.
  • Corrected two errors in EXAMPLE.md: the main-path output was shown in uppercase, and the AB branch line was missing.
  • README now lists lexy as a dependency.
  • CHANGELOG has an Unreleased entry.

Testing

  • Windows / MSVC (VS 18), windows-msvc-debug-developer-mode preset (ASan, /W4): 55/55 tests pass, no compiler warnings; the textPipeline output matches EXAMPLE.md.
  • New tests cover execution semantics, GraphOutputs, Void graphs, nesting, every build-time check, and parser diagnostics: 20 malformed inputs with exact expected messages, run through both parsers.
  • The README code snippets were compiled and run separately.
  • Not built locally with GCC or Clang; CI covers linux-gcc.

Notes

  • GraphLang.hpp (and therefore DslFilterGraph.hpp) now includes lexy. Consumers of the CMake target are unaffected, since lexy is an external include there. Projects adding lexy's headers as a regular include path at /W4 will see MSVC warning C4310 from inside lexy.
  • Known limitation: stage arguments are flat key=value pairs; nested objects and lists are not expressible yet.

… format

- DslFilterGraph<In, Out> builds, type-checks and runs named-edge DSL graphs:
  fan-out copies, fan-in merges with holes for dropped paths, drop
  propagation and dead ends; GraphOutputs for keyed multi-output graphs and
  Void for sink-only graphs
- MergeFilter / registerMergeFilter for fan-in stages
- validateDslGraph and GraphError report every build problem with line:column
- lexy parser: located, descriptive syntax errors identical to the
  hand-written parser, real term columns, decimals, escapes and '#' in strings
- both parsers: one syntax error per statement, no follow-on errors; checks
  for malformed numbers, empty fan-in groups and unterminated strings
- toMermaid: generated node ids, output labels and stage arguments
- README, EXAMPLE.md and apps/textPipeline lead with the DSL; the JSON format
  stays supported
… hand-written parser

- dsl::parseGraphProgram (and so DslFilterGraph and validateDslGraph) now
  runs on the lexy-based parser, which moved into GraphLang.hpp; results and
  diagnostics are unchanged
- the hand-written tokenizer/parser moved to GraphLangHandwritten.hpp as
  [[deprecated]] dsl::parseGraphProgramHandwritten, to be removed later
- GraphLangLexy.hpp keeps parseGraphProgramLexy as an alias
- parity tests compare the lexy parser against the deprecated one
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant