feat(engine): support { comments: true } for ctx.ast() Ruby

.archgate/adrs/ARCH-022-ast-aware-rule-context.md

@@ -63,12 +63,13 @@ This method dispatches internally based on `language`, and the dispatch mechanis
 - All four guardrails still run, in order, inside `ast()` for a base parse: path safety (`safePath` on the original path — which also yields the repo-relative form `git show` needs), language plausibility (`AST_LANGUAGE_EXTENSIONS` on the original path), interpreter probe, guarded invocation.
 - TypeScript/JavaScript base source is parsed in-process by the same `meriyah` path (`parseTsOrJsSource`, factored into `src/engine/js-parser.ts` and shared with the working-tree branch). Python/Ruby base content is not on disk, so it is written to a throwaway OS-temp file — outside the project tree, therefore outside any cwd-derived load path — and handed to the **same, unchanged** `PYTHON_AST_PROGRAM`/`RUBY_AST_PROGRAM` with the **same mandatory `-I` isolation**. The `python-subprocess-isolated` invariant is unchanged.
 
-**Comment access (`{ comments: true }`).** `AstOptions` carries a second field, `comments?: boolean`. With `{ comments: true }`, the returned tree carries a `comments` array of `CommentToken` (`{ type: "line" | "block"; value: string; loc: { start, end } }`) — structured comment data for comment-governance rules (comment length, style, content), in place of the line-by-line regex heuristics that were the only option while the AST exposed no comments at all. `typescript`, `javascript`, and `python` are supported; `ast(path, "ruby", { comments: true })` throws a clear "not supported yet" error (Ripper.lex support is a deferred follow-up). This capability MUST fold into `ast()` — attaching `comments` to the returned tree — rather than a separate method, so it runs inside the same four-guardrail flow and `single-ast-method` stays satisfied (no new method, the catch-all signature unchanged).
+**Comment access (`{ comments: true }`).** `AstOptions` carries a second field, `comments?: boolean`. With `{ comments: true }`, the returned tree carries a `comments` array of `CommentToken` (`{ type: "line" | "block"; value: string; loc: { start, end } }`) — structured comment data for comment-governance rules (comment length, style, content), in place of the line-by-line regex heuristics that were the only option while the AST exposed no comments at all. All four languages are supported (Ruby comments come from a second `Ripper.lex` pass — see below). This capability MUST fold into `ast()` — attaching `comments` to the returned tree — rather than a separate method, so it runs inside the same four-guardrail flow and `single-ast-method` stays satisfied (no new method, the catch-all signature unchanged).
 
 - `value` has its delimiters removed (`//`, `/* … *​/`, `#`); `type` is `"line"` or `"block"`. Python has only `"line"` comments — `#`; it has no block comments, and `"""` docstrings are string expressions in the `ast` tree, NOT comments.
 - **TS/JS comments are extracted from the ORIGINAL source, not the tree** (`extractJsComments` in `js-parser.ts`). This is a direct consequence of the transpiled-`loc` note above: `Bun.Transpiler` strips comments before `meriyah` ever sees them, so they cannot come from the ESTree tree. Scanning the pre-transpile source has a genuine advantage — a comment's `loc` is accurate against the original `.ts` file, unlike the tree's own transpiled-relative `loc`. The scanner is string/template-literal aware (a `//` or `/*` inside a string is not a comment) but does NOT track regex literals, so a comment delimiter inside a regex literal (e.g. `/foo\/\//`) is a known blind spot, consistent with the scanner in `source-positions.ts`.
 - **Python comments come from the `tokenize` module** (the `ast` tree has none), via a second serializer, `PYTHON_AST_WITH_COMMENTS_PROGRAM`, that prints `{ _tree, comments }`; the engine unwraps it to the module tree with `comments` attached. That serializer shares the same `convert()` preamble as the base program and runs under the same mandatory `-I` isolation — the `python-subprocess-isolated` invariant is unchanged. Tokenizer errors on otherwise-parseable source degrade to an empty comment list rather than failing the parse.
-- **No new subprocess site and no new guardrail.** Comment support rides entirely inside `ast()`'s existing four-guardrail flow: the only additions are an in-process source scan for TS/JS and an alternate Python serializer program selected inside the same guarded invocation with the same `-I`. Because it folds into `ast()` rather than a new method, `ast-guardrail-ordering` already covers it.
+- **Ruby comments come from a second `Ripper.lex` pass** (the `Ripper.sexp` tree has none), via `RUBY_AST_WITH_COMMENTS_PROGRAM`, which prints the same `{ _tree, comments }` envelope; the engine unwraps it with `comments` riding on the sexp array as a non-index property. `#` comments are `"line"` tokens; each `=begin`/`=end` region is ONE `"block"` token whose `value` is the inner content (marker lines stripped) with line endings normalized to LF, so a CRLF file yields the same value on every OS. Ripper reports columns as byte offsets; comment `loc` columns are converted to character offsets to match the Python/TS convention — the sexp tree's own node positions stay byte-based, as Ripper emits them. Lex errors on otherwise-parseable source degrade to an empty comment list, matching the Python fallback.
+- **No new subprocess site and no new guardrail.** Comment support rides entirely inside `ast()`'s existing four-guardrail flow: the only additions are an in-process source scan for TS/JS and alternate Python/Ruby serializer programs selected inside the same guarded invocation (Python keeping the same `-I`). Because it folds into `ast()` rather than a new method, `ast-guardrail-ordering` already covers it.
 
 **Explicit non-goal: cross-language AST shape unification.** `ctx.ast()` unifies the call site and the failure contract across languages. It does NOT unify the shape of the returned tree. TypeScript/JavaScript returns ESTree-shaped nodes (via `meriyah`); Python returns whatever the standard `ast` module's own node schema produces; Ruby returns `Ripper`'s native s-expression shape. A rule author writing a Python check and a rule author writing a Ruby check are working against two different, language-native grammars, and must know the target language's own AST vocabulary. This ADR accepts that trade explicitly in exchange for avoiding the dependency and distribution cost of a unifying parser (see the tree-sitter alternatives above); it is not a limitation to be silently discovered later.
 
@@ -107,7 +108,7 @@ This method dispatches internally based on `language`, and the dispatch mechanis
 - **DON'T** read a base revision by spawning git or an interpreter from a `.rules.ts` file — that is exactly the sandbox escape [ARCH-024](./ARCH-024-rule-file-sandbox-boundary.md) blocks; `ast(path, language, { rev: "base" })` and `fileAtBase()` are the sanctioned path.
 - **DON'T** return `null` from `ast({ rev: "base" })` for the no-base or added-file cases — those MUST throw, distinguishably; only `fileAtBase()` reports them as `null`.
 - **DON'T** expect a `comments` array without `{ comments: true }` — it is opt-in and absent otherwise.
-- **DON'T** request `{ comments: true }` for `"ruby"` yet — it throws; Ruby comment support (Ripper.lex) is a deferred follow-up.
+- **DON'T** expect Ruby comment `loc` columns to line up with the sexp tree's own node positions — comment columns are character offsets; Ripper's node positions are byte offsets.
 - **DON'T** treat the TS/JS comment scan as regex-literal aware — a comment delimiter inside a regex literal is a known blind spot.
 
 ## Consequences
@@ -156,7 +157,7 @@ This method dispatches internally based on `language`, and the dispatch mechanis
 
 The base-revision surface (`ast(path, language, { rev: "base" })` and `fileAtBase()`) is covered by the four rules above unchanged — it adds no new subprocess site and no new guardrail — plus behavioural coverage in `tests/engine/runner-ast-base.test.ts` (base parsing per language, comment-only structural equivalence, and the throw-vs-null semantics of `ast({ rev: "base" })` versus `fileAtBase()`).
 
-The comment surface (`ast(path, language, { comments: true })`) is likewise covered by the four rules unchanged — it folds into `ast()`, adds no new subprocess site, and reuses the same guarded invocation — plus behavioural coverage in `tests/engine/runner-ast-comments.test.ts` (TS line/block extraction with delimiter-stripped values and original-source `loc`, string-literal awareness, JavaScript, the opt-in-only absence of `comments`, the Ruby "not supported yet" throw, and Python `tokenize` extraction including the `#`-inside-a-string exclusion and composition with `{ rev: "base" }`).
+The comment surface (`ast(path, language, { comments: true })`) is likewise covered by the four rules unchanged — it folds into `ast()`, adds no new subprocess site, and reuses the same guarded invocation — plus behavioural coverage in `tests/engine/runner-ast-comments.test.ts` (TS line/block extraction with delimiter-stripped values and original-source `loc`, string-literal awareness, JavaScript, the opt-in-only absence of `comments`, Ruby `Ripper.lex` extraction (line/block tokens), and Python `tokenize` extraction including the `#`-inside-a-string exclusion and composition with `{ rev: "base" }`). The Ruby character-offset columns (versus Ripper's byte offsets) and LF-normalized block values are pinned at the serializer level in `tests/engine/ast-support.test.ts`.
 
 ### Manual Enforcement
 

docs/public/llms-full.txt

@@ -5397,7 +5397,7 @@ Reach for `fileAtBase()` first when you need to detect the no-base or added-file
 
 ##### Collecting comments (`{ comments: true }`)
 
-With `{ comments: true }`, the returned tree carries a `comments` array -- structured comment data for comment-governance rules, in place of line-by-line regex. Supported for `typescript`, `javascript`, and `python`; requesting it for `ruby` throws.
+With `{ comments: true }`, the returned tree carries a `comments` array -- structured comment data for comment-governance rules, in place of line-by-line regex. Supported for all four languages. For `ruby`, the returned tree is an array (`Ripper.sexp` output), so the `comments` array rides on it as a non-index property.
 
 ```typescript
 interface CommentToken {
@@ -5424,7 +5424,7 @@ for (const comment of tree.comments ?? []) {
 }
 ```
 
-Comment positions are accurate against the **original source**, even for TypeScript. This is a deliberate advantage over the tree's own `loc`, which is transpiled-relative for TypeScript (see [AstNode](#astnode)): comments are scanned from the pre-transpile source, so their `loc` never drifts. Python comments are always `type: "line"` (`#`) -- Python has no block comments, and `"""` docstrings are string expressions in the tree, not comments. The TypeScript/JavaScript scanner is string- and template-literal-aware, but does not track regular-expression literals, so a comment delimiter inside a regex literal is a known blind spot.
+Comment positions are accurate against the **original source**, even for TypeScript. This is a deliberate advantage over the tree's own `loc`, which is transpiled-relative for TypeScript (see [AstNode](#astnode)): comments are scanned from the pre-transpile source, so their `loc` never drifts. Python comments are always `type: "line"` (`#`) -- Python has no block comments, and `"""` docstrings are string expressions in the tree, not comments. Ruby `#` comments are `type: "line"`; each `=begin`/`=end` documentation block is a single `type: "block"` token whose `value` is the inner content (the `=begin`/`=end` marker lines stripped, analogous to `/* */` delimiter stripping) and whose `loc` spans the `=begin` line through the `=end` line. Ruby comment `loc` columns are character offsets, consistent with the other languages (Ripper's own sexp node positions are byte offsets), and block `value` line endings are normalized to LF regardless of the source file's line endings. For Python and Ruby, comment extraction is a second pass over the same source (`tokenize` / `Ripper.lex`); a tokenizer failure on otherwise-parseable source degrades to an empty comment list rather than failing the parse. The TypeScript/JavaScript scanner is string- and template-literal-aware, but does not track regular-expression literals, so a comment delimiter inside a regex literal is a known blind spot.
 
 ##### Failure behavior
 
@@ -5538,7 +5538,7 @@ type AstLanguage = "typescript" | "javascript" | "python" | "ruby";
 type AstNode = Record<string, unknown> | unknown[];
 ```
 
-When parsed with [`{ comments: true }`](#ast), the root node also carries a `comments: CommentToken[]` array (`typescript`, `javascript`, and `python` only). It is absent otherwise.
+When parsed with [`{ comments: true }`](#ast), the root node also carries a `comments: CommentToken[]` array (all four languages; for `ruby` it is a non-index property on the root sexp array). It is absent otherwise.
 
 | Language     | Backing parser                                                                                                     | Returned shape                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | ------------ | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

docs/src/content/docs/nb/reference/rule-api.mdx

@@ -243,7 +243,7 @@ Bruk `fileAtBase()` først når du trenger å oppdage tilfellene uten basis elle
 
 ##### Samle inn kommentarer (`{ comments: true }`)
 
-Med `{ comments: true }` bærer det returnerte treet en `comments`-matrise -- strukturert kommentardata for kommentarstyringsregler, i stedet for linje-for-linje regulære uttrykk. Støttet for `typescript`, `javascript` og `python`; å be om det for `ruby` kaster.
+Med `{ comments: true }` bærer det returnerte treet en `comments`-matrise -- strukturert kommentardata for kommentarstyringsregler, i stedet for linje-for-linje regulære uttrykk. Støttet for alle fire språk. For `ruby` er det returnerte treet en matrise (`Ripper.sexp`-utdata), så `comments`-matrisen ligger på den som en ikke-indeksert egenskap.
 
 ```typescript
 interface CommentToken {
@@ -270,7 +270,7 @@ for (const comment of tree.comments ?? []) {
 }
 ```
 
-Kommentarposisjoner er nøyaktige mot **originalkilden**, selv for TypeScript. Dette er en bevisst fordel fremfor treets egen `loc`, som er transpilert-relativ for TypeScript (se [AstNode](#astnode)): kommentarer skannes fra kildekoden før transpilering, så deres `loc` driver aldri. Python-kommentarer er alltid `type: "line"` (`#`) -- Python har ingen blokkommentarer, og `"""`-dokstrenger er strengeuttrykk i treet, ikke kommentarer. TypeScript/JavaScript-skanneren er streng- og malstreng-bevisst, men sporer ikke regulært-uttrykk-literaler, så et kommentarskilletegn inne i en regex-literal er et kjent blindpunkt.
+Kommentarposisjoner er nøyaktige mot **originalkilden**, selv for TypeScript. Dette er en bevisst fordel fremfor treets egen `loc`, som er transpilert-relativ for TypeScript (se [AstNode](#astnode)): kommentarer skannes fra kildekoden før transpilering, så deres `loc` driver aldri. Python-kommentarer er alltid `type: "line"` (`#`) -- Python har ingen blokkommentarer, og `"""`-dokstrenger er strengeuttrykk i treet, ikke kommentarer. Ruby `#`-kommentarer er `type: "line"`; hver `=begin`/`=end`-dokumentasjonsblokk er ett enkelt `type: "block"`-token der `value` er det indre innholdet (markørlinjene `=begin`/`=end` er fjernet, tilsvarende fjerningen av `/* */`-skilletegn) og der `loc` spenner fra `=begin`-linjen til og med `=end`-linjen. Kolonnene i `loc` for Ruby-kommentarer er tegnposisjoner, i samsvar med de andre språkene (Rippers egne sexp-nodeposisjoner er byteposisjoner), og linjeskift i `value` for blokker normaliseres til LF uavhengig av kildefilens linjeskift. For Python og Ruby er kommentaruthentingen et eget pass over samme kilde (`tokenize` / `Ripper.lex`); en tokeniseringsfeil på ellers parsbar kode degraderer til en tom kommentarliste i stedet for å feile hele parsingen. TypeScript/JavaScript-skanneren er streng- og malstreng-bevisst, men sporer ikke regulært-uttrykk-literaler, så et kommentarskilletegn inne i en regex-literal er et kjent blindpunkt.
 
 ##### Feiloppførsel
 
@@ -385,7 +385,7 @@ type AstLanguage = "typescript" | "javascript" | "python" | "ruby";
 type AstNode = Record<string, unknown> | unknown[];
 ```
 
-Når parset med [`{ comments: true }`](#ast), bærer rotnoden også en `comments: CommentToken[]`-matrise (kun `typescript`, `javascript` og `python`). Den er fraværende ellers.
+Når parset med [`{ comments: true }`](#ast), bærer rotnoden også en `comments: CommentToken[]`-matrise (alle fire språk; for `ruby` er den en ikke-indeksert egenskap på rot-sexp-matrisen). Den er fraværende ellers.
 
 | Språk        | Underliggende parser                                                                                                     | Returnert form                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | ------------ | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |