Skip to content

Skip ERB comments and <%% literals when converting ERB to Ruby - #66

Merged
dduugg merged 4 commits into
rubyatscale:mainfrom
iMacTia:fix-erb-multiline-comments
Oct 1, 2026
Merged

dduugg merged 4 commits into
rubyatscale:mainfrom
iMacTia:fix-erb-multiline-comments

Conversation

@iMacTia

@iMacTia iMacTia commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #42.

pks turns an ERB file into Ruby by joining the contents of its tags, and it treats <%# %> comments like any other tag. Only the first line of a multi-line comment ends up as a Ruby comment, and the rest is parsed as code. Depending on the text, that loses the references before the comment, the ones after it, or the whole file, with no error. A constant named on a later comment line is reported as a reference.

Approach

Comment tags are now dropped before the join, which is what packwerk's ERB parser does. The check is on the whole tag, not the captured contents, because only a tag that opens with exactly <%# is a comment. <%-# and <% # are code tags in Erubi and in packwerk, and their later lines are real Ruby. The default and experimental parsers both go through this one function.

Where to look

  • Dropping single-line comments changes one case, which the issue says should behave as today. When a file's first tag is an encoding comment such as <%# encoding: iso-8859-1 %>, it used to become the first line of the Ruby source, where lib-ruby-parser read it as a magic comment. For any encoding but UTF-8 or ASCII-8BIT, pks then reported nothing for the file. Those files are now checked, so they can show new violations. Rails strips that tag before compiling and packwerk ignores it, so this matches both. The CHANGELOG says so and there's a test for it.
  • Tags opening with <%%, ERB's escape for a literal <%, are dropped as well. pks used to parse their contents as Ruby, which lost the references after them and sometimes all of them. This follows Erubi, which Rails renders with: it outputs the whole tag as text. packwerk parses the literal's contents as Ruby, so a file with <%%= fails to parse and other literals can report constants from inside them. This is one place pks deliberately doesn't match it.
  • hash_after_hyphen_or_space_is_code_not_comment passes on main. It fails if the check moves to the capture, which would treat <%-# and <% # as comments.

Notes

The cache used to reuse an entry whenever the file's content matched, so after an upgrade an ERB file that hadn't changed would keep its old result until someone ran pks delete-cache. Cache entries now include the pks version in their content digest, so the first run after an upgrade is a cold run for every file, not just ERB. That's a separate commit with its own CHANGELOG entry.

Sent with Claude Code

convert_erb_to_ruby_without_sourcemaps joins the contents of every ERB
tag into one Ruby source. The first line of a `<%#` comment starts with
`#`, so it became a Ruby comment, but every later line became Ruby code.
What that did depended on the text: a trailing period turned the next
tag into a method call, an apostrophe opened a string that swallowed the
rest of the file, and `def x` made the whole file fail to parse. So
references were missed with no error, and a constant named on a later
comment line was reported as a reference.

Comment tags are now dropped before the join, as packwerk's ERB parser
does. Only a tag that opens with exactly `<%#` is a comment. `<%-#` and
`<% #` are code tags in Erubi and in packwerk, and their later lines are
Ruby, which is why the check is on the whole match and not the capture.
Both ERB parsers go through this function, so both are fixed.

Single-line comments are dropped too, which changes one case. A leading
encoding comment such as `<%# encoding: iso-8859-1 %>` used to become
the first line of the Ruby source, where lib-ruby-parser read it as a
magic comment, and for any encoding but UTF-8 or ASCII-8BIT pks reported
nothing for the file. Rails strips that tag before compiling
(ActionView's ENCODING_TAG) and packwerk ignores it, so those files are
now checked. Every other single-line comment was already a Ruby comment,
so dropping it changes nothing. ERB references are all reported at 0:0,
so the shorter Ruby source doesn't move any location.

The first three new tests fail on main. The fourth passes on main and
fails if the check moves to the capture.

Cached results aren't tied to the pks version, so an unchanged ERB file
keeps its old result after an upgrade. The CHANGELOG entry tells users
to run `pks delete-cache`.

Fixes rubyatscale#42.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iMacTia
iMacTia requested a review from a team as a code owner September 28, 2026 17:35

@dduugg dduugg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a clean fix, and the reasoning in the description checks out against the upstream code. BetterHtml, which packwerk's ERB parser uses, and Erubi both treat a tag as a comment only when # comes straight after <%. capture[0].starts_with("<%#") gives the same answer as both for <%#, <%-#, <% #, <%=#, <%#= and <%#-.

I ran each of the seven texts in the first test against main on its own, and every one fails there. That includes "second (line" and "class foo", which lose the references before the comment. The first three tests fail on main, and the fourth fails if the filter moves to the capture. The encoding explanation holds up too. Without a custom decoder, lib-ruby-parser accepts only UTF-8, ASCII-8BIT and BINARY, pks doesn't set a decoder, and a missing AST becomes an empty result for the whole file. The repro from #42 misses the privacy violation on main and reports it here, with the default parser and with --experimental-parser. clippy, fmt and CI are clean, and it merges cleanly with #65.

A note on the CHANGELOG and a suggestion for the cache.

CHANGELOG

The entry only says "may show new violations" for the encoding case, but the multi-line fix changes results in both directions too. References that used to be missed are now reported. A constant named on a later line of a comment that someone recorded in package_todo.yml is no longer found, so that entry goes stale and pks check fails until pks update removes it. One sentence covering both, with pks update as the remedy, would help anyone upgrading.

Cache

I'd rather not ask everyone to run pks delete-cache. Anyone who keeps tmp/cache between CI runs, which is common because of bootsnap, won't see the note, and the next parser fix would need the same paragraph. Adding the pks version to the stored content digest in EmptyCacheEntry::new (src/packs/caching/mod.rs:32) handles it:

let file_contents_digest = format!(
    "{}-{}",
    file_content_digest(filepath)?,
    env!("CARGO_PKG_VERSION")
);

After an upgrade, every existing entry fails the digest comparison in PerFileCache::get and is rewritten in place. The cache file name is the md5 of the path, so nothing is left orphaned. The cost is one cold run per upgrade across every file type, which seems fine for pks. I tried it on top of this branch. The full suite passes, and on a cache populated by main, the #42 repro reports the violation without a delete-cache.

I'm happy to have it here or in a follow-up that lands before the release. Either way, the cache paragraph in the CHANGELOG can then go.

Test

Nothing guards the single-line half of "nothing inside an ERB comment is reported". <%# Foo %> followed by <%= Bar %> gives [Bar] on main and on this branch, so it hasn't regressed, but one more case in multiline_comment_contents_are_not_references would pin it.

iMacTia and others added 3 commits September 29, 2026 17:51
A single-line `<%# %>` comment already yielded no references on main,
but nothing pinned it. `comment_contents_are_not_references` now
includes one, and is renamed since it no longer covers only multi-line
comments. It fails if a change strips the `#` instead of dropping the
tag.

The CHANGELOG now says the fix changes results in both directions. A
todo entry recorded for a constant named inside a comment is no longer
found, so `pks check` reports it as stale until `pks update` removes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cache entry was reused whenever the file's content matched, whatever
version of pks wrote it. So after an upgrade, a file that hadn't changed
kept the result the old version produced, which hid parser fixes such as
the ERB one until the user ran `pks delete-cache`.

EmptyCacheEntry::new now appends CARGO_PKG_VERSION to the content
digest. PerFileCache::get compares the stored digest with it, so the
first run after an upgrade misses on every entry and rewrites it at the
same path (the file name is the md5 of the file path). The digest isn't
read anywhere else. The cost is one cold run per upgrade.

The new test writes an entry, checks it's served, then replaces its
digest with the bare content digest (what earlier versions stored) and
with another version's, and checks both are misses. It fails without
this change. The CHANGELOG's `pks delete-cache` paragraph is replaced by
an entry for this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`<%%` is ERB's escape for a literal `<%`. Erubi, which Rails renders
with, outputs the whole tag up to the first `%>` as text: `<%%= x %>`
renders `<%= x %>`, and nothing in it is evaluated. pks passed its
contents on as Ruby, `%= x` or `% if x`, which broke the parse, so a
file using it lost some or all of its references.

Tags opening with `<%%` are now dropped before the join, like comments.
`%%>` needs no handling. Erubi gives it no special meaning, and a tag
ends at the first `%>` in Erubi and in pks, so `<%% x %%>` is the text
`<% x %%>`.

This is where pks deliberately stops matching packwerk. packwerk parses
a literal's contents as Ruby along with the rest of the file, so a file
with `<%%=` fails to parse, and other literals can report constants from
inside them. pks follows Erubi, which evaluates none of it.

The new test fails on the previous commit, which finds the reference
before the literal and loses the one after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iMacTia iMacTia changed the title Skip ERB comments when converting ERB to Ruby Skip ERB comments and <%% literals when converting ERB to Ruby Sep 30, 2026
@iMacTia

iMacTia commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review! I've added three commits on top. For your three points:

  • The CHANGELOG now says the fix changes results both ways, and that a todo entry for a constant named inside a comment goes stale until pks update removes it.
  • I took the cache suggestion here rather than in a follow-up. EmptyCacheEntry::new now adds the pks version to the digest, and a new test checks that an entry with the old bare digest or another version's is a miss. I checked that nothing else reads the digest. The delete-cache paragraph is gone, and the cache change has its own CHANGELOG entry.
  • Added the single-line case to the comment test, and renamed it to comment_contents_are_not_references since it's not only about multi-line comments anymore.

I also fixed <%% here rather than leaving it for later. It's ERB's escape for a literal <%, and pks parsed the rest of the tag as Ruby, so a file using it lost some or all of its references. It's now treated as text, the way Erubi and Rails do. packwerk parses the literal as Ruby, so it fails on any <%%= and can report constants from inside other literals. pks deliberately differs from it there. That's in the description and has its own CHANGELOG entry.

I've updated the description to match.

Sent with Claude Code

@dduugg
dduugg merged commit c2fe84f into rubyatscale:main Oct 1, 2026
15 checks passed
dduugg added a commit that referenced this pull request Oct 2, 2026
* Check the pks version before taking the cache's stat fast path

c2fe84f (#66) added the pks version to each cache entry's content digest
so that an upgrade would invalidate every entry. But the stat fast path
from 96f7fb0 (#59) returns an entry as soon as its file's mtime and
length match, without comparing the digest. So for any file that hadn't
changed, an entry written by another version of pks was still served,
and the CHANGELOG's "Upgrading pks invalidates cached results" didn't
hold for any entry that recorded a stat.

The fast path now also requires the entry's digest to end with this
version's suffix. An entry from another version falls through to the
digest comparison, misses, and is rewritten. The suffix is one constant,
used both to build the digest and to check it, so the two can't drift
apart.

Neither change has shipped. v0.5.0 predates both, and its entries have
no stat, so they already miss on the digest. The bug affects builds of
main since #59, and would have affected every upgrade starting from the
next release. The CHANGELOG entry already describes the fixed behavior,
so it's unchanged.

test_entry_from_another_pks_version_is_a_miss now also covers an entry
carrying the file's real stat. Its comment said a real stat would take
the fast path instead, which was the bug. A new integration test
rewrites every entry as if another version had written it, with its
references removed and its stat still matching, and checks that the
violations are still reported. Both tests fail without the fix.

* Tighten comments and tests around the fast path's version check

The fast-path comment said a matching stat meant the file "cannot have
changed in any way we care about", which the version paragraph right
below it contradicts: the file is unchanged, but its entry can still be
stale. The slow-path comment now lists an entry from another version
among the reasons for getting there.

The unit test now asserts that the file has a usable stat. On a
filesystem with whole-second mtimes it has none, and the loop's second
pass would quietly repeat the first instead of reaching the fast path.
The integration test strips the version with split_once, since an md5
hex digest has no hyphens and a prerelease version can.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Multi-line ERB comments are parsed as Ruby, so pks misses references

2 participants