Repository navigation
Skip ERB comments and <%% literals when converting ERB to Ruby - #66
Conversation
convert_erb_to_ruby_without_sourcemaps joins the contents of every ERB tag into one Ruby source. The first line of a `<%#` comment starts with `#`, so it became a Ruby comment, but every later line became Ruby code. What that did depended on the text: a trailing period turned the next tag into a method call, an apostrophe opened a string that swallowed the rest of the file, and `def x` made the whole file fail to parse. So references were missed with no error, and a constant named on a later comment line was reported as a reference. Comment tags are now dropped before the join, as packwerk's ERB parser does. Only a tag that opens with exactly `<%#` is a comment. `<%-#` and `<% #` are code tags in Erubi and in packwerk, and their later lines are Ruby, which is why the check is on the whole match and not the capture. Both ERB parsers go through this function, so both are fixed. Single-line comments are dropped too, which changes one case. A leading encoding comment such as `<%# encoding: iso-8859-1 %>` used to become the first line of the Ruby source, where lib-ruby-parser read it as a magic comment, and for any encoding but UTF-8 or ASCII-8BIT pks reported nothing for the file. Rails strips that tag before compiling (ActionView's ENCODING_TAG) and packwerk ignores it, so those files are now checked. Every other single-line comment was already a Ruby comment, so dropping it changes nothing. ERB references are all reported at 0:0, so the shorter Ruby source doesn't move any location. The first three new tests fail on main. The fourth passes on main and fails if the check moves to the capture. Cached results aren't tied to the pks version, so an unchanged ERB file keeps its old result after an upgrade. The CHANGELOG entry tells users to run `pks delete-cache`. Fixes rubyatscale#42. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dduugg
left a comment
There was a problem hiding this comment.
Thanks, this is a clean fix, and the reasoning in the description checks out against the upstream code. BetterHtml, which packwerk's ERB parser uses, and Erubi both treat a tag as a comment only when # comes straight after <%. capture[0].starts_with("<%#") gives the same answer as both for <%#, <%-#, <% #, <%=#, <%#= and <%#-.
I ran each of the seven texts in the first test against main on its own, and every one fails there. That includes "second (line" and "class foo", which lose the references before the comment. The first three tests fail on main, and the fourth fails if the filter moves to the capture. The encoding explanation holds up too. Without a custom decoder, lib-ruby-parser accepts only UTF-8, ASCII-8BIT and BINARY, pks doesn't set a decoder, and a missing AST becomes an empty result for the whole file. The repro from #42 misses the privacy violation on main and reports it here, with the default parser and with --experimental-parser. clippy, fmt and CI are clean, and it merges cleanly with #65.
A note on the CHANGELOG and a suggestion for the cache.
CHANGELOG
The entry only says "may show new violations" for the encoding case, but the multi-line fix changes results in both directions too. References that used to be missed are now reported. A constant named on a later line of a comment that someone recorded in package_todo.yml is no longer found, so that entry goes stale and pks check fails until pks update removes it. One sentence covering both, with pks update as the remedy, would help anyone upgrading.
Cache
I'd rather not ask everyone to run pks delete-cache. Anyone who keeps tmp/cache between CI runs, which is common because of bootsnap, won't see the note, and the next parser fix would need the same paragraph. Adding the pks version to the stored content digest in EmptyCacheEntry::new (src/packs/caching/mod.rs:32) handles it:
let file_contents_digest = format!(
"{}-{}",
file_content_digest(filepath)?,
env!("CARGO_PKG_VERSION")
);After an upgrade, every existing entry fails the digest comparison in PerFileCache::get and is rewritten in place. The cache file name is the md5 of the path, so nothing is left orphaned. The cost is one cold run per upgrade across every file type, which seems fine for pks. I tried it on top of this branch. The full suite passes, and on a cache populated by main, the #42 repro reports the violation without a delete-cache.
I'm happy to have it here or in a follow-up that lands before the release. Either way, the cache paragraph in the CHANGELOG can then go.
Test
Nothing guards the single-line half of "nothing inside an ERB comment is reported". <%# Foo %> followed by <%= Bar %> gives [Bar] on main and on this branch, so it hasn't regressed, but one more case in multiline_comment_contents_are_not_references would pin it.
A single-line `<%# %>` comment already yielded no references on main, but nothing pinned it. `comment_contents_are_not_references` now includes one, and is renamed since it no longer covers only multi-line comments. It fails if a change strips the `#` instead of dropping the tag. The CHANGELOG now says the fix changes results in both directions. A todo entry recorded for a constant named inside a comment is no longer found, so `pks check` reports it as stale until `pks update` removes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cache entry was reused whenever the file's content matched, whatever version of pks wrote it. So after an upgrade, a file that hadn't changed kept the result the old version produced, which hid parser fixes such as the ERB one until the user ran `pks delete-cache`. EmptyCacheEntry::new now appends CARGO_PKG_VERSION to the content digest. PerFileCache::get compares the stored digest with it, so the first run after an upgrade misses on every entry and rewrites it at the same path (the file name is the md5 of the file path). The digest isn't read anywhere else. The cost is one cold run per upgrade. The new test writes an entry, checks it's served, then replaces its digest with the bare content digest (what earlier versions stored) and with another version's, and checks both are misses. It fails without this change. The CHANGELOG's `pks delete-cache` paragraph is replaced by an entry for this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`<%%` is ERB's escape for a literal `<%`. Erubi, which Rails renders with, outputs the whole tag up to the first `%>` as text: `<%%= x %>` renders `<%= x %>`, and nothing in it is evaluated. pks passed its contents on as Ruby, `%= x` or `% if x`, which broke the parse, so a file using it lost some or all of its references. Tags opening with `<%%` are now dropped before the join, like comments. `%%>` needs no handling. Erubi gives it no special meaning, and a tag ends at the first `%>` in Erubi and in pks, so `<%% x %%>` is the text `<% x %%>`. This is where pks deliberately stops matching packwerk. packwerk parses a literal's contents as Ruby along with the rest of the file, so a file with `<%%=` fails to parse, and other literals can report constants from inside them. pks follows Erubi, which evaluates none of it. The new test fails on the previous commit, which finds the reference before the literal and loses the one after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
<%% literals when converting ERB to Ruby
|
Thanks for the careful review! I've added three commits on top. For your three points:
I also fixed I've updated the description to match. Sent with Claude Code |
* Check the pks version before taking the cache's stat fast path c2fe84f (#66) added the pks version to each cache entry's content digest so that an upgrade would invalidate every entry. But the stat fast path from 96f7fb0 (#59) returns an entry as soon as its file's mtime and length match, without comparing the digest. So for any file that hadn't changed, an entry written by another version of pks was still served, and the CHANGELOG's "Upgrading pks invalidates cached results" didn't hold for any entry that recorded a stat. The fast path now also requires the entry's digest to end with this version's suffix. An entry from another version falls through to the digest comparison, misses, and is rewritten. The suffix is one constant, used both to build the digest and to check it, so the two can't drift apart. Neither change has shipped. v0.5.0 predates both, and its entries have no stat, so they already miss on the digest. The bug affects builds of main since #59, and would have affected every upgrade starting from the next release. The CHANGELOG entry already describes the fixed behavior, so it's unchanged. test_entry_from_another_pks_version_is_a_miss now also covers an entry carrying the file's real stat. Its comment said a real stat would take the fast path instead, which was the bug. A new integration test rewrites every entry as if another version had written it, with its references removed and its stat still matching, and checks that the violations are still reported. Both tests fail without the fix. * Tighten comments and tests around the fast path's version check The fast-path comment said a matching stat meant the file "cannot have changed in any way we care about", which the version paragraph right below it contradicts: the file is unchanged, but its entry can still be stale. The slow-path comment now lists an entry from another version among the reasons for getting there. The unit test now asserts that the file has a usable stat. On a filesystem with whole-second mtimes it has none, and the loop's second pass would quietly repeat the first instead of reaching the fast path. The integration test strips the version with split_once, since an md5 hex digest has no hyphens and a prerelease version can.
Fixes #42.
pks turns an ERB file into Ruby by joining the contents of its tags, and it treats
<%# %>comments like any other tag. Only the first line of a multi-line comment ends up as a Ruby comment, and the rest is parsed as code. Depending on the text, that loses the references before the comment, the ones after it, or the whole file, with no error. A constant named on a later comment line is reported as a reference.Approach
Comment tags are now dropped before the join, which is what packwerk's ERB parser does. The check is on the whole tag, not the captured contents, because only a tag that opens with exactly
<%#is a comment.<%-#and<% #are code tags in Erubi and in packwerk, and their later lines are real Ruby. The default and experimental parsers both go through this one function.Where to look
<%# encoding: iso-8859-1 %>, it used to become the first line of the Ruby source, where lib-ruby-parser read it as a magic comment. For any encoding but UTF-8 or ASCII-8BIT, pks then reported nothing for the file. Those files are now checked, so they can show new violations. Rails strips that tag before compiling and packwerk ignores it, so this matches both. The CHANGELOG says so and there's a test for it.<%%, ERB's escape for a literal<%, are dropped as well. pks used to parse their contents as Ruby, which lost the references after them and sometimes all of them. This follows Erubi, which Rails renders with: it outputs the whole tag as text. packwerk parses the literal's contents as Ruby, so a file with<%%=fails to parse and other literals can report constants from inside them. This is one place pks deliberately doesn't match it.hash_after_hyphen_or_space_is_code_not_commentpasses on main. It fails if the check moves to the capture, which would treat<%-#and<% #as comments.Notes
The cache used to reuse an entry whenever the file's content matched, so after an upgrade an ERB file that hadn't changed would keep its old result until someone ran
pks delete-cache. Cache entries now include the pks version in their content digest, so the first run after an upgrade is a cold run for every file, not just ERB. That's a separate commit with its own CHANGELOG entry.Sent with Claude Code