Eliminate templated STRING_TYPE and use a factory

Currently there are several templates that must be customized to either use java.lang.String or org.jruby.RubySymbol for various string or identifier representations. This is done via the JAVA_STRING_TYPE constant which is customized in template.rb to the specific types used by TruffleRuby and JRuby. Despite the name, this appears to only be used for representing identifiers, e.g. for variables, methods, and class names.

I'd like to eliminate this variability in the Java API by instead providing a factory class that can construct appropriate representations.

The interface might look something like this:

public interface IdentitierFactory<IDType> {
    public IDType newIdentifier();
}

The parser would be set up to accept such a factory on instantiation, and indentifiable nodes it produces would aggregate values of T rather than the templated String or RubySymbol that they do now. This would not change usage of the API significantly, other than requiring TruffleRuby and JRuby to both pass in a factory object for creating identifiers.

Nodes that aggregate identifiers (or aggregate nodes that aggregate identifiers) would be generified to propagate this IDType parameter.

public class Nodes {
    ...
    public static class LocalVariableReadNode<IDType> extends Node {
         ...
         public final IDType name;

This would allow a single Java API to be published that all JVM consumers of the Prism API could use, with only the backends differing (initially publishing a WASM version and later a multi-platform native backend).

As this will require some discussion I'm putting it here as an issue and will for now proceed with publishing a JRuby-specific API artifact so we can ship that in JRuby 10.1.

What are concrete issues with the current templated approach?
I don't see a clear problem/motivation in the issue description.

My concerns with this proposal are:

Slower performance due to extra checkcast on every usage of a generic type
More verbose node definitions, making it look needlessly complicated and boilerplate-Java-like
More verbose creation of nodes (at least TruffleRuby uses that to desugar some nodes into simpler ones)

IOW, this sounds a perfect use case for templating given we already have templating.

What about always using java.lang.String as the JAVA_STRING_TYPE?
From your messages on Slack I thought that would be the proposal.
That's a neutral type, makes the API significantly easier to use than a factory, and should work fine under JRuby too.

I agree it doesn't make much sense to publish under org.ruby_lang if it's still JRuby-specific (would be better under a JRuby Maven namespace then).

@eregon

I had a separate discussion with @eregon to clarify some of this and wanted to summarize here.

JRuby cannot (or would rather not) move to String for identifiers.

We need to represent all identifiers as original bytes + encoding, because this does matter for metaprogramming when literal symbols and identifier symbols don't match. We had bug reports from real users prior to making that move, including 3rd-party APIs that failed due to the mismatch.

Other string representations in the Prism Java AST use byte[] + source encoding, so I believe that may be the best common path forward.

This would be the simplest option overall, I believe. JRuby would be able to consume the byte[] directly (without copying), preserving the original characters. It's a bit less convenient for TR, which will still need to turn them into java.lang.String, but @eregon says "it'd work and be fairly neutral".

Another point in its favor is that the deserialization process will end up creating byte[] anyway, so if we just have it stop there it's not really any additional overhead.

My goal with this Maven artifact is to produce an official Java API that JVM-based consumers can use for Ruby tooling.

Think things like JetBrains's RubyMine, which has been notoriously bad about keeping up with language changes. It is a backburnered project at this point, but if we could produce Prism as an official API they might make the move. There are other potential consumers out there as well.

Publishing this as an official API means it should live under the org.ruby-lang namespace.

Also, a note on versioning Prism versus versioning the JVM artifacts...

Because the parser artifact for JRuby (JVM WASM or native wrapper with location info removed) would be a self-contained release, with no external dependency on the Prism gem or this repository, I think we can version it however we like. We can also do official releases of that artifact on-demand (for now, at least). That frees both sides of the project to handle versioning however they like and release on independent schedules.

The parser artifact for the gem (JVM WASM-based library with full location info) would be locked to the Prism version and released along with the gem at the same time, but since it's hidden within the gem's API there's less concern about any exposure to JVM users.

The bits and pieces are falling into place in my head, but I think we have a path forward.

One more thought on this is for Loader#loadConstant:

prism/templates/java/org/ruby_lang/prism/Loader.java.erb

Lines 216 to 218 in aa6bce6

    
           private <%= string_type %> loadConstant() { 
        
               return constantPool.get(buffer, loadVarUInt()); 
        
           }

We might want, in TruffleRuby at least, to return just the constant index or maybe the constant index + a byte[] wrapped in some object.
The idea there is it seems needlessly inefficient to do hashmap lookups to find the frame offset from the local variable name (multiple times for the same variable, once per usage), when we could just have a constant index->frame offset mapping as an int[] (one int[] by scope) by looking the names only once.

If we go for returning just the constant index, that would still be neutral/universal and we'd then need a way to make it easier to use the constant pool with custom conversion or to build your own (currently it's a private class, doesn't expose the size of it, etc).
It would be less convenient than Strings or byte[] for general Java usage, but maybe not by much.
I think general Java/IDE usage java.lang.String would still be the best (as it is currently).

Because this is not figured out yet, if you want to make progress for this I would suggest to support JAVA_STRING_TYPE to be byte[] and it could even be the default. And we'd keep j.l.String for TruffleRuby for now, to avoid a probably useless migration to byte[] when we'd actually want constant indexes instead for TruffleRuby.

I'll try to experiment with returning just the constant index when I have some time.

BTW probably we should rename JAVA_STRING_TYPE to JAVA_IDENTIFIER_TYPE (as you said).
Could also be JAVA_CONSTANT_TYPE since that's the Prism concept for that, but might be rather unclear.

Also, a note on versioning Prism versus versioning the JVM artifacts...

Because the parser artifact for JRuby (JVM WASM or native wrapper with location info removed) would be a self-contained release, with no external dependency on the Prism gem or this repository, I think we can version it however we like. We can also do official releases of that artifact on-demand (for now, at least). That frees both sides of the project to handle versioning however they like and release on independent schedules.

It would still be very useful to know which Prism version is in which jar though, without consulting a manually-maintained mapping for it. The easiest would be to use the same version, or maybe with an extra component or so.

Probably a good idea to do some 0.x releases though until things are more fledged out and it's actually ready for general Java usage.

The easiest would be to use the same version, or maybe with an extra component or so.

Yeah, something like we've been doing for our ActiveRecord adapters (two-digit Rails version followed by other components) might make sense, if that fits Prism's versioning.

I think we can deal with that when we're ready to release a version > 0.x for more general usage.

@headius

@headius whatever y'all agree upon is fine with me

@kddnewton

@kddnewton I will finalize the maven PR today so that can get merged and do a separate PR to switch the identifier type. Then release number 0.0.1!

[ruby/prism] Swich identifiers to byte[]

Reopen temporarily while we work out the broken part of #4016 in #4023.

I'm OK with #4023 but I wonder now what it achieves:

Isn't a identifier type of org.jruby.RubySymbol (as it was) more convenient for JRuby? And you could use the byte[] + offset + length optimization from here to avoid an extra byte[] easily. But with Switch identifiers to byte[] #4023 it's basically impossible now to have this optimization for JRuby.
The default was j.l.String, which I think is still the most convenient by far for general usages of the Java API. Like I can't imagine IntelliJ/RubyMine using anything else but j.l.String when referring to identifiers like local variables (though I haven't looked at the code, not sure that code is open source). jruby-parser also uses j.l.String for identifiers, and that has been used in IDEs.
There are some edge cases like you mentioned, but those probably only matter for runtimes and are very rare too, i.e. I think no tool needs to care about that. In theory there could be missing charsets, in practice only a handful of encodings are used as source encoding, and extra Charsets can be registered. Removing the encodingCharset is really just a nice-to-have cleanup but there was never a requirement for it, I just wanted to make it clear adding a JCoding dependency is not necessary in Release official Java API #4018.
I'm not* against byte[] but it seems inconvenient for all usages, be it JRuby, TruffleRuby (but since we keep j.l.String it's fine) or tools.

Like I said I'm OK with the change because it's still convenient & clean for TruffleRuby and the byte[] + offset + length optimization is feasible before & after for j.l.String, but for other cases (JRuby, tools) it seems we might have actually lost something.

I'm thinking to make a PR for the byte[] + offset + length optimization, that will probably make it clearer why having org.jruby.RubySymbol as a possible identifier type would be beneficial.

Isn't a identifier type of org.jruby.RubySymbol (as it was) more convenient for JRuby?

It is more convenient but the Java library must not depend upon JRuby. You said you were against using generics to represent the identifier type, so the only remaining option that doesn't force Ruby source code bytes through char[] is a lowest common denominator byte[] + encoding (or byte[] + offset + length + encoding).

The default was j.l.String, which I think is still the most convenient by far for general usages of the Java API.

If you still need to use an encoding or charset to turn this back into byte[] (as is the case in JRuby), it is not the most convenient. If you lose the original representation of identifier characters in the Ruby source (not all encodings will round-trip through char[]) it is not the most convenient.

The most convenient for preserving the original source bytes is byte[] or byte[] + offset + length. RubySymbol aggregated these for us, but we can just construct it lazily at compile time.

java.lang.String is a nonstarter if you want to preserve the original source bytes in the AST. We could certainly add a "getNameString" method to transcode the byte[] to char[] and return a String for convenience.

I can't imagine IntelliJ/RubyMine using anything else but j.l.String

They would if they wanted to ensure the source bytes were preserved. I also do not know how they parse Ruby source currently. They used to use JRuby's parse.y-based parser, I believe.

I'm thinking to make a PR for the byte[] + offset + length optimization, that will probably make it clearer why having org.jruby.RubySymbol as a possible identifier type would be beneficial.

In #4028 We discussed the possibility of having the ConstantPool be the source of these identifier objects, potentially generified to hold whatever we want, with current identifier byte[] references just being indexes into the constant pool. What about doing that PR instead?

It is more convenient but the Java library must not depend upon JRuby.

Those are not exclusive, you could have:

A Maven artifact for generic Java usage of Prism, using byte[] or j.l.String
A Maven artifact for usage as the JRuby parser, which uses org.jruby.RubySymbol (when setting PRISM_JAVA_BACKEND=jruby at template time)

Given the first one would probably want non-semantic fields too, and we already agreed there are two builds (only semantic fields, all fields) I think that'd make a lot of sense.

It seems you want a single artifact, but I think that cannot work because of the 2 builds with two different set of fields.

I can't imagine IntelliJ/RubyMine using anything else but j.l.String

IntelliJ does use j.l.String for representing AST identifiers, see PsiNamedElement in https://dev.to/pyltsinm/how-to-write-your-own-language-plugin-for-idea-part-2-50g5.

I had a typo in #4009 (comment), fixed it now, the correction is I'm not* against byte[] ...

In #4028 We discussed the possibility of having the ConstantPool be the source of these identifier objects, potentially generified to hold whatever we want, with current identifier byte[] references just being indexes into the constant pool. What about doing that PR instead?

True, that would address it (we discussed it in #4018 (comment) not #4028). However it would make the API vastly more difficult to use for general usage and in JRuby & TruffleRuby. For starters we wouldn't be able to even print local variable/method/other identifier names when printing the Java AST, that seems pretty bad.

Also the optimization I was thinking in #4009 (comment) was to have a int[] per scope from constant index to index in the variables for that scope. Sounds nice but might be not great because the constant pool can be quite big and typically much bigger than the number of locals per scope, so memory footprint wise it's unclear if it's a good idea.

It seems you want a single artifact

I have already said multiple times that I accept there will be multiple artifacts:

#4009 (comment)

#3962 (comment)

I also spelled out a possible plan for how the artifacts would be structured:

6c9710d

Since that's a commit message, I repeat it here:

This library will eventually be split into a few pieces:

The direct Parser API, probably called prism-parser-api

The native backend, probably called prism-parser-native

The WASM backend, probably called prism-parser-wasm

The Prism gem backend (with line info etc), name TBD

Given the likely need for semantic info in the general-purpose Java API, there could be two versions of that API, one without semantic info.

IntelliJ does use j.l.String for representing AST identifiers

That article is about writing plugins for IntelliJ and the example languages have narrow Unicode source encoding requirements.

If you click through to actual IntelliJ documentation, you'll see that "symbols" used for identifiers are represented with a much more complex set of abstractions... for example representing a PsiSymbolDeclaration as a TextRange over the original source.

Also,

A Maven artifact for generic Java usage of Prism, using byte[] or j.l.String

I don't think it's a good idea to publish two otherwise-identical Maven artifacts that differ only by the types of identifier fields. If there's a need for a better API that can support byte[] and String[] and user-defined identifier types, let's design that.

However it would make the API vastly more difficult to use for general usage

Then we're back to aggregating the identifiers in the AST, and avoiding a JRuby dependency, and avoiding generics, and avoiding a round-trip through char[]. So that's byte[] + supporting metadata.

Sounds nice but might be not great because the constant pool can be quite big and typically much bigger than the number of locals per scope, so memory footprint wise it's unclear if it's a good idea.

I believe we are supporting two different use cases here.

For a use case that keeps the entire AST alive in memory, the ConstantPool is just a separate place to store the same data that would be on the AST, and I wouldn't expect the memory use to be significantly different.

For a use case that only keeps parts of the AST in memory that have not been processed (compiled, translated to internal form), holding on to the entire ConstantPool may be a concern (e.g. typical Rails source with dozens of methods that are never called). In this case a per-scope ConstantPool could make sense.

Or we just continue aggregating the full identifier data + metadata in the AST and figure out what form that should take to be most generically useful.

I don't think it's a good idea to publish two otherwise-identical Maven artifacts that differ only by the types of identifier fields. If there's a need for a better API that can support byte[] and String[] and user-defined identifier types, let's design that.

That's where we see it differently I think.
For me that would make perfect sense, we do have 2-3 different types and whether published or not (for TruffleRuby) there is already at least 2 different types (byte[] & j.l.String).

I think it would make most sense to have one of them with org.jruby.RubySymbol to be used by JRuby's compiler. But if you prefer to use byte[] in JRuby and accept the extra copying overhead of that, that's your choice.

There is already variability at template time, hiding it or moving it to runtime doesn't seem good, better be explicitly separate Maven artifacts with clear purpose, than something very generic but inconvenient for all usages.

I think it is clear we will need 2 top-level artifacts anyway, one with semantic fields only and one with all fields.
The former would be used by JRuby, the latter for generic Prism Java API usage.
They would depend on distinct artifacts for libprism (whether native or wasm), since they need to generate different serialized output based on the fields.

whether published or not (for TruffleRuby)

You have said multiple times that TruffleRuby will not use the artifacts I'm building, so I don't see why that should have any influence on what's in them.

But if you prefer to use byte[] in JRuby and accept the extra copying overhead of that, that's your choice.

There's zero extra copying. The old RubySymbol had to copy the bytes out of the serialized data just like the new byte[] does, and the byte[] can just be used directly for the RubySymbol we eventually create

There is already variability at template time

With a factory and generics that variability could be entirely dependent on the consumer of the API. But you said you don't want generics. So there has to be a single type per published API.

I think it is clear we will need 2 top-level artifacts anyway, one with semantic fields only and one with all fields.

If we go your way, instead of 2 top-level artifacts we'd have up to six: [semantic, non-semantic] X [byte[], RubySymbol, String]. That seems like a massive overcomplication when everyone can generate their preferred representation of identifiers from byte[].

You can certainly publish your own Maven artifact that templates everything exactly the way TruffleRuby wants it. I don't plan to do so.

There's zero extra copying.

There is copying from the byte[] serialized to byte[]. Could be easily avoided with org.jruby.RubySymbol by having bytesToName() being pass byte[] serialized, int offset, int length. Can't be done if the type is byte[], which forces an extra copy there.
(that copying was already there before this PR, but this PR prevents to remove it)

If we go your way, instead of 2 top-level artifacts we'd have up to six: [semantic, non-semantic] X [byte[], RubySymbol, String]. That seems like a massive overcomplication when everyone can generate their preferred representation of identifiers from byte[].

No, we'd only have (semantic, RubySymbol) and (non-semantic, String), as detailed in #4031 (comment).
There would be (semantic, String) for TruffleRuby but not needed as a Maven artifact.

There is copying from the byte[] serialized to byte[]. Could be easily avoided with org.jruby.RubySymbol by having bytesToName() being pass byte[] serialized, int offset, int length. Can't be done if the type is byte[], which forces an extra copy there.

If the identifier is a RubySymbol directly referencing the serialized byte[], it would mean those bytes live as long as the symbol. Identifiers in JRuby are hard-referenced and kept alive for the lifetime of the JRuby runtime instance.

I mentioned this concern in #4028 when discussing options to avoid copying data out of the serialized bytes.

Copying once to the AST byte[], and then using that byte[] directly only produces a single extra object and does not keep the original serialized bytes alive.

Right. We once had the same concern in TruffleRuby for Symbols, etc but found a solution of only doing the copy when inserting in the cache, that way the lookup is done without any copy: https://github.com/truffleruby/truffleruby/blob/8429d80abcbc6a9f4061060d2e2ad23cfa905cfd/src/main/java/org/truffleruby/core/string/TBytesKey.java#L117-L138
This only helps when the cache is hit though.

That copy might not matter much here, I haven't measured it.
For the TruffleRuby case with j.l.String, passing the offset & length was a natural fit as the other side already supports that.

Navigation Menu

Repository navigation

Eliminate templated STRING_TYPE and use a factory #4009

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	private <%= string_type %> loadConstant() {
	return constantPool.get(buffer, loadVarUInt());
	}

Eliminate templated STRING_TYPE and use a factory #4009

Description

Activity

eregon commented on Mar 17, 2026

eregon commented on Mar 17, 2026

headius commented on Mar 17, 2026

headius commented on Mar 17, 2026

eregon commented on Mar 18, 2026

eregon commented on Mar 18, 2026

headius commented on Mar 18, 2026

kddnewton commented on Mar 19, 2026

headius commented on Mar 19, 2026

headius commented on Mar 20, 2026

1 remaining item

eregon commented on Mar 22, 2026

eregon commented on Mar 22, 2026

headius commented on Mar 23, 2026

headius commented on Mar 23, 2026

eregon commented on Mar 23, 2026

eregon commented on Mar 23, 2026

eregon commented on Mar 23, 2026

headius commented on Mar 23, 2026

headius commented on Mar 23, 2026

headius commented on Mar 23, 2026

eregon commented on Mar 23, 2026

headius commented on Mar 23, 2026

eregon commented on Mar 23, 2026

headius commented on Mar 30, 2026

eregon commented on Mar 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions