Serialize using post-order traversal

docs/serialization.md

@@ -124,8 +124,14 @@ The header is structured like the following table:
 | varuint | content pool size |
 
 After the header comes the body of the serialized string.
-The body consists of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
-Each node is structured like the following table:
+The body consists of a sequence of nodes that is built using a postfix traversal order of the syntax tree: the children of each node come before the node itself, so the root node comes last.
+The body ends at the content pool offset specified in the header.
+
+The children of a node are the values of its `node`, `node?`, and `node[]` fields, which are serialized in the order in which the fields are listed in `config.yml`.
+A `node` field is serialized as its node, a `node?` field is serialized as its node or as a single `0` byte if the node is not present, and a `node[]` field is serialized as each of its nodes in order.
+A loader can therefore keep a stack of loaded nodes: a `0` byte pushes an absent node, and each node pops its children off the top of the stack (they are in field order) and then pushes itself.
+
+After its children, each node is structured like the following table:
 
 | # bytes | field |
 | --- | --- |
@@ -134,12 +140,11 @@ Each node is structured like the following table:
 | location | node location |
 | varuint | node flags |
 
-Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
+Every other field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
 
 * `double` - A field that is a `double`. This is structured as a sequence of 8 bytes in native endian order.
-* `node` - A field that is a node. This is structured just as like parent node.
-* `node?` - A field that is a node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
-* `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
+* `node`, `node?` - Nothing is appended here, since the node has already been serialized as a child.
+* `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, which is the number of children that belong to this field.
 * `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string bytes (_without_ a trailing null byte).
 * `constant` - A variable-length integer that represents an index in the constant pool.
 * `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
@@ -149,6 +154,14 @@ Every field on the node is then appended to the serialized string. The fields ca
 * `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
 * `uint32` - A field that is a 32-bit unsigned integer. This is structured as a variable-length integer.
 
+The `body` field of a `DefNode` is lazy, which means that loaders can skip it and load it later. The children of a `DefNode` are preceded by the following, which loaders that do not skip lazy fields can ignore:
+
+| # bytes | field |
+| --- | --- |
+| `1` | `255` |
+| `4` | the byte length of the children that come before the `body` child |
+| `4` | the byte length of the `body` child |
+
 After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Every constant is embedded in the serialization. Each constant is structured as follows:
 
 | # bytes | field |

include/prism/compiler/branch.h

@@ -0,0 +1,26 @@
+/**
+ * @file compiler/branch.h
+ */
+#ifndef PRISM_COMPILER_BRANCH_H
+#define PRISM_COMPILER_BRANCH_H
+
+/**
+ * Branch prediction hints. These tell the compiler which side of a branch to
+ * predicate the fast path through, so it can lay out instructions to favour
+ * that side and steer the static branch predictor.
+ */
+#if defined(__GNUC__) || defined(__clang__)
+    /** The compiler should predicate that this branch will be taken. */
+    #define PRISM_LIKELY(x) __builtin_expect(!!(x), 1)
+
+    /** The compiler should predicate that this branch will not be taken. */
+    #define PRISM_UNLIKELY(x) __builtin_expect(!!(x), 0)
+#else
+    /** Void because this platform does not support branch prediction hints. */
+    #define PRISM_LIKELY(x)   (x)
+
+    /** Void because this platform does not support branch prediction hints. */
+    #define PRISM_UNLIKELY(x) (x)
+#endif
+
+#endif

javascript/test.js

@@ -427,3 +427,11 @@ suite("location", () => {
     assert(node.receiver.location.endCodeUnitsColumn("utf-8") === 5);
   });
 });
+
+suite("serialization", () => {
+  test("deeply nested trees", () => {
+    // 20,000 nodes deep, which is deeper than a walk of the tree can recurse.
+    const result = parse("1" + "+1".repeat(20000));
+    assert(result.value instanceof nodes.ProgramNode);
+  });
+});

prism.gemspec

@@ -49,6 +49,7 @@ Gem::Specification.new do |spec|
     "include/prism/compiler/accel.h",
     "include/prism/compiler/align.h",
     "include/prism/compiler/assume.h",
+    "include/prism/compiler/branch.h",
     "include/prism/compiler/exported.h",
     "include/prism/compiler/fallthrough.h",
     "include/prism/compiler/filesystem.h",

rakelib/serialization.rake

@@ -62,6 +62,14 @@ task "test:java_loader:internal" => :compile do
   require 'java'
   require_relative '../tmp/prism.jar'
 
+  # A loader that skips the bodies of DefNodes, which are then loaded on demand
+  # by DefNode#getNonLazy.
+  class LazyDefNodeLoader < org.ruby_lang.prism.Loader
+    def loadDefNodesLazily
+      true
+    end
+  end
+
   Dir["**/*.txt", base: fixtures].each do |relative|
     path = "#{fixtures}/#{relative}"
     puts
@@ -71,15 +79,38 @@ task "test:java_loader:internal" => :compile do
     puts parse_result.value
   end
 
+  # Check that loading the bodies of DefNodes lazily and then loading them with
+  # getNonLazy gives the same tree as loading eagerly.
+  describe = -> (node, force) {
+    return "nil" if node.nil?
+    node = node.getNonLazy if force && node.is_a?(org.ruby_lang.prism.Nodes::DefNode)
+    "#{node.getClass.getSimpleName}@#{node.startOffset}(#{node.childNodes.map { |child| describe.call(child, force) }.join(", ")})"
+  }
+
+  lazy_failures = []
+  Dir["**/*.txt", base: fixtures].each do |relative|
+    path = "#{fixtures}/#{relative}"
+    serialized = Prism.dump_file(path).unpack('c*')
+
+    eager = org.ruby_lang.prism.Loader.load(serialized).value
+    lazy = LazyDefNodeLoader.new(serialized.to_java(:byte)).java_send(:load).value
+    lazy_failures << path if describe.call(eager, false) != describe.call(lazy, true)
+  end
+
+  abort "Lazily loaded DefNodes differ from eagerly loaded ones in:\n#{lazy_failures.join("\n")}" unless lazy_failures.empty?
+
   # Check that MarkNewlinesVisitor.java (run by Loader.load() above) marks
   # exactly the same lines as Ruby's newlines.rb, using the fixtures generated
   # by the test:java_loader:newline_fixtures task.
   newline_fixtures = File.expand_path("../java/newline_fixtures.txt", __dir__)
   raise "no #{newline_fixtures}, run the test:java_loader task to generate it" unless File.exist?(newline_fixtures)
 
+  # A lazy DefNode carries its own newline flag, while the DefNode returned by
+  # getNonLazy carries the newline flags of its descendants.
   collect_newlines = -> (node, source, lines) {
     return if node.nil?
     lines << source.line(node.startOffset) if node.hasNewLineFlag
+    node = node.getNonLazy if node.is_a?(org.ruby_lang.prism.Nodes::DefNode)
     node.childNodes.each { |child| collect_newlines.call(child, source, lines) }
   }
 
@@ -88,16 +119,21 @@ task "test:java_loader:internal" => :compile do
   File.readlines(newline_fixtures, chomp: true).each do |fixture|
     path, expected = fixture.split("\t", 2)
 
-    serialized = Prism.dump_file(path)
-    parse_result = org.ruby_lang.prism.Loader.load(serialized.unpack('c*'))
-
-    lines = []
-    collect_newlines.call(parse_result.value, parse_result.source, lines)
-    actual = lines.sort.join(" ")
+    serialized = Prism.dump_file(path).unpack('c*')
+    parse_results = {
+      "java" => org.ruby_lang.prism.Loader.load(serialized),
+      "java (lazy)" => LazyDefNodeLoader.new(serialized.to_java(:byte)).java_send(:load)
+    }
 
     checked += 1
-    if actual != (expected || "")
-      failures << "#{path}\n  ruby: #{expected}\n  java: #{actual}"
+    parse_results.each do |name, parse_result|
+      lines = []
+      collect_newlines.call(parse_result.value, parse_result.source, lines)
+      actual = lines.sort.join(" ")
+
+      if actual != (expected || "")
+        failures << "#{path}\n  ruby: #{expected}\n  #{name}: #{actual}"
+      end
     end
   end
 

src/prism.c

@@ -1,4 +1,5 @@
 #include "prism/compiler/accel.h"
+#include "prism/compiler/branch.h"
 #include "prism/compiler/fallthrough.h"
 #include "prism/compiler/unused.h"
 
@@ -68,24 +69,6 @@
 #   define PM_STATIC_ASSERT(line, condition, message) typedef char PM_CONCATENATE(static_assert_, line)[(condition) ? 1 : -1]
 #endif
 
-/**
- * Support PRISM_LIKELY and PRISM_UNLIKELY to help the compiler optimize its
- * branch predication.
- */
-#if defined(__GNUC__) || defined(__clang__)
-    /** The compiler should predicate that this branch will be taken. */
-    #define PRISM_LIKELY(x) __builtin_expect(!!(x), 1)
-
-    /** The compiler should predicate that this branch will not be taken. */
-    #define PRISM_UNLIKELY(x) __builtin_expect(!!(x), 0)
-#else
-    /** Void because this platform does not support branch prediction hints. */
-    #define PRISM_LIKELY(x)   (x)
-
-    /** Void because this platform does not support branch prediction hints. */
-    #define PRISM_UNLIKELY(x) (x)
-#endif
-
 /**
  * The prism version and the serialization format.
  */