Skip to content

Replace lua shared with require, and fix Lua error line mapping - #144

Merged
vinniefalco merged 7 commits into
cppalliance:masterfrom
vinniefalco:master
Oct 14, 2026
Merged

vinniefalco merged 7 commits into
cppalliance:masterfrom
vinniefalco:master

Conversation

@vinniefalco

Copy link
Copy Markdown
Member

This branch lands two plans and one doc fix, in 12 commits. Shared Lua moves from an automatic replay into an explicit require, and runtime errors map to the right prompt line in more cases.

  • require replaces lua shared. A prompt used to hold one lua shared block under the H1, and the Engine replayed it into every section before the section's own code, whether the section used it or not. Now require '## heading' runs a section as a library in the calling section's own Lua environment, inside the calling block's coroutine, and returns its value. A library runs once per section, can make suspending calls such as store, model, and tool calls, gets its own prose with substitutions from the caller, and reports errors at its own prompt lines. require reaches further than call and jump: a section's direct children, its siblings, and the siblings of each of its ancestors, so a nested tasks.fanout worker can require a top-level library. Nothing is inherited, so each section requires what it uses. The lua shared fence is now a parse error that points to require, and three library-load events that name the library replace the three shared-load events.
  • Require run debt removal. A debt review of the require work found that prompt-line mapping identified a block by its full display name inside Lua's chunk id. Lua cuts chunk names of 45 bytes or more, so a long or shared section title lost its mapping, and every H1 Lua block shared one name, so one H1 block's error could map through another's start line. Prompt blocks now compile under a short line N chunk name, and the mapper writes the display name back. LuaProgram::map_runtime_error, which lost its last production caller, is now test-only.
  • Doc links. Three doc link problems in the Lua crate's docs, which no standard doc command builds: a link to the test-only run_chunk, a link to VfsError::Conflict with no target, and a redundant explicit target on SortKey.

Plans:

  • vibe/2026-10-12-1-require-replaces-lua-shared.md
  • vibe/2026-10-12-2-require-run-debt-removal.md

Changes

require replaces lua shared:

  • Add library-load events that name the loaded library
  • Add a require that runs library sections in the caller
  • Give library blocks their own prose and map their errors
  • Add require reach lookup and library construction
  • Install require in section setup
  • Move engine tests and fixtures onto require
  • Remove the lua shared fence and its section replay

Require run debt removal:

  • Map Lua errors through short line-based chunk names
  • Gate map_runtime_error behind test-support

Doc links (no plan):

  • Fix broken and redundant doc links in the Lua crate

The two Close plan commits retire each plan's active marker, and the second also marks its plan record's todos completed.

Behavior changes

For prompt authors:

  • Breaking: the lua shared fence is gone. A prompt that still has one, under the H1 or anywhere else, fails to parse with "the lua shared fence form was removed; put shared code in a section and load it with require '## heading'". Move the shared code into an ordinary section and require it from each section that uses it. Shared setup such as models.default or tools.always_offer moves into a library the H1 requires. Both still set prompt-wide state, so later sections see it without requiring the library.
  • The error for the already-removed lua prompt fence now reads "the lua prompt fence form was removed; use lua for a live H1 block and require for shared code".
  • require '## heading' names a section the same way call does. A library's blocks run in order. The first block that returns a value stops the library, and require hands back that value, whatever its type. A block with no return, or with return nil, lets the next block run, and a library where no block returns anything gives true.
  • A second require of the same library in a section returns the saved value without running it again, false included. A library that raises while loading stays out of the cache, so the next require runs it again, and globals it set before the error stay set.
  • Globals a library defines stay visible to the caller, and its local variables stay inside it. Load-time code, such as a model call, runs once in each section that requires the library.
  • A require made while a library loads resolves from the library's own position, so a library can pull in its own nested helpers. Everything else in library code resolves from the calling section: call, jump, prompt.section, prompt.list, tasks.spawn, tasks.fanout, and any require a library function makes later.
  • prompt.section and prompt.list keep the narrower reach of call, so from a nested worker require '## house_style' works while prompt.section '## house_style' reads nil.
  • A heading out of reach raises at the call and lists only the reachable sections. A cycle raises require cycle: ## a -> ## b -> ## a. jump raises while a library loads ("jump is unavailable while a library loads: ..."), and a library function that calls jump after the load acts on the caller as usual. A non-string heading raises "require takes a section heading string such as '## helpers', got ...".
  • A library that fails while the H1 requires it ends the run as RequirementsUnmet, like any uncaught H1 error. The old shared replay ran outside the H1's code and kept the Lua error kind.
  • A required section is still an ordinary section, and the walk runs it when it reaches it. A globals-only library runs harmlessly, but a library that returns a table fails the run, and one that returns a scalar ends the chain with that value. Place a returning library after a section that returns, or nest it under the section that uses it.
  • require is a reserved Engine global, so the reserved list grows to 61 and a Plugin prelude can't define it. A prelude that calls require fails to load with Lua's "attempt to call a nil value (global 'require')", because require installs after the preludes.
  • Middle Lua blocks of a section are now named section `X` lua N, and when two sections share a title their block names add the heading line, as in section `Helpers` (line 18) prologue. Mapped positions read location:line: instead of [string "location"]:line:. An error that stays unmapped, such as one raised in a function one block defined and another block called, shows [string "line 120"]:3:: the prompt line the block starts on, then the line inside the block.
  • Errors in sections with long titles, or with shared titles past about line 100, now map to their prompt-file lines. Before, Lua cut their chunk names and the lines stayed chunk-relative. An H1 block's error is no longer mapped through another H1 block's start line.

For hosts:

  • Breaking: Event::LuaSharedLoadStarted, LuaSharedLoadSucceeded, and LuaSharedLoadFailed and their lifecycle constants are gone. Event::LuaLibraryLoadStarted, LuaLibraryLoadSucceeded, and LuaLibraryLoadFailed replace them, each with a library field beside the usual coordinates that holds the library's heading with its markers, such as ## shared_stuff. Every section used to emit the shared pair even with no shared code. The new events appear only on a library's first load in a section, and a cached require emits nothing. crates/promptforge/public-api.txt changes to match.

For contributors:

  • LuaProgram::compile now compiles prompt blocks under the chunk name line {source_line}, and location() still returns the display name. compile_internal keeps its explicit names.
  • LuaProgram::map_runtime_error now compiles only under test or the test-support feature, and promptforge-parser takes promptforge-lua with test-support as a dev-dependency for its tests. Production mapping goes through map_runtime_error_among, which maps through the started block and every library loaded into the VM.
  • The removed replay symbols (replay_shared, section_vm_with_shared, the three LuaSharedLoad* variants, and their LUA_SHARED_LOAD_* constants) are in RETIRED_SEEDS.

Known limits

  • A library's load-time work repeats in each section that requires it, so an expensive load, such as a model call, costs once per section.
  • Keeping a returning library away from the walk is the author's job. Nothing marks a section as a library.
  • An error raised in a function one block defined and another block called stays unmapped. Mapping covers the block that started and the libraries loaded into its VM.
  • Library-load events name a library only by heading, so two same-heading libraries loaded through nested requires in one section look alike apart from their nesting order.
  • A cached require still resolves the heading and copies the library's programs before it finds the saved value. The copies are reclaimed, but they sit outside the Lua memory limit until the collector runs.
  • If a memory error hits the loader at one narrow point, its bookkeeping can stay set, so later require calls in that section resolve from the library's position and jump keeps refusing. It needs the heap at its limit at that instant plus a pcall around require.
  • crates/promptforge-internal/lua/src/tests/require.rs sits at 498 of its 500 allowed lines, so its next change has to move something out first.
  • The separate promptforge-docs repository still describes lua shared in its language guide and needs its own update.

Testing

  • Every step of both plans ran its focused tests, and the steps that end a work item ran the whole workspace suite. The last step of each plan passed the full gates: the build, the format check, clippy with warnings denied on both partitions, the headless gateway check, the doc builds with warnings denied, the workspace and Workshop nextest runs, and the docs-claims scan. The require plan's last step also passed the facade API check.
  • At the debt plan's last step, the workspace nextest run passed 5,057 tests and the Workshop run passed 311. The doc-link commit changes only doc comments. It passed all four Lua crate doc builds (with and without features, public and private items) with warnings denied, clippy for the Lua crate with warnings denied, and the format check, but not a second full suite.
  • A search for lua shared outside vibe/ finds only the parser's rejection, its docs, and its tests.
  • The Lua-level suites in lua/src/tests/require.rs and lua/src/tests/require/ cover returns, caching, a saved false, cycles, a retry after a failed load, resolver keys during and after a load, the jump refusal, suspending calls in library blocks, prose swap and restore, error mapping, and the load events.
  • The whole-prompt suites in engine/src/execute/tests/suite/require/ cover each rule end to end: a sibling library's table, a nested fanout worker requiring a top-level library, store, model, and tool calls during a load, an H1 setup library's prompt-wide state, the walk reaching a library, error lines during and after a load, the load events, a child starting without its parent's libraries, and each raise.
  • Tests that used lua shared moved to require, including models.default setup, which moved into a ## setup library the H1 requires. The fixture shared-library.md became require-library.md. a_prelude_write_across_two_arms_ends_the_run_with_determinism_caught_or_not pins that a prelude's store conflict ends the run as a determinism violation whether or not the prelude catches it, now that the conflict check sits right after the preludes.
  • Five regression tests each failed at 67be02738 and pass now: every_h1_block_maps_only_its_own_errors, a_location_longer_than_lua_keeps_in_a_chunk_name_still_maps, a_long_unique_section_title_maps_its_prologue_line, a_shared_title_whose_heading_line_reaches_100_maps_its_prologue_line, and a_raise_while_a_long_titled_library_loads_reports_the_library_line. A check in lua/src/tests/require/errors.rs that the chunk rename made vacuous now matches any unmapped [string " marker.

The Workshop server no longer has a build mode without the web UI. That mode skipped the UI build and swapped the embedded asset layer for one that served nothing, so server-only tests could run without Node.js or the UI bundle, but it needed its own test run in CI. The build script now always bundles the UI, the asset routes always serve the embedded bundle, and the interface that chose between the two asset layers folds into the embedded one.

- `EmbeddedAssets` is now the whole asset layer. The `AssetServer` trait and `NoopAssets` are deleted, and `get` and `manifest` become associated functions with no `self`.
- `ui_asset` drops its `server: &dyn AssetServer` parameter and reads through `EmbeddedAssets::get`. Every route handler loses the matching argument, `logical_asset` calls `assets::EmbeddedAssets::manifest()` directly, and `asset_server()` is deleted.
- `crates/workshop/server/build.rs` always runs the UI build after the ceiling check, since the `CARGO_FEATURE_HEADLESS` branch and `empty_asset_dir` are gone. Building `workshop-server` therefore always needs Node.js 22 and one `npm ci` per checkout.
- `the_server_boots_and_serves_the_ui_with_an_unreachable_gateway` now always requests the UI index and asserts `reqwest::StatusCode::OK`, with no feature gate around that check.
- `crates/workshop/server/Cargo.toml` no longer defines `headless`, and `.github/workflows/ci.yml` drops the `Test (workshop-server headless)` step that ran it.
- `crates/workshop/server/src/routes/assets-headless-tests.rs` and `the_noop_asset_layer_serves_nothing` are deleted along with the feature they covered. The remaining asset tests run under plain `#[cfg(test)]`.

Design: removes feature-flag @ crates/workshop/server/Cargo.toml::headless
Design: removes strategy @ crates/workshop/server/src/assets.rs::AssetServer
Plan: vibe/2026-10-13-3-remove-hakari.md
The Windows Workshop job now runs all of its tests from a single nextest command instead of three separate test steps. A filter keeps coverage as it was: every test in the four Workshop crates, including the workspace jail's Windows path tests, plus the five Gateway process-ownership race tests selected by name. Default features are turned off for every selected package and only the Gateway's test fixtures are turned on. The Linux Workshop job drops its copy of the race tests because the main Linux test job already runs the Gateway suites with every feature enabled.

- `Test (Windows)` replaces `Test (workshop, concurrent via nextest)`, `Test (workshop-workspace, Windows jail)`, and the Windows `Test Gateway process ownership races` step with one six-package `cargo nextest run`. Because it is one cargo command, dependency features unify across all six packages, so the Workshop crates now build with the union of features that all six request.
- `--no-default-features --features gateway/test-fixtures` applies to every selected package, with the fixtures feature enabled on the Gateway alone.
- `-E` selects whole suites for `workshop`, `workshop-server`, `workshop-server-api`, and `workshop-workspace`, plus tests matching the five race-test names in `gateway-api-discovery` and the gateway `it` binary. nextest matches `package()` against the whole name, so `package(workshop)` does not pull in the other Workshop crates.
- `cargo test` no longer runs the five race tests one at a time; they now run under nextest in parallel with the Workshop suites.
- `--all-features` no longer applies to `workshop-workspace`; the merged command builds it with default features off.
- `Test Gateway process ownership races` is gone from the Linux Workshop job. The main Linux test job already runs every `gateway` and `gateway-api-discovery` test with `--all-features`, so Linux keeps race-test coverage.

Design: removes clone-block @ .github/workflows/ci.yml
Plan: vibe/2026-10-13-3-remove-hakari.md
This change deletes the hakari-generated feature-unification crate that nearly every workspace member depended on, along with its configuration and the CI steps that installed and verified it. That crate put test-only features into the normal dependencies of every member that used it, and it let crates compile against features their own manifests never declared. Feature unification now falls back to cargo's default, where a crate gets the features its manifest declares, unified only with the other packages the same command selects. The crate whose tests relied on a supplied feature now declares it, and the comments that described the old arrangement are reworded.

- `crates/workspace-hack` is deleted with `.config/hakari.toml`, its root workspace dependency entry, and the `workspace-hack.workspace = true` line in each of the 49 member manifests that carried it. It was the most-depended-on package in the workspace, so one generated feature list no longer decides what those 49 crates compile against.
- `crates/harness-internal/runner/Cargo.toml` adds `test-util` to its tokio dev-dependency, the feature behind the controllable test clock its suites pace performers with. The removed crate had supplied it through its normal tokio line, so it now stays out of builds without dev-dependencies.
- `Cargo.lock` only loses lines: the `workspace-hack` package block and the `"workspace-hack",` entry in each member's dependency list. No third-party package leaves the lockfile and no version changes.
- `.github/workflows/ci.yml` no longer installs `cargo-hakari` or runs `cargo hakari verify`. The one-crypto-backend gateway check is unchanged, and its comment drops hakari regeneration as a way for `ring` to come back.
- `tauri-plugin-updater` keeps `default-features = false`. Its comment now says the plugin's `rustls-tls` default would put `ring` into the Workshop binary, instead of into every member through the removed crate.
- `system-proxy` reached reqwest in every member through the removed crate's reqwest line, and no reqwest line in this change declares it in its place. The root reqwest pin still lists only `json`, `query`, `rustls`, and `stream`.

Design: removes god-module @ crates/workspace-hack/Cargo.toml
Design: removes hidden-dependency @ crates/harness-internal/runner/Cargo.toml
Plan: vibe/2026-10-13-3-remove-hakari.md
Turn on system proxy support in the workspace's shared HTTP client dependency. With hakari's unified feature set gone, that dependency entry has to ask for the feature itself, or the gateway would stop honoring proxy settings from the Windows registry and macOS system configuration. Default features stay off, and the rest of the feature list is unchanged.

- `reqwest` gains `system-proxy` on the root workspace pin rather than on a gateway crate's own dependency line, so every member that inherits the pin gets system proxy lookup, not only the gateway.

Plan: vibe/2026-10-13-3-remove-hakari.md
With hakari feature unification retired, the build tooling drops every allowance it made for hakari's crate. Outside their own family, Harness crates may now name only the Engine's two public crates, and Plugin crates may name only the Plugin contract, shared crates, and outside libraries. The build-script ceiling check loses its separate line-count path for a crate with no build script, so every crate meets the same wiring rule. Module docs and guard comments that listed the crate as an allowed dependency now match the narrower rules.

- `WORKSPACE_HACK` leaves both `product.rs` and `tidy-wiring.rs`, and `LINE_LIMIT` and `workspace_hack_violations` go with it.
- `ceiling_wiring_violations` runs the former `else` arm for every walked crate with no branch. Each crate has its `Cargo.toml` read and goes through `crate_violation`, so a crate without a build script meets the same `build.rs` rule as the rest.
- `boundary_breach` makes the `(Family::Harness, Family::Unaffiliated)` arm unconditional, so any unaffiliated dependency of a Harness crate is a breach. That arm and the `(Family::Harness, Family::Build)` arm now say Harness crates may name only `promptforge` and `promptforge-plugin` outside their family.
- `plugin_breach` allows `[PLUGIN_CONTRACT, package.package.as_str()]` plus `shared-*` crates, and its breach message no longer names `workspace-hack`.
- `a_harness_crate_depending_on_workspace_hack_passes` and `a_workspace_hack_file_over_the_limit_fails` go away. `a_plugin_crate_naming_the_contract_and_a_shared_crate_passes` replaces the old passing Plugin test without its `workspace-hack` line, and the remaining assertions expect the new messages.
- `## Invariants` sections in the Plugin crates, `promptforge-plugin`, and `promptforge-types` drop `workspace-hack` from their allowed dependencies. The guard comments in `engine_deps.rs`, `harness_bans.rs`, and `test_support_leak.rs` keep only that each check reads declared dependencies.

Plan: vibe/2026-10-13-3-remove-hakari.md
Every exit criterion for the hakari removal now passes on Windows, Linux, and macOS, and this records that. The checks needed no fixes, so the change touches no code, manifests, or workflows.

Plan: vibe/2026-10-13-3-remove-hakari.md
Plan: vibe/2026-10-13-3-remove-hakari.md
@vinniefalco
vinniefalco merged commit 7476ed8 into cppalliance:master Oct 14, 2026
41 checks passed

This branch was successfully deployed

1 active deployment
github-pages — 7476ed81 Deployed Oct 14, 2026 by vinniefalco via deploy #112
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant