Skip to content

feat(services): export runtime variables from streaming targets - #77

Open
ark-archastro wants to merge 6 commits into
mainfrom
feature/target-variable-exports
Open

ark-archastro wants to merge 6 commits into
mainfrom
feature/target-variable-exports

Conversation

@ark-archastro

@ark-archastro ark-archastro commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What this enables for Aster clients

This PR lets an Aster-managed service discover configuration at runtime and publish it to selected dependent services. Clients no longer need an external launcher to start a producer, scrape a value, export it into the shell, and then hand the remaining lifecycle back to Aster.

A representative use case is a credentials helper whose token is not known until the helper is running:

Aster starts the credentials producer
  → the producer publishes its first variable snapshot
  → Aster considers the producer ready
  → Aster starts dependent services with their allowlisted values
  → later value changes restart only consumers whose effective environment changed

This supports runtime-discovered credentials, dynamically allocated endpoints, local emulators, tunnels, and similar long-running tools while keeping startup ordering, readiness, restarts, logs, and teardown under one Aster supervisor.

Client configuration

The following is one end-to-end example spread across three aster.toml files.

1. Producer target — services/credentials/aster.toml

[targets.dev]
command = "./publish-credentials"

# Run this target as a long-lived process under the service supervisor.
stream = true

# Aster gives this process an ASTER_EXPORT_VAR_PATH FIFO. The process writes
# complete JSONL snapshots such as {"DISCOVERED_TOKEN":"example"} to that FIFO.
exports_vars = true

2. Consumer target — services/api/aster.toml

[targets.dev]
command = "./start-api"
stream = true

# This direct dependency makes the API eligible to consume variables published
# by //services/credentials:dev. It also starts credentials before the API.
depends_on = ["//services/credentials:dev"]

3. Service wiring — workspace-root aster.toml

# Start the target that discovers and publishes DISCOVERED_TOKEN.
[dev.services.credentials]
target = "//services/credentials:dev"

# Start the API target after the credentials producer is ready.
[dev.services.api]
target = "//services/api:dev"

# Of all variables published by the API's direct dependencies, copy only
# DISCOVERED_TOKEN into the API process environment.
inherit_env = ["DISCOVERED_TOKEN"]

The resulting data flow is:

//services/credentials:dev
  writes {"DISCOVERED_TOKEN":"..."} to ASTER_EXPORT_VAR_PATH
    → Aster validates and stores the snapshot
      → //services/api:dev receives DISCOVERED_TOKEN in its process environment

The producer writes newline-delimited JSON objects containing complete string-to-string snapshots. The capability is available through aster services up; exporter targets are rejected by aster run, aster watch, and direct target execution because those paths do not provide the service-supervisor lifecycle contract.

Client-visible behavior

  • Consumers wait for the producer's first valid snapshot instead of starting with missing runtime configuration.
  • Only direct target dependencies can receive exports, and each service must opt into individual names with inherit_env.
  • Exports override ambient and env-file values. Explicit service env, resolved ports, Aster internals, and leading command assignments remain final.
  • A changed effective environment restarts affected consumers. Identical, unused, or higher-precedence-masked changes do not cause restarts.
  • If a producer exits or publishes invalid data, Aster stops consumers using its values and holds them until a healthy snapshot is available.
  • Nested exporters are reconciled in dependency order, so downstream generations cannot publish or consume stale state.
  • Publication records are bounded to 64 KiB, malformed payloads are not logged, and each producer generation gets a private mode-0600 FIFO.

The result is a least-privilege, fail-closed mechanism for feeding runtime-discovered values into startup-only process environments without splitting ownership across Aster and a wrapper process.

Implementation notes

Streaming service targets can set exports_vars = true; aliases preserve that capability during target resolution. The service supervisor tracks producer health and desired/applied snapshots, fences events by producer and consumer generation, coalesces rapid changes, and restarts only consumers whose resolved environment changes.

Nested exporters are a stricter lifecycle boundary than ordinary consumers. If an upstream input changes, Aster immediately holds the nested exporter and its downstream tree. A publication is accepted only when it comes from a running generation whose applied values, sources, and upstream epochs are still current. This prevents output produced from stale inputs from reaching downstream services while a restart is queued.

Before the supervisor first becomes ready, every exporter must remain alive and healthy. An exporter that publishes once and then exits while another exporter is still starting now fails startup instead of leaving the supervisor permanently pending.

The daemon protocol is 5 so ready-deadline refresh records are not treated as ready by a v4 daemon.

Scope and limitations

  • Unix service supervision only; Windows transport is not included.
  • The transport is intentionally local to Aster's private runtime directory.
  • Snapshots contain strings only and replace the producer's previous complete snapshot.
  • This PR provides the generic Aster primitive. Repository-specific producer wrappers and service wiring remain client work.

Verification

  • cargo fmt --all -- --check
  • cargo clippy --locked --all-targets --all-features -- -D warnings
  • cargo test --locked --all-features --lib (512 unit tests)
  • cargo test --locked --all-features --test dev_services export (four exporter lifecycle and CLI integration tests)
  • cargo test --locked --all-features --test integration --test watch_tests --test config_bug_bash
  • All GitHub CI checks pass on commit 42cf7ef: Ubuntu, macOS, quality, MSRV, dependency audit, packaging, and CodeQL.
  • The dependency-audit failure was RUSTSEC-2026-0285, newly reported against rustls 0.23.43. The lockfile now uses patched rustls 0.23.45 and rustls-webpki 0.103.15; the audit rerun passes.

The regression coverage includes initial-snapshot gating, effective-value restarts, direct-dependency scoping, nested-exporter epoch fencing, failure when an exporter exits before global readiness, and rejection from execution modes outside aster services up.

Follow-up

Extract the exporter lifecycle decisions from run_dev into a pure supervisor state machine. The state model should own service and producer generations, desired-versus-applied snapshots, readiness deadlines, holds, and restart queues, then emit Start, Stop, Hold, and PublishReady actions for the existing runner to execute. This is intentionally left as a behavior-preserving follow-up so it does not add broad refactoring risk to this feature PR.

Checklist

  • Tests cover the behavior change
  • Documentation is updated where needed
  • No secrets, private fixtures, or generated artifacts are included
  • Formatting, clippy, unit tests, focused integration tests, and platform CI pass

Streaming service targets can set exports_vars to publish JSON snapshots
over a private FIFO. Direct dependents consume allowlisted names through
inherit_env, and exporter targets are restricted to services up.
- Exporter start failure after ready aborted the supervisor — hold
  consumers instead, matching process-exit after the first snapshot.
- 20s first-snapshot timer included prerequisites and later generations —
  start the clock when the process is running and abort only before ready.
- Colliding export keys unwound the event loop — treat collisions as a
  hold for the consumer, not a supervisor fatal error.
- aster run (and watch/executor spawn) could still execute exporters —
  reject at the CLI closure and at the spawn seams.
- Daemon ready deadline was a single 25s window for serial exporters —
  refresh it as the supervisor starts each generation.
- No tests for the services-up-only gate — added CLI and spawn coverage.

Reviewers: grok-native, review-principles
- Ready-deadline refresh used protocol 4 records an old daemon treats as
  ready — bump to protocol 5 so mixed binaries replace the daemon.
- Invalid snapshot reasons forwarded serde text that can include payload
  values — keep parser errors generic so secrets stay out of logs.
- Invalid data after a first snapshot left ready hung with no 20s abort —
  re-arm the wait and time out on unhealthy, not only missing values.
- Daemon deadline was not refreshed while a start generation ran —
  keep extending it while a start is in flight.
- Non-Unix rejected any workspace exporter, not just selected services —
  gate Unix support on the selected plan.
- --no-deps still walked exporter dependencies — skip that closure when
  the exporter would not run.

Reviewers: grok-native, codex-cli, review-principles
- Homogeneous --no-deps still collected same-project exporter deps and
  failed the run — skip dependency expansion in execute_internal too.
- Pre-ready abort still said "initial snapshot" after invalid-after-
  snapshot — report that the exporter did not become healthy.

Reviewers: grok-native, review-principles
@ark-archastro
ark-archastro requested a review from a team September 3, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant