Skip to content

feat(pull): add pull-based streams adapter - #367

Draft
tryangul wants to merge 9 commits into
mainfrom
rbroughan/pull-based-runtime
Draft

feat(pull): add pull-based streams adapter #367
tryangul wants to merge 9 commits into
mainfrom
rbroughan/pull-based-runtime

Conversation

@tryangul

@tryangul tryangul commented Aug 7, 2026

Copy link
Copy Markdown

What

PullBasedAdapter — new streams adapter backed by the arroyo pull-based runtime.

Depends on arroyo#559.

How

  • PullBasedAdapter — drop-in StreamAdapter replacement for RustArroyoAdapter. Translates DSL steps (HeadersFilter, Batch, Map, GCSSink, etc.) into pull-based operators. Registered as "pull" in loader.py.
  • PullConsumer — the runtime entry point. Rebuilds fresh stages on each rebalance (matching the push model's destroy/recreate pattern). Parameterless run() with SIGINT/SIGTERM handling.
  • PipelineValue — type-erased payload (Raw/Rust/Python) so the adapter can compose stages dynamically from the DSL without compile-time generics for every pipeline shape.
  • PyCallableStage — bridges Rust and Python. Wraps input in a streams Message object before calling so existing callables (batch_msg_parser, ItemsSpanProcessor, serialize_to_parquet) work unchanged.
  • GcsClient — injectable GCS upload client, separated from sink handler for testability.

Why

Steel thread for the pull-based runtime — proves the arroyo primitives work end-to-end with the streams DSL and real pipeline callables (protobuf parsing, parquet serialization).

Tests (11)

  • 6 Rust — stage unit tests, consumer e2e, graceful shutdown
  • 5 Python — adapter translation, e2e with Message wrapping, real BatchParser (protobuf via sentry_kafka_schemas) + ParquetSerializer (polars) through the pipeline

Notes

  • Stages rebuilt fresh on each rebalance — no stale batch state
  • No test infrastructure in production code — Python e2e tests use a capturing Map step
  • Concurrent GCS uploads not yet supported — straightforward via .apply_concurrent()

Next steps

  • Deploy to SBC items_span to benchmark
  • Batch timeout flush
  • GCS upload retry (backon)

STREAM-1668

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant