Skip to content

Add streaming shared subplan example #24780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nathanb9 wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add streaming shared subplan example #24780

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c760798
docs: add streaming shared subplan example
nathanb9 Aug 30, 2026
cc6816a
Merge remote-tracking branch 'origin/main' into HEAD
nathanb9 Aug 30, 2026
287af36
fix: use channel capacity for fanout metric
nathanb9 Aug 30, 2026
38cc480
refactor: separate streaming fanout consumers
nathanb9 Aug 30, 2026
a5ce077
docs: format streaming example entry
nathanb9 Aug 30, 2026
eea8c09
docs: explain streaming sharing extension points
nathanb9 Aug 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions datafusion-examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,16 +178,17 @@ cargo run --example dataframe -- dataframe

#### Category: Single Process

| Subcommand | File Path | Description |
| -------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------ |
| analyzer_rule | [`query_planning/analyzer_rule.rs`](examples/query_planning/analyzer_rule.rs) | Custom AnalyzerRule to change query semantics |
| expr_api | [`query_planning/expr_api.rs`](examples/query_planning/expr_api.rs) | Create, execute, analyze, and coerce Exprs |
| optimizer_rule | [`query_planning/optimizer_rule.rs`](examples/query_planning/optimizer_rule.rs) | Replace predicates via a custom OptimizerRule |
| parse_sql_expr | [`query_planning/parse_sql_expr.rs`](examples/query_planning/parse_sql_expr.rs) | Parse SQL into DataFusion Expr |
| plan_to_sql | [`query_planning/plan_to_sql.rs`](examples/query_planning/plan_to_sql.rs) | Generate SQL from expressions or plans |
| planner_api | [`query_planning/planner_api.rs`](examples/query_planning/planner_api.rs) | APIs for logical and physical plan manipulation |
| pruning | [`query_planning/pruning.rs`](examples/query_planning/pruning.rs) | Use pruning to skip irrelevant files |
| thread_pools | [`query_planning/thread_pools.rs`](examples/query_planning/thread_pools.rs) | Configure custom thread pools for DataFusion execution |
| Subcommand | File Path | Description |
| ------------------------ | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| analyzer_rule | [`query_planning/analyzer_rule.rs`](examples/query_planning/analyzer_rule.rs) | Custom AnalyzerRule to change query semantics |
| expr_api | [`query_planning/expr_api.rs`](examples/query_planning/expr_api.rs) | Create, execute, analyze, and coerce Exprs |
| optimizer_rule | [`query_planning/optimizer_rule.rs`](examples/query_planning/optimizer_rule.rs) | Replace predicates via a custom OptimizerRule |
| parse_sql_expr | [`query_planning/parse_sql_expr.rs`](examples/query_planning/parse_sql_expr.rs) | Parse SQL into DataFusion Expr |
| plan_to_sql | [`query_planning/plan_to_sql.rs`](examples/query_planning/plan_to_sql.rs) | Generate SQL from expressions or plans |
| planner_api | [`query_planning/planner_api.rs`](examples/query_planning/planner_api.rs) | APIs for logical and physical plan manipulation |
| pruning | [`query_planning/pruning.rs`](examples/query_planning/pruning.rs) | Use pruning to skip irrelevant files |
| streaming_shared_subplan | [`query_planning/streaming_shared_subplan.rs`](examples/query_planning/streaming_shared_subplan.rs) | Stream one subplan into multiple consumers |
| thread_pools | [`query_planning/thread_pools.rs`](examples/query_planning/thread_pools.rs) | Configure custom thread pools for DataFusion execution |

## Relation Planner Examples

Expand Down
10 changes: 9 additions & 1 deletion datafusion-examples/examples/query_planning/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
//!
//! ## Usage
//! ```bash
//! cargo run --example query_planning -- [all|analyzer_rule|expr_api|optimizer_rule|parse_sql_expr|plan_to_sql|planner_api|pruning|thread_pools]
//! cargo run --example query_planning -- [all|analyzer_rule|expr_api|optimizer_rule|parse_sql_expr|plan_to_sql|planner_api|pruning|streaming_shared_subplan|thread_pools]
//! ```
//!
//! Each subcommand runs a corresponding example:
Expand All @@ -48,6 +48,9 @@
//! - `pruning`
//! (file: pruning.rs, desc: Use pruning to skip irrelevant files)
//!
//! - `streaming_shared_subplan`
//! (file: streaming_shared_subplan.rs, desc: Stream one subplan into multiple consumers)
//!
//! - `thread_pools`
//! (file: thread_pools.rs, desc: Configure custom thread pools for DataFusion execution)

Expand All @@ -58,6 +61,7 @@ mod parse_sql_expr;
mod plan_to_sql;
mod planner_api;
mod pruning;
mod streaming_shared_subplan;
mod thread_pools;

use datafusion::error::{DataFusionError, Result};
Expand All @@ -75,6 +79,7 @@ enum ExampleKind {
PlanToSql,
PlannerApi,
Pruning,
StreamingSharedSubplan,
ThreadPools,
}

Expand All @@ -100,6 +105,9 @@ impl ExampleKind {
ExampleKind::PlanToSql => plan_to_sql::plan_to_sql_examples().await?,
ExampleKind::PlannerApi => planner_api::planner_api().await?,
ExampleKind::Pruning => pruning::pruning()?,
ExampleKind::StreamingSharedSubplan => {
streaming_shared_subplan::streaming_shared_subplan().await?
}
ExampleKind::ThreadPools => thread_pools::thread_pools().await?,
}
Ok(())
Expand Down
Loading