diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index 5bba39625f606..8c82de8e3a45f 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -172,12 +172,14 @@ cat <<'EOF' >> "$TARGET_FILE" # Tuning Guide -## Short Queries +## Short Queries and Point Lookups By default DataFusion will attempt to maximize parallelism and use all cores -- For example, if you have 32 cores, each plan will split the data into 32 -partitions. However, if your data is small, the overhead of splitting the data -to enable parallelization can dominate the actual computation. +partitions. However, if your data is small, or the query only reads a few rows +(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the +overhead of planning, opening files, and splitting the data to enable +parallelization can dominate the actual computation. You can find out how many cores are being used via the [`EXPLAIN`] command and look at the number of partitions in the plan. @@ -190,7 +192,7 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; @@ -198,6 +200,28 @@ SET datafusion.execution.target_partitions = '1'; [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +For point lookups, the following techniques also help: + +- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). + This makes techniques such as Parquet min/max metadata pruning more effective. +- Enable `datafusion.execution.parquet.pushdown_filters` so filters are + evaluated while decoding Parquet. +- Set the Parquet metadata cache size via + `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the + files you query repeatedly. +- Reuse prepared statements to avoid re-planning each query. + +Here is how to enable pushdown filters: + +```sql +SET datafusion.execution.parquet.pushdown_filters = true; +``` + +See the [Best Practices for Single Row Lookups (Point Queries)] blog post for +more details. + +[Best Practices for Single Row Lookups (Point Queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/ + ## Memory-limited Queries When executing a memory-consuming query under a tight memory limit, DataFusion diff --git a/docs/source/user-guide/concepts-readings-events.md b/docs/source/user-guide/concepts-readings-events.md index 2024dc6474caf..5d7a066b3e658 100644 --- a/docs/source/user-guide/concepts-readings-events.md +++ b/docs/source/user-guide/concepts-readings-events.md @@ -37,6 +37,8 @@ This is a list of DataFusion related blog posts, articles, and other resources. Please open a PR to add any new resources you create or find +- **2026-10-24** [Blog: Best Practices for Single Row Lookups (Point Queries)](https://datafusion.apache.org/blog/2026/10/24/point-queries/) + - **2026-10-01** [Blog: How we extended Apache DataFusion to execute one query across many machines](https://www.datadoghq.com/blog/engineering/distributed-datafusion/) - **2026-07-29** [Blog: Bringing DuckLake to DataFusion](https://ducklake.select/2026/07/29/bringing-ducklake-to-datafusion/) diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md index ca81398508b17..4b9fa14c6759a 100644 --- a/docs/source/user-guide/configs.md +++ b/docs/source/user-guide/configs.md @@ -259,12 +259,14 @@ The following runtime configuration settings are available: # Tuning Guide -## Short Queries +## Short Queries and Point Lookups By default DataFusion will attempt to maximize parallelism and use all cores -- For example, if you have 32 cores, each plan will split the data into 32 -partitions. However, if your data is small, the overhead of splitting the data -to enable parallelization can dominate the actual computation. +partitions. However, if your data is small, or the query only reads a few rows +(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the +overhead of planning, opening files, and splitting the data to enable +parallelization can dominate the actual computation. You can find out how many cores are being used via the [`EXPLAIN`] command and look at the number of partitions in the plan. @@ -277,7 +279,7 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; @@ -285,6 +287,28 @@ SET datafusion.execution.target_partitions = '1'; [`listingtable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +For point lookups, the following techniques also help: + +- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). + This makes techniques such as Parquet min/max metadata pruning more effective. +- Enable `datafusion.execution.parquet.pushdown_filters` so filters are + evaluated while decoding Parquet. +- Set the Parquet metadata cache size via + `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the + files you query repeatedly. +- Reuse prepared statements to avoid re-planning each query. + +Here is how to enable pushdown filters: + +```sql +SET datafusion.execution.parquet.pushdown_filters = true; +``` + +See the [Best Practices for Single Row Lookups (Point Queries)] blog post for +more details. + +[best practices for single row lookups (point queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/ + ## Memory-limited Queries When executing a memory-consuming query under a tight memory limit, DataFusion