Repository navigation
Conversation
|
Do we want to expand on with configuration as well? We'd can have this blog as ballista py introduction? |
Sure! Do you want to make some suggestions? |
|
cc @timsaucer |
| ctx.sql("create external table t stored as parquet location 'data/t.parquet'") | ||
|
|
||
|
|
||
| def is_null(array: pa.Array) -> pa.Array: |
There was a problem hiding this comment.
Can we craft better function here ?
| `ballista` Python package. Until then, you can build the package from source by following the instructions in | ||
| the [Python client README]. The [example script] starts a local test cluster and walks through both the | ||
| DataFrame and SQL forms shown above. | ||
|
|
There was a problem hiding this comment.
Maybe we should mention that we're going to do further testing and improvements
|
Once we merge the remaining PR we can mention how to convert your current df Python script to Ballista. |
Replace the is_null UDF with one that encodes the TPC-H Query 19 discount rules in a Python dictionary and evaluates them with pyarrow.compute kernels, so batches stay in Arrow. - Show the UDF from both the DataFrame API and SQL - Add steps to generate data with tpchgen-cli and start a local scheduler and executor from the ballista Python package - Print from the UDF so readers can see it run in the executor process Tested against a local Ballista cluster built from main. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since Ballista now returns standard DataFusion Python objects, point readers to the DataFusion Python user guide and API reference, and contrast with the old wrapper, where methods it did not intercept ran locally. Note that DataFrame.cache() is not yet supported because its in-memory table cannot be serialized into the plan. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Explain the concrete cost of the Ballista-specific DataFrame: running on the cluster depended on re-wrapping every returned DataFrame, and write options were silently ignored. Note that tracking the DataFusion Python API caused churn in the Ballista repository, such as mirroring the Parquet writer options field by field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only the main execution methods were intercepted, not every method that executed a plan, which is why others ran locally. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
I had a bunch of recommendations so I opened a PR targeting your branch instead of a bunch of inline comments. andygrove#3 |
timsaucer
left a comment
timsaucer
left a comment
There was a problem hiding this comment.
I posted the bulk of my suggestions in the PR, but there were a couple of things here I didn't edit.
|
|
||
| ## Trying it out | ||
|
|
||
| Python UDF support is on the `main` branch of Ballista and will be included in the next release of the |
There was a problem hiding this comment.
How close are we to releasing the 55.0.0 python bindings? If we hold this off until then we can give instructions with released wheels.
There was a problem hiding this comment.
I would like to merge apache/datafusion-ballista#2637 before we release it, I won't be able to work on it until weekend. Also, I don't think we're in rush
| ## The building blocks | ||
|
|
||
| Most of the work behind this feature happened upstream, in DataFusion Python and the `datafusion-ffi` crate. | ||
| [Tim Saucer] has led the DataFusion FFI effort for a long time, and over the DataFusion 54 and 55 cycles he |
There was a problem hiding this comment.
FWIW it's not important at all to me that I get recognition for this. I wouldn't mind if we just list the work.
There was a problem hiding this comment.
If someone needs to get recognition for this work it's you @timsaucer this would not be possible without you.
Use TPC-H Q19 as the Python UDF example
Thank you. I merged this. |
Adds a blog post announcing Python UDF support in the Ballista Python client (apache/datafusion-ballista#2619), along with the move to a native DataFusion Python integration through the FFI query planner (apache/datafusion-ballista#2588).
The post covers:
datafusion.SessionContextinstead of wrapping itThe date in the filename and front matter will be updated to match the merge date.