Repository navigation
Parallel Processing #1460
Description
Flow is amazing, but the true power would come with a parallel processing of datasets.
The absolute minimum we need to prepare for that is:
- parallel data extraction
- dataframe serialization
- data shuffling (to move data between workers)
We also need to define the communication protocol between workers/nodes (something like thrift or protobuf)
Activity
norberttech commentedon Oct 2, 2025
For the last few weeks I have been mostly enjoying vacations (that's why there was a gap in releases). Before that I started working on the playground project, but during my break I have been thinking quite a lot about the project.
Long story short I think I'm going to shift the priorities slightly and delay the playground a bit (probably until next year) and focus on parallel processing which seems the be the biggest missing piece.
What is parallel processing?
Reading datasets in chunks (splitting them into X chunks and processing on Y processors) but also writing them in chunks.
For now the idea is to us child processes with a communication protocol based on Apache Thrift
The thrift protocol would require following building blocks definitions (to serialize them and send across processes)
In the first phase only following data sources are going to be supported:
In the second phase we should add:
The most tricky part is going to be data shuffeling between processors for operations like: Sorting / Grouping / Aggregating / Pivoting
Those algorithms usually requires access to the entire dataset (at least at some pooint), other than that most of the pieces are already in place.
So in general there are going to be 2 independant protocols used here:
There is something called MPI https://pl.wikipedia.org/wiki/Message_Passing_Interface - which is exactly what I'm looking for but unfortunately there are no implementaitons in PHP.
Apache Spark for example uses https://netty.io/ (a lib for client/server communication), which is on the conceptual level very similar to MPI,
So it seems I have no other choice than design something very similar, a Client / Server communication library that would be an abstraction layer.
As for the implementation I'm thinking about creating adapters for HTTP2.0 and TCP and as for the implementation:
mkoskl commentedon Sep 25, 2026
@norberttech what's the status with parallel processing?
norberttech commentedon Sep 25, 2026
Well it's hard to summarize it here, as it's a maaasive task that has many dependencies that needs to happen first.
Many of them are already in place, like:
Right now I'm working on changing the Rows by making it lazy materialized.
Our biggest bottleneck right now is that while reading data from any source, regardless if the columns are used or not, we are still materializing all the Values in order to build Row which is necessary to build Rows.
The new apporach, changes Rows into a proper binary data container, when values are materialized only when accessed.
This means that csv dataset with 25 columns where only 3 of them are used for transformations will no longer need to go through the full process of building all 25 columns for all rows, it will just build those that are used.
This creates new optimization angle that Optimizer will be able to leverage.
Only when those things are done, I'm going to move to parallel execution - for which I already have quite a lot designed (thats how I figured out the other things, from a design).
But to be completely fair, I'm not sure if flow-php/parallel will be released under open source MIT license, I might need to put it behind the paywall (this is not yet a final decision, but an option I'm exploring).
The best way to stay on top of the latest progress it through Flow Discord Server - It's a form of digital garden where I publish my thoughts/progress/designs
mkoskl commentedon Sep 25, 2026
@norberttech ok, thanks for the answer. I appreciate you putting effort to give detailed answer. Thanks!