fix: keyset pagination rejects NULL and duplicate keys at every batch size

.github/actions/codecov-report/action.yml

@@ -96,7 +96,7 @@ runs:
 
     - name: Upload commit SHA as artifact
       if: ${{ github.event_name == 'push' && inputs.php-version == '8.3' && inputs.dependencies == 'locked' }}
-      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+      uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
       with:
         name: codecov-base-commit-sha-${{ steps.baseline.outputs.branch_name }}
         path: codecov-base-commit-sha-${{ steps.baseline.outputs.branch_name }}.txt

.github/workflows/baseline.yml

@@ -58,7 +58,7 @@ jobs:
           ./build/flow.phar --version
 
       - name: "Upload Flow PHAR for Docker build"
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: "docker-phar"
           path: "build/flow.phar"
@@ -72,7 +72,7 @@ jobs:
           BUILD_TAG=${GITHUB_SHA:0:7}
           echo "BUILD_TAG=$BUILD_TAG" >> "$GITHUB_ENV"
 
-      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: flow-${{ env.BUILD_TAG }}.phar
           path: build/flow.phar

.github/workflows/job-docker-image.yml

@@ -103,7 +103,7 @@ jobs:
           touch "${DIGESTS_DIR}/${DIGEST#sha256:}"
 
       - name: "Upload digest"
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: "digests-${{ env.PLATFORM_PAIR }}"
           path: "${{ runner.temp }}/digests/*"

.github/workflows/release.yml

@@ -65,7 +65,7 @@ jobs:
           ./build/flow.phar
 
       - name: "Upload Flow PHAR for Docker build"
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: "docker-phar"
           path: "build/flow.phar"

.github/workflows/test-website.yml

@@ -46,7 +46,7 @@ jobs:
         run: "just phar"
 
       - name: "Upload Flow PHAR"
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: "website-flow-phar"
           path: "web/landing/assets/wasm/tools/flow.phar"

.nix/pkgs/php-arrow-ext/package.nix

@@ -5,7 +5,7 @@
   rustPlatform,
   clang,
   llvmPackages,
-  arrow-ext-version ? "0.45.0-dev",
+  arrow-ext-version ? "0.46.0-dev",
 }:
 
 let

.nix/pkgs/php-flow-php-ext/package.nix

@@ -5,7 +5,7 @@
   rustPlatform,
   clang,
   llvmPackages,
-  flow-php-ext-version ? "0.45.0-dev",
+  flow-php-ext-version ? "0.46.0-dev",
 }:
 
 let

documentation/components/adapters/doctrine.md

@@ -226,6 +226,41 @@ data_frame()
 
 The query runs once per parameter set, three times here.
 
+## Extractor - Keyset Pagination
+
+`from_dbal_key_set_qb()` reads a query builder page by page, asking for the rows after the last key it saw.
+
+The keys must be **indexed, non-null and unique together**. A single non-unique column such as `created_at` is not a
+key; add the primary key. `pagination_key_set()` takes the keys from least to most significant, so the last key is the
+primary sort:
+
+```php
+use function Flow\ETL\Adapter\Doctrine\{from_dbal_key_set_qb, pagination_key_asc, pagination_key_set};
+use function Flow\ETL\DSL\{data_frame, to_output};
+
+data_frame()
+    ->read(from_dbal_key_set_qb(
+        $connection,
+        $connection->createQueryBuilder()->select('id', 'created_at', 'total')->from('orders'),
+        pagination_key_set(pagination_key_asc('id'), pagination_key_asc('created_at')), // ORDER BY created_at, id
+    )->withBatchSize(1000))
+    ->limit(10_000)
+    ->write(to_output())
+    ->run();
+```
+
+- Before the first page the extractor sends one check query, the builder plus `WHERE (created_at IS NULL) OR (id IS
+  NULL) LIMIT 1`, and throws when it finds a row. On an indexed key this is an index lookup; on an unindexed key it is
+  one full scan.
+- Two rows with the same key throw `Keyset pagination requires unique keys`.
+- A next page that does not start at the row the previous page looked ahead to throws. This happens when the database
+  compares two different values as equal (a case- or accent-insensitive collation such as MySQL's default, `numeric`
+  `1.0` and `1.00`) or when rows change between pages.
+- Known limit: two such values on the same page are both read without an error. Nothing is lost.
+- The query builder must not have an ORDER BY, `setMaxResults()` or `setFirstResult()`; use `withMaximum()` or
+  `DataFrame::limit()`.
+- A page shorter than the batch size ends the read without another query.
+
 ## Schema Converter
 
 With `to_dbal_schema_table()` function we can convert any Flow Schema (which represents a dataset)

documentation/components/adapters/postgresql.md

@@ -119,7 +119,7 @@ data_frame()
 | Extractor                 | Best For                            | Memory                 | ORDER BY Required |
 |---------------------------|-------------------------------------|------------------------|-------------------|
 | `from_pgsql_cursor`       | Very large datasets, true streaming | Lowest (server-side)   | No                |
-| `from_pgsql_key_set`      | Large datasets with indexed keys    | Medium (page buffered) | Auto-generated    |
+| `from_pgsql_key_set`      | Large datasets with indexed keys    | Medium (page buffered) | Equal to the keys |
 | `from_pgsql_limit_offset` | Small-medium datasets               | Medium (page buffered) | Yes               |
 
 ## Extractor - LIMIT/OFFSET Pagination
@@ -180,8 +180,29 @@ data_frame()
 The `from_pgsql_key_set` extractor uses keyset pagination (also known as cursor-based pagination). This provides
 consistent performance regardless of how deep you paginate, making it ideal for large datasets.
 
-> **Note:** The ORDER BY clause is automatically generated from the keyset configuration. You only need to define
-> the sort order once using `pgsql_pagination_key_asc()` or `pgsql_pagination_key_desc()`.
+### Key rule
+
+The keys must be **indexed, non-null and unique together**. A single non-unique column such as `created_at` is not a
+key; add the primary key as the least significant key:
+
+```php
+pgsql_pagination_key_set(
+    pgsql_pagination_key_asc('created_at'),
+    pgsql_pagination_key_asc('id'),
+);
+```
+
+- Before the first page the extractor sends one check query,
+  `SELECT 1 FROM (<query>) _flow_keyset_nulls WHERE created_at IS NULL OR id IS NULL LIMIT 1`, and throws when it
+  finds a row. On an indexed key this is an index lookup; on an unindexed key it is one full scan.
+- Two rows with the same key throw `Keyset pagination requires unique keys`.
+- A next page that does not start at the row the previous page looked ahead to throws. This happens when the database
+  compares two different values as equal (a case- or accent-insensitive collation, `numeric` `1.0` and `1.00`) or when
+  rows change between pages.
+- Known limit: two such values on the same page are both read without an error. Nothing is lost.
+- ORDER BY is generated from the keys; a query's own ORDER BY must equal them. The query must not have its own
+  LIMIT or OFFSET; use `withMaximum()` or `DataFrame::limit()`.
+- A page shorter than the batch size ends the read without another query.
 
 ### Basic Usage
 

documentation/components/libs/postgresql.md

@@ -494,15 +494,23 @@ $page1 = sql_to_keyset_query($sql, limit: 100, columns: $columns, cursor: null);
 // SELECT * FROM audit_log ORDER BY created_at DESC, id DESC LIMIT 100
 
 $page2 = sql_to_keyset_query($sql, limit: 100, columns: $columns, cursor: ['2025-01-15 14:30:00', 1000]);
-// SELECT * FROM audit_log WHERE created_at < $1 OR (created_at = $1 AND id < $2) ORDER BY created_at DESC, id DESC LIMIT 100
+// SELECT * FROM audit_log WHERE (created_at, id) < ($1, $2) ORDER BY created_at DESC, id DESC LIMIT 100
 ```
 
 The cursor values come from the last row of the previous page. Keyset pagination:
 
 - Uses O(log n) index lookups instead of O(n) row scanning
-- Handles mixed ASC/DESC sort orders correctly
+- Uses a row comparison `(a, b) > ($1, $2)` when all keys sort the same direction, and an `OR` expansion when
+  directions are mixed
 - Works with existing WHERE conditions (combined with AND)
 
+Rules:
+
+- Key columns must be non-null and unique together; a NULL cursor value throws `PaginationException`
+- ORDER BY is optional; when present it must list exactly the keys, in order, with no NULLS clause, ordinal,
+  expression or USING
+- The query must not have its own LIMIT or OFFSET; pass the page size as `limit`
+
 ---
 
 ## Advanced Features
@@ -678,7 +686,7 @@ $query->traverse(new KeysetPaginationModifier(new KeysetPaginationConfig(
     cursor: ['2025-01-15', 42]
 )));
 echo $query->deparse();
-// SELECT * FROM users WHERE created_at > $1 OR (created_at = $1 AND id > $2) ORDER BY created_at, id LIMIT 10
+// SELECT * FROM users WHERE (created_at, id) > ($1, $2) ORDER BY created_at, id LIMIT 10
 ```
 
 `ExplainModifier` wraps every statement before any of them is visited, so run it in its own `traverse()` after the

documentation/upgrading.md

@@ -9,6 +9,97 @@ Upgrading from older versions: [0.35.x to 0.40.x](/documentation/upgrading-0.35-
 
 ---
 
+## Upgrading from 0.45.x to 0.46.x
+
+### 1) `flow-php/etl-adapter-postgresql`, `flow-php/etl-adapter-doctrine` - keyset reads reject NULL keys up front
+
+| Before                                                                                 | After                                                                                                         |
+|----------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|
+| a NULL key: rows lost or `NULL value found in column ...`, depending on the batch size | one check query before the first page, then `RuntimeException` `Keyset pagination requires non-null keys ...` |
+| no query before the first page                                                         | `SELECT 1 FROM (<query>) ... WHERE k IS NULL LIMIT 1` (DBAL: the builder plus `IS NULL` conditions)           |
+
+Filter NULL keys out with `IS NOT NULL`, or page by non-null columns. Index the keys - on an unindexed key the check is one full scan.
+
+### 2) `flow-php/etl-adapter-postgresql`, `flow-php/etl-adapter-doctrine` - keyset reads reject duplicate keys
+
+| Before                                                                                              | After                                                                                                                  |
+|-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
+| two rows with the same key: rows at a page boundary silently skipped                                | `RuntimeException` `Keyset pagination requires unique keys ...`                                                        |
+| keys equal in the database but different in PHP (`'a'`/`'A'` under a `_ci` collation, `1.0`/`1.00`) | `RuntimeException` `Keyset pagination expected the next page to start at the key ...` when a page boundary splits them |
+
+Add a unique column (for example the primary key) to the key set as its least significant key.
+
+### 3) `flow-php/etl-adapter-postgresql`, `flow-php/etl-adapter-doctrine` - a short keyset page ends the read
+
+| Before                                                | After                                                                                             |
+|-------------------------------------------------------|---------------------------------------------------------------------------------------------------|
+| each page `LIMIT n`; reading stopped on an empty page | each page `LIMIT n + 1` (the extra row is not yielded); a page of `n` rows or fewer ends the read |
+| one trailing empty `SELECT`                           | none                                                                                              |
+
+Nothing to change unless you count queries: a read sends the NULL check plus the page queries.
+
+### 4) `flow-php/postgresql`, `flow-php/etl-adapter-postgresql` - keyset pagination rejects the query's own LIMIT/OFFSET
+
+| Before                                                                               | After                                                             |
+|--------------------------------------------------------------------------------------|-------------------------------------------------------------------|
+| `sql_to_keyset_query('SELECT ... LIMIT 5', 10, ...)` - LIMIT silently replaced by 10 | `PaginationException` `Keyset pagination sets its own LIMIT; ...` |
+| `from_pgsql_key_set($client, 'SELECT ... LIMIT 5', ...)`                             | same `PaginationException`, before any query                      |
+
+Drop LIMIT/OFFSET from the query; use `withMaximum()` or `DataFrame::limit()`.
+
+### 5) `flow-php/etl-adapter-doctrine` - `from_dbal_key_set_qb()` rejects `setMaxResults()` / `setFirstResult()`
+
+| Before                                                          | After                                                                             |
+|-----------------------------------------------------------------|-----------------------------------------------------------------------------------|
+| `$qb->setMaxResults(5)` - silently overwritten by the page size | `InvalidArgumentException` `Keyset pagination sets its own LIMIT and OFFSET, ...` |
+| `$qb->setFirstResult(5)` - applied to every page                | same `InvalidArgumentException`                                                   |
+
+Remove them from the builder; use `withMaximum()` or `DataFrame::limit()`.
+
+### 6) `flow-php/postgresql`, `flow-php/etl-adapter-postgresql` - a query's own ORDER BY must equal the keys
+
+| Before                                                                                 | After                                                                                          |
+|----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------|
+| `'SELECT ... ORDER BY name'` with key `id` - kept, pages wrong                         | `PaginationException` `Keyset pagination requires ORDER BY item #1 to be the key "id ASC" ...` |
+| `ORDER BY id NULLS FIRST`, `ORDER BY 1`, `ORDER BY lower(name)`, `ORDER BY id USING <` | same `PaginationException`                                                                     |
+
+Drop the ORDER BY (it is generated from the keys) or make it list exactly the keys, in order and direction.
+
+### 7) `flow-php/postgresql` - `KeysetPaginationConfig` / `sql_to_keyset_query()` reject a NULL cursor value
+
+| Before                                                                                 | After                                                                                |
+|----------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| `new KeysetPaginationConfig(10, $columns, [null, 7])` - predicate that matches nothing | `PaginationException` `Keyset cursor value #1 is NULL; key columns must be non-null` |
+
+Page by non-null key columns.
+
+### 8) `flow-php/postgresql` - same-direction composite keys emit a row comparison
+
+| Before                                                       | After                         |
+|--------------------------------------------------------------|-------------------------------|
+| `created_at > $1 OR (created_at = $1 AND id > $2)`           | `(created_at, id) > ($1, $2)` |
+| all DESC: `created_at < $1 OR (created_at = $1 AND id < $2)` | `(created_at, id) < ($1, $2)` |
+
+Nothing to change; the row comparison is an index range bound. Mixed ASC/DESC keys keep the `OR` form. Update assertions on the generated SQL.
+
+### 9) `flow-php/etl-adapter-doctrine` - a non-scalar keyset value is rejected
+
+| Before                                                                       | After                                                                                        |
+|------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
+| a key value that is not `string`/`int`/`float`/`bool` - bound as a parameter | `RuntimeException` `Unsupported value type "<type>" in column "<key>" for keyset pagination` |
+
+Page by scalar columns.
+
+### 10) `flow-php/etl-adapter-doctrine` - `pagination_key_set()` rejects zero keys
+
+| Before                                                                                                             | After                                                                                       |
+|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
+| `pagination_key_set()` built; `from_dbal_key_set_qb()` threw `KeySet must contain at least one key for pagination` | `pagination_key_set()` throws `InvalidArgumentException` `KeySet requires at least one key` |
+
+Pass at least one key.
+
+---
+
 ## Upgrading from 0.44.x to 0.45.x
 
 ### 1) `flow-php/etl-adapter-postgresql` - a failed `from_pgsql_cursor()` read throws its own error and rolls back