andygrove · Oct 28, 2026
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -16,7 +16,9 @@ ruleset in `.asf.yaml`. That splits CI into three tiers:
 
 - **PR tier** (`pr`): fast feedback while a change is being iterated on.
   The Linux build, with the Comet test suites run against the default Spark
-  profile (4.1) only, and nothing else.
+  profile (4.1) only, and nothing else. For a pull request from a fork this
+  tier runs on the contributor's fork instead of here; see
+  [Fork runs](#fork-runs).
 - **Queue tier** (`queue`): the authoritative gate. The PR tier plus the
   macOS build, the benchmark compile check, the Delta contrib build gate, the
   PyArrow UDF suite, Spark SQL on Spark 4.1 and Iceberg 1.11, evaluated
@@ -198,6 +200,57 @@ still needs its label, so a `labeled` run there adds `spark_3_4` or nothing.
 `check-ci-config.py` pins that wiring, because a dropped variable reads as an
 empty string and would route the pull request as if it targeted `main`.
 
+## Fork runs
+
+Nearly every pull request comes from a fork, and the PR tier is the
+largest share of the runner-hours CI spends (about 60% in the week to
+2026-10-28, label runs included). A fork pull request therefore does not run
+the PR tier on ASF's runners. The contributor's fork runs it, and the merge
+queue, which stays on ASF, runs everything on the merge result before anything
+lands. The design follows `apache/datafusion` (#25591) and `apache/arrow-rs`
+(#11788). Three files carry it:
+
+| File                 | Role                                                                                                                                                       |
+| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ci_fork.yml`        | Fires on a push to any branch of a fork except `main` and calls `ci.yml`. Upstream its only job is skipped.                                                |
+| `compute-changes.py` | `is_fork_push` gives a fork the `pr` tier. `is_fork_pull_request` takes the `pr` tier away upstream, and leaves every `run-*` label and the `labeled` run. |
+| `fork_ci_status.yml` | Mirrors the fork's run onto the pull request as the `CI (fork)` check, on `workflow_run` and every 15 minutes.                                             |
+
+Things that follow from that, and are easy to break:
+
+- **The fork result is advisory.** `Required Checks` does not depend on
+  `CI (fork)`, because a contributor controls the workflow that produced it. A
+  green fork run that never ran the tests costs one queue slot, not a bad merge.
+- **The fork's push to `main` uses `ci.yml` directly,** since `ci.yml` already
+  triggers on push to `main`, and `ci_fork.yml` ignores `main` so the two never
+  both fire. That run is the PR tier for whatever the push changed, which also
+  writes the fork's cache entries. A branch restores caches saved on its own
+  ref or on the fork's default branch only, so a fork whose `main` has never
+  run starts every branch cold (about 27 minutes for the native build alone).
+  The bridge reads `ci.yml` for a pull request opened from a fork's `main`, and
+  `ci_fork.yml` for every other branch.
+- **`ci.yml` must not trigger on every push.** A skipped job still publishes a
+  check run under its own name, so a push to a branch of this repository that
+  lands on a pull request's commit would put a skipped `Required Checks` next to
+  the real one. `ci_fork.yml` is a separate workflow so the only name that can
+  be skipped is its own, and `check-ci-config.py` pins its trigger and the
+  `github.repository != 'apache/datafusion-comet'` guard.
+- **A fork push diffs against the fork's default branch.** There is no pull
+  request to diff against, and a fork does not know what upstream `main` is, so
+  `Detect changes` uses `git diff origin/<default branch>...<head>`. A fork that
+  has not synced diffs against an older `main`, which selects more suites, never
+  fewer.
+- **Exceptions stay upstream:** a pull request from a branch of this
+  repository (a committer's, dependabot's), one that targets a release branch,
+  and the opt-in `run-*` suites. A fork cannot see the labels on its pull
+  request, and a release branch has no queue to catch what a fork run misses.
+- **The bridge never checks out or runs anything from the pull request.** It
+  runs with `checks: write` from the default branch and reads the fork's public
+  run list. Its logic is tested by `dev/ci/test-fork-ci-status.cjs`, which
+  Preflight runs, against mocked GitHub APIs. `workflow_run` and `schedule`
+  events use the copy on the default branch, so a change to the bridge cannot be
+  exercised by the pull request that makes it.
+
 ### Label events
 
 `ci_label.yml` fires on `pull_request.types: [labeled]` and calls `ci.yml`
@@ -300,6 +353,8 @@ umbrella doesn't watch, or operate independently of the rest of CI:
 | `take.yml`             | Issue-comment trigger for `take` / `untake`.                                                         |
 | `label_new_issues.yml` | Issue trigger to apply `requires-triage`.                                                            |
 | `label_prs.yml`        | Runs on `pull_request_target` so it can label pull requests opened from forks.                       |
+| `ci_fork.yml`          | Runs `ci.yml` on a push to a fork's branch; skipped upstream. See [Fork runs](#fork-runs).           |
+| `fork_ci_status.yml`   | Mirrors a fork's run onto its pull request as `CI (fork)`. See [Fork runs](#fork-runs).              |
 
 ## Reusable workflows (called by `ci.yml`)
 

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -73,9 +73,11 @@ name: Comet CI
 # each other. Every other run is keyed on its event name: opened, synchronize
 # and reopened all map to `pull_request`, so a new push still supersedes its
 # predecessor, while a scheduled run at the tip of main never cancels that
-# commit's push run.
+# commit's push run. A push to a fork (ci_fork.yml) is keyed on its branch
+# instead of its commit, so the next push to the branch cancels the run in
+# flight.
 concurrency:
-  group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}-${{ github.event.action == 'labeled' && github.event.label.name || github.event_name }}
+  group: ${{ github.repository }}-${{ github.head_ref || (github.event_name == 'push' && github.repository != 'apache/datafusion-comet' && github.ref) || github.sha }}-${{ github.workflow }}-${{ github.event.action == 'labeled' && github.event.label.name || github.event_name }}
   cancel-in-progress: true
 
 # No `labeled` here: ci_label.yml handles that event and calls this file
@@ -194,6 +196,9 @@ jobs:
       - name: Check pull request type labeling
         run: node --test dev/ci/pr-type-label.test.mjs
 
+      - name: Check fork CI status bridge
+        run: node --test dev/ci/test-fork-ci-status.cjs
+
       - name: Check Iceberg shard inventory validation
         run: python3 dev/ci/test-iceberg-shards.py
 
@@ -282,6 +287,10 @@ jobs:
           LABEL_NAME:   ${{ github.event.label.name }}
           PR_LABELS:    ${{ toJSON(github.event.pull_request.labels.*.name) }}
           PR_BASE_REF:  ${{ github.event.pull_request.base.ref }}
+          PR_HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
+          REPOSITORY:   ${{ github.repository }}
+          DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
+          PUSH_REF:     ${{ github.ref }}
           PR_BASE_SHA:  ${{ github.event.pull_request.base.sha }}
           PR_HEAD_SHA:  ${{ github.event.pull_request.head.sha }}
           MQ_BASE_SHA:  ${{ github.event.merge_group.base_sha }}
@@ -329,8 +338,22 @@ jobs:
             # merge_group runs; the queue's ALLGREEN strategy is what makes
             # that enough (see the Merge Queue ruleset in .asf.yaml).
             git diff --name-only "$MQ_BASE_SHA"..."$MQ_HEAD_SHA" > changed_files.txt
+          elif [[ "$REPOSITORY" != "apache/datafusion-comet" && "$PUSH_REF" != "refs/heads/$DEFAULT_BRANCH" ]]; then
+            # A push to a feature branch of a contributor's fork (ci_fork.yml).
+            # There is no pull request to diff against, and the fork cannot know
+            # what upstream main is, so diff against the fork's own default
+            # branch. A fork that has not synced yet diffs against an older
+            # main, which can only select more suites than the pull request
+            # needs, never fewer. Without a default branch to diff against, or
+            # without a merge base, run everything the tier selects.
+            if git merge-base "origin/$DEFAULT_BRANCH" HEAD >/dev/null 2>&1; then
+              git diff --name-only "origin/$DEFAULT_BRANCH"..."$PUSH_AFTER" > changed_files.txt
+            else
+              git ls-tree -r --name-only "$PUSH_AFTER" > changed_files.txt
+            fi
           else
-            # push to main; first push to a branch has all-zero before sha
+            # push to main, upstream or a fork's own; the first push to a
+            # branch has an all-zero before sha
             if [[ "$PUSH_BEFORE" =~ ^0+$ ]]; then
               git ls-tree -r --name-only "$PUSH_AFTER" > changed_files.txt
             else

diff --git a/.github/workflows/ci_fork.yml b/.github/workflows/ci_fork.yml
@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Fork runs: a push to a branch of a contributor's fork runs the pull request
+# tier on the contributor's own GitHub runners, so a fork pull request does not
+# spend ASF's shared capacity on it. ci.yml upstream skips the PR tier for a
+# fork pull request into main (see `is_fork_pull_request` in
+# dev/ci/compute-changes.py), keeping only the preflight checks and the opt-in
+# `run-*` suites, and fork_ci_status.yml mirrors this run onto the pull request
+# as the informational `CI (fork)` check. The merge queue runs every suite on
+# the merge result, so a fork run is feedback and never a gate.
+#
+# This is a separate workflow, and not a wider `push` trigger on ci.yml, for the
+# same reason ci_label.yml is one. A skipped job still publishes a check run
+# under its own name, and a push to a branch of this repository can land on the
+# very commit a pull request is testing. A skipped `Required Checks` from that
+# push would compete with the pull request's real verdict. Here the only job
+# that can be skipped is named after this workflow's own run, and upstream it is
+# always skipped.
+#
+# `main` is excluded because ci.yml already runs on push to main, in a fork as
+# upstream. There it refreshes the fork's cache entries and runs the PR tier
+# for what the push changed, which is what makes a fork's branch runs fast: a
+# branch can restore caches saved on its own ref or on the default branch only.
+name: Comet CI (fork run)
+
+# No `concurrency` here. ci.yml declares its own group, which for a push to a
+# fork is keyed on the branch, so the next push to the branch supersedes this
+# one. The same group on the caller as well would deadlock the two.
+on:
+  push:
+    branches-ignore:
+      - main
+      - asf-site
+      # Merge queue branches exist only in this repository; the guard below
+      # would skip them anyway.
+      - 'gh-readonly-queue/**'
+
+permissions:
+  contents: read
+
+jobs:
+  ci:
+    name: Fork run
+    # Upstream, a push to any branch other than main is a pull request's own
+    # business: the pull request run tests it, and a second run here would
+    # test it twice.
+    if: github.repository != 'apache/datafusion-comet'
+    # The union of what ci.yml's jobs request, for the same reason as in
+    # ci_label.yml: GitHub checks a called workflow's permissions against
+    # this ceiling for every job when the run starts, including the ones this
+    # event skips.
+    permissions:
+      actions: read
+      contents: write
+      issues: write
+    uses: ./.github/workflows/ci.yml