Repository navigation
feat: make release artifacts and images reproducible - #8812
Open
parshimers wants to merge 4 commits into
Open
parshimers wants to merge 4 commits into
parshimers wants to merge 4 commits into
Conversation
github-actions
Bot
added
feature
dependencies
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8812 +/- ##
=========================================
Coverage 91.98% 91.99%
- Complexity 5083 5084 +1
=========================================
Files 1285 1285
Lines 56094 56103 +9
Branches 7137 7137
=========================================
+ Hits 51599 51610 +11
+ Misses 2742 2741 -1
+ Partials 1753 1752 -1
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 482 | 0.294 | 20,283/25,090/25,090 us | 🔴 +10.4% / 🔴 +79.0% |
| 🔴 | bs=100 sw=10 sl=64 | 1,150 | 0.702 | 86,107/116,474/116,474 us | 🔴 +6.0% / 🔴 +21.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,425 | 0.87 | 704,239/747,741/747,741 us | ⚪ within ±5% / 🟢 -22.4% |
Baseline details
Latest main 92e86d4 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 482 tuples/sec | 533 tuples/sec | 905.73 tuples/sec | -9.6% | -46.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.294 MB/s | 0.325 MB/s | 0.553 MB/s | -9.5% | -46.8% |
| bs=10 sw=10 sl=64 | p50 | 20,283 us | 18,365 us | 11,331 us | +10.4% | +79.0% |
| bs=10 sw=10 sl=64 | p95 | 25,090 us | 26,811 us | 14,032 us | -6.4% | +78.8% |
| bs=10 sw=10 sl=64 | p99 | 25,090 us | 26,811 us | 17,338 us | -6.4% | +44.7% |
| bs=100 sw=10 sl=64 | throughput | 1,150 tuples/sec | 1,206 tuples/sec | 1,205 tuples/sec | -4.6% | -4.5% |
| bs=100 sw=10 sl=64 | MB/s | 0.702 MB/s | 0.736 MB/s | 0.735 MB/s | -4.6% | -4.5% |
| bs=100 sw=10 sl=64 | p50 | 86,107 us | 81,255 us | 89,237 us | +6.0% | -3.5% |
| bs=100 sw=10 sl=64 | p95 | 116,474 us | 113,621 us | 96,280 us | +2.5% | +21.0% |
| bs=100 sw=10 sl=64 | p99 | 116,474 us | 113,621 us | 111,344 us | +2.5% | +4.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,425 tuples/sec | 1,457 tuples/sec | 1,239 tuples/sec | -2.2% | +15.0% |
| bs=1000 sw=10 sl=64 | MB/s | 0.87 MB/s | 0.889 MB/s | 0.756 MB/s | -2.1% | +15.1% |
| bs=1000 sw=10 sl=64 | p50 | 704,239 us | 685,244 us | 877,826 us | +2.8% | -19.8% |
| bs=1000 sw=10 sl=64 | p95 | 747,741 us | 736,937 us | 919,412 us | +1.5% | -18.7% |
| bs=1000 sw=10 sl=64 | p99 | 747,741 us | 736,937 us | 963,000 us | +1.5% | -22.4% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,415.32,200,128000,482,0.294,20282.79,25089.98,25089.98
1,100,10,64,20,1738.68,2000,1280000,1150,0.702,86107.12,116474.04,116474.04
2,1000,10,64,20,14032.82,20000,12800000,1425,0.870,704239.43,747740.67,747740.67
parshimers
force-pushed
the
feat/reproducible-builds
branch
from
October 8, 2026 20:08
889a6b0 to
1cbb5d9
Compare
parshimers
changed the title
parshimers
force-pushed
the
feat/reproducible-builds
branch
from
October 8, 2026 20:34
1cbb5d9 to
9ac8c5d
Compare
parshimers
marked this pull request as ready for review
October 9, 2026 02:30
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
1 open finding
What changed in this PR
This PR aims to make release artifacts and container images reproducible so rebuilding the same commit yields byte-identical outputs across environments.
Changes:
- Make frontend build numbers deterministic via
SOURCE_DATE_EPOCHand remove Sass randomness affecting CSS output. - Pin Docker base images by digest, introduce snapshot-based apt/pip installs, and clamp image timestamps during builds/pushes.
- Make RC tarball + Docker Compose bundle generation reproducible and document reproduction steps (incl. Renovate support for digest refreshes).
| File | Description |
|---|---|
| frontend/src/build-version.spec.ts | Adds tests for SOURCE_DATE_EPOCH-derived build numbers and reproducibility. |
| frontend/src/app/workspace/component/code-editor-dialog/code-editor.component.scss | Removes nondeterministic Sass random() to stabilize CSS output. |
| frontend/build-version.js | Implements deterministic build number derivation from SOURCE_DATE_EPOCH and exports it. |
| frontend/build-version.d.ts | Adds typings for the new defaultBuildNumber API. |
| file-service/src/main/resources/rustfs-config.yml | Pins RustFS image tag to stable 1.0.0. |
| file-service/src/main/resources/docker-compose.yml | Pins RustFS image tag to stable 1.0.0. |
| file-service/build.sbt | Replaces play-json milestone dependency with stable 3.0.6. |
| file-service/LICENSE-binary | Updates binary NOTICE entries for the stable Play JSON jars. |
| config-service/build.sbt | Replaces play-json milestone dependency with stable 3.0.6. |
| config-service/LICENSE-binary | Updates binary NOTICE entries for the stable Play JSON jars. |
| common/workflow-core/src/test/scala/org/apache/texera/service/util/RustFSContainer.scala | Updates test container image tag to 1.0.0. |
| bin/y-websocket-server/Dockerfile | Pins Node image patch version and switches to npm ci for reproducible installs. |
| bin/single-node/docker-compose.yml | Pins several images (RustFS/nginx/alpine/curl) for reproducibility. |
| bin/k8s/values.yaml | Pins RustFS Helm chart image tag to 1.0.0. |
| bin/k8s/values-development.yaml | Pins RustFS Helm chart image tag to 1.0.0. |
| bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml | Pins curl initContainer image to a concrete version. |
| bin/k8s/templates/base/lakefs/lakefs-setup-job.yaml | Pins curl image to a concrete version. |
| bin/k8s/Chart.yaml | Pins RustFS chart dependency version to 1.0.0. |
| bin/dockerfiles/workflow-computing-unit-managing-service.dockerfile | Pins base images by digest and uses snapshot installer + SOURCE_DATE_EPOCH args. |
| bin/dockerfiles/workflow-compiling-service.dockerfile | Pins base images by digest and uses snapshot installer + SOURCE_DATE_EPOCH args. |
| bin/dockerfiles/texera-web-application.dockerfile | Pins base images by digest, uses snapshot installer, and switches Yarn install to immutable. |
| bin/dockerfiles/snapshot/test_install.sh | Adds regression tests for snapshot installer logic (apt/pip split/cutoff). |
| bin/dockerfiles/snapshot/pip-bootstrap-requirements.txt | Pins a pip version used to bootstrap --uploaded-prior-to support. |
| bin/dockerfiles/snapshot/install.sh | Introduces snapshot-based apt/pip installation logic for reproducible dependency resolution. |
| bin/dockerfiles/notebook-migration-service.dockerfile | Pins base images by digest and uses snapshot installer + SOURCE_DATE_EPOCH args. |
| bin/dockerfiles/mounter.dockerfile | Pins Debian base by digest and routes apt installs through snapshot installer. |
| bin/dockerfiles/jupyter.dockerfile | Pins Jupyter base image by digest. |
| bin/dockerfiles/file-service.dockerfile | Pins base images by digest and uses snapshot installer + SOURCE_DATE_EPOCH args. |
| bin/dockerfiles/config-service.dockerfile | Pins base images by digest and uses snapshot installer + SOURCE_DATE_EPOCH args. |
| bin/dockerfiles/computing-unit-worker.dockerfile | Pins base images by digest and uses snapshot installer; removes silent fallback installs. |
| bin/dockerfiles/computing-unit-master.dockerfile | Pins base images by digest and uses snapshot installer for apt/pip. |
| bin/dockerfiles/agent-service.dockerfile | Pins bun image by digest and removes date-dependent bun cache + /etc/shadow variability. |
| bin/dockerfiles/access-control-service.dockerfile | Pins base images by digest and uses snapshot installer + SOURCE_DATE_EPOCH args. |
| bin/build-images.sh | Passes reproducibility args and uses rewrite-timestamp=true registry outputs. |
| bin/README.md | Documents reproducible image build inputs and manual rebuild commands. |
| .gitignore | Un-ignores bin/y-websocket-server/package-lock.json to support npm ci. |
| .github/workflows/create-release-candidate.yml | Uses reproducible tarball script from tag and threads IMAGE_TAG into email templating. |
| .github/workflows/build.yml | Updates RustFS container/image references to 1.0.0 and macOS artifact URL. |
| .github/workflows/build-and-push-images.yml | Derives and passes SOURCE_DATE_EPOCH/PACKAGE_SNAPSHOT and enables timestamp rewriting on push. |
| .github/scripts/test_create_release_tarballs.sh | Adds regression tests ensuring tarballs are byte-identical and metadata is normalized. |
| .github/scripts/create-release-tarballs.sh | New reproducible tarball + compose bundle generator used by RC workflow and voters. |
| .github/renovate.json5 | Enables Dockerfile manager and groups digest/tag bumps for pinned base images. |
| .github/release/vote-email-template.md | Documents how voters can reproduce tarballs and adds a checklist item. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copilot stopped reviewing on behalf of
mengw15 due to an error
October 9, 2026 14:28
parshimers
force-pushed
the
feat/reproducible-builds
branch
2 times, most recently
from
October 16, 2026 19:47
7f627c0 to
ccc5da1
Compare
Previously, a variety of things caused Texera images built from the same source to not have the same binary output from invocation to invocation. Having this kind of binary build reproducibility is a requirement when performing Apache releases not on developer-owned hardware, because the release has to be verified on developer owned hardware by rebuilding it and getting the exact same artifact. This change allows all release artifacts to be binary reproducible with effectively one piece of information, though in two different formats. These build arguments contain the date: SOURCE_DATE_EPOCH and PACKAGE_SNAPSHOT. The first sets a uniform timestamp wherever a timestamp is present, and the second sets the snapshot of the apt repositories used in image builds. One other piece of information is implicit, but important in practice. Binary reproducibility is inherently dependent on the versions of tools used to produce said binaries. Most of them are pinned within the container build files, but the container builder itself is another piece of necessary information (e.g. buildah vs. buildx) for reproducing a build from its source and time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parshimers
force-pushed
the
feat/reproducible-builds
branch
from
October 19, 2026 18:41
ccc5da1 to
43b9170
Compare
Member
Author
|
I'm waiting on #9072 to be fixed before rebasing and adding backport tags, that is what is causing the CI chaos here. I don't think it's related. |
Contributor
|
Hi @parshimers, What is the status of this PR? |
Member
Author
|
Was blocked by the CI chaos but now it should be fine. Let's see if it passes now. |
mengw15
reviewed
Oct 29, 2026
| raw="$work/raw" | ||
| bundle="$work/compose/$compose_name" | ||
| mkdir -p "$raw" "$bundle" | ||
| git archive --format=tar "$commit" -- bin/single-node/ sql/ NOTICE licenses/LICENSE-BSD-3-Clause.txt | "$gnu_tar" -x -C "$raw" |
Contributor
There was a problem hiding this comment.
The merge with main dropped what #5377 added to the bundle step in create-release-candidate.yml: bin/observability/ shipped as observability/, and ../observability rewritten to ./observability in docker-compose.yml. Without them the bundle's observability profiles point outside the bundle, e.g. COMPOSE_PROFILES=observability-profiles docker compose config fails with env file .../observability/parca/parca-agent.env not found. Running without profiles is unaffected.
| @@ -360,6 +360,15 @@ jobs: | |||
| - name: Set up Docker Buildx | |||
| uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 | |||
Contributor
There was a problem hiding this comment.
The BuildKit that builds the published images isn't pinned (setup-buildx-action uses its default image), and image digests can change between BuildKit versions. Pinning it, for example driver-opts: image=moby/buildkit:v0.33.1 (the version used for the rebuilds in #8807), would let voters reproduce these images later. Same at line 460.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Building the same commit twice now gives byte-identical release artifacts: the source tarball, the Docker Compose bundle, the jars and dist zips, the frontend bundle, and the container images. Before, each of them picked up the build machine's clock, user, umask, or whatever the package archives held that day, so a voter or downstream packager could not check a release against its source.
.github/scripts/create-release-tarballs.sh: sorted, 0:0, commit-time mtimes,gzip -n; the RC workflow and voters run the tag's copyThe day, not the commit time, keys the package snapshot so commits from the same day still share cached apt/pip layers. The PyTorch index publishes no upload times, so pip refuses a cutoff while it is in play; the helper installs the one pin that needs it (
torch==2.13.0+cpu, a local version PyPI cannot host) exactly and without dependencies, then resolves everything else, torch's dependencies included, from PyPI under the cutoff..gitshipped only forOPVersion, which has looked upamber/src/main/scala/...since the module moved tocommon/workflow-operatorin #3878. No commit ever touched that path, so every operator version has been"N/A"in every deployment; it stays"N/A"without.git. The worker image also dropped a fallback that, when the operator requirements failed to install, silently shippedwordcloud==1.9.2and nothing else; such a build now fails.Non-stable versions are replaced: play-json
3.1.0-M1->3.0.6(same runtime dependencies; neither service imports it), RustFS1.0.0-rc.6->1.0.0(chart unchanged but for its version), andalpine:latest,nginx:alpine,curlimages/curl:latestandoven/bun:1-alpine-> exact versions. y-websocket-server gains a lockfile andnpm ci.betterproto==2.0.0b7stays: betterproto 2 has no stable release.Any related issues, documentation, discussions?
Closes #8809. Discussion: #8807.
How was this PR tested?
bin/README.mddocuments what is pinned and how to rebuild an image; the vote email gives voters the command that rebuilds both tarballs.Each artifact was built twice from clean and compared byte for byte.
bash .github/scripts/test_create_release_tarballs.sh(14 checks): identical across umask, clock, TZ, cwd and tag kind; the old recipe differed run to run and leaked uid/usernameThe original mounter Dockerfile, rebuilt eight seconds apart with the same flags, gave different images. Not verified end to end: the web-application, computing-unit-master and -worker images, which exceed the test host's memory and disk, and arm64. Their parts (bundle, dist, Python layers) are each covered above.
Was this PR authored or co-authored using generative AI tooling?
Yes, Claude Opus 5.5