Skip to content

Externalise cache keys to prevent unneeded invalidation #3922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pljones wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Externalise cache keys to prevent unneeded invalidation #3922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/autobuild/ios-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash

# Values are consumed by ios.sh and the dependency cache key.
# shellcheck disable=SC2034
export AQTINSTALL_VERSION="${AQTINSTALL_VERSION:-3.3.0}"
export QT_VERSION="${QT_VERSION:-5.15.2}"
10 changes: 6 additions & 4 deletions .github/autobuild/ios.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,12 @@

set -eu

PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
readonly PROJECT_DIR
# shellcheck disable=SC1091
source "${PROJECT_DIR}/.github/autobuild/ios-dependencies.sh"

QT_DIR=/opt/qt
# The following version pinnings are semi-automatically checked for
# updates. Verify .github/workflows/bump-dependencies.yaml when changing those manually:
AQTINSTALL_VERSION=3.3.0

if [[ ! ${QT_VERSION:-} =~ [0-9]+\.[0-9]+\..* ]]; then
echo "Environment variable QT_VERSION must be set to a valid Qt version"
Expand All @@ -63,7 +65,7 @@ if [[ ! ${JAMULUS_BUILD_VERSION:-} =~ [0-9]+\.[0-9]+\.[0-9]+ ]]; then
fi

setup() {
if [[ -d "${QT_DIR}" ]]; then
if [[ -x "${QT_DIR}/${QT_VERSION}/ios/bin/qmake" ]]; then
echo "Using Qt installation from previous run (actions/cache)"
else
echo "Installing Qt"
Expand Down
7 changes: 7 additions & 0 deletions .github/autobuild/mac-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/bash

# Values are consumed by mac.sh and the dependency cache key.
# shellcheck disable=SC2034
export AQTINSTALL_VERSION="${AQTINSTALL_VERSION:-3.3.0}"
export QT_VERSION="${QT_VERSION:-6.10.2}"
export QT_LEGACY_VERSION=5.15.2
11 changes: 7 additions & 4 deletions .github/autobuild/mac.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,12 @@

set -eu

PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
readonly PROJECT_DIR
# shellcheck disable=SC1091
source "${PROJECT_DIR}/.github/autobuild/mac-dependencies.sh"

QT_DIR=~/qt
# The following version pinnings are semi-automatically checked for
# updates. Verify .github/workflows/bump-dependencies.yaml when changing those manually:
AQTINSTALL_VERSION=3.3.0

TARGET_ARCHS="${TARGET_ARCHS:-}"

Expand All @@ -65,7 +67,8 @@ if [[ ! ${JAMULUS_BUILD_VERSION:-} =~ [0-9]+\.[0-9]+\.[0-9]+ ]]; then
fi

setup() {
if [[ -d "${QT_DIR}" ]]; then
if [[ -x "${QT_DIR}/${QT_VERSION}/macos/bin/qmake" &&
-x "${QT_DIR}/${QT_VERSION}/macos/bin/macdeployqt" ]]; then
echo "Using Qt installation from previous run (actions/cache)"
else
echo "Installing Qt..."
Expand Down
8 changes: 8 additions & 0 deletions .github/autobuild/windows-dependencies.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Values are consumed by windows.ps1 and the dependency cache key.
$Qt32Version = "5.15.2"
$Qt64Version = "6.10.2"
$AqtinstallVersion = "3.3.0"
$JackVersion = "1.9.22"
$Msvc32Version = "win32_msvc2019"
$Msvc64Version = "win64_msvc2022_64"
$JomVersion = "1.1.2"
13 changes: 3 additions & 10 deletions .github/autobuild/windows.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,15 +68,7 @@ $ProgressPreference = 'SilentlyContinue'
$QtDir = 'C:\Qt'
$ChocoCacheDir = 'C:\ChocoCache'
$DownloadCacheDir = 'C:\AutobuildCache'
# The following version pinnings are semi-automatically checked for
# updates. Verify .github/workflows/bump-dependencies.yaml when changing those manually:
$Qt32Version = "5.15.2"
$Qt64Version = "6.10.2"
$AqtinstallVersion = "3.3.0"
$JackVersion = "1.9.22"
$Msvc32Version = "win32_msvc2019"
$Msvc64Version = "win64_msvc2022_64"
$JomVersion = "1.1.2"
. "$PSScriptRoot\windows-dependencies.ps1"

# Compose JACK download urls
$JackBaseUrl = "https://github.com/jackaudio/jack2-releases/releases/download/v${JackVersion}/jack2-win"
Expand Down Expand Up @@ -150,7 +142,8 @@ Function Install-Qt

Function Ensure-Qt
{
if ( Test-Path -Path $QtDir )
if ( (Test-Path -Path "$QtDir\$Qt32Version\bin\qmake.exe" -PathType Leaf) -and
(Test-Path -Path "$QtDir\$Qt64Version\bin\qmake.exe" -PathType Leaf) )
{
echo "Using Qt installation from previous run (actions/cache)"
return
Expand Down
16 changes: 14 additions & 2 deletions .github/workflows/autobuild.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,7 @@ jobs:
# Disable CodeQL on mac as it interferes with signing the binaries (signing hangs, see #2563 and #2564)
run_codeql: false
xcode_version: 16.3.0
dependency_cache_key: qt6
is_main_build_target: true

# Reminder: If Legacy is removed, be sure to add a dedicated job for CodeQL again.
Expand All @@ -243,6 +244,7 @@ jobs:
# https://developer.apple.com/support/xcode/
# https://xcodereleases.com/
xcode_version: 16.3.0
dependency_cache_key: qt5
is_main_build_target: true

- config_name: iOS (artifacts)
Expand All @@ -254,19 +256,22 @@ jobs:
# Therefore, disable it:
run_codeql: false
xcode_version: 26.2
dependency_cache_key: ios

- config_name: Windows (artifact+codeQL)
target_os: windows
building_on_os: windows-2025
base_command: powershell .\.github\autobuild\windows.ps1 -Stage
run_codeql: true
dependency_cache_key: asio
is_main_build_target: true

- config_name: Windows JACK (artifact)
target_os: windows
building_on_os: windows-2025
base_command: powershell .\.github\autobuild\windows.ps1 -BuildOption jackonwindows -Stage
run_codeql: false
dependency_cache_key: jack

# This injects the build_all_targets information into each matrix output:
build_all_targets:
Expand Down Expand Up @@ -300,7 +305,14 @@ jobs:
path: |
~/qt
~/Library/Cache/jamulus-dependencies
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/workflows/autobuild.yml', '.github/autobuild/mac.sh', 'mac/deploy_mac.sh') }}-${{ matrix.config.base_command }}
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/autobuild/mac-dependencies.sh') }}-${{ matrix.config.dependency_cache_key }}

@softins softins Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hash still needs to contain mac/deploy_mac.sh, because that script contains CREATEDMG_VERSION, which is updated by bump-dependencies

@mcfnord mcfnord Aug 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI: Confirmed, and it is the only one of its kind here. Running all seven bump components against this head, every component moves the cache key of each platform whose version it changes — except create-dmg, whose single pin is CREATEDMG_VERSION in mac/deploy_mac.sh, which main hashes today.

The failure mode is milder than the ASIO/NSIS one: create-dmg unpacks into a version-namespaced folder, so a stale cache cannot install the wrong version. The cost is that after a bump the key is unchanged, the exact hit means the post step saves nothing, and create-dmg is downloaded and make installed again on every macOS run.

Suggested change
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/autobuild/mac-dependencies.sh') }}-${{ matrix.config.dependency_cache_key }}
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/autobuild/mac-dependencies.sh', 'mac/deploy_mac.sh') }}-${{ matrix.config.dependency_cache_key }}


- name: Cache iOS dependencies
if: matrix.config.target_os == 'ios'
uses: actions/cache@v6
with:
path: /opt/qt
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/autobuild/ios-dependencies.sh') }}-${{ matrix.config.dependency_cache_key }}

- name: Cache Windows dependencies
if: matrix.config.target_os == 'windows'
Expand All @@ -312,7 +324,7 @@ jobs:
C:\AutobuildCache
${{ github.workspace }}\libs\NSIS\NSIS-source
${{ github.workspace }}\libs\ASIOSDK2
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/workflows/autobuild.yml', '.github/autobuild/windows.ps1', 'windows/deploy_windows.ps1') }}-${{ matrix.config.base_command }}
key: ${{ matrix.config.target_os }}-${{ hashFiles('.github/autobuild/windows-dependencies.ps1', 'windows/deploy_windows.ps1') }}-${{ matrix.config.dependency_cache_key }}

- name: Cache Android dependencies
if: matrix.config.target_os == 'android'
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/bump-dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ jobs:
- name: aqt
# not Changelog-worthy
get_upstream_version: GH_REPO=miurahr/aqtinstall gh release view --json tagName --jq .tagName | sed -re 's/^v//'
# The following regexps capture both the *nix and the Windows variable syntax (different case, underscore):
local_version_regex: (.*AQTINSTALL_?VERSION\s*=\s*"?)([0-9.]*)("?.*)
# Capture both plain assignments and shell parameter-expansion defaults.
local_version_regex: (.*AQTINSTALL_?VERSION\s*=\s*"?[^0-9]*)([0-9.]+)(.*)

@softins softins Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regex and the one below look ok to me by eye, but have you tested them in situ somehow?

@pljones pljones Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It runs... But I'm not sure how to check it actually picks up new versions. It's syntactically stronger...

I'd have to mock the returns from the version probes, I guess, and check it creates the bump. I'll give that a go in my repo.

- name: create-dmg
changelog_name: create-dmg (macOS)
get_upstream_version: GH_REPO=create-dmg/create-dmg gh release view --json tagName --jq .tagName | sed -re 's/^v//'
Expand All @@ -50,8 +50,8 @@ jobs:
get_upstream_version: |
latest_minor="$(curl -s https://download.qt.io/official_releases/qt/ | grep -oP 'href="\K[0-9.]+(?=/")' | sort --reverse --version-sort | head -n1)";
curl -s https://download.qt.io/official_releases/qt/"${latest_minor}"/ | grep -oP 'href="\K[0-9.]+(?=/")' | sort --reverse --version-sort | head -n1
# The following regex captures both the *nix and the Windows variable syntax (different case, underscore):
local_version_regex: (.*QT[0-9_]+VERSION\s*=\s*"?)(6\.[0-9.]+)("?.*)
# Capture both plain assignments and shell parameter-expansion defaults.
local_version_regex: (.*QT[0-9_]+VERSION\s*=\s*"?[^0-9]*)(6\.[0-9.]+)(.*)

@softins softins Aug 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment above


- name: jack
changelog_name: bundled JACK (Windows-only)
Expand Down
Loading