diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1937d97cc70..2bcafaba0ce 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -407,6 +407,8 @@ publish-artifacts-to-s3: image: registry.ddbuild.io/images/mirror/amazon/aws-cli:2.4.29 stage: publish needs: [ build ] + before_script: + - yum install -y -q git unzip zip script: - source upstream.env - export VERSION="${UPSTREAM_TRACER_VERSION%~*}" # remove ~githash from the end of version @@ -416,8 +418,15 @@ publish-artifacts-to-s3: - aws s3 cp workspace/dd-java-agent/build/libs/dd-java-agent-${VERSION}.jar s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent.jar - aws s3 cp workspace/dd-trace-api/build/libs/dd-trace-api-${VERSION}.jar s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-trace-api.jar - aws s3 cp workspace/dd-trace-ot/build/libs/dd-trace-ot-${VERSION}.jar s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-trace-ot.jar + - export AGENT_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-${VERSION}.jar" + - export PATCHED_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-profiler-SNAPSHOT.jar" + - git clone --depth 1 https://github.com/DataDog/java-profiler.git java-profiler + - curl -fLo ddprof.jar https://github.com/DataDog/java-profiler/releases/latest/download/ddprof.jar + - DD_AGENT_JAR="${AGENT_JAR}" DDPROF_JAR="$(pwd)/ddprof.jar" OUTPUT_JAR="${PATCHED_JAR}" bash java-profiler/utils/patch-dd-java-agent.sh + - aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_COMMIT_REF_NAME}/dd-java-agent-profiler-SNAPSHOT.jar + - aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent-profiler-SNAPSHOT.jar - | - cat << EOF > links.json + cat << EOF > links.json { "S3 Links": [ { @@ -425,6 +434,12 @@ publish-artifacts-to-s3: "label": "Public Link to dd-java-agent.jar", "url": "https://s3.us-east-1.amazonaws.com/dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent.jar" } + }, + { + "external_link": { + "label": "Public Link to dd-java-agent-profiler-SNAPSHOT.jar", + "url": "https://s3.us-east-1.amazonaws.com/dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent-profiler-SNAPSHOT.jar" + } } ] } diff --git a/doc/design-publish-profiler-patched-agent.md b/doc/design-publish-profiler-patched-agent.md new file mode 100644 index 00000000000..be844ebd081 --- /dev/null +++ b/doc/design-publish-profiler-patched-agent.md @@ -0,0 +1,323 @@ +# Design: Publish Profiler-Patched Agent to S3 + +## Problem Statement + +The existing `publish-artifacts-to-s3` job (`.gitlab-ci.yml:406`) uploads the stock +`dd-java-agent.jar` to S3 on every master push. We want to also publish a variant +that has been patched with the **latest released `java-profiler`** native libraries, +producing `dd-java-agent-profiler.jar` alongside the original artifact. + +--- + +## Background + +### How ddprof is currently embedded + +The stock agent embeds ddprof at **build time**: +- `gradle/libs.versions.toml:12` pins `ddprof = "1.44.0"` +- `dd-java-agent/ddprof-lib/build.gradle` declares `com.datadoghq:ddprof` as a + dependency and shadows its content (native `.so` files, services metadata) + into the agent jar under the `META-INF/native-libs/` namespace. + +The patching workflow replaces that embedded native library with a newer +(post-release) version **without rebuilding** the agent, which is the fast path +used for profiler-forward testing. + +### `patch-dd-java-agent.sh` + +This script lives in the **`DataDog/java-profiler`** GitHub repository. It takes: +- an existing `dd-java-agent.jar` +- a `ddprof.jar` (the artifact released by java-profiler) + +and produces a patched jar with the new native libraries swapped in. The exact +signature needs to be confirmed by reading the script after cloning, but the +canonical invocation is expected to be something like: + +```sh +./patch-dd-java-agent.sh +``` + +--- + +## Design + +### 1. New GitLab CI job: `publish-profiler-patched-agent-to-s3` + +A **new, independent job** in `.gitlab-ci.yml` in the `publish` stage, parallel +to (not extending) the existing `publish-artifacts-to-s3` job. + +Keeping it separate means: +- A failure in patching does not block the primary S3 upload. +- It can be retried independently. +- The responsibility boundary is clear. + +### 2. Trigger condition + +Same as `publish-artifacts-to-s3`: no explicit `rules` block → runs on every +pipeline. If restriction to master + release branches is desired, add: + +```yaml +rules: + - if: '$POPULATE_CACHE' + when: never + - *master_only +``` + +The `&master_only` anchor is defined at `.gitlab-ci.yml` line 130: + +```yaml +.master_only: &master_only + - if: $CI_COMMIT_BRANCH == "master" + when: on_success +``` + +Mirror the existing `deploy_snapshot_with_ddprof_snapshot` pattern for release +branches if needed later. + +### 3. Image + +The existing `amazon/aws-cli:2.4.29` image does not have `git` or `gh`. + +Options (in preference order): + +| Option | Image | Notes | +|--------|-------|-------| +| A (preferred) | `registry.ddbuild.io/images/base/gbi-ubuntu_2204:release` | Already used by `deploy_artifacts_to_github` (line 1328). Has `git`, `gh`, `curl`, `unzip`. Add `awscli` via `pip` or a side-step with `aws` already in PATH via the CI base. | +| B | `registry.ddbuild.io/images/dd-octo-sts-ci-base:2025.06-1` | Has `gh` + AWS support. | +| C | Custom image with both `aws-cli` and `git` | Requires a new image PR in DataDog/images. More work. | + +**Recommendation: Option A** — `gbi-ubuntu_2204:release` is already in use for +GitHub publishing jobs in this file and avoids a new image dependency. + +### 4. Cloning `DataDog/java-profiler` + +Use a **shallow clone** (depth 1) to get just the tip of the default branch: + +```sh +git clone --depth 1 https://github.com/DataDog/java-profiler.git java-profiler +``` + +If the repo is private, authentication is required. Use a GitLab CI variable +(e.g. `$GITHUB_TOKEN`, already used in other jobs in this file) with HTTPS: + +```sh +git clone --depth 1 https://oauth2:${GITHUB_TOKEN}@github.com/DataDog/java-profiler.git java-profiler +``` + +**Verify the script path** after first clone: likely `java-profiler/patch-dd-java-agent.sh` +or `java-profiler/tools/patch-dd-java-agent.sh`. + +### 5. Getting the latest `ddprof.jar` + +Two options: + +#### Option A — GitHub Releases of `java-profiler` (recommended) + +```sh +# Select the jar matching the current runner architecture (x86_64 or arm64). +ARCH=$(uname -m) +case "${ARCH}" in + x86_64) DDPROF_ARCH="linux-x86_64" ;; + aarch64) DDPROF_ARCH="linux-arm64" ;; + *) echo "Unsupported architecture: ${ARCH}"; exit 1 ;; +esac +gh release download --repo DataDog/java-profiler \ + --pattern "ddprof*${DDPROF_ARCH}*.jar" \ + --dir ./ddprof-release +DDPROF_JAR=$(ls ./ddprof-release/ddprof*${DDPROF_ARCH}*.jar) +# Fail loudly if no jar or more than one jar was matched. +[ "$(echo "${DDPROF_JAR}" | wc -l)" -eq 1 ] || { echo "Expected exactly one ddprof jar for ${DDPROF_ARCH}, got: ${DDPROF_JAR}"; exit 1; } +``` + +This is the canonical release artifact that `patch-dd-java-agent.sh` is designed +to consume. It is version-aligned with the script in the same repo tag. + +`GITHUB_TOKEN` is already set in CI for other jobs. The `gh` CLI accepts both +`GH_TOKEN` and `GITHUB_TOKEN`; to be unambiguous the job exports `GH_TOKEN` +from `GITHUB_TOKEN` in `before_script` (see job sketch below). + +#### Option B — Maven Central + +```sh +DDPROF_VERSION=$(curl -s "https://search.maven.org/solrsearch/select?q=g:com.datadoghq+AND+a:ddprof&rows=1&wt=json" \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['response']['docs'][0]['latestVersion'])") +curl -Lo ddprof.jar \ + "https://repo1.maven.org/maven2/com/datadoghq/ddprof/${DDPROF_VERSION}/ddprof-${DDPROF_VERSION}.jar" +``` + +Drawback: Maven Central lags GitHub releases by hours/days. The unclassified jar +(`ddprof-${DDPROF_VERSION}.jar`) is a POM-only artifact; the actual native-library +jar requires a classifier. For the amd64 CI runner use `-linux-x86_64`: + +```sh +curl -Lo ddprof.jar \ + "https://repo1.maven.org/maven2/com/datadoghq/ddprof/${DDPROF_VERSION}/ddprof-${DDPROF_VERSION}-linux-x86_64.jar" +``` + +For arm64 runners substitute `-linux-arm64`. Because the correct classifier depends +on the runner architecture and must be kept in sync with future runner changes, +Option A (GitHub releases with `--pattern "ddprof*.jar"`) is strongly preferred. + +**Recommendation: Option A** — GitHub releases are the primary distribution +channel for java-profiler and are what the patch script is built around. + +### 6. Running the patch script + +`upstream.env` is produced by the `build` job (`.gitlab-ci.yml`, `build:` block). +That job appends `UPSTREAM_TRACER_VERSION=` to `upstream.env` and lists **both** +`upstream.env` and `workspace/dd-java-agent/build/libs/*.jar` under `artifacts.paths`, +making them available to downstream jobs. Because the new job declares both `needs: [ build ]` +and `dependencies: [ build ]`, GitLab downloads the `build` artifacts before the script +runs, so both `upstream.env` and the agent jar are present on disk without any additional +configuration. (Without an explicit `dependencies:` key, GitLab's default is to download +artifacts from **all** jobs listed in `needs:`, which is the same outcome here, but the +explicit key makes the intent unambiguous.) + +**Note**: the `build` job unpacks the agent jar under the `workspace/` subdirectory +(`.gitlab-ci.yml` line 347: `workspace/dd-java-agent/build/libs/*.jar`), not directly +under `CI_PROJECT_DIR`. The `AGENT_JAR` path below must therefore be adjusted to +`${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-${VERSION}.jar`. + +```sh +source upstream.env +VERSION="${UPSTREAM_TRACER_VERSION%~*}" +# ${CI_PROJECT_DIR} is the absolute path to the cloned repository in a GitLab CI job +# (see https://docs.gitlab.com/ee/ci/variables/predefined_variables.html). +# The `build` job artifact path is workspace/dd-java-agent/build/libs/*.jar +# (.gitlab-ci.yml artifacts.paths line 347), so the jar is under the workspace/ subdir. +AGENT_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-${VERSION}.jar" +PATCHED_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-profiler.jar" + +bash java-profiler/patch-dd-java-agent.sh "${AGENT_JAR}" "${DDPROF_JAR}" "${PATCHED_JAR}" +``` + +**Caveat**: the exact signature of `patch-dd-java-agent.sh` must be confirmed +from the actual script after cloning. Update the invocation accordingly. + +### 7. S3 upload + +Upload `dd-java-agent-profiler.jar` to the same two paths as the existing agent: + +```sh +aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_COMMIT_REF_NAME}/dd-java-agent-profiler.jar +aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent-profiler.jar +``` + +Add an annotation link in `links-profiler.json` for the profiler-patched jar. + +--- + +## Full job sketch + +```yaml +publish-profiler-patched-agent-to-s3: + image: registry.ddbuild.io/images/base/gbi-ubuntu_2204:release + stage: publish + needs: [ build ] + dependencies: [ build ] + timeout: 10 minutes # shallow clone + single release asset download; 10 min is ample + # No explicit rules → runs on every pipeline (same behaviour as + # publish-artifacts-to-s3). Add the following block if restriction is needed: + # rules: + # - *master_only + variables: + # AWS credentials required for `aws s3 cp` to s3://dd-trace-java-builds/. + # These mirror the variables used by the existing `publish-artifacts-to-s3` job. + # They must be defined as GitLab CI/CD project variables (Settings → CI/CD → + # Variables) and are injected automatically by GitLab when the job runs. + AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY + AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION # e.g. "us-east-1" + before_script: + # Install AWS CLI only if not already present in the image (resolves OQ 5) + - aws --version 2>/dev/null || pip install --quiet awscli==1.36.40 + # Export GH_TOKEN so that `gh` CLI is authenticated regardless of which variable + # name the CI environment exposes (gh accepts both GH_TOKEN and GITHUB_TOKEN, + # but being explicit avoids silent auth failures). + - export GH_TOKEN="${GITHUB_TOKEN}" + script: + # --- 1. Derive agent version from build output --- + # upstream.env is produced by the `build` job and inherited via `needs: [ build ]` + # artifacts. It contains UPSTREAM_TRACER_VERSION set to the agent jar version string. + - source upstream.env + - export VERSION="${UPSTREAM_TRACER_VERSION%~*}" + # CI_PROJECT_DIR is set by GitLab to the absolute path of the cloned repository. + # The build job artifact path is workspace/dd-java-agent/build/libs/*.jar, so the + # jar is unpacked under the workspace/ subdirectory of CI_PROJECT_DIR. + - export AGENT_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-${VERSION}.jar" + - export PATCHED_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-profiler.jar" + + # --- 2. Shallow-clone java-profiler for the patch script --- + - git clone --depth 1 https://oauth2:${GITHUB_TOKEN}@github.com/DataDog/java-profiler.git java-profiler + + # --- 3. Download latest ddprof.jar from java-profiler GitHub releases --- + # Narrow the pattern to the runner's architecture to avoid silently picking the + # wrong jar when multiple per-arch jars are present in the release. + - ARCH=$(uname -m); case "${ARCH}" in x86_64) DDPROF_ARCH="linux-x86_64" ;; aarch64) DDPROF_ARCH="linux-arm64" ;; *) echo "Unsupported architecture ${ARCH}"; exit 1 ;; esac + - gh release download --repo DataDog/java-profiler --pattern "ddprof*${DDPROF_ARCH}*.jar" --dir ./ddprof-release + - export DDPROF_JAR=$(ls ./ddprof-release/ddprof*${DDPROF_ARCH}*.jar) + - '[ "$(echo "${DDPROF_JAR}" | wc -l)" -eq 1 ] || { echo "Expected exactly one ddprof jar for ${DDPROF_ARCH}, got: ${DDPROF_JAR}"; exit 1; }' + + # --- 4. Patch the agent --- + # NOTE: confirm exact script path and argument order from the cloned repo + - bash java-profiler/patch-dd-java-agent.sh "${AGENT_JAR}" "${DDPROF_JAR}" "${PATCHED_JAR}" + + # --- 5. Upload to S3 --- + - aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_COMMIT_REF_NAME}/dd-java-agent-profiler.jar + - aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent-profiler.jar + + # --- 6. Annotate pipeline with public links --- + # printf is used instead of a heredoc to avoid shell-vs-YAML variable + # expansion ambiguity: GitLab CI executes `- |` blocks via bash, so + # unquoted heredoc delimiters do expand CI variables, but printf makes + # the intent explicit and sidesteps any editor/linter confusion. + - | + printf '{"S3 Links":[{"external_link":{"label":"Public Link to dd-java-agent-profiler.jar","url":"https://s3.us-east-1.amazonaws.com/dd-trace-java-builds/%s/dd-java-agent-profiler.jar"}}]}' \ + "${CI_PIPELINE_ID}" > links-profiler.json + artifacts: + paths: + - ${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-profiler.jar + reports: + annotations: + - links-profiler.json +``` + +--- + +## Open Questions + +1. **Script path in java-profiler**: Is it `patch-dd-java-agent.sh` at the repo + root, or in a subdirectory (e.g. `tools/`, `scripts/`)? Confirm after cloning. + +2. **Script argument order**: Confirm whether the signature is + `script ` or different (e.g. `--agent`, `--profiler`, + `--output` flags). + +3. **ddprof release artifact name**: Does the GitHub release publish `ddprof-*.jar` + or a differently named artifact? Check `gh release view --repo DataDog/java-profiler`. + +4. **Authentication for java-profiler**: Is the repo public or private? If private, + `GITHUB_TOKEN` needs read access to it (scope: `repo` or `contents:read`). + +5. **`gbi-ubuntu_2204:release` has AWS CLI?**: Confirm via `aws --version` in a + test job, or switch to a combined image. Alternatively keep the `aws-cli` image + and install `git`+`gh` as a `before_script` step (only two packages). + +6. **Multiple native architectures**: `ddprof` ships per-arch jars (x86_64, arm64). + The current upload target is `arch:amd64`. If arm64 patching is needed later, + a matrix job should be introduced. + +7. **Rules alignment**: Should this job also run on release branches (`v*.*.* `)? + The current `publish-artifacts-to-s3` has no explicit rules restriction. + +--- + +## Recommended Implementation Order + +1. Read `patch-dd-java-agent.sh` from the java-profiler repo to confirm the + interface (resolves OQ 1 & 2). +2. Confirm the ddprof release artifact name (`gh release view`) (OQ 3). +3. Confirm `GITHUB_TOKEN` access (OQ 4). +4. Choose image and confirm AWS CLI availability (OQ 5). +5. Add the job to `.gitlab-ci.yml` after `publish-artifacts-to-s3`. +6. Test on a branch before merging to master. diff --git a/doc/plan-publish-profiler-patched-agent.md b/doc/plan-publish-profiler-patched-agent.md new file mode 100644 index 00000000000..23c19025c57 --- /dev/null +++ b/doc/plan-publish-profiler-patched-agent.md @@ -0,0 +1,104 @@ +# Implementation Plan: publish-profiler-patched-agent-to-s3 + +Source design: `doc/design-publish-profiler-patched-agent.md` + +--- + +## Scope + +One file changes: `.gitlab-ci.yml`. + +Insert a new job `publish-profiler-patched-agent-to-s3` immediately after the +`publish-artifacts-to-s3` job (currently ending at line 435). No other files +are modified. + +--- + +## Tasks + +### Task 1 — Insert the new job into `.gitlab-ci.yml` + +**File**: `.gitlab-ci.yml` +**After**: line 435 (the closing line of `publish-artifacts-to-s3`) +**Action**: insert the following YAML block, preserving two blank lines between +jobs (consistent with the rest of the file). + +```yaml +publish-profiler-patched-agent-to-s3: + image: registry.ddbuild.io/images/base/gbi-ubuntu_2204:release + stage: publish + needs: [ build ] + dependencies: [ build ] + timeout: 10 minutes + variables: + AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY + AWS_DEFAULT_REGION: us-east-1 + before_script: + - aws --version 2>/dev/null || pip install --quiet awscli==1.36.40 + - export GH_TOKEN="${GITHUB_TOKEN}" + script: + - source upstream.env + - export VERSION="${UPSTREAM_TRACER_VERSION%~*}" + - export AGENT_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-${VERSION}.jar" + - export PATCHED_JAR="${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-profiler.jar" + - git clone --depth 1 https://oauth2:${GITHUB_TOKEN}@github.com/DataDog/java-profiler.git java-profiler + - ARCH=$(uname -m); case "${ARCH}" in x86_64) DDPROF_ARCH="linux-x86_64" ;; aarch64) DDPROF_ARCH="linux-arm64" ;; *) echo "Unsupported architecture ${ARCH}"; exit 1 ;; esac + - gh release download --repo DataDog/java-profiler --pattern "ddprof*${DDPROF_ARCH}*.jar" --dir ./ddprof-release + - export DDPROF_JAR=$(ls ./ddprof-release/ddprof*${DDPROF_ARCH}*.jar) + - '[ "$(echo "${DDPROF_JAR}" | wc -l)" -eq 1 ] || { echo "Expected exactly one ddprof jar for ${DDPROF_ARCH}, got: ${DDPROF_JAR}"; exit 1; }' + - bash java-profiler/patch-dd-java-agent.sh "${AGENT_JAR}" "${DDPROF_JAR}" "${PATCHED_JAR}" + - aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_COMMIT_REF_NAME}/dd-java-agent-profiler.jar + - aws s3 cp "${PATCHED_JAR}" s3://dd-trace-java-builds/${CI_PIPELINE_ID}/dd-java-agent-profiler.jar + - | + printf '{"S3 Links":[{"external_link":{"label":"Public Link to dd-java-agent-profiler.jar","url":"https://s3.us-east-1.amazonaws.com/dd-trace-java-builds/%s/dd-java-agent-profiler.jar"}}]}' \ + "${CI_PIPELINE_ID}" > links-profiler.json + artifacts: + paths: + - ${CI_PROJECT_DIR}/workspace/dd-java-agent/build/libs/dd-java-agent-profiler.jar + reports: + annotations: + - links-profiler.json +``` + +#### Verification checklist + +- [ ] Job appears after `publish-artifacts-to-s3` in the `publish` stage +- [ ] `needs: [ build ]` and `dependencies: [ build ]` are both present +- [ ] `timeout: 10 minutes` is set +- [ ] `before_script` installs `awscli` conditionally and exports `GH_TOKEN` +- [ ] Arch detection uses `uname -m` and covers `x86_64` and `aarch64`; exits 1 for unknown arch +- [ ] Exact-one-jar guard is present after `ls` +- [ ] Both S3 upload paths (`${CI_COMMIT_REF_NAME}` and `${CI_PIPELINE_ID}`) are present +- [ ] `artifacts.paths` lists the patched jar +- [ ] `artifacts.reports.annotations` lists `links-profiler.json` +- [ ] YAML is syntactically valid (run `yamllint` or `gitlab-ci-lint`) + +--- + +## Open Questions (must resolve before merging) + +These are carried over from the design document and block a clean merge: + +| # | Question | Where to look | +|---|----------|---------------| +| OQ1 | Exact path of `patch-dd-java-agent.sh` in java-profiler | `git clone --depth 1` the repo and `find . -name patch-dd-java-agent.sh` | +| OQ2 | Script argument order / flags | Read the script header/usage comment | +| OQ3 | Release artifact name pattern (`ddprof*${ARCH}*.jar` correct?) | `gh release view --repo DataDog/java-profiler` | +| OQ4 | `DataDog/java-profiler` public or private | `gh repo view DataDog/java-profiler` | +| OQ5 | Does `gbi-ubuntu_2204:release` already have `aws`? | Run `aws --version` in a throwaway job or inspect the image manifest | + +The `before_script` conditional (`aws --version 2>/dev/null || pip install`) means +OQ5 is handled safely at runtime regardless. OQ1–OQ3 must be resolved by manually +inspecting the repo before this PR is merged. + +--- + +## Validation + +1. **YAML lint**: `yamllint .gitlab-ci.yml` — must pass with no errors. +2. **GitLab CI lint**: push to a feature branch and use the GitLab CI Lint UI + (`/ci/lint`) to confirm the job graph is valid and the new job appears in the + `publish` stage with `build` as its dependency. +3. **Dry-run on branch**: trigger the pipeline on a non-master branch; confirm + the job runs, the patched jar is produced, and S3 uploads succeed.