CLDSRV-940: pin CI Node to 22.23.1 to fix node-fetch@2 premature-close regression by leif-scality · Pull Request #6209 · scality/cloudserver

leif-scality · 2026-06-26T15:06:03Z

CI functional‑test jobs (sur-tests, file-ft-tests, mongo-*-ft-tests) started failing intermittently around 2026‑06‑22 on ~12–15 bucket quota and rate‑limit tests, always with FetchError: … Premature close rather than an assertion. The root cause is a regression in the Node.js 22.23.0 / 24.17.0 security releases (nodejs/node#63989, node-fetch/node-fetch#1767) that breaks node-fetch@2 on reused keep‑alive pooled sockets under load, surfacing as a premature close while reading the response body. It hit globally and on the same day because the GitHub ubuntu-24.04 runner image bumped Node 22.22.3 → 22.23.0 and our CI selects Node via actions/setup-node with node-version: '22' (which floats to the latest 22.x), so every branch's runner picked up the broken release at once; only the bespoke node-fetch@2‑based quota/rate‑limit test tooling is affected (the AWS SDK v3 tests use a different HTTP stack, and the server itself is uninvolved — every request returns 200 and the process never crashes).

Pin the node-version to 22.23.1 to fix the issue.

bert-e · 2026-06-26T15:06:07Z

Hello leif-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-06-26T15:06:13Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-940 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.2.42
9.3.12
9.4.0

Please check the Fix Version/s of CLDSRV-940, or the target
branch of this pull request.

codecov · 2026-06-26T15:17:02Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.50%. Comparing base (f30dee2) to head (7b2dbfb).
✅ All tests successful. No failed tests found.

Additional details and impacted files

see 3 files with indirect coverage changes

@@               Coverage Diff                @@
##           development/9.2    #6209   +/-   ##
================================================
  Coverage            84.49%   84.50%           
================================================
  Files                  204      204           
  Lines                13217    13217           
================================================
+ Hits                 11168    11169    +1     
+ Misses                2049     2048    -1

Flag	Coverage Δ
file-ft-tests	`68.12% <ø> (ø)`
kmip-ft-tests	`28.41% <ø> (ø)`
mongo-v0-ft-tests	`69.41% <ø> (+0.03%)`	⬆️
mongo-v1-ft-tests	`69.35% <ø> (ø)`
multiple-backend	`35.68% <ø> (ø)`
sur-tests	`35.96% <ø> (-0.04%)`	⬇️
sur-tests-inflights	`37.81% <ø> (-0.03%)`	⬇️
unit	`70.34% <ø> (ø)`
utapi-v2-tests	`34.69% <ø> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…e regression Node 22.23.0 (from GitHub runner image 20260622) introduced a regression breaking node-fetch@2: reading a response body throws 'FetchError: Premature close' (nodejs/node#63989, node-fetch#1767), fixed in 22.23.1. CI floated node-version '22' and so picked up the broken release on every branch, failing the quota/rate-limit functional tooling. Pin to 22.23.1 (keeps the 22.23.0 security patches). Note: development/9.4 has a diverged CI layout (separate lint.yaml plus 4 node-version refs in tests.yaml) that this cascade will not fully cover; pin those 6 refs manually when bert-e raises the 9.4 cascade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

francoisferrand

nit: would be nice to have a way to store this value in just one place (like a single environment variable)... but probably overkill here, no simple way accross multiple actions

leif-scality · 2026-06-26T17:11:43Z

ping

leif-scality · 2026-06-26T17:14:11Z

/approve

bert-e · 2026-06-26T17:14:20Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-940 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.2.42
9.3.12
9.4.0

Please check the Fix Version/s of CLDSRV-940, or the target
branch of this pull request.

The following options are set: approve

leif-scality · 2026-06-26T17:16:40Z

ping

bert-e · 2026-06-26T17:16:51Z

Conflict

A conflict has been raised during the creation of
integration branch w/9.3/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close with contents from bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close
and development/9.3.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 git fetch
 git checkout -B w/9.3/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close origin/development/9.3
 git merge origin/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close
 # <intense conflict resolution>
 git commit
 git push -u origin w/9.3/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close

The following options are set: approve

leif-scality · 2026-06-26T17:29:29Z

ping

bert-e · 2026-06-26T17:29:42Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close into
development/9.2
w/9.3/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close will be merged into development/9.3
w/9.4/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close will be merged into development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e · 2026-06-26T17:45:45Z

Build failed

The build for commit did not succeed in branch w/9.3/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close

The following options are set: approve

bert-e · 2026-06-26T18:05:39Z

Build failed

The build for commit did not succeed in branch w/9.4/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close

The following options are set: approve

bert-e · 2026-06-26T18:10:44Z

Build failed

The build for commit did not succeed in branch w/9.3/bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close

The following options are set: approve

leif-scality force-pushed the bugfix/CLDSRV-940-fix-CI-fetch-error-premature-close branch from 9e45aa0 to 7b2dbfb Compare June 26, 2026 16:10

leif-scality changed the title ~~CLDSRV-940: harden quota/rate-limit functional tooling by adding retry and remove keep-alive~~ CLDSRV-940: pin CI Node to 22.23.1 to fix node-fetch@2 premature-close regression Jun 26, 2026

francoisferrand approved these changes Jun 26, 2026

View reviewed changes

jonathan-gramain approved these changes Jun 26, 2026

View reviewed changes

DarkIsDude approved these changes Jun 26, 2026

View reviewed changes

Uh oh!

Conversation

leif-scality commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Jun 26, 2026

Hello leif-scality,

Uh oh!

bert-e commented Jun 26, 2026

Incorrect fix version

Uh oh!

codecov Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

francoisferrand left a comment

Choose a reason for hiding this comment

Uh oh!

leif-scality commented Jun 26, 2026

Uh oh!

leif-scality commented Jun 26, 2026

Uh oh!

bert-e commented Jun 26, 2026

Incorrect fix version

Uh oh!

leif-scality commented Jun 26, 2026

Uh oh!

bert-e commented Jun 26, 2026

Conflict

Uh oh!

leif-scality commented Jun 26, 2026

Uh oh!

bert-e commented Jun 26, 2026

Integration data created

Uh oh!

bert-e commented Jun 26, 2026

Build failed

Uh oh!

bert-e commented Jun 26, 2026

Build failed

Uh oh!

bert-e commented Jun 26, 2026

Build failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

leif-scality commented Jun 26, 2026 •

edited

Loading

codecov Bot commented Jun 26, 2026 •

edited

Loading