Skip to content

Move CI provisioning from v1 workspaces to v2 clusters - #134

Open
kesmit13 wants to merge 12 commits into
mainfrom
ci-workspaces-to-clusters
Open

kesmit13 wants to merge 12 commits into
mainfrom
ci-workspaces-to-clusters

Conversation

@kesmit13

@kesmit13 kesmit13 commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Retargets the CI provisioning scripts and workflows from the v1
workspace-group flow at v2 clusters, which is what management.version
already defaults to.

Provisioning

resources/create_test_cluster.py is now a single manage_clusters() /
create_cluster() call. wait_on_active=True covers ACTIVE, then the
endpoint, then the firewall, which replaces both polling loops and a bare
time.sleep(10). --expires was parsed and then never used, so CI
clusters had no expiry at all; it is wired to expires_at now.
resources/drop_test_cluster.py takes a cluster ID and terminates it.

The admin password needs care. POST /v2/clusters accepts an
adminPassword and ignores it, generating its own, and PATCH does the
same, so the password can only be read off the create response and is
None after any refresh. That value cannot reach a consuming job: it has
to be masked, and the runner drops any output whose value matches a mask
-- "Skip output 'cluster-password' since it may contain secret" -- so
masking it and passing it between jobs are mutually exclusive.

So the generated password is traded for secrets.CLUSTER_PASSWORD over
SQL, once, while the script still holds it: ALTER USER 'admin'@'%' IDENTIFIED BY ... (SET PASSWORD wants a pre-hashed value and rejects a
literal with 1372: Password hash should be a 41-digit hexadecimal number). No password is reported at all after that, and every consuming
job reads the secret it already had. Keeping the credential a secret
rather than a job output is also why SINGLESTOREDB_URL interpolates it
raw: a percent-encoded copy is a different string from the mask, so
emitting one would put a live credential past the runner's masking. The
secret is maintainer-set, so keeping it alphanumeric is a cheaper
constraint than that; both consuming sites say so.

The ID, host, port and database name are reported as soon as
create_cluster() returns, before that password reset and before the SQL
load. Both can fail against a cluster that is already running and already
billing, and a teardown job with an empty cluster-id would send its
DELETE to /v2/clusters/ and leak it. The two shutdown steps refuse an
empty ID outright now.

secrets.CLUSTER_USER and vars.CLUSTER_PROJECT are gone. A cluster has
exactly one user, admin, and no route creates another, so the secret
could only ever hold that value; the project is named in the workflow so
the deployment target is visible next to the create call. Once this
merges, CLUSTER_USER can be deleted from repository settings.

A datetime bug this uncovered

expires_at read back as None on a cluster that demonstrably had an
expiry set. expiresAt is the one field the API returns in Go's
time.Time.String() format -- 2026-09-17 14:42:41.445984 +0000 UTC --
while every other timestamp is RFC 3339. to_datetime split on . to
pad the fraction, produced 445984 +0000 UTC, and parsing returned
None. Pre-existing, and it also left the expiry column blank in
SHOW CLUSTERS. Fixed with tests for both shapes.

Accepting that shape then exposed its zero value: an unset Go
time.Time renders as 0001-01-01 00:00:00 +0000 UTC, and only the
RFC 3339 spelling of that sentinel was recognized, so a resource that
does not expire reported an expiry in year 1. Both to_datetime and
to_datetime_strict now test the parsed value for January 1 of year 1,
which covers every spelling, and do it before the UTC shift, which can
fall below MINYEAR on a year-1 value.

Workflow linting

There was no linter over .github/workflows/, so nothing checked
expression syntax, needs.*.outputs.* references or the shell in run:
blocks. actionlint is now a pre-commit hook -- a pip wrapper, so no Go
toolchain or Docker -- and the 17 pre-existing findings it reported are
fixed: stale action majors, a step named after ${{ matrix.python-version }} in a job whose matrix only defines os, and
six unquoted $GITHUB_OUTPUT redirects.

Every action pin is also off the Node.js 20 runtime now. The first pass
bumped to checkout@v4 / setup-python@v5, which satisfied actionlint
but still declared node20; the pins are the lowest major of each action
that declares node24, read from action.yml at the tag. Verified by the
run annotations disappearing between 6c22959 and 8828326.

Verified

  • pre-commit clean, including actionlint on all six workflows, on 3.9-3.13
  • live create: ACTIVE in 5:54, test.sql loaded, expires_at exactly
    1h after created_at
  • test_basics against that cluster: 30 passed, 2 skipped
  • live teardown both ways: drop_test_cluster.py to TERMINATED in 2.8s,
    and the workflow's curl DELETE /v2/clusters/{id}?force=true returning
    200. No clusters leaked
  • marker partition exact: 905 selected + 65 deselected = 970
  • the zero-time sentinel in both shapes and both helpers, by unit test

Not verified

The reordered reporting block is not covered by a test, and the live
create above predates it. The publish.yml artifact-action bumps only
execute on a tag or release, so they land untested here.

🤖 Generated with Claude Code


Note

Medium Risk
Changes CI cluster lifecycle, credential handling, and release smoke paths against live cloud APIs; datetime parsing affects all management clients reading cluster expiry.

Overview
Retargets live CI from the deprecated v1 workspace-group API to v2 clusters, updating create_test_cluster.py and drop_test_cluster.py to use manage_clusters() / create_cluster() with project selection, cluster naming rules, expires_at, and an ALTER USER step so jobs can keep using a known CLUSTER_PASSWORD (the API ignores adminPassword). Workflows now connect as admin, tear down via DELETE /v2/clusters/{id}?force=true, and pass secrets through env vars where shell interpolation would leak them.

Fixes management timestamp parsing so expiresAt from GET /v2/clusters/{id} (Go time.Time.String() form) no longer becomes None; to_datetime normalizes RFC 3339 and Go shapes and returns naive UTC, with new unit tests.

Partitions deprecated v1 tests: PR/coverage runs use -m 'not management_v1'; coverage.yml adds a nightly management-v1-tests job for -m 'management_v1'. code-check.yml uses full git history (fetch-depth: 0) so change detection against origin/main works on PRs, stops swallowing git diff failures, and bumps Actions pins.

Adds actionlint to pre-commit and upgrades workflow actions (checkout/setup-python v7, artifacts, QEMU). Smoke tests add Python 3.14; docs build rewrites cluster.Stage links like workspace stage.

Reviewed by Cursor Bugbot for commit 1378da4. Bugbot is set up for automated code reviews on this repo. Configure here.

kesmit13 and others added 5 commits September 17, 2026 09:40
DEFAULT_MANAGEMENT_VERSION is v2 and the test suite already deploys
clusters, but CI still drove v1 workspace groups and tore down through a
hardcoded /v1/workspaces URL, emitting a DeprecationWarning on every run.

resources/create_test_cluster.py now makes one create_cluster() call.
wait_on_active covers ACTIVE, the endpoint and the firewall, replacing
both polling loops and the bare sleep, and closing a gap: the script
never waited for the firewall, which the API applies outside the state
machine. The shared "Python Client Testing" group -- never deleted by CI
-- is gone with the flat cluster model.

Fixes a live leak: --expires was parsed and never passed to the API, so
CI clusters had no expiry and a failed shutdown job leaked a billable
deployment indefinitely. It now reaches expires_at=.

POST /v2/clusters generates the admin password and ignores what is sent;
PATCH ignores it too (docs/management-api-audit.md item 9), so
create-then-PATCH is not available and secrets.CLUSTER_PASSWORD can no
longer be the password of the cluster CI just made. The generated value
is read off the create response and propagated as a cluster-password job
output. Job outputs are not secrets, so it is ::add-mask::-ed where it is
emitted and again in every consuming job -- the mask does not cross job
boundaries, and omitting the re-mask would leak a live credential.

The generated password is drawn from the full printable set (one observed
value: {:D}TK*[F3Ll}Ups2pNv), so a percent-encoded cluster-password-url
output is emitted alongside it for the SINGLESTOREDB_URL and
CIBW_ENVIRONMENT call sites; the raw form reaches drop_db.py through the
environment rather than being interpolated into a shell word. Verified
that the encoded form round-trips through the SDK's own URL parser.

Cluster names are cleaned to [a-z0-9]([a-z0-9-]*[a-z0-9])? and truncated
to 32 characters -- CI passes a workflow name that can overrun the limit.
Regions are matched to a Region object, since v2 regions have no ID, and
the pattern is tried against both the display and provider region names.
A project is required by the API and does not auto-resolve in a
multi-project org, so --project was added, defaulting to the
STANDARD-edition project and pinnable with the CLUSTER_PROJECT variable.

drop_test_cluster.py now takes a cluster ID, matching what the create
script emits; its old contract said workspace-id but slugified the
argument into a name. The workflow teardown stays curl, so the shutdown
job needs no install, with the URL moved to DELETE /v2/clusters/{id}.

Also makes the deprecated v1 suite a nightly gate rather than a per-PR
cost: code-check.yml and coverage.yml deselect management_v1, and
coverage.yml gains a job that runs it. Verified the two selections
partition the suite exactly, 905 + 65 of 970.

Two docs gaps closed alongside: Cluster.update()'s admin_password lacked
the warning create_cluster() carries, an asymmetry that invites the
create-then-PATCH dead end; and build_docs.py rewrote workspace.Stage
but not cluster.Stage, which the v2 shim now re-exports.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v2 clusters API returns expiresAt in Go's time.Time.String()
format -- '2026-09-17 14:42:41.445984 +0000 UTC' -- while every other
timestamp is RFC 3339. to_datetime split the string on '.' to pad the
fractional seconds, produced '445984 +0000 UTC' as the fraction, and
datetime_fromisoformat then returned None. Cluster.expires_at read as
None on a cluster that had an expiry set, and SHOW CLUSTERS printed a
blank expiry column.

Normalize both shapes before parsing: pad or truncate the fraction to
six digits, keep a numeric offset, and drop the trailing zone
abbreviation and Go's monotonic-clock reading. An offset-aware result
is converted to UTC and returned naive, matching what the RFC 3339
path already produced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A v2 cluster has exactly one user, admin, and no route creates another,
so secrets.CLUSTER_USER could only ever hold that one value. Drop the
secret and name admin directly.

Name the deployment project in the workflow too, rather than reading it
from vars.CLUSTER_PROJECT: the target is then visible next to the
create call and an unset repository variable cannot quietly change
where CI deploys. create_cluster resolves a project name against
GET /v2/projects and raises with the org's project list if it matches
none, so a renamed project fails loudly at setup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The workflow files had no linter, so nothing checked expression syntax,
needs.*.outputs.* references or the shell inside run: blocks. actionlint
covers all three and ships as a pip wrapper, so the hook needs no Go
toolchain or Docker.

It reported 17 pre-existing problems, all fixed here so the hook lands
green:

- actions/checkout@v3, actions/setup-python@v4 and
  docker/setup-qemu-action@v2 run on node runtimes GitHub is retiring.
  Bumped to v4, v5 and v3, which is what the newer workflows already
  pin.
- publish.yml named a step after ${{ matrix.python-version }} in a job
  whose matrix defines only os, so the name rendered with nothing after
  it. That job pins 3.10 as the host interpreter for cibuildwheel, so
  the reference was never going to resolve; dropped it.
- code-check.yml left $GITHUB_OUTPUT unquoted on six redirects. Quoted.
  The one remaining sed is prefixing every line, which parameter
  expansion cannot do, so SC2001 is suppressed in place with a reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit bumped checkout to v4 and setup-python to v5 to
satisfy actionlint, but both of those majors still declare node20, so
the runners kept reporting them as forced onto node24. actionlint 1.7.7
does not know about that deprecation, so it had nothing to say.

Pin the lowest major of each action that declares node24, taken from
action.yml at the tag rather than from the release notes: checkout v5,
setup-python v6, upload-artifact v6, setup-qemu-action v4, and
download-artifact v7 -- v5 and v6 of download-artifact are still node20,
so it is the one that has to skip further ahead. cibuildwheel and
gh-action-pypi-publish are composite actions and never had a node
runtime to move.

Choosing the lowest node24 major rather than the newest keeps the
behavioural change to a minimum; there is no other reason to jump to
checkout v7 today.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows the node24 move rather than stopping at the lowest major that
cleared it: checkout v7, setup-python v7, upload-artifact v7,
download-artifact v8, setup-qemu-action v4. One deprecation cycle
instead of two.

Checked the breaking changes in the majors this skips over:

- setup-python dropped its default Python version, so a step that names
  neither python-version nor python-version-file now fails. All nine
  call sites name python-version, so none are affected.
- download-artifact v5 changed the path layout for single downloads by
  ID. publish.yml downloads by name, so it is untouched.
- download-artifact v8 stopped unzipping unconditionally, checking
  Content-Type first, and now errors on a hash mismatch. The artifacts
  here are ordinary zipped directory uploads, so both apply harmlessly.

cibuildwheel and gh-action-pypi-publish stay where they are: both are
composite actions, so neither was part of the node problem, and moving
cibuildwheel two majors is a build-behaviour change that does not belong
in this PR.

The artifact pins are the ones with no coverage here -- publish.yml runs
only on a tag or a release, so they are first exercised by the next
release build.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Go renders the offset without a separator -- the expiresAt values come
back as '+0000' -- and datetime.fromisoformat only accepts that spelling
on Python 3.11 and later. On 3.9 and 3.10 it raised, the converter
returned the string unchanged, and to_datetime turned that into None:
exactly the silent unset expiration the Go-format handling was added to
fix, just on the interpreters the previous commit did not cover.

Normalize the offset to +00:00. Verified that every shape the normalizer
emits parses on 3.8, 3.10 and 3.11.

The new test asserts on the normalized string rather than on a parsed
datetime. The six existing tests were correct and still passed on 3.11,
which is how this reached CI; a string comparison fails the same way on
every version.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7a0f3ba. Configure here.

Comment thread .github/workflows/smoke-test.yml Outdated
POST /v2/clusters generates its own admin password and ignores any that
is sent, so create_test_cluster.py was reading the generated one back off
the create response and reporting it as a job output. That cannot work:
the value has to be masked, and the runner drops any output whose value
matches a mask -- "Skip output 'cluster-password' since it may contain
secret" -- so the test jobs received an empty password and failed with
"1045: Access denied for user 'admin'@... (using password: NO)".

Take a --password instead and hand it to the new cluster over SQL once it
is active, so every job reads the credential from secrets.CLUSTER_PASSWORD
and nothing crosses a job boundary. ALTER USER is the statement that
works; SET PASSWORD wants a 41-digit hash and rejects a literal. The
percent-encoded variant and the per-job ::add-mask:: steps both go away
with the output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
code-check.yml checks singlestoredb/management and singlestoredb/fusion
for changes and picks between a run that includes the v2 management tests
and one that excludes them. It has always picked the second. The checkout
was shallow, fetch-depth: 2, which creates no origin/main, so every diff
against it died -- "fatal: bad revision 'origin/main'" appears twice in
each run log -- and the `|| true` turned that into an empty file list,
which reads as "nothing changed". The step that runs -m 'not management_v1'
was unreachable and the -m 'not management' one always won, so the 37 live
v2 management tests never ran on a pull request; only the nightly
coverage.yml covered them.

Fetch the full history so origin/main exists, which also repairs the
branch-push path that probed origin/main and origin/master and fell
through to HEAD~1. Drop the `|| true` as well: a git failure means the
comparison did not happen, and swallowing it silently downgrades the run
rather than reporting the breakage.

Expect this job to get slower on any PR touching those directories -- the
tests it now selects deploy real clusters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A job's createdAt came back as '2026-09-18T12:39:20.43888Z'. The zone
group in _GO_DATETIME_RE demanded whitespace ahead of it, so a bare Z
never matched and the value fell through to the escape hatch, which
strips the Z and skips the fractional-second padding. Only Python 3.11
and later read a fraction that is neither 3 nor 6 digits, so on 3.10 the
converter handed back the string and to_datetime_strict raised
ValueError, taking down TestJobsFusion.test_run_wait_drop_job in CI.

Recognize Z as an offset so RFC 3339 goes down the same path as the Go
shape and gets its fraction padded, and spell the offset out as +00:00
for the same reason the numeric ones grew a colon: nothing before 3.11
parses the short form. _as_naive_utc shifts it back off, so parsed
results are unchanged. Verified on a real 3.10 that every shape the
normalizer emits parses, and that the old output for this value does not.

The tests assert on the normalized string, not on a parsed datetime, so
they fail on 3.11 as well -- the same blind spot that let the offset bug
reach CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The matrix stopped at 3.13 while 3.14 has been final since October 2025,
so the newest interpreter the package claims to support -- requires-python
is >=3.9 with no ceiling -- went untested. Crossed with the driver axis
this adds two jobs, mysql and https.

3.15 is left out on purpose: it is at rc2 today with GA planned for
2026-10-01, and setup-python will not resolve a bare "3.15" until then.
The condition for adding it is recorded above the matrix.

Not touched: the include: block still pins macOS and Windows to 3.11, so
3.14 is covered on Linux only, and publish.yml builds one abi3 wheel from
cp39, which needs no change for a new minor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Raw password URLs can fail for valid secrets, and setup failures can leave created clusters without a cleanup ID.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Retargets CI provisioning from v1 workspaces to v2 clusters, while improving timestamp parsing and workflow validation.

Changes:

  • Adds v2 cluster creation, expiration, project selection, password setup, and teardown.
  • Normalizes RFC 3339 and Go-formatted timestamps with regression tests.
  • Updates workflows, action versions, test markers, documentation transforms, and actionlint integration.
File summaries
File Description
resources/create_test_cluster.py Creates and initializes v2 test clusters.
resources/drop_test_cluster.py Terminates clusters by ID.
singlestoredb/management/utils.py Normalizes management timestamps.
singlestoredb/tests/test_management_utils.py Tests timestamp parsing behavior.
.github/workflows/smoke-test.yml Runs v2 cluster smoke tests.
.github/workflows/publish.yml Uses v2 clusters for wheel tests.
.github/workflows/coverage.yml Separates v1 nightly coverage.
.github/workflows/code-check.yml Improves change detection and test selection.
.github/workflows/pre-commit.yml Updates workflow actions.
.github/workflows/fusion-docs.yml Updates documentation workflow actions.
.pre-commit-config.yaml Adds actionlint.
resources/build_docs.py Normalizes cluster stage links.
singlestoredb/management/v2/cluster.py Documents ignored admin-password updates.
Review details

Suppressed comments (1)

.github/workflows/smoke-test.yml:154

  • The HTTPS matrix uses the same raw password interpolation as the MySQL matrix, so a secret containing +, @, /, ?, #, or % is not the password the client receives after URL parsing. Encode this userinfo component or supply the credential outside the URL rather than relying on an unenforced alphanumeric secret convention.
          SINGLESTOREDB_URL: "${{ matrix.driver }}://admin:${{ secrets.CLUSTER_PASSWORD }}@${{ needs.setup-database.outputs.cluster-host }}:443/${{ needs.setup-database.outputs.cluster-database }}?pure_python=${{ matrix.pure-python }}&buffered=${{ matrix.buffered }}"
  • Files reviewed: 13/13 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/publish.yml
Comment thread .github/workflows/smoke-test.yml
Comment thread resources/create_test_cluster.py Outdated
Comment thread singlestoredb/management/utils.py
Comment thread resources/create_test_cluster.py
Two findings from review.

resources/create_test_cluster.py reported the cluster ID after the
password reset and the SQL load, both of which can fail against a
cluster that is already running and already billing. A CI teardown job
would then have an empty cluster-id output and send its DELETE to
/v2/clusters/, leaking the cluster. Everything the reporting block
prints is known as soon as create_cluster() returns, so it now runs
there. The two workflow shutdown steps also refuse to issue a DELETE
with an empty ID, and --fail-with-body makes a refused one fail the step.

to_datetime read Go's zero time as year 1 whenever it arrived in the Go
shape -- '0001-01-01 00:00:00 +0000 UTC' -- because only the RFC 3339
spelling was compared against. That reports an expiry on a resource that
does not expire. Both helpers now test the parsed value for January 1 of
year 1, which covers every spelling including the offset, zone name and
monotonic reading, and the check runs before the UTC shift, which can
fall below MINYEAR on a year-1 value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants