diff --git a/.claude/rules/bff-go.md b/.claude/rules/bff-go.md index a5fa926..769b531 100644 --- a/.claude/rules/bff-go.md +++ b/.claude/rules/bff-go.md @@ -66,7 +66,7 @@ route, so handlers do not import chi. The OpenShell Go SDK is the source of truth: ```go -import openshell "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1" +import openshell "github.com/ai-openshell/sdk-go/openshell/v1" ``` Handlers never hold `openshell.ClientInterface` directly. `NewApp` resolves the @@ -188,7 +188,12 @@ answers a request without a bearer with `unauthorized` (`pkg/auth/proxy.go`). ## SDK updates -The SDK is pinned to the commit of an OpenShell **release tag** and moves +The SDK is its own module, `github.com/ai-openshell/sdk-go` +([ai-openshell/sdk-go](https://github.com/ai-openshell/sdk-go)); until +OpenShell 0.1.8 it was `github.com/NVIDIA/OpenShell/sdk/go`, with the same +packages and APIs, and a go.mod still on that path is moved off it, imports +included, by the next bump. It is pinned to sdk-go's tag of an OpenShell +**release** (`vX.Y.Z` for core `vX.Y.Z`) and moves together with the gateway the branch pins, in one commit (ADR 0005, decision 7; hard facts 17 and 20 in `openshell-api.md`). The Follow upstream workflow makes that commit, on a `bump/openshell-X.Y.Z` branch with a pull request into diff --git a/.claude/rules/openshell-api.md b/.claude/rules/openshell-api.md index 233992e..1d0b2d2 100644 --- a/.claude/rules/openshell-api.md +++ b/.claude/rules/openshell-api.md @@ -6,7 +6,7 @@ alwaysApply: false # OpenShell API Reference -**THE OPENSHELL GO SDK IS THE SOURCE OF TRUTH.** The BFF calls `github.com/NVIDIA/OpenShell/sdk/go` directly, and this repository holds no copy of the gateway's proto files and no generated stubs. Before writing any handler or TypeScript type, read the actual RPC/message definitions in `$(go env GOMODCACHE)/github.com/!n!v!i!d!i!a/!open!shell/sdk/go@/openshell/v1/` (hand-written client wrappers with doc comments; `types/*.go` for message shapes) or via `go doc`. Never invent RPCs, fields, or lifecycle states. If a UI idea has no backing RPC, flag it — do not fabricate an endpoint. The SDK pin moves only together with the gateway the branch pins, to the commit of an upstream *release tag*, and the Follow upstream workflow makes that move on a `bump/openshell-X.Y.Z` branch (hard facts 17 and 20). Never run `go get github.com/NVIDIA/OpenShell/sdk/go@latest`. +**THE OPENSHELL GO SDK IS THE SOURCE OF TRUTH.** The BFF calls the SDK directly. It is its own module, `github.com/ai-openshell/sdk-go` (repository [ai-openshell/sdk-go](https://github.com/ai-openshell/sdk-go)), whose tag `vX.Y.Z` is the SDK for OpenShell `vX.Y.Z`; until OpenShell 0.1.8 it was the `sdk/go` directory of NVIDIA/OpenShell, `github.com/NVIDIA/OpenShell/sdk/go`, and a `backend/go.mod` still on that path moves off it in the next bump (the bump rewrites the imports). Packages and APIs are the same on both paths, and this repository holds no copy of the gateway's proto files and no generated stubs. Before writing any handler or TypeScript type, read the actual RPC/message definitions in `$(go env GOMODCACHE)/github.com/ai-openshell/sdk-go@/openshell/v1/` (or `github.com/!n!v!i!d!i!a/!open!shell/sdk/go@/` while go.mod is still on the old path) (hand-written client wrappers with doc comments; `types/*.go` for message shapes) or via `go doc`. Never invent RPCs, fields, or lifecycle states. If a UI idea has no backing RPC, flag it — do not fabricate an endpoint. The SDK pin moves only together with the gateway the branch pins, to sdk-go's tag of an upstream *release*, and the Follow upstream workflow makes that move on a `bump/openshell-X.Y.Z` branch (hard facts 17 and 20). Never run `go get github.com/ai-openshell/sdk-go@latest`. **There are three direct gRPC calls**, all in `backend/pkg/clients`, each for a named gap in the SDK and all on the proto client the SDK ships. `rawexec.go` does binary-safe file uploads, because the public `Exec().Run(...)` API takes no stdin and `Exec().Interactive(...)` always asks for a PTY (hard fact 22). `rawprovider.go` reads the keys of the credentials a provider holds, because the SDK's `ProviderFromProto` drops the map the gateway returns them in (hard fact 8); `rawprovider_allworkspaces.go` is the same read for the all-workspaces list. `rawprofile.go` reads and writes provider profiles, because the SDK's profile endpoint type keeps a host, a port and a protocol and drops everything else the gateway stores (hard fact 24). Do not add another unless the SDK has a gap you can name, and delete each one when its gap closes. @@ -47,7 +47,7 @@ Skip `GatewayInterceptor`, `SupervisorMiddleware`, `ComputeDriver` — internal/ 17. **Three links, each proven separately — and the gateway and the SDK move together, on a bump branch (ADR 0005, decisions 6 to 8 and the amendment of 2026-11-06).** The chain is gateway → SDK → BFF → UI. - **Wire** (gateway ↔ SDK) is proven only by running `backend/test/compat` against a real gateway; it is never inferred from version numbers. **Source** (SDK ↔ BFF) is proven by the compiler, `go vet` and the unit tests. BFF and UI ship together. - **Say which link failed.** A compile error after an SDK move is a source migration, never "the gateway is incompatible". - - **A branch pins ONE upstream release** in `deploy/ci/gateway-pins.json` — the gateway and supervisor images and the SDK at that release's tag — and every change is tested against that gateway. + - **A branch pins ONE upstream release** in `deploy/ci/gateway-pins.json` — the gateway and supervisor images and the SDK at sdk-go's tag of that release — and every change is tested against that gateway. - **Moving to another release is one commit that moves all of it:** the pins file, `backend/go.mod`, `backend/go.sum`, the built-in release line when the minor changes, and the README's generated block. Nobody makes that commit by hand. `follow-upstream.yml` reads upstream's tags once a day and, for a new stable release, makes it on `bump/openshell-X.Y.Z`: `main` plus that one commit, titled `fix: move to OpenShell X.Y.Z`, with a pull request into `main`. A person can run it with a `version` to bump to a given release, a pre-release included. The bot creates a bump branch once and never rebases, updates or force-pushes it; after that it is an ordinary branch. Merging the pull request moves `main` and releases nothing: a person cuts the release (`publish.yml`), the next patch, or `X.Y.0` when the move starts a new gateway minor. - **Upstream pre-releases are tested ahead of the stable release in two ways:** *check A*, the BFF as `main` builds it against the target gateway (a failure is the one issue labelled `compat-migrate`), and CI on the bump pull request, the dashboard on the target's SDK against the target's gateway. - **A change that needs the upcoming release** — an SDK method or an RPC the released gateway does not have — goes on that release's bump branch; before upstream ships it, on a prep branch made by running Follow upstream with the pre-release as `version`. Everything else goes to `main`. @@ -67,7 +67,7 @@ Skip `GatewayInterceptor`, `SupervisorMiddleware`, `ComputeDriver` — internal/ - **The pins file.** `deploy/ci/gateway-pins.json` holds `release` (without the `v`), that release's gateway and supervisor images pinned by digest (`tag@sha256:…`, the tag being the release), `sdk`, `sandbox_image` and `config_schema`. - **Which branch pins what.** `main` and `release/**` pin a stable release. A bump branch pins the release in its name; only a prep branch a person asked for pins a pre-release, and `ci.yml`'s `pins a stable release` check refuses a pre-release in every pull request, so a prep branch cannot merge until upstream ships. That check is deliberately not `compat`: it means "not released upstream yet", not "does not work". - **Never pin a `dev` build, with or without a digest.** A dev gateway pulls `ghcr.io/nvidia/openshell/sandbox:dev` at runtime, so it drifts with upstream HEAD anyway. - - **The SDK pin.** The pin is the SDK at the commit the release tag points at, written identically in `backend/go.mod` and in the `sdk` field. Up to 0.1.6 the SDK module had no tags of its own, so the pin is a pseudo-version of that commit (`v0.1.3` → `sdk/go@v0.0.0-20261009050449-e1f3c82caa3e`). From 0.1.7 upstream tags the module itself (`sdk/go/v0.1.7`), `go get` of the release commit resolves to that tag, and the pin is `v0.1.7`; the bump checks with `go list -m -json` that the tag is the release commit. CI fails when they differ, when the pins file is malformed, or when it is not in canonical form (`python3 deploy/ci/upstream/follow.py format-pins` rewrites it). CI cannot tell a release tag's commit from any other commit; the Follow upstream workflow can, and its run summary says so when `main`'s SDK is not at the tag of the release it pins. + - **The SDK pin.** The pin is the SDK for the pinned release, written identically in `backend/go.mod` and in the `sdk` field. The SDK is `github.com/ai-openshell/sdk-go` and its tag `vX.Y.Z` is the SDK for core `vX.Y.Z`, so the pin is that tag (`v0.1.9`); the bump reads the tag's commit from sdk-go with `git ls-remote` and checks with `go list -m -json` that `go get` landed on it. sdk-go cuts the tag after core releases (its `core-watch.yml` opens a bump per core tag, `release-tag.yml` tags it once that merges), so for a while a core release has no SDK: the bump then waits, says so in the run summary, and pins nothing else in its place, while check A, which needs no SDK, still runs. Before that, the SDK was the `sdk/go` directory of NVIDIA/OpenShell, `github.com/NVIDIA/OpenShell/sdk/go`: up to 0.1.6 pinned as a pseudo-version of the release commit (`v0.1.3` → `v0.0.0-20261009050449-e1f3c82caa3e`), at 0.1.7 and 0.1.8 also tagged there (`sdk/go/v0.1.8` is the last). A branch still on that path is valid; the bump that moves it to sdk-go rewrites the BFF's imports to the new path in the same commit. CI fails when they differ, when the pins file is malformed, or when it is not in canonical form (`python3 deploy/ci/upstream/follow.py format-pins` rewrites it). CI cannot tell a release tag's commit from any other commit; the Follow upstream workflow can, and its run summary says so when `main`'s SDK is not at the tag of the release it pins. - **The line in the BFF.** One more thing restates the pins: the gateway release line compiled into the BFF (`BuiltInGatewayReleaseLine` in `backend/pkg/models/gateway_release_line.go`), so that an image knows its line with no build args and no environment. It must be the major.minor of the pinned release, and `node scripts/gateway-range.mjs --check` fails in CI when it is not; the pin move changes the constant when the minor changes. 21. **Gateway settings are typed and type-checked.** `SettingValue` is a oneof of string, bool, int and bytes. Each registered key takes exactly one kind and the gateway refuses any other (`setting 'ocsf_json_enabled' expects bool value`). Gateways 0.1.0 to 0.1.2 register the same four keys: `ocsf_json_enabled` and `agent_policy_proposals_enabled` are bools, `ocsf_schema_version` and `proposal_approval_mode` are strings with a fixed set of values. `GetGatewayConfig` lists every registered key, and one that was never set comes back with an *empty* value: no kind. The API does not expose the registry (the CLI and TUI link it in), so the type of an unset setting cannot be read. The BFF mirrors all of this: a setting is a JSON string, boolean or integer in both directions, absent when unset, and nothing is coerced — the Settings page edits a value in the type it has and asks for the type when it has none. diff --git a/.claude/skills/proto-sync/SKILL.md b/.claude/skills/proto-sync/SKILL.md index 709981c..e60ee36 100644 --- a/.claude/skills/proto-sync/SKILL.md +++ b/.claude/skills/proto-sync/SKILL.md @@ -13,7 +13,7 @@ moves to the new release, and the handlers and models it affects are audited. ## Before you start You do not move the SDK. It moves together with the gateway the branch pins, -to the commit of an upstream release tag, and a workflow makes that move +to ai-openshell/sdk-go's tag of an upstream release, and a workflow makes that move (ADR 0005, decision 7; hard facts 17 and 20 in `.claude/rules/openshell-api.md`): @@ -49,9 +49,13 @@ git diff origin/main origin/bump/openshell- -- deploy/ci/gateway-pins.j ``` Then compare the two SDK versions: -- `sdk/go/openshell/v1/` for public client methods and resource shapes -- `sdk/go/openshell/v1/types/` for request/response field details -- `sdk/go/proto/sandboxv1/` only when the frontend policy protojson contract is involved +- `openshell/v1/` for public client methods and resource shapes +- `openshell/v1/types/` for request/response field details +- `proto/sandboxv1/` only when the frontend policy protojson contract is involved + +These are paths in the SDK module, `github.com/ai-openshell/sdk-go` (in +`sdk/go/` of NVIDIA/OpenShell while go.mod is still on the old path, +`github.com/NVIDIA/OpenShell/sdk/go`). ### 2. Check out the bump branch diff --git a/.github/workflows/follow-upstream.yml b/.github/workflows/follow-upstream.yml index 24e3c67..5a5d277 100644 --- a/.github/workflows/follow-upstream.yml +++ b/.github/workflows/follow-upstream.yml @@ -16,8 +16,10 @@ # # bump Opens a bump: the branch bump/openshell-, which is main # plus ONE commit that moves the pin (gateway and supervisor -# images, and the Go SDK at the tag's commit, together), and a -# pull request into main. A scheduled run bumps only to a stable +# images, and the Go SDK at ai-openshell/sdk-go's tag of the same +# version, together), and a pull request into main. sdk-go tags +# the SDK after core releases; until it has, there is no bump and +# the run summary says so. A scheduled run bumps only to a stable # release; a version asked for by hand is bumped to as it is, so # a pre-release gets a prep branch. The bot makes a bump branch # ONCE: when the branch exists, or any pull request has come from diff --git a/deploy/ci/upstream/README.md b/deploy/ci/upstream/README.md index 5832c65..faf25ca 100644 --- a/deploy/ci/upstream/README.md +++ b/deploy/ci/upstream/README.md @@ -17,14 +17,30 @@ python3 deploy/ci/upstream/follow.py require-stable # fail if it names a p python3 deploy/ci/upstream/follow.py plan --out /tmp/plan.json # what is upstream ahead of this branch? ``` -`plan` asks the real upstream (`git ls-remote`, and `docker buildx imagetools -inspect` for the target's image digests). It only reads; it starts nothing. +`plan` asks the real upstream (`git ls-remote` of NVIDIA/OpenShell and of +ai-openshell/sdk-go, and `docker buildx imagetools inspect` for the target's +image digests). It only reads; it starts nothing. + +## The SDK + +The Go SDK is its own module, `github.com/ai-openshell/sdk-go`. Its tag +`vX.Y.Z` is the SDK for core `vX.Y.Z`, and sdk-go cuts it after core releases. +So a target can have its images and no SDK yet: `plan` then sets the target's +`sdk` to null and `sdk_waiting_for` to why, the `bump` output is `false`, and +the run summary says the bump waits. Check A still runs. Nothing else is +pinned in the SDK's place; the next run looks again. + +Until OpenShell 0.1.8 the SDK was the `sdk/go` directory of the core +repository, `github.com/NVIDIA/OpenShell/sdk/go`. The pins check accepts a +go.mod on either path, and the pin move takes a tree off the old one: it +rewrites `backend/**/*.go` to the new import path before `go mod tidy`, and +commits those files with the pin. ## One release per branch `deploy/ci/gateway-pins.json` names the one OpenShell release a branch is built on. The gateway and supervisor images and the Go SDK all come from that -release's tag and move together, in one commit. +release (the SDK from sdk-go's tag of it) and move together, in one commit. | Branch | Pins | Who moves it | |---|---|---| @@ -151,7 +167,8 @@ gets a branch of its own. | File | What moves | |---|---| | `deploy/ci/gateway-pins.json` | `release`, `gateway_image`, `supervisor_image`, `sdk` | -| `backend/go.mod`, `backend/go.sum` | the SDK, with `go get @` and `go mod tidy`. Never `@latest` | +| `backend/go.mod`, `backend/go.sum` | the SDK, with `go get github.com/ai-openshell/sdk-go@vX.Y.Z` and `go mod tidy`, then `go list -m -json` to check the tag is the commit `plan` read. Never `@latest` | +| `backend/**/*.go` | only in the move off `github.com/NVIDIA/OpenShell/sdk/go`: the import path | | `backend/pkg/models/gateway_release_line.go` | the built-in line, only when the minor changes | | `README.md` | the generated block, with `scripts/readme-gateway-range.mjs --write` | diff --git a/deploy/ci/upstream/bump.py b/deploy/ci/upstream/bump.py index 1331475..0aaec2c 100644 --- a/deploy/ci/upstream/bump.py +++ b/deploy/ci/upstream/bump.py @@ -80,14 +80,16 @@ def maybe(self, *args): return done.stdout.strip() if done.returncode == 0 else None -def _stage_pin_move(git): +def _stage_pin_move(git, rewritten=()): """Stage the pin move's files. True when there is something to commit. - The move may change PIN_PATHS and nothing else. A change anywhere else - means a tool did something it was not asked to, and it is not committed. + The move may change PIN_PATHS, and the Go files it says it rewrote off the + SDK's old path, and nothing else. A change anywhere else means a tool did + something it was not asked to, and it is not committed. """ + allowed = set(pinmove.PIN_PATHS) | set(rewritten or ()) changed = [text[3:] for text in git.run("status", "--porcelain").stdout.splitlines() if text.strip()] - foreign = [path for path in changed if path not in pinmove.PIN_PATHS] + foreign = [path for path in changed if path not in allowed] if foreign: raise GitError( "the pin move changed files it must not: %s. Nothing was committed." % ", ".join(sorted(foreign)) @@ -132,7 +134,7 @@ def prepare(repo, target, mover, message, main_ref="HEAD"): try: moved = mover(tree, target) outcome["moved"] = moved - if not _stage_pin_move(work): + if not _stage_pin_move(work, (moved or {}).get("rewritten", ())): outcome["reason"] = "main already pins OpenShell %s" % target["version"] return outcome _commit(work, message(target, moved)) diff --git a/deploy/ci/upstream/follow.py b/deploy/ci/upstream/follow.py index f7fd004..8621553 100644 --- a/deploy/ci/upstream/follow.py +++ b/deploy/ci/upstream/follow.py @@ -87,7 +87,9 @@ def _load_valid_pins(path, go_mod=None): if go_mod: go_mod_version = pins.go_mod_sdk(_read(go_mod)) if go_mod_version is None: - raise pins.PinsError(["%s does not require %s" % (go_mod, pins.SDK_MODULE)]) + raise pins.PinsError( + ["%s does not require %s (nor %s, its old path)" % (go_mod, pins.SDK_MODULE, pins.LEGACY_SDK_MODULE)] + ) problems = pins.validate(doc, schemas=pins.known_schemas(path), go_mod_version=go_mod_version) if problems: raise pins.PinsError(problems) @@ -154,8 +156,8 @@ def cmd_plan(args, upstream=None): "ready": "true" if target.get("ready") else "false", "target": target.get("version", ""), "same_line": "true" if target.get("same_line") else "false", - # Something to open a bump branch for, with images to pin. - "bump": "true" if target.get("bump") and target.get("ready") else "false", + # Something to open a bump branch for, with images and an SDK tag to pin. + "bump": "true" if target.get("bump") and target.get("ready") and target.get("sdk") else "false", "branch": target.get("branch", ""), "gateway_image": target.get("gateway_image") or "", "supervisor_image": target.get("supervisor_image") or "", @@ -193,7 +195,7 @@ def cmd_bump(args, run=None): repo, target, mover=lambda tree, to: pinmove.apply(tree, to, tools=tools, run=run, echo=_echo), - message=lambda to, moved: report.commit_message(to, made["pinned"]), + message=lambda to, moved: report.commit_message(to, made["pinned"], migrated=moved.get("migrated")), main_ref=args.main_ref, ) decided = publish_module.build(made, outcome, pulls=pulls, branch_sha=branch_sha, repo_url=args.repo_url) diff --git a/deploy/ci/upstream/pinmove.py b/deploy/ci/upstream/pinmove.py index 10e5ee8..b96ec14 100644 --- a/deploy/ci/upstream/pinmove.py +++ b/deploy/ci/upstream/pinmove.py @@ -5,13 +5,16 @@ brought back in step: deploy/ci/gateway-pins.json release, images, sdk - backend/go.mod, backend/go.sum the SDK at the tag's commit + backend/go.mod, backend/go.sum the SDK at its tag for the release backend/pkg/models/gateway_release_line.go only when the minor changes README.md the generated block -The SDK has no tags of its own: it is a directory of the upstream repository. -So it is pinned to the commit the release tag points at, with `go get -@` and `go mod tidy`. Never `@latest`. +The SDK is its own module, github.com/ai-openshell/sdk-go, and its tag vX.Y.Z +is the SDK for core vX.Y.Z. It is pinned at that tag with `go get +@vX.Y.Z` and `go mod tidy`, never `@latest`, and `go list -m -json` +confirms the tag is the commit the plan read from sdk-go. A tree still on the +SDK's old path (github.com/NVIDIA/OpenShell/sdk/go, a directory of the core +repository until 0.1.8) has its imports rewritten in the same move. What runs here is `go get`, `go mod tidy`, `go list -m` and this repository's own README script. None of them builds or executes anything from upstream, which is why @@ -25,7 +28,7 @@ import pins import versions -from upstream import SDK_MODULE +from upstream import LEGACY_SDK_MODULE, SDK_MODULE PINS = "deploy/ci/gateway-pins.json" GO_MOD = "backend/go.mod" @@ -35,7 +38,8 @@ README_SCRIPT = "scripts/readme-gateway-range.mjs" # Everything a pin move may change. The commit is made from these paths and -# no others. +# no others, plus the Go files rewrite_imports() changed in the one move that +# takes a tree off the SDK's old path. PIN_PATHS = (PINS, GO_MOD, GO_SUM, LINE_SOURCE, README) LINE_CONSTANT = "BuiltInGatewayReleaseLine" @@ -76,22 +80,45 @@ def must(run, command, cwd, echo): raise PinMoveError("`%s` failed in %s:\n%s" % (" ".join(command), cwd, output.rstrip())) -def sdk_commands(commit): - """The two commands that move the SDK, in the order they run.""" - return [["go", "get", "%s@%s" % (SDK_MODULE, commit)], ["go", "mod", "tidy"]] +def sdk_commands(version): + """`go get` of the SDK at its tag. `go mod tidy` runs after it, once imports are right.""" + return [["go", "get", "%s@%s" % (SDK_MODULE, version)]] -def _tagged_commit(run, backend, version, echo): - """The commit a tagged SDK version was cut from, or None. +TIDY = ["go", "mod", "tidy"] - From OpenShell 0.1.7 upstream tags the SDK module itself (sdk/go/vX.Y.Z), - so `go get @` resolves to that tag rather than to a - pseudo-version, and the version no longer names the commit. `go list -m - -json` reads the module's metadata, which records the commit the tag - points at; it builds and runs nothing. + +def rewrite_imports(tree): + """Move the BFF's imports from the SDK's old path to its own module. + + The SDK left the core repository with OpenShell 0.1.7: the same packages + under github.com/ai-openshell/sdk-go instead of + github.com/NVIDIA/OpenShell/sdk/go. A tree that still imports the old path + is rewritten once, in the move that takes it onto the new module, so the + bump branch builds. Returns the paths it changed, relative to the tree. + """ + changed = [] + backend = os.path.join(tree, "backend") + for directory, subdirs, files in os.walk(backend): + subdirs[:] = sorted(name for name in subdirs if not name.startswith(".") and name != "vendor") + for name in sorted(files): + if not name.endswith(".go"): + continue + path = os.path.join(directory, name) + text = _read(path) + if LEGACY_SDK_MODULE not in text: + continue + _write(path, text.replace(LEGACY_SDK_MODULE, SDK_MODULE)) + changed.append(os.path.relpath(path, tree).replace(os.sep, "/")) + return changed + + +def _sdk_origin(run, backend, version, echo): + """The commit the SDK's tag points at, as the module metadata records it, or None. + + `go list -m -json` reads the module's metadata, which records the commit + the tag was cut from; it builds and runs nothing. """ - if pins.sdk_commit(version) or not version: - return None command = ["go", "list", "-m", "-json", "%s@%s" % (SDK_MODULE, version)] if echo: echo("$ " + " ".join(command)) @@ -104,23 +131,25 @@ def _tagged_commit(run, backend, version, echo): raise PinMoveError("`%s` printed no module metadata:\n%s" % (" ".join(command), output.rstrip())) origin = found.get("Origin") or {} commit = origin.get("Hash") - return commit[:12] if isinstance(commit, str) and len(commit) >= 12 else None + return commit if isinstance(commit, str) and len(commit) >= 12 else None def at_target(doc, tree, target): - """Is this tree already on the target: its release, its images and its SDK commit? + """Is this tree already on the target: its release, its images and its SDK tag? doc: the tree's pins. The image digests count, not only the version: a pre-release tag that upstream pushed again is a different gateway under the same name. """ - commit = pins.sdk_commit(pins.go_mod_sdk(_read(os.path.join(tree, GO_MOD))) or "") + module, version = pins.go_mod_sdk_module(_read(os.path.join(tree, GO_MOD))) + sdk = target.get("sdk") or {} return ( doc.get("release") == target["version"] and doc.get("gateway_image") == target["gateway_image"] and doc.get("supervisor_image") == target["supervisor_image"] - and bool(commit) - and target["commit"].startswith(commit) + and module == SDK_MODULE + and bool(sdk.get("version")) + and version == sdk["version"] ) @@ -145,21 +174,49 @@ def apply(tree, target, tools, run=None, echo=None): if not isinstance(doc, dict): raise PinMoveError("%s in the tree being moved is not a JSON object" % PINS) before = doc.get("release") - result = {"moved": False, "from": before, "to": target["version"], "sdk": doc.get("sdk"), "line": None} + result = { + "moved": False, + "from": before, + "to": target["version"], + "sdk": doc.get("sdk"), + "line": None, + "migrated": False, + "rewritten": [], + } if at_target(doc, tree, target): return result + sdk_target = target.get("sdk") or {} + if not sdk_target.get("version") or not sdk_target.get("commit"): + raise PinMoveError( + "the plan names no SDK for OpenShell %s: %s has not tagged %s yet. Nothing is moved " + "without it." % (target["version"], SDK_MODULE, target["tag"]) + ) backend = os.path.join(tree, "backend") - for command in sdk_commands(target["commit"]): + go_mod_path = os.path.join(tree, GO_MOD) + module_before, _ = pins.go_mod_sdk_module(_read(go_mod_path)) + if module_before is None: + raise PinMoveError("%s requires neither %s nor %s" % (GO_MOD, SDK_MODULE, LEGACY_SDK_MODULE)) + for command in sdk_commands(sdk_target["version"]): must(run, command, backend, echo) - sdk = pins.go_mod_sdk(_read(os.path.join(tree, GO_MOD))) - if sdk is None: - raise PinMoveError("%s no longer requires %s after the move" % (GO_MOD, SDK_MODULE)) - commit = pins.sdk_commit(sdk) or _tagged_commit(run, backend, sdk, echo) - if not commit or not target["commit"].startswith(commit): + rewritten = rewrite_imports(tree) if module_before == LEGACY_SDK_MODULE else [] + if echo and rewritten: + echo("rewrote the SDK's imports from %s to %s in %d files" % (LEGACY_SDK_MODULE, SDK_MODULE, len(rewritten))) + must(run, TIDY, backend, echo) + + go_mod = _read(go_mod_path) + module, sdk = pins.go_mod_sdk_module(go_mod) + if module != SDK_MODULE or sdk is None: + raise PinMoveError("%s does not require %s after the move" % (GO_MOD, SDK_MODULE)) + if re.search(r"^\s*(?:require\s+)?%s\s" % re.escape(LEGACY_SDK_MODULE), go_mod, re.M): + raise PinMoveError( + "%s still requires %s after the move: something still imports the SDK's old path" % (GO_MOD, LEGACY_SDK_MODULE) + ) + commit = _sdk_origin(run, backend, sdk, echo) if sdk == sdk_target["version"] else None + if not commit or not sdk_target["commit"].startswith(commit[:12]): raise PinMoveError( - "%s requires SDK %s after `go get`, which is not commit %s of %s. Nothing is pinned " - "to an SDK other than the one at the release tag." % (GO_MOD, sdk, target["commit"][:12], target["tag"]) + "%s requires SDK %s after `go get`, which is not %s at %s (%s). Nothing is pinned to an SDK " + "other than the one tagged for the release." % (GO_MOD, sdk, SDK_MODULE, sdk_target["version"], sdk_target["commit"][:12]) ) moved = pins.move(doc, target["version"], target["gateway_image"], target["supervisor_image"], sdk) @@ -197,5 +254,5 @@ def apply(tree, target, tools, run=None, echo=None): tree, echo, ) - result.update(moved=True, sdk=sdk) + result.update(moved=True, sdk=sdk, migrated=module_before == LEGACY_SDK_MODULE, rewritten=rewritten) return result diff --git a/deploy/ci/upstream/pins.py b/deploy/ci/upstream/pins.py index 72f9c8d..9841fd6 100644 --- a/deploy/ci/upstream/pins.py +++ b/deploy/ci/upstream/pins.py @@ -36,9 +36,13 @@ # A Go pseudo-version (what `go get module@` writes) or a tagged # release. Never a query such as "latest": a query is not a pin. SDK_PSEUDO_RE = re.compile(r"^v\d+\.\d+\.\d+-(?:[0-9A-Za-z.]+\.)?\d{14}-[0-9a-f]{12}$") -SDK_TAGGED_RE = re.compile(r"^v\d+\.\d+\.\d+$") +SDK_TAGGED_RE = re.compile(r"^v\d+\.\d+\.\d+(?:-pre\.\d+)?$") PSEUDO_COMMIT_RE = re.compile(r"[-.]\d{14}-([0-9a-f]{12})$") -SDK_MODULE = "github.com/NVIDIA/OpenShell/sdk/go" +SDK_MODULE = "github.com/ai-openshell/sdk-go" +# The SDK's path until OpenShell 0.1.8. A go.mod that still requires it is +# valid until the next pin move takes it off it (pinmove.py). +LEGACY_SDK_MODULE = "github.com/NVIDIA/OpenShell/sdk/go" +SDK_MODULES = (SDK_MODULE, LEGACY_SDK_MODULE) # Tags that move. An image reference carrying one of these is not a release # even when a digest is attached, because the gateway inside still resolves @@ -83,10 +87,22 @@ def known_schemas(pins_path): return found +def go_mod_sdk_module(go_mod_text): + """(module, version) of the SDK backend/go.mod requires, or (None, None). + + The SDK at its own path, github.com/ai-openshell/sdk-go, when go.mod + requires it; else at the path it had inside the core repository. + """ + for module in SDK_MODULES: + m = re.search(r"^\s*(?:require\s+)?%s\s+(\S+)" % re.escape(module), go_mod_text, re.M) + if m: + return module, m.group(1) + return None, None + + def go_mod_sdk(go_mod_text): """The SDK version required by backend/go.mod, or None.""" - m = re.search(r"^\s*(?:require\s+)?%s\s+(\S+)" % re.escape(SDK_MODULE), go_mod_text, re.M) - return m.group(1) if m else None + return go_mod_sdk_module(go_mod_text)[1] def sdk_commit(version): diff --git a/deploy/ci/upstream/plan.py b/deploy/ci/upstream/plan.py index 1522383..6645355 100644 --- a/deploy/ci/upstream/plan.py +++ b/deploy/ci/upstream/plan.py @@ -18,7 +18,7 @@ import pins import versions -from upstream import GATEWAY_REPOSITORY, REGISTRY, SUPERVISOR_REPOSITORY +from upstream import GATEWAY_REPOSITORY, REGISTRY, SDK_MODULE, SDK_URL, SUPERVISOR_REPOSITORY class PlanError(Exception): @@ -36,6 +36,23 @@ def _images(upstream, version): return refs[0], refs[1], None +def _sdk(upstream, version): + """(the SDK to pin for this release, why there is none yet). + + The SDK for core vX.Y.Z is ai-openshell/sdk-go's tag vX.Y.Z. sdk-go cuts it + after core releases, so a release can have no SDK tag for a while. Then + there is nothing to bump to yet: never another SDK version, never a commit. + """ + tag = "v" + version + commit = versions.parse_tags(upstream.sdk_tags()).get(tag) + if not commit: + return None, ( + "%s has not tagged %s yet. It releases the SDK after core, and the bump waits for it" + % (SDK_URL.split("github.com/")[1], tag) + ) + return {"module": SDK_MODULE, "version": tag, "commit": commit}, None + + def _pin_on_tag(doc, tags): """Is the pinned SDK the commit the pinned release's tag points at? @@ -115,6 +132,7 @@ def build(doc, upstream, requested=None): return plan gateway, supervisor, waiting_for = _images(upstream, target["version"]) + sdk, sdk_waiting_for = _sdk(upstream, target["version"]) new_line = versions.line(target["version"]) != versions.line(pinned) plan["target"] = { "version": target["version"], @@ -142,7 +160,11 @@ def build(doc, upstream, requested=None): "branch": branch_name(target["version"]), "gateway_image": gateway, "supervisor_image": supervisor, + # Images to test and pin. Check A needs these and nothing else. "ready": waiting_for is None, "waiting_for": waiting_for, + # The SDK to pin. A bump needs it as well. + "sdk": sdk, + "sdk_waiting_for": sdk_waiting_for, } return plan diff --git a/deploy/ci/upstream/publish.py b/deploy/ci/upstream/publish.py index d28b18a..2d98075 100644 --- a/deploy/ci/upstream/publish.py +++ b/deploy/ci/upstream/publish.py @@ -64,6 +64,8 @@ def blocked(plan, pulls=None, branch_sha=None): return "there is nothing to bump to" if not target.get("ready"): return target.get("waiting_for") or "the release's images are not published yet" + if not target.get("sdk"): + return target.get("sdk_waiting_for") or "the SDK for this release is not tagged yet" own = own_pulls(pulls, branch) if own: first = min(own, key=lambda pull: pull["number"]) @@ -93,7 +95,7 @@ def build(plan, outcome, pulls=None, branch_sha=None, repo_url=""): return publish publish["push"] = {"sha": outcome["sha"]} - publish["pr"] = {"title": report.pr_title(target["version"]), "body": report.pr_body(plan, repo_url)} + publish["pr"] = {"title": report.pr_title(target["version"]), "body": report.pr_body(plan, repo_url, migrated=bool((outcome.get("moved") or {}).get("migrated")))} if repo_url: publish["compare"] = "%s/compare/%s...%s?expand=1" % (repo_url.rstrip("/"), BASE, branch) return publish diff --git a/deploy/ci/upstream/report.py b/deploy/ci/upstream/report.py index d22e322..57ec3f4 100644 --- a/deploy/ci/upstream/report.py +++ b/deploy/ci/upstream/report.py @@ -8,7 +8,7 @@ import bump import versions -from upstream import UPSTREAM_URL +from upstream import LEGACY_SDK_MODULE, SDK_MODULE, UPSTREAM_URL ADR_PATH = "docs/adrs/0005-gateway-compatibility-and-releases.md" WORKFLOW_PATH = ".github/workflows/follow-upstream.yml" @@ -65,6 +65,18 @@ def _tag_link(tag): return "[%s](%s/releases/tag/%s)" % (tag, UPSTREAM_URL, tag) +def _sdk_module(target): + return (target.get("sdk") or {}).get("module", SDK_MODULE) + + +def _sdk_version(target): + return (target.get("sdk") or {}).get("version", target["tag"]) + + +def _sdk_commit(target): + return ((target.get("sdk") or {}).get("commit") or "")[:12] + + # --- the pin commit -------------------------------------------------------- @@ -72,14 +84,19 @@ def pr_title(release): return PR_TITLE % release -def commit_message(target, pinned): +def commit_message(target, pinned, migrated=False): """The pin commit's message. The trailer is how the commit is found again.""" version = target["version"] paragraphs = [ "Moves the OpenShell release this branch is built on from %s to %s: the gateway and " - "supervisor images the compatibility suite runs against, and the Go SDK at the commit " - "upstream tagged %s (%s)." % (pinned["release"], version, target["tag"], target["commit"][:12]) + "supervisor images the compatibility suite runs against, and the Go SDK at %s %s (%s)." + % (pinned["release"], version, _sdk_module(target), _sdk_version(target), _sdk_commit(target)) ] + if migrated: + paragraphs.append( + "The SDK is now its own module, %s, so the move also rewrites the BFF's imports from %s " + "to it. Its packages and APIs are unchanged." % (SDK_MODULE, LEGACY_SDK_MODULE) + ) if target["line"] != pinned["line"]: paragraphs.append( "%s starts gateway release line %s, so the line compiled into the BFF moves from %s " @@ -99,7 +116,7 @@ def commit_message(target, pinned): # --- the bump pull request ----------------------------------------------- -def pr_body(plan, repo_url=""): +def pr_body(plan, repo_url="", migrated=False): """The description of a bump pull request. Written once, when it is opened.""" target, pinned = plan["target"], plan["pinned"] version = target["version"] @@ -125,8 +142,14 @@ def pr_body(plan, repo_url=""): "| release | %s |" % _code(version), "| gateway | %s |" % _code(target["gateway_image"]), "| supervisor | %s |" % _code(target["supervisor_image"]), - "| Go SDK | at %s |" % _code(target["commit"][:12]), + "| Go SDK | %s %s (%s) |" % (_code(_sdk_module(target)), _code(_sdk_version(target)), _code(_sdk_commit(target))), ] + if migrated: + lines += [ + "", + "**The SDK moved.** It is now its own module, %s, and this commit also rewrites the BFF's " + "imports from %s to it. Its packages and APIs are unchanged." % (_code(SDK_MODULE), _code(LEGACY_SDK_MODULE)), + ] if target.get("skipped"): lines += [ "", @@ -339,6 +362,16 @@ def plan_summary(plan, dry_run=False): else: lines.append("- gateway %s" % _code(target["gateway_image"])) lines.append("- supervisor %s" % _code(target["supervisor_image"])) + if target.get("sdk"): + lines.append( + "- Go SDK %s %s (%s)" + % (_code(target["sdk"]["module"]), _code(target["sdk"]["version"]), _code(target["sdk"]["commit"][:12])) + ) + elif target["bump"]: + lines.append( + "- **No bump yet:** %s. Check A does not need the SDK and still runs. The next run looks again." + % target["sdk_waiting_for"] + ) return "\n".join(lines) + "\n" diff --git a/deploy/ci/upstream/tests/support.py b/deploy/ci/upstream/tests/support.py index d1f84a0..e7634b1 100644 --- a/deploy/ci/upstream/tests/support.py +++ b/deploy/ci/upstream/tests/support.py @@ -49,6 +49,16 @@ def commit_of(version): return hashlib.sha1(("commit of " + version).encode()).hexdigest() +def sdk_commit_of(version): + """A made-up commit for ai-openshell/sdk-go's tag of a release (no leading v).""" + return hashlib.sha1(("sdk-go commit of " + version).encode()).hexdigest() + + +def sdk_of(version): + """The plan's SDK for a release: sdk-go's tag of it.""" + return {"module": pins.SDK_MODULE, "version": "v" + version, "commit": sdk_commit_of(version)} + + def image(name, version, salt=""): return "ghcr.io/nvidia/openshell/%s:%s@%s" % (name, version, digest(name + version + salt)) @@ -101,6 +111,8 @@ def target(version, salt="", stable=None): "supervisor_image": image("supervisor", version, salt), "ready": True, "waiting_for": None, + "sdk": sdk_of(version), + "sdk_waiting_for": None, } @@ -114,6 +126,11 @@ def __init__(self, data): def release_tags(self): return "\n".join(self.data["tags"]) + "\n" + def sdk_tags(self): + if "sdk_tags" in self.data.get("outage", []): + raise upstream.UpstreamError("git ls-remote %s failed: simulated" % upstream.SDK_REPO) + return "\n".join(self.data["sdk_tags"]) + "\n" + def image_digest(self, repository, tag): key = "%s:%s" % (repository.rsplit("/", 1)[-1], tag) self.digest_lookups.append(key) @@ -122,11 +139,21 @@ def image_digest(self, repository, tag): return self.data["images"].get(key) -def upstream_data(tags=None, images=None, outage=None, drop=()): - """The real upstream of 2026-10-09, with a test's additions on top.""" +def upstream_data(tags=None, images=None, outage=None, drop=(), sdk_missing=()): + """The real upstream of 2026-10-09, with a test's additions on top. + + ai-openshell/sdk-go has a tag for every core release tag, except the + versions in sdk_missing (no leading v): the ones it has not cut yet. + """ data = copy.deepcopy(fixture("upstream.json")) data["tags"] = [text for text in data["tags"] if text.split("refs/tags/")[1] not in drop] data["tags"] += ["%s\trefs/tags/%s" % (commit_of(name), name) for name in (tags or [])] + names = sorted({text.split("refs/tags/")[1] for text in data["tags"] if not text.endswith("^{}")}) + data["sdk_tags"] = [ + "%s\trefs/tags/%s" % (sdk_commit_of(name[1:]), name) + for name in names + if name.startswith("v") and name[1:] not in sdk_missing + ] data["images"].update(images or {}) data["outage"] = list(outage or []) return data @@ -159,12 +186,11 @@ def readme_block(doc): class FakeTools(object): """Runs in place of pinmove.run_command, and records what was asked of it.""" - def __init__(self, fail=None, stray=None, tagged=None): + def __init__(self, fail=None, stray=None, origin=None): self.calls = [] - # {release commit: tag}: `go get` of that commit resolves to the SDK's - # own tag, as it does upstream from 0.1.7 on. - self.tagged = tagged or {} - self.tag_commits = {tag: commit for commit, tag in self.tagged.items()} + # {SDK version: commit}: what the module metadata says a tag points + # at, when a test wants it to differ from sdk-go's real tag. + self.origin = origin or {} self.fail = fail # a command prefix that fails, e.g. ("go", "get") self.stray = stray # a path `go mod tidy` also writes, which it must not @@ -173,19 +199,28 @@ def __call__(self, command, cwd): if self.fail and tuple(command[: len(self.fail)]) == tuple(self.fail): return 1, "%s: simulated failure" % " ".join(command) if command[:2] == ["go", "get"]: - module, _, commit = command[2].partition("@") + module, _, version = command[2].partition("@") path = os.path.join(cwd, "go.mod") text = _read(path) - version = self.tagged.get(commit) or sdk_version(commit) - text = re.sub(r"(?m)^\t%s \S+$" % re.escape(module), "\t%s %s" % (module, version), text) + line = re.compile(r"(?m)^\t%s \S+$" % re.escape(module)) + if line.search(text): + text = line.sub("\t%s %s" % (module, version), text) + else: + text = text.replace("require (\n", "require (\n\t%s %s\n" % (module, version), 1) _write(path, text) return 0, "" if command[:3] == ["go", "list", "-m"]: - # A tagged SDK: the module metadata names the commit the tag points at. module, _, version = command[-1].partition("@") - return 0, json.dumps({"Path": module, "Version": version, "Origin": {"Hash": self.tag_commits.get(version, "")}}) + commit = self.origin.get(version, sdk_commit_of(version[1:])) + return 0, json.dumps({"Path": module, "Version": version, "Origin": {"Hash": commit, "Ref": "refs/tags/" + version}}) if command[:3] == ["go", "mod", "tidy"]: - _write(os.path.join(cwd, "go.sum"), go_sum_for(_read(os.path.join(cwd, "go.mod")))) + path = os.path.join(cwd, "go.mod") + text = _read(path) + if not _imports(cwd, pins.LEGACY_SDK_MODULE): + # tidy drops a requirement nothing imports. + text = re.sub(r"(?m)^\t%s \S+\n" % re.escape(pins.LEGACY_SDK_MODULE), "", text) + _write(path, text) + _write(os.path.join(cwd, "go.sum"), go_sum_for(text)) if self.stray: _write(os.path.join(cwd, self.stray), "left behind\n") return 0, "" @@ -207,6 +242,14 @@ def _read(path): return fh.read() +def _imports(directory, module): + for root, _, files in os.walk(directory): + for name in files: + if name.endswith(".go") and module in _read(os.path.join(root, name)): + return True + return False + + def _write(path, text): directory = os.path.dirname(path) if directory: @@ -240,6 +283,14 @@ def _write(path, text): ) """ +SDK_USER = "backend/pkg/services/sandbox.go" +SDK_USER_SOURCE = """package services + +import openshell "%s/openshell/v1" + +var _ openshell.ClientInterface +""" + LINE_SOURCE = """package models // The gateway minor release line this build is for. @@ -254,32 +305,38 @@ class Repo(object): # test that asks: a dozen git commands cost more than copying their result. _templates = {} - def __init__(self, release="0.1.3"): + def __init__(self, release="0.1.3", legacy=False): + """legacy: on the SDK's old path, as main was until it moved past OpenShell 0.1.8.""" self.root = tempfile.mkdtemp(prefix="upstream-test-") self.origin = os.path.join(self.root, "origin.git") self.path = os.path.join(self.root, "work") - if release not in Repo._templates: - self._build(release) + key = (release, legacy) + if key not in Repo._templates: + self._build(release, legacy) template = tempfile.mkdtemp(prefix="upstream-template-") atexit.register(shutil.rmtree, template, ignore_errors=True) for name in ("origin.git", "work"): shutil.copytree(os.path.join(self.root, name), os.path.join(template, name), symlinks=True) - Repo._templates[release] = template + Repo._templates[key] = template return for name in ("origin.git", "work"): - shutil.copytree(os.path.join(Repo._templates[release], name), os.path.join(self.root, name), symlinks=True) + shutil.copytree(os.path.join(Repo._templates[key], name), os.path.join(self.root, name), symlinks=True) - def _build(self, release): + def _build(self, release, legacy): self._git(self.root, "init", "-q", "--bare", self.origin) self._git(self.root, "init", "-q", "-b", "main", self.path) # Relative, so a copy of the pair still points at its own origin. self.git("remote", "add", "origin", "../origin.git") - doc = pins_doc(release) - go_mod = GO_MOD % (pins.SDK_MODULE, doc["sdk"]) + if legacy: + module, doc = pins.LEGACY_SDK_MODULE, pins_doc(release) + else: + module, doc = pins.SDK_MODULE, pins_doc(release, sdk="v" + release) + go_mod = GO_MOD % (module, doc["sdk"]) self.write(pinmove.PINS, pins.dump(doc)) self.write(pinmove.GO_MOD, go_mod) self.write(pinmove.GO_SUM, go_sum_for(go_mod)) self.write(pinmove.LINE_SOURCE, LINE_SOURCE % ".".join(release.split(".")[:2])) + self.write(SDK_USER, SDK_USER_SOURCE % module) self.write(pinmove.README, "# Dashboard\n\n%s\n\nMore prose.\n" % readme_block(doc)) package = json.dumps({"dependencies": {"react": "18.0.0"}}, indent=2) + "\n" self.write("frontend/package.json", package) diff --git a/deploy/ci/upstream/tests/test_bump.py b/deploy/ci/upstream/tests/test_bump.py index 4e2a9b0..d7d8b90 100644 --- a/deploy/ci/upstream/tests/test_bump.py +++ b/deploy/ci/upstream/tests/test_bump.py @@ -55,5 +55,24 @@ def test_the_working_copy_is_left_as_it_was(self): self.assertEqual(self.repo.git("worktree", "list").count("\n"), 0, "the scratch worktree is gone") +class OffTheOldPath(unittest.TestCase): + prepare = Prepare.prepare + + def setUp(self): + self.repo = support.Repo("0.1.3", legacy=True) + self.addCleanup(self.repo.close) + + def test_the_rewritten_imports_are_committed_with_the_pins(self): + outcome = self.prepare(support.target("0.1.9")) + changed = set(self.repo.git("diff", "--name-only", "origin/main", outcome["sha"]).splitlines()) + self.assertIn(support.SDK_USER, changed) + self.assertLessEqual(changed, set(pinmove.PIN_PATHS) | {support.SDK_USER}) + self.assertIn("github.com/ai-openshell/sdk-go/openshell/v1", self.repo.read(support.SDK_USER, ref=outcome["sha"])) + + def test_a_go_file_the_move_did_not_rewrite_is_still_refused(self): + with self.assertRaisesRegex(bump.GitError, "changed files it must not"): + self.prepare(support.target("0.1.9"), tools=support.FakeTools(stray="pkg/other.go")) + + if __name__ == "__main__": unittest.main() diff --git a/deploy/ci/upstream/tests/test_cli.py b/deploy/ci/upstream/tests/test_cli.py index 906d8b6..f9f08e2 100644 --- a/deploy/ci/upstream/tests/test_cli.py +++ b/deploy/ci/upstream/tests/test_cli.py @@ -188,6 +188,21 @@ def test_a_version_asked_for_reaches_the_plan_and_the_outputs(self): self.assertEqual(code, 0) self.assertEqual((outputs["target"], outputs["bump"], outputs["branch"]), ("0.1.4-pre.1", "true", "bump/openshell-0.1.4-pre.1")) + def test_a_release_with_no_sdk_tag_yet_is_tested_but_not_bumped(self): + images = {"gateway:0.1.4": support.digest("g"), "supervisor:0.1.4": support.digest("s")} + _, outputs, made = self.plan(tags=["v0.1.4"], images=images, sdk_missing=("0.1.4",)) + # Check A runs on the images; the bump waits for ai-openshell/sdk-go to tag v0.1.4. + self.assertEqual((outputs["target"], outputs["ready"], outputs["bump"]), ("0.1.4", "true", "false")) + summary = self.read("plan.md") + self.assertIn("No bump yet", summary) + self.assertIn("ai-openshell/sdk-go has not tagged v0.1.4 yet", summary) + + def test_a_release_with_its_sdk_tag_is_bumped(self): + images = {"gateway:0.1.4": support.digest("g"), "supervisor:0.1.4": support.digest("s")} + _, outputs, _ = self.plan(tags=["v0.1.4"], images=images) + self.assertEqual((outputs["target"], outputs["ready"], outputs["bump"]), ("0.1.4", "true", "true")) + self.assertIn("Go SDK `github.com/ai-openshell/sdk-go` `v0.1.4`", self.read("plan.md")) + def test_a_schedule_does_not_bump_to_a_pre_release(self): _, outputs, _ = self.plan() self.assertEqual((outputs["target"], outputs["bump"]), ("0.1.4-pre.2", "false")) diff --git a/deploy/ci/upstream/tests/test_pinmove.py b/deploy/ci/upstream/tests/test_pinmove.py index 5f05636..8bd04fe 100644 --- a/deploy/ci/upstream/tests/test_pinmove.py +++ b/deploy/ci/upstream/tests/test_pinmove.py @@ -41,8 +41,10 @@ def test_says_what_moved(self): "moved": True, "from": "0.1.3", "to": "0.1.4-pre.2", - "sdk": support.sdk_version(self.to["commit"]), + "sdk": "v0.1.4-pre.2", "line": None, + "migrated": False, + "rewritten": [], }, ) @@ -51,17 +53,22 @@ def test_the_pins_name_the_release_its_images_and_its_sdk(self): self.assertEqual(doc["release"], "0.1.4-pre.2") self.assertEqual(doc["gateway_image"], self.to["gateway_image"]) self.assertEqual(doc["supervisor_image"], self.to["supervisor_image"]) - self.assertEqual(doc["sdk"], support.sdk_version(self.to["commit"])) + self.assertEqual(doc["sdk"], "v0.1.4-pre.2") self.assertTrue(pins.is_canonical(self.repo.read(pinmove.PINS))) - def test_go_mod_is_on_the_sdk_at_the_tags_commit(self): - self.assertEqual(pins.go_mod_sdk(self.repo.read(pinmove.GO_MOD)), self.pins()["sdk"]) - self.assertIn(self.to["commit"][:12], self.repo.read(pinmove.GO_SUM)) + def test_go_mod_is_on_the_sdk_go_module_at_the_releases_tag(self): + go_mod = self.repo.read(pinmove.GO_MOD) + self.assertEqual(pins.go_mod_sdk_module(go_mod), (pins.SDK_MODULE, self.pins()["sdk"])) + self.assertIn("%s v0.1.4-pre.2" % pins.SDK_MODULE, self.repo.read(pinmove.GO_SUM)) - def test_the_sdk_is_fetched_at_the_commit_and_tidied_never_at_latest(self): + def test_the_sdk_is_fetched_at_its_tag_tidied_and_its_tag_checked_never_at_latest(self): self.assertEqual( - self.tools.calls[:2], - [["go", "get", "%s@%s" % (pins.SDK_MODULE, self.to["commit"])], ["go", "mod", "tidy"]], + self.tools.calls[:3], + [ + ["go", "get", "%s@v0.1.4-pre.2" % pins.SDK_MODULE], + ["go", "mod", "tidy"], + ["go", "list", "-m", "-json", "%s@v0.1.4-pre.2" % pins.SDK_MODULE], + ], ) for command in self.tools.calls: self.assertNotIn("@latest", " ".join(command)) @@ -88,7 +95,7 @@ def test_nothing_else_in_the_tree_changes(self): def test_only_go_get_go_mod_tidy_and_the_readme_script_are_run(self): # Nothing that builds or executes what was downloaded. for command in self.tools.calls: - self.assertIn(tuple(command[:2]), (("go", "get"), ("go", "mod"), ("node", command[1])), command) + self.assertIn(tuple(command[:2]), (("go", "get"), ("go", "mod"), ("go", "list"), ("node", command[1])), command) self.assertNotIn(command[1], ("build", "test", "vet", "run", "install", "generate")) @@ -138,7 +145,8 @@ def test_at_target_needs_the_release_the_digests_and_the_sdk_commit(self): self.apply(to) self.assertTrue(at(to)) self.assertFalse(at(support.target("0.1.4-pre.2", salt="rebuilt"))) - self.assertFalse(at(dict(to, commit="f" * 40))) + self.assertFalse(at(dict(to, sdk=support.sdk_of("0.1.4")))) + self.assertFalse(at(dict(to, sdk=None))) class WhatStopsIt(PinMove): @@ -155,34 +163,31 @@ def test_a_failing_tidy_or_readme_script_is_an_error(self): with self.assertRaises(pinmove.PinMoveError): self.apply(support.target("0.1.4"), support.FakeTools(fail=prefix)) - def test_an_sdk_that_did_not_land_on_the_tags_commit_is_refused(self): - # go.mod ends up somewhere other than the commit that was asked for. + def test_an_sdk_that_did_not_land_on_the_tag_is_refused(self): + # go.mod ends up on some other version than the one that was asked for. def elsewhere(command, cwd): if command[:2] == ["go", "get"]: - command = ["go", "get", "%s@%s" % (pins.SDK_MODULE, "e" * 40)] + command = ["go", "get", "%s@v0.1.3" % pins.SDK_MODULE] return support.FakeTools()(command, cwd) with self.assertRaises(pinmove.PinMoveError) as caught: self.apply(support.target("0.1.4"), elsewhere) - self.assertIn("Nothing is pinned to an SDK other than the one at the release tag", str(caught.exception)) - - def test_a_tagged_sdk_is_pinned_when_its_tag_is_the_release_commit(self): - # From OpenShell 0.1.7 upstream tags the SDK module, and `go get` of - # the release commit resolves to that tag. - to = support.target("0.1.4") - tools = support.FakeTools(tagged={to["commit"]: "v0.1.4"}) - result = self.apply(to, tools) - self.assertEqual((result["moved"], result["sdk"]), (True, "v0.1.4")) - self.assertEqual(self.pins()["sdk"], "v0.1.4") - self.assertIn(["go", "list", "-m", "-json", "%s@v0.1.4" % pins.SDK_MODULE], tools.calls) - - def test_a_tagged_sdk_whose_tag_is_another_commit_is_refused(self): - to = support.target("0.1.4") - tools = support.FakeTools(tagged={to["commit"]: "v0.1.4"}) - tools.tag_commits["v0.1.4"] = "e" * 40 + self.assertIn("Nothing is pinned to an SDK other than the one tagged for the release", str(caught.exception)) + + def test_a_tag_that_is_not_the_commit_the_plan_read_is_refused(self): + # sdk-go's tag was moved between the plan and the bump. + tools = support.FakeTools(origin={"v0.1.4": "e" * 40}) with self.assertRaises(pinmove.PinMoveError) as caught: - self.apply(to, tools) - self.assertIn("Nothing is pinned to an SDK other than the one at the release tag", str(caught.exception)) + self.apply(support.target("0.1.4"), tools) + self.assertIn("Nothing is pinned to an SDK other than the one tagged for the release", str(caught.exception)) + + def test_a_target_with_no_sdk_tag_yet_is_refused_and_nothing_is_run(self): + before = self.repo.read(pinmove.PINS) + with self.assertRaises(pinmove.PinMoveError) as caught: + self.apply(dict(support.target("0.1.4"), sdk=None)) + self.assertIn("has not tagged v0.1.4 yet", str(caught.exception)) + self.assertEqual(self.tools.calls, []) + self.assertEqual(self.repo.read(pinmove.PINS), before) def test_a_target_whose_image_is_of_another_release_is_refused(self): to = dict(support.target("0.1.4"), gateway_image=support.image("gateway", "0.1.3")) @@ -213,5 +218,56 @@ def test_a_missing_tool_is_an_error_that_names_it(self): self.assertIn("no-such-tool-anywhere", output) +class OffTheOldPath(unittest.TestCase): + """A tree that still imports github.com/NVIDIA/OpenShell/sdk/go, as main did up to 0.1.8.""" + + def setUp(self): + self.repo = support.Repo("0.1.3", legacy=True) + self.addCleanup(self.repo.close) + self.tools = support.FakeTools() + self.to = support.target("0.1.9") + self.result = pinmove.apply(self.repo.path, self.to, tools=TOOLS, run=self.tools) + + def test_the_move_lands_on_the_new_module_at_the_tag(self): + go_mod = self.repo.read(pinmove.GO_MOD) + self.assertEqual(pins.go_mod_sdk_module(go_mod), (pins.SDK_MODULE, "v0.1.9")) + self.assertNotIn(pins.LEGACY_SDK_MODULE, go_mod) + self.assertEqual(pins.load(os.path.join(self.repo.path, pinmove.PINS))["sdk"], "v0.1.9") + + def test_the_imports_are_rewritten_and_reported(self): + source = self.repo.read(support.SDK_USER) + self.assertIn('"%s/openshell/v1"' % pins.SDK_MODULE, source) + self.assertNotIn(pins.LEGACY_SDK_MODULE, source) + self.assertEqual((self.result["migrated"], self.result["rewritten"]), (True, [support.SDK_USER])) + + def test_imports_are_rewritten_before_tidy_so_tidy_drops_the_old_path(self): + self.assertEqual( + [command[:3] for command in self.tools.calls[:2]], + [["go", "get", "%s@v0.1.9" % pins.SDK_MODULE], ["go", "mod", "tidy"]], + ) + + def test_only_the_pin_paths_and_the_rewritten_go_files_change(self): + out = self.repo.git("status", "--porcelain") + changed = sorted(text.split()[-1] for text in out.splitlines()) + self.assertEqual(changed, sorted([pinmove.PINS, pinmove.GO_MOD, pinmove.GO_SUM, pinmove.README, support.SDK_USER])) + + def test_an_import_left_on_the_old_path_is_refused(self): + repo = support.Repo("0.1.3", legacy=True) + self.addCleanup(repo.close) + + def keeps_the_old_import(command, cwd): + code, output = support.FakeTools()(command, cwd) + if command[:3] == ["go", "mod", "tidy"]: + # Something outside backend/**.go still imports the old path. + path = os.path.join(cwd, "go.mod") + text = support._read(path).replace("require (\n", "require (\n\t%s v0.0.1\n" % pins.LEGACY_SDK_MODULE, 1) + support._write(path, text) + return code, output + + with self.assertRaises(pinmove.PinMoveError) as caught: + pinmove.apply(repo.path, self.to, tools=TOOLS, run=keeps_the_old_import) + self.assertIn("still requires %s" % pins.LEGACY_SDK_MODULE, str(caught.exception)) + + if __name__ == "__main__": unittest.main() diff --git a/deploy/ci/upstream/tests/test_pins.py b/deploy/ci/upstream/tests/test_pins.py index 839c65c..525bfa9 100644 --- a/deploy/ci/upstream/tests/test_pins.py +++ b/deploy/ci/upstream/tests/test_pins.py @@ -259,6 +259,28 @@ def test_go_mod_sdk_reads_both_forms_of_require(self): self.assertEqual(pins.go_mod_sdk(single), "v0.1.3") self.assertIsNone(pins.go_mod_sdk("module example.com/x\n")) + def test_go_mod_sdk_reads_the_new_module_and_the_old_path(self): + new = "require (\n\t%s v0.1.9\n)\n" % pins.SDK_MODULE + old = "require (\n\t%s v0.0.0-20261027154453-ac7f615cd12d\n)\n" % pins.LEGACY_SDK_MODULE + self.assertEqual(pins.SDK_MODULE, "github.com/ai-openshell/sdk-go") + self.assertEqual(pins.go_mod_sdk_module(new), (pins.SDK_MODULE, "v0.1.9")) + self.assertEqual(pins.go_mod_sdk_module(old), (pins.LEGACY_SDK_MODULE, "v0.0.0-20261027154453-ac7f615cd12d")) + self.assertEqual(pins.go_mod_sdk_module("module x\n"), (None, None)) + # While a move is half done both may be listed; the new module is the pin. + self.assertEqual(pins.go_mod_sdk(new + old), "v0.1.9") + + def test_the_sdk_field_must_equal_go_mod_on_either_path(self): + doc = support.pins_doc("0.1.9", sdk="v0.1.9") + self.assertEqual(pins.validate(doc, go_mod_version=pins.go_mod_sdk("require %s v0.1.9\n" % pins.SDK_MODULE)), []) + problems = pins.validate(doc, go_mod_version=pins.go_mod_sdk("require %s v0.1.8\n" % pins.SDK_MODULE)) + self.assertTrue(any("must move in the same commit" in problem for problem in problems)) + + def test_a_tagged_pre_release_sdk_is_a_pin(self): + doc = support.pins_doc("0.1.9-pre.4", sdk="v0.1.9-pre.4") + doc["gateway_image"] = support.image("gateway", "0.1.9-pre.4") + doc["supervisor_image"] = support.image("supervisor", "0.1.9-pre.4") + self.assertEqual(pins.validate(doc), []) + def test_sdk_commit(self): self.assertEqual(pins.sdk_commit("v0.0.0-20261009050449-e1f3c82caa3e"), "e1f3c82caa3e") self.assertEqual(pins.sdk_commit("v0.1.4-0.20261009050449-e1f3c82caa3e"), "e1f3c82caa3e") diff --git a/deploy/ci/upstream/tests/test_plan.py b/deploy/ci/upstream/tests/test_plan.py index 4b2914d..04a07a5 100644 --- a/deploy/ci/upstream/tests/test_plan.py +++ b/deploy/ci/upstream/tests/test_plan.py @@ -136,6 +136,40 @@ def test_a_missing_supervisor_image_is_not_ready_either(self): self.assertIn("supervisor image", made["target"]["waiting_for"]) +class TheSdkTag(unittest.TestCase): + """The SDK for core vX.Y.Z is ai-openshell/sdk-go's tag vX.Y.Z, cut after core releases.""" + + IMAGES = images_for("0.1.4") + + def test_the_target_pins_sdk_go_at_the_releases_tag(self): + made, _ = build(tags=["v0.1.4"], images=self.IMAGES) + target = made["target"] + self.assertTrue(target["bump"]) + self.assertEqual( + target["sdk"], + {"module": "github.com/ai-openshell/sdk-go", "version": "v0.1.4", "commit": support.sdk_commit_of("0.1.4")}, + ) + self.assertIsNone(target["sdk_waiting_for"]) + + def test_a_release_sdk_go_has_not_tagged_yet_waits(self): + made, _ = build(tags=["v0.1.4"], images=self.IMAGES, sdk_missing=("0.1.4",)) + target = made["target"] + # Still the target, and ready for check A: its images are out. + self.assertEqual((target["version"], target["ready"], target["bump"]), ("0.1.4", True, True)) + # But there is no SDK to pin, and nothing else is pinned in its place. + self.assertIsNone(target["sdk"]) + self.assertIn("ai-openshell/sdk-go has not tagged v0.1.4 yet", target["sdk_waiting_for"]) + + def test_an_older_sdk_tag_is_not_taken_instead(self): + # sdk-go has v0.1.3 but not v0.1.4: the plan waits rather than pin v0.1.3. + made, _ = build(tags=["v0.1.4"], images=self.IMAGES, sdk_missing=("0.1.4", "0.1.4-pre.2")) + self.assertIsNone(made["target"]["sdk"]) + + def test_an_sdk_go_outage_is_an_error_not_a_missing_tag(self): + with self.assertRaises(upstream.UpstreamError): + build(tags=["v0.1.4"], images=self.IMAGES, outage=["sdk_tags"]) + + class WhatMustStopTheRun(unittest.TestCase): def test_a_registry_outage_is_an_error_not_a_missing_image(self): with self.assertRaises(upstream.UpstreamError): diff --git a/deploy/ci/upstream/tests/test_publish.py b/deploy/ci/upstream/tests/test_publish.py index 548171d..c1291d3 100644 --- a/deploy/ci/upstream/tests/test_publish.py +++ b/deploy/ci/upstream/tests/test_publish.py @@ -51,6 +51,10 @@ def test_images_not_out_yet(self): waiting = dict(STABLE, ready=False, waiting_for="upstream has tagged v0.1.4 but has not published its gateway image yet") self.assertIn("has not published", publish.blocked(plan_for(waiting))) + def test_sdk_not_tagged_yet(self): + waiting = dict(STABLE, sdk=None, sdk_waiting_for="ai-openshell/sdk-go has not tagged v0.1.4 yet") + self.assertIn("has not tagged v0.1.4", publish.blocked(plan_for(waiting))) + def test_any_pull_request_from_the_branch_in_this_repository(self): for state in ("OPEN", "CLOSED", "MERGED"): self.assertEqual( diff --git a/deploy/ci/upstream/tests/test_report.py b/deploy/ci/upstream/tests/test_report.py index 3469943..adf3a5a 100644 --- a/deploy/ci/upstream/tests/test_report.py +++ b/deploy/ci/upstream/tests/test_report.py @@ -50,7 +50,16 @@ def test_it_ends_with_the_trailer_the_commit_is_found_by(self): def test_it_says_where_from_where_to_and_which_commit(self): text = " ".join(self.message(TODAY).split()) self.assertIn("from 0.1.3 to 0.1.4-pre.2", text) - self.assertIn("upstream tagged v0.1.4-pre.2 (4c1b16a4a104)", text) + self.assertIn( + "the Go SDK at github.com/ai-openshell/sdk-go v0.1.4-pre.2 (%s)" % support.sdk_commit_of("0.1.4-pre.2")[:12], text + ) + self.assertNotIn("rewrites the BFF's imports", text) + + def test_the_move_off_the_old_path_says_so(self): + text = " ".join(report.commit_message(TODAY["target"], TODAY["pinned"], migrated=True).split()) + self.assertIn("rewrites the BFF's imports from github.com/NVIDIA/OpenShell/sdk/go to it", text) + self.assertIn("The SDK moved", report.pr_body(TODAY, REPO_URL, migrated=True)) + self.assertNotIn("The SDK moved", report.pr_body(TODAY, REPO_URL)) def test_a_pre_release_says_it_cannot_merge_and_a_stable_release_does_not(self): self.assertIn("pre-release", self.message(TODAY)) diff --git a/deploy/ci/upstream/tests/test_upstream.py b/deploy/ci/upstream/tests/test_upstream.py index 5d05dc4..7ae27ae 100644 --- a/deploy/ci/upstream/tests/test_upstream.py +++ b/deploy/ci/upstream/tests/test_upstream.py @@ -99,7 +99,14 @@ def test_a_failing_git_is_an_error_not_an_empty_tag_list(self): def test_the_repository_followed_is_openshell(self): self.assertEqual(upstream.UPSTREAM_REPO, "https://github.com/NVIDIA/OpenShell.git") - self.assertEqual(upstream.SDK_MODULE, "github.com/NVIDIA/OpenShell/sdk/go") + self.assertEqual(upstream.SDK_MODULE, "github.com/ai-openshell/sdk-go") + self.assertEqual(upstream.SDK_REPO, "https://github.com/ai-openshell/sdk-go.git") + self.assertEqual(upstream.LEGACY_SDK_MODULE, "github.com/NVIDIA/OpenShell/sdk/go") + + def test_sdk_tags_are_read_from_sdk_go(self): + with mock.patch("subprocess.check_output", return_value="") as check_output: + upstream.Upstream().sdk_tags() + self.assertEqual(check_output.call_args[0][0], ["git", "ls-remote", "--tags", upstream.SDK_REPO]) self.assertEqual(upstream.GATEWAY_REPOSITORY, "nvidia/openshell/gateway") self.assertEqual(upstream.SUPERVISOR_REPOSITORY, "nvidia/openshell/supervisor") diff --git a/deploy/ci/upstream/upstream.py b/deploy/ci/upstream/upstream.py index 5993600..200321e 100644 --- a/deploy/ci/upstream/upstream.py +++ b/deploy/ci/upstream/upstream.py @@ -14,7 +14,15 @@ UPSTREAM_REPO = "https://github.com/NVIDIA/OpenShell.git" UPSTREAM_URL = "https://github.com/NVIDIA/OpenShell" -SDK_MODULE = "github.com/NVIDIA/OpenShell/sdk/go" +# The Go SDK is its own module, in its own repository, since OpenShell 0.1.7 +# (ai-openshell/sdk-go). Its tag vX.Y.Z is the SDK for core vX.Y.Z, and it is +# cut after core releases, so for a while a core release has no SDK tag yet. +SDK_MODULE = "github.com/ai-openshell/sdk-go" +SDK_REPO = "https://github.com/ai-openshell/sdk-go.git" +SDK_URL = "https://github.com/ai-openshell/sdk-go" +# Where the SDK lived until then: a directory of the core repository. A +# go.mod that still requires it is moved off it by the next pin move. +LEGACY_SDK_MODULE = "github.com/NVIDIA/OpenShell/sdk/go" REGISTRY = "ghcr.io" GATEWAY_REPOSITORY = "nvidia/openshell/gateway" SUPERVISOR_REPOSITORY = "nvidia/openshell/supervisor" @@ -55,18 +63,25 @@ class Upstream(object): def __init__(self, timeout=120): self.timeout = timeout + def _ls_remote_tags(self, repo): + try: + return subprocess.check_output( + ["git", "ls-remote", "--tags", repo], universal_newlines=True, timeout=self.timeout + ) + except (subprocess.CalledProcessError, subprocess.TimeoutExpired, OSError) as err: + raise UpstreamError("git ls-remote %s failed: %s" % (repo, err)) + def release_tags(self): """Raw `git ls-remote --tags` output for the upstream repository. Deliberately without --refs: an annotated tag is listed twice, and only - the peeled `^{}` line names the commit. `go get` needs the commit. + the peeled `^{}` line names the commit. """ - try: - return subprocess.check_output( - ["git", "ls-remote", "--tags", UPSTREAM_REPO], universal_newlines=True, timeout=self.timeout - ) - except (subprocess.CalledProcessError, subprocess.TimeoutExpired, OSError) as err: - raise UpstreamError("git ls-remote %s failed: %s" % (UPSTREAM_REPO, err)) + return self._ls_remote_tags(UPSTREAM_REPO) + + def sdk_tags(self): + """Raw `git ls-remote --tags` output for the SDK's repository, ai-openshell/sdk-go.""" + return self._ls_remote_tags(SDK_REPO) def image_digest(self, repository, tag): """'sha256:...' for repository:tag on ghcr.io, or None when the tag is not published. diff --git a/docs/adrs/0003-gateway-api-openshell-sdk.md b/docs/adrs/0003-gateway-api-openshell-sdk.md index 4394040..17296a1 100644 --- a/docs/adrs/0003-gateway-api-openshell-sdk.md +++ b/docs/adrs/0003-gateway-api-openshell-sdk.md @@ -9,7 +9,7 @@ The dashboard is a client of the OpenShell gateway's gRPC API. Two things go wro ## Decisions -1. **The BFF reaches the gateway through the OpenShell Go SDK,** `github.com/NVIDIA/OpenShell/sdk/go`, required in `backend/go.mod` at the commit of the OpenShell release the branch is built on ([ADR 0005](0005-gateway-compatibility-and-releases.md)). The repository holds no copy of the gateway's proto files and no generated stubs. +1. **The BFF reaches the gateway through the OpenShell Go SDK,** `github.com/ai-openshell/sdk-go` (until OpenShell 0.1.8, `github.com/NVIDIA/OpenShell/sdk/go`), required in `backend/go.mod` at the tag of the OpenShell release the branch is built on ([ADR 0005](0005-gateway-compatibility-and-releases.md)). The repository holds no copy of the gateway's proto files and no generated stubs. 2. **The gateway's API defines what exists.** The dashboard invents no RPC, field, lifecycle state or object. Sandbox is the object the gateway has; labels categorize sandboxes, and there is no Agent object. An idea with no RPC behind it is raised with OpenShell, not simulated in the dashboard. 3. **The BFF translates nothing.** Request and response bodies mirror the gateway's messages. When the gateway expects something in a particular form, the UI sends it in that form. The OpenShell CLI and TUI are the reference for how a client calls an RPC. Beyond that the BFF adds little: its bootstrap and health routes, the compatibility verdict (ADR 0005, decision 9) and a summary of pending proposals. 4. **A direct gRPC call is the exception, for a gap in the SDK that can be named.** It lives in `backend/pkg/clients`, uses the proto client the SDK ships, states the gap at the top of its file, and is deleted when the SDK closes the gap. There are three today: running a command with stdin and no terminal (binary-safe upload), reading the names of the credentials a provider holds, and reading and writing a provider profile's endpoints in full. diff --git a/docs/adrs/0005-gateway-compatibility-and-releases.md b/docs/adrs/0005-gateway-compatibility-and-releases.md index 67e1463..eaadc9c 100644 --- a/docs/adrs/0005-gateway-compatibility-and-releases.md +++ b/docs/adrs/0005-gateway-compatibility-and-releases.md @@ -65,6 +65,8 @@ The repository moved from a personal account to the `ai-openshell` organization - **A6. A tagged SDK is pinned at its tag.** From OpenShell 0.1.7 upstream tags the SDK module itself (`sdk/go/vX.Y.Z`), and `go get` of the release commit resolves to that tag. The pin is then the tag (`v0.1.7`), and the bump checks with `go list -m -json` that the tag is the release commit; before 0.1.7 it is a pseudo-version of that commit. +- **A7. The SDK comes from ai-openshell/sdk-go, and the bump waits for its tag.** *(2026-11-17)* The Go SDK left the core repository: it is the module `github.com/ai-openshell/sdk-go`, whose tag `vX.Y.Z` is the SDK for core `vX.Y.Z`, cut after core releases by sdk-go's own bump. Follow upstream pins that tag, reading its commit from sdk-go and checking it with `go list -m -json`. While sdk-go has not tagged the target release, the run makes no bump and says so in its summary; it never pins another SDK version or a commit instead. Check A does not need the SDK and runs as before. A branch still on the old path, `github.com/NVIDIA/OpenShell/sdk/go`, stays valid, and the bump that moves it to sdk-go rewrites the BFF's imports in the same commit, the only Go files a pin move may change. The pin is moved by the bump, never by hand, so the import rewrite lands with it. + Decisions 1 to 5 and 9 to 12, and what decisions 6 to 8 say about the pins file, the pin move and check A, still hold. ### Consequences diff --git a/docs/architecture.md b/docs/architecture.md index 9a02b5b..be5128b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,7 +14,7 @@ The proxy and the BFF are two containers in one pod: the BFF listens on `127.0.0 **BFF** (`backend/`). A Go HTTP server that serves the REST API under `/api/v1` and the built UI. Routes are declared in `pkg/server/app.go`, with one handler struct per resource in `pkg/handlers/`. Handlers call the interfaces in `pkg/services/`, and `pkg/models/` converts between the SDK's types and the JSON the UI reads. The BFF keeps no state. -**SDK.** The BFF reaches the gateway through the OpenShell Go SDK, `github.com/NVIDIA/OpenShell/sdk/go`, which `backend/go.mod` requires at the commit of the OpenShell release the branch is built on. +**SDK.** The BFF reaches the gateway through the OpenShell Go SDK, `github.com/ai-openshell/sdk-go`, which `backend/go.mod` requires at the tag of the OpenShell release the branch is built on (sdk-go `vX.Y.Z` is the SDK for OpenShell `vX.Y.Z`). Until OpenShell 0.1.8 the SDK was the `sdk/go` directory of the core repository, `github.com/NVIDIA/OpenShell/sdk/go`; the Follow upstream bump moves a branch off that path. The directory layout is in [development.md](development.md#layout). diff --git a/scripts/gateway-range.mjs b/scripts/gateway-range.mjs index 3e01a7d..b0a84cb 100755 --- a/scripts/gateway-range.mjs +++ b/scripts/gateway-range.mjs @@ -37,7 +37,11 @@ const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..'); export const PINS_FILE = 'deploy/ci/gateway-pins.json'; export const PINS_PATH = join(repoRoot, PINS_FILE); export const GO_MOD_PATH = join(repoRoot, 'backend', 'go.mod'); -export const SDK_MODULE = 'github.com/NVIDIA/OpenShell/sdk/go'; +// The Go SDK's own module (ai-openshell/sdk-go), and the path it had inside the +// core repository until OpenShell 0.1.8. A go.mod still on the old path is +// moved off it by the next Follow upstream bump. +export const SDK_MODULE = 'github.com/ai-openshell/sdk-go'; +export const LEGACY_SDK_MODULE = 'github.com/NVIDIA/OpenShell/sdk/go'; // Where the BFF's built-in line is written, and the name it is written under. export const LINE_SOURCE_FILE = 'backend/pkg/models/gateway_release_line.go'; export const LINE_SOURCE_PATH = join(repoRoot, LINE_SOURCE_FILE); @@ -112,12 +116,14 @@ export function readGatewayLine(pinsPath = PINS_PATH) { /** The SDK version backend/go.mod actually builds against. */ export function sdkInGoMod(goModPath = GO_MOD_PATH) { const goMod = readFileSync(goModPath, 'utf8'); - const escaped = SDK_MODULE.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - const match = goMod.match(new RegExp(`^\\s*(?:require\\s+)?${escaped}\\s+(\\S+)`, 'm')); - if (!match) { - throw new Error(`${goModPath} does not require ${SDK_MODULE}`); + for (const module of [SDK_MODULE, LEGACY_SDK_MODULE]) { + const escaped = module.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + const match = goMod.match(new RegExp(`^\\s*(?:require\\s+)?${escaped}\\s+(\\S+)`, 'm')); + if (match) { + return match[1]; + } } - return match[1]; + throw new Error(`${goModPath} does not require ${SDK_MODULE} (nor ${LEGACY_SDK_MODULE}, its old path)`); } /** The release line compiled into the BFF, as its Go source states it. */ @@ -170,8 +176,9 @@ function check(declared, pinsPath) { if (sdk !== declared.sdk) { throw new Error( `${pinsPath} says "sdk": ${JSON.stringify(declared.sdk)}, but backend/go.mod builds against ${sdk}. ` + - 'The sdk field must equal the SDK version in backend/go.mod: the SDK at the commit upstream ' + - `tagged ${declared.release} with. Bring the stale one of the two in line.`, + 'The sdk field must equal the SDK version in backend/go.mod: the SDK for the pinned release ' + + `(${SDK_MODULE} v${declared.release}, or upstream tagged ${declared.release} with on the old path). ` + + 'Bring the stale one of the two in line.', ); } } diff --git a/scripts/gateway-range.test.mjs b/scripts/gateway-range.test.mjs index 42a1333..7be1c82 100644 --- a/scripts/gateway-range.test.mjs +++ b/scripts/gateway-range.test.mjs @@ -16,6 +16,8 @@ import { lineOf, lineOfLanes, readGatewayLine, + LEGACY_SDK_MODULE, + SDK_MODULE, sdkInGoMod, } from './gateway-range.mjs'; import { BEGIN, END, README_PATH, renderBlock, replaceBlock } from './readme-gateway-range.mjs'; @@ -120,6 +122,21 @@ test('the committed pins give a line, and name the SDK that backend/go.mod build assert.equal(declared.sdk, sdkInGoMod()); }); +// The SDK is its own module, ai-openshell/sdk-go, since OpenShell 0.1.7. A +// go.mod on the path it had inside the core repository is still read, until +// the bump that moves it off. +test('the SDK is read from go.mod on its own module and on its old path', (t) => { + const dir = tempDir(t); + const goMod = join(dir, 'go.mod'); + assert.equal(SDK_MODULE, 'github.com/ai-openshell/sdk-go'); + writeFileSync(goMod, `module x\n\nrequire (\n\t${SDK_MODULE} v0.1.9\n)\n`); + assert.equal(sdkInGoMod(goMod), 'v0.1.9'); + writeFileSync(goMod, `module x\n\nrequire ${LEGACY_SDK_MODULE} ${SDK}\n`); + assert.equal(sdkInGoMod(goMod), SDK); + writeFileSync(goMod, 'module x\n'); + assert.throws(() => sdkInGoMod(goMod), /does not require github\.com\/ai-openshell\/sdk-go/); +}); + // The BFF has the line compiled in, so that an image knows it whatever built // it. That constant restates the pins, and this is what holds it to them. test('the line compiled into the BFF is the line the committed pins give', () => {