Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .claude/rules/bff-go.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ route, so handlers do not import chi.
The OpenShell Go SDK is the source of truth:

```go
import openshell "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1"
import openshell "github.com/ai-openshell/sdk-go/openshell/v1"
```

Handlers never hold `openshell.ClientInterface` directly. `NewApp` resolves the
Expand Down Expand Up @@ -188,7 +188,12 @@ answers a request without a bearer with `unauthorized` (`pkg/auth/proxy.go`).

## SDK updates

The SDK is pinned to the commit of an OpenShell **release tag** and moves
The SDK is its own module, `github.com/ai-openshell/sdk-go`
([ai-openshell/sdk-go](https://github.com/ai-openshell/sdk-go)); until
OpenShell 0.1.8 it was `github.com/NVIDIA/OpenShell/sdk/go`, with the same
packages and APIs, and a go.mod still on that path is moved off it, imports
included, by the next bump. It is pinned to sdk-go's tag of an OpenShell
**release** (`vX.Y.Z` for core `vX.Y.Z`) and moves
together with the gateway the branch pins, in one commit (ADR 0005, decision 7;
hard facts 17 and 20 in `openshell-api.md`). The Follow upstream workflow makes
that commit, on a `bump/openshell-X.Y.Z` branch with a pull request into
Expand Down
6 changes: 3 additions & 3 deletions .claude/rules/openshell-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ alwaysApply: false

# OpenShell API Reference

**THE OPENSHELL GO SDK IS THE SOURCE OF TRUTH.** The BFF calls `github.com/NVIDIA/OpenShell/sdk/go` directly, and this repository holds no copy of the gateway's proto files and no generated stubs. Before writing any handler or TypeScript type, read the actual RPC/message definitions in `$(go env GOMODCACHE)/github.com/!n!v!i!d!i!a/!open!shell/sdk/go@<version>/openshell/v1/` (hand-written client wrappers with doc comments; `types/*.go` for message shapes) or via `go doc`. Never invent RPCs, fields, or lifecycle states. If a UI idea has no backing RPC, flag it — do not fabricate an endpoint. The SDK pin moves only together with the gateway the branch pins, to the commit of an upstream *release tag*, and the Follow upstream workflow makes that move on a `bump/openshell-X.Y.Z` branch (hard facts 17 and 20). Never run `go get github.com/NVIDIA/OpenShell/sdk/go@latest`.
**THE OPENSHELL GO SDK IS THE SOURCE OF TRUTH.** The BFF calls the SDK directly. It is its own module, `github.com/ai-openshell/sdk-go` (repository [ai-openshell/sdk-go](https://github.com/ai-openshell/sdk-go)), whose tag `vX.Y.Z` is the SDK for OpenShell `vX.Y.Z`; until OpenShell 0.1.8 it was the `sdk/go` directory of NVIDIA/OpenShell, `github.com/NVIDIA/OpenShell/sdk/go`, and a `backend/go.mod` still on that path moves off it in the next bump (the bump rewrites the imports). Packages and APIs are the same on both paths, and this repository holds no copy of the gateway's proto files and no generated stubs. Before writing any handler or TypeScript type, read the actual RPC/message definitions in `$(go env GOMODCACHE)/github.com/ai-openshell/sdk-go@<version>/openshell/v1/` (or `github.com/!n!v!i!d!i!a/!open!shell/sdk/go@<version>/` while go.mod is still on the old path) (hand-written client wrappers with doc comments; `types/*.go` for message shapes) or via `go doc`. Never invent RPCs, fields, or lifecycle states. If a UI idea has no backing RPC, flag it — do not fabricate an endpoint. The SDK pin moves only together with the gateway the branch pins, to sdk-go's tag of an upstream *release*, and the Follow upstream workflow makes that move on a `bump/openshell-X.Y.Z` branch (hard facts 17 and 20). Never run `go get github.com/ai-openshell/sdk-go@latest`.

**There are three direct gRPC calls**, all in `backend/pkg/clients`, each for a named gap in the SDK and all on the proto client the SDK ships. `rawexec.go` does binary-safe file uploads, because the public `Exec().Run(...)` API takes no stdin and `Exec().Interactive(...)` always asks for a PTY (hard fact 22). `rawprovider.go` reads the keys of the credentials a provider holds, because the SDK's `ProviderFromProto` drops the map the gateway returns them in (hard fact 8); `rawprovider_allworkspaces.go` is the same read for the all-workspaces list. `rawprofile.go` reads and writes provider profiles, because the SDK's profile endpoint type keeps a host, a port and a protocol and drops everything else the gateway stores (hard fact 24). Do not add another unless the SDK has a gap you can name, and delete each one when its gap closes.

Expand Down Expand Up @@ -47,7 +47,7 @@ Skip `GatewayInterceptor`, `SupervisorMiddleware`, `ComputeDriver` — internal/
17. **Three links, each proven separately — and the gateway and the SDK move together, on a bump branch (ADR 0005, decisions 6 to 8 and the amendment of 2026-11-06).** The chain is gateway → SDK → BFF → UI.
- **Wire** (gateway ↔ SDK) is proven only by running `backend/test/compat` against a real gateway; it is never inferred from version numbers. **Source** (SDK ↔ BFF) is proven by the compiler, `go vet` and the unit tests. BFF and UI ship together.
- **Say which link failed.** A compile error after an SDK move is a source migration, never "the gateway is incompatible".
- **A branch pins ONE upstream release** in `deploy/ci/gateway-pins.json` — the gateway and supervisor images and the SDK at that release's tag — and every change is tested against that gateway.
- **A branch pins ONE upstream release** in `deploy/ci/gateway-pins.json` — the gateway and supervisor images and the SDK at sdk-go's tag of that release — and every change is tested against that gateway.
- **Moving to another release is one commit that moves all of it:** the pins file, `backend/go.mod`, `backend/go.sum`, the built-in release line when the minor changes, and the README's generated block. Nobody makes that commit by hand. `follow-upstream.yml` reads upstream's tags once a day and, for a new stable release, makes it on `bump/openshell-X.Y.Z`: `main` plus that one commit, titled `fix: move to OpenShell X.Y.Z`, with a pull request into `main`. A person can run it with a `version` to bump to a given release, a pre-release included. The bot creates a bump branch once and never rebases, updates or force-pushes it; after that it is an ordinary branch. Merging the pull request moves `main` and releases nothing: a person cuts the release (`publish.yml`), the next patch, or `X.Y.0` when the move starts a new gateway minor.
- **Upstream pre-releases are tested ahead of the stable release in two ways:** *check A*, the BFF as `main` builds it against the target gateway (a failure is the one issue labelled `compat-migrate`), and CI on the bump pull request, the dashboard on the target's SDK against the target's gateway.
- **A change that needs the upcoming release** — an SDK method or an RPC the released gateway does not have — goes on that release's bump branch; before upstream ships it, on a prep branch made by running Follow upstream with the pre-release as `version`. Everything else goes to `main`.
Expand All @@ -67,7 +67,7 @@ Skip `GatewayInterceptor`, `SupervisorMiddleware`, `ComputeDriver` — internal/
- **The pins file.** `deploy/ci/gateway-pins.json` holds `release` (without the `v`), that release's gateway and supervisor images pinned by digest (`tag@sha256:…`, the tag being the release), `sdk`, `sandbox_image` and `config_schema`.
- **Which branch pins what.** `main` and `release/**` pin a stable release. A bump branch pins the release in its name; only a prep branch a person asked for pins a pre-release, and `ci.yml`'s `pins a stable release` check refuses a pre-release in every pull request, so a prep branch cannot merge until upstream ships. That check is deliberately not `compat`: it means "not released upstream yet", not "does not work".
- **Never pin a `dev` build, with or without a digest.** A dev gateway pulls `ghcr.io/nvidia/openshell/sandbox:dev` at runtime, so it drifts with upstream HEAD anyway.
- **The SDK pin.** The pin is the SDK at the commit the release tag points at, written identically in `backend/go.mod` and in the `sdk` field. Up to 0.1.6 the SDK module had no tags of its own, so the pin is a pseudo-version of that commit (`v0.1.3` → `sdk/go@v0.0.0-20261009050449-e1f3c82caa3e`). From 0.1.7 upstream tags the module itself (`sdk/go/v0.1.7`), `go get` of the release commit resolves to that tag, and the pin is `v0.1.7`; the bump checks with `go list -m -json` that the tag is the release commit. CI fails when they differ, when the pins file is malformed, or when it is not in canonical form (`python3 deploy/ci/upstream/follow.py format-pins` rewrites it). CI cannot tell a release tag's commit from any other commit; the Follow upstream workflow can, and its run summary says so when `main`'s SDK is not at the tag of the release it pins.
- **The SDK pin.** The pin is the SDK for the pinned release, written identically in `backend/go.mod` and in the `sdk` field. The SDK is `github.com/ai-openshell/sdk-go` and its tag `vX.Y.Z` is the SDK for core `vX.Y.Z`, so the pin is that tag (`v0.1.9`); the bump reads the tag's commit from sdk-go with `git ls-remote` and checks with `go list -m -json` that `go get` landed on it. sdk-go cuts the tag after core releases (its `core-watch.yml` opens a bump per core tag, `release-tag.yml` tags it once that merges), so for a while a core release has no SDK: the bump then waits, says so in the run summary, and pins nothing else in its place, while check A, which needs no SDK, still runs. Before that, the SDK was the `sdk/go` directory of NVIDIA/OpenShell, `github.com/NVIDIA/OpenShell/sdk/go`: up to 0.1.6 pinned as a pseudo-version of the release commit (`v0.1.3` → `v0.0.0-20261009050449-e1f3c82caa3e`), at 0.1.7 and 0.1.8 also tagged there (`sdk/go/v0.1.8` is the last). A branch still on that path is valid; the bump that moves it to sdk-go rewrites the BFF's imports to the new path in the same commit. CI fails when they differ, when the pins file is malformed, or when it is not in canonical form (`python3 deploy/ci/upstream/follow.py format-pins` rewrites it). CI cannot tell a release tag's commit from any other commit; the Follow upstream workflow can, and its run summary says so when `main`'s SDK is not at the tag of the release it pins.
- **The line in the BFF.** One more thing restates the pins: the gateway release line compiled into the BFF (`BuiltInGatewayReleaseLine` in `backend/pkg/models/gateway_release_line.go`), so that an image knows its line with no build args and no environment. It must be the major.minor of the pinned release, and `node scripts/gateway-range.mjs --check` fails in CI when it is not; the pin move changes the constant when the minor changes.

21. **Gateway settings are typed and type-checked.** `SettingValue` is a oneof of string, bool, int and bytes. Each registered key takes exactly one kind and the gateway refuses any other (`setting 'ocsf_json_enabled' expects bool value`). Gateways 0.1.0 to 0.1.2 register the same four keys: `ocsf_json_enabled` and `agent_policy_proposals_enabled` are bools, `ocsf_schema_version` and `proposal_approval_mode` are strings with a fixed set of values. `GetGatewayConfig` lists every registered key, and one that was never set comes back with an *empty* value: no kind. The API does not expose the registry (the CLI and TUI link it in), so the type of an unset setting cannot be read. The BFF mirrors all of this: a setting is a JSON string, boolean or integer in both directions, absent when unset, and nothing is coerced — the Settings page edits a value in the type it has and asks for the type when it has none.
Expand Down
12 changes: 8 additions & 4 deletions .claude/skills/proto-sync/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ moves to the new release, and the handlers and models it affects are audited.
## Before you start

You do not move the SDK. It moves together with the gateway the branch pins,
to the commit of an upstream release tag, and a workflow makes that move
to ai-openshell/sdk-go's tag of an upstream release, and a workflow makes that move
(ADR 0005, decision 7; hard facts 17 and 20 in
`.claude/rules/openshell-api.md`):

Expand Down Expand Up @@ -49,9 +49,13 @@ git diff origin/main origin/bump/openshell-<release> -- deploy/ci/gateway-pins.j
```

Then compare the two SDK versions:
- `sdk/go/openshell/v1/` for public client methods and resource shapes
- `sdk/go/openshell/v1/types/` for request/response field details
- `sdk/go/proto/sandboxv1/` only when the frontend policy protojson contract is involved
- `openshell/v1/` for public client methods and resource shapes
- `openshell/v1/types/` for request/response field details
- `proto/sandboxv1/` only when the frontend policy protojson contract is involved

These are paths in the SDK module, `github.com/ai-openshell/sdk-go` (in
`sdk/go/` of NVIDIA/OpenShell while go.mod is still on the old path,
`github.com/NVIDIA/OpenShell/sdk/go`).

### 2. Check out the bump branch

Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/follow-upstream.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@
#
# bump Opens a bump: the branch bump/openshell-<version>, which is main
# plus ONE commit that moves the pin (gateway and supervisor
# images, and the Go SDK at the tag's commit, together), and a
# pull request into main. A scheduled run bumps only to a stable
# images, and the Go SDK at ai-openshell/sdk-go's tag of the same
# version, together), and a pull request into main. sdk-go tags
# the SDK after core releases; until it has, there is no bump and
# the run summary says so. A scheduled run bumps only to a stable
# release; a version asked for by hand is bumped to as it is, so
# a pre-release gets a prep branch. The bot makes a bump branch
# ONCE: when the branch exists, or any pull request has come from
Expand Down
25 changes: 21 additions & 4 deletions deploy/ci/upstream/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,30 @@ python3 deploy/ci/upstream/follow.py require-stable # fail if it names a p
python3 deploy/ci/upstream/follow.py plan --out /tmp/plan.json # what is upstream ahead of this branch?
```

`plan` asks the real upstream (`git ls-remote`, and `docker buildx imagetools
inspect` for the target's image digests). It only reads; it starts nothing.
`plan` asks the real upstream (`git ls-remote` of NVIDIA/OpenShell and of
ai-openshell/sdk-go, and `docker buildx imagetools inspect` for the target's
image digests). It only reads; it starts nothing.

## The SDK

The Go SDK is its own module, `github.com/ai-openshell/sdk-go`. Its tag
`vX.Y.Z` is the SDK for core `vX.Y.Z`, and sdk-go cuts it after core releases.
So a target can have its images and no SDK yet: `plan` then sets the target's
`sdk` to null and `sdk_waiting_for` to why, the `bump` output is `false`, and
the run summary says the bump waits. Check A still runs. Nothing else is
pinned in the SDK's place; the next run looks again.

Until OpenShell 0.1.8 the SDK was the `sdk/go` directory of the core
repository, `github.com/NVIDIA/OpenShell/sdk/go`. The pins check accepts a
go.mod on either path, and the pin move takes a tree off the old one: it
rewrites `backend/**/*.go` to the new import path before `go mod tidy`, and
commits those files with the pin.

## One release per branch

`deploy/ci/gateway-pins.json` names the one OpenShell release a branch is
built on. The gateway and supervisor images and the Go SDK all come from that
release's tag and move together, in one commit.
release (the SDK from sdk-go's tag of it) and move together, in one commit.

| Branch | Pins | Who moves it |
|---|---|---|
Expand Down Expand Up @@ -151,7 +167,8 @@ gets a branch of its own.
| File | What moves |
|---|---|
| `deploy/ci/gateway-pins.json` | `release`, `gateway_image`, `supervisor_image`, `sdk` |
| `backend/go.mod`, `backend/go.sum` | the SDK, with `go get <module>@<tag's commit>` and `go mod tidy`. Never `@latest` |
| `backend/go.mod`, `backend/go.sum` | the SDK, with `go get github.com/ai-openshell/sdk-go@vX.Y.Z` and `go mod tidy`, then `go list -m -json` to check the tag is the commit `plan` read. Never `@latest` |
| `backend/**/*.go` | only in the move off `github.com/NVIDIA/OpenShell/sdk/go`: the import path |
| `backend/pkg/models/gateway_release_line.go` | the built-in line, only when the minor changes |
| `README.md` | the generated block, with `scripts/readme-gateway-range.mjs --write` |

Expand Down
12 changes: 7 additions & 5 deletions deploy/ci/upstream/bump.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,16 @@ def maybe(self, *args):
return done.stdout.strip() if done.returncode == 0 else None


def _stage_pin_move(git):
def _stage_pin_move(git, rewritten=()):
"""Stage the pin move's files. True when there is something to commit.

The move may change PIN_PATHS and nothing else. A change anywhere else
means a tool did something it was not asked to, and it is not committed.
The move may change PIN_PATHS, and the Go files it says it rewrote off the
SDK's old path, and nothing else. A change anywhere else means a tool did
something it was not asked to, and it is not committed.
"""
allowed = set(pinmove.PIN_PATHS) | set(rewritten or ())
changed = [text[3:] for text in git.run("status", "--porcelain").stdout.splitlines() if text.strip()]
foreign = [path for path in changed if path not in pinmove.PIN_PATHS]
foreign = [path for path in changed if path not in allowed]
if foreign:
raise GitError(
"the pin move changed files it must not: %s. Nothing was committed." % ", ".join(sorted(foreign))
Expand Down Expand Up @@ -132,7 +134,7 @@ def prepare(repo, target, mover, message, main_ref="HEAD"):
try:
moved = mover(tree, target)
outcome["moved"] = moved
if not _stage_pin_move(work):
if not _stage_pin_move(work, (moved or {}).get("rewritten", ())):
outcome["reason"] = "main already pins OpenShell %s" % target["version"]
return outcome
_commit(work, message(target, moved))
Expand Down
10 changes: 6 additions & 4 deletions deploy/ci/upstream/follow.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,9 @@ def _load_valid_pins(path, go_mod=None):
if go_mod:
go_mod_version = pins.go_mod_sdk(_read(go_mod))
if go_mod_version is None:
raise pins.PinsError(["%s does not require %s" % (go_mod, pins.SDK_MODULE)])
raise pins.PinsError(
["%s does not require %s (nor %s, its old path)" % (go_mod, pins.SDK_MODULE, pins.LEGACY_SDK_MODULE)]
)
problems = pins.validate(doc, schemas=pins.known_schemas(path), go_mod_version=go_mod_version)
if problems:
raise pins.PinsError(problems)
Expand Down Expand Up @@ -154,8 +156,8 @@ def cmd_plan(args, upstream=None):
"ready": "true" if target.get("ready") else "false",
"target": target.get("version", ""),
"same_line": "true" if target.get("same_line") else "false",
# Something to open a bump branch for, with images to pin.
"bump": "true" if target.get("bump") and target.get("ready") else "false",
# Something to open a bump branch for, with images and an SDK tag to pin.
"bump": "true" if target.get("bump") and target.get("ready") and target.get("sdk") else "false",
"branch": target.get("branch", ""),
"gateway_image": target.get("gateway_image") or "",
"supervisor_image": target.get("supervisor_image") or "",
Expand Down Expand Up @@ -193,7 +195,7 @@ def cmd_bump(args, run=None):
repo,
target,
mover=lambda tree, to: pinmove.apply(tree, to, tools=tools, run=run, echo=_echo),
message=lambda to, moved: report.commit_message(to, made["pinned"]),
message=lambda to, moved: report.commit_message(to, made["pinned"], migrated=moved.get("migrated")),
main_ref=args.main_ref,
)
decided = publish_module.build(made, outcome, pulls=pulls, branch_sha=branch_sha, repo_url=args.repo_url)
Expand Down
Loading
Loading