Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/rfc/stovepipe/steps/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,10 +230,10 @@ Several shapes for sharing the `BuildRunner` contract across domains were raised
- **Shared `Status`/`Cancel` via a `platform/extension/buildrunner.StatusCanceller` sub-interface, with `BuildID`/`BuildStatus`/`BuildMetadata` promoted to `platform/base`.** An earlier draft of this doc adopted exactly this: since both domains poll and cancel by the same opaque, runner-minted id with the same async semantics, `Status`/`Cancel` moved to a shared interface embedded in each domain's `BuildRunner`, with the supporting types promoted to `platform/base` so both sides used the same Go types (a dual-implementing backend would then satisfy both interfaces through one embedded method set).

Trade-offs: set aside on review — splitting one conceptual contract (`Trigger` + `Status` + `Cancel`) across two packages (`platform/extension/buildrunner` for two of the three methods, `{domain}/extension/buildrunner` for the third) fragments a single interface across an ownership boundary for a resemblance that isn't yet load-bearing: SubmitQueue is the only existing consumer of the "shared" half today, and the promotion cost — migrating SubmitQueue's already-shipped controllers, storage, and protobuf mappings onto the shared type — bought less than keeping each domain's `BuildRunner` whole and pushing reuse down to the implementation layer instead, per the option below.
- **Shared backend under `platform`, thin per-domain contracts (adopted).** House the Buildkite / CI-gateway implementation once under `platform/extension` and let each domain define its own contract over it:
- **Shared backend under `platform`, thin per-domain contracts (adopted).** House the Buildkite / CI-gateway implementation once under `platform/` and let each domain define its own contract over it. It sits at `platform/{backend}`, beside `platform/http`, rather than under `platform/extension/` — precisely because this option declines to define a shared interface, the package is a vendor client with no interface, `Config`, or `Factory`, so it is platform plumbing rather than an extension. Only the rejected alternatives above would have earned a `platform/extension/buildrunner` package.

```go
// platform/extension/buildrunner/buildkite — shared HTTP client, auth, poll loop
// platform/buildkite — shared HTTP client, auth, poll loop
package buildkite

type Client struct{ /* ... */ }
Expand Down
3 changes: 2 additions & 1 deletion platform/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ Cross-domain packages shared by SubmitQueue, Stovepipe, and other services in th
- **metrics/** — Tally helpers with error-aware tagging via `platform/errs`.
- **consumer/** — Queue consumer framework (`consumer.Controller`, registry, DLQ wiring).
- **http/** — Small HTTP client helpers (e.g. base-URL `RoundTripper`). Go import path: `github.com/uber/submitqueue/platform/http`; package name is `http`. Callers that also import `net/http` should import this package with an alias (for example `phttp "github.com/uber/submitqueue/platform/http"`) and use `phttp.NewClient`.
- **buildkite/**, **githubactions/** — Vendor CI clients over `platform/http`: the REST calls and provider-specific vocabulary (state strings, id encoding) shared by every domain's `BuildRunner` backend. They deliberately define no interface, so they are plumbing rather than extensions — each domain keeps its own `BuildRunner` contract under `{domain}/extension/buildrunner` and adapts these clients to it.
- **base/** — Shared domain entities (`change`, `messagequeue`, and related subpackages). Root package `base` is documentation-only.
- **extension/** — Shared extension interfaces and implementations reused across domains (`counter`, `messagequeue`, and backends such as `mysql`).
- **extension/** — Shared extension interfaces and implementations reused across domains (`counter`, `messagequeue`, and backends such as `mysql`). A package belongs here only if it defines a behavioral interface with its `Config` and `Factory` interface; a vendor client with no interface belongs directly under `platform/`.

Domain-scoped infrastructure and extensions stay under each domain (for example `submitqueue/core/`, `submitqueue/extension/`).
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ load("@rules_go//go:def.bzl", "go_library", "go_test")
go_library(
name = "go_default_library",
srcs = ["client.go"],
importpath = "github.com/uber/submitqueue/platform/extension/buildrunner/buildkite",
importpath = "github.com/uber/submitqueue/platform/buildkite",
visibility = ["//visibility:public"],
deps = ["//platform/http:go_default_library"],
)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Buildkite client

Shared HTTP client and Buildkite-specific facts for every domain's Buildkite-backed `BuildRunner`. There is no `BuildRunner` interface here by design — each domain (`submitqueue`, `stovepipe`, ...) defines its own `BuildRunner` and its own `BuildStatus`, and adapts this package's `State` to it. See [`doc/rfc/stovepipe/steps/build.md`](../../../../doc/rfc/stovepipe/steps/build.md#alternatives-considered-for-sharing-the-contract) for why the contract stays per-domain while the backend is shared.
Shared HTTP client and Buildkite-specific facts for every domain's Buildkite-backed `BuildRunner`. There is no `BuildRunner` interface here by design — each domain (`submitqueue`, `stovepipe`, ...) defines its own `BuildRunner` and its own `BuildStatus`, and adapts this package's `State` to it. See [`doc/rfc/stovepipe/steps/build.md`](../../doc/rfc/stovepipe/steps/build.md#alternatives-considered-for-sharing-the-contract) for why the contract stays per-domain while the backend is shared.

## What lives here

Expand All @@ -17,7 +17,7 @@ Both wrap a `*Client` built at the wiring layer; this package never constructs o

## How the same `Client` stays safe to share across two different checkout strategies

SubmitQueue's build materializes state that doesn't exist yet — the runner resolves a batch DAG into composite base/head commits by applying patches. Stovepipe's build checks out a commit that already exists on trunk and, for an incremental build, diffs it against a baseline. These are different problems at the CI-pipeline level (see [build.md's "Why separate contracts"](../../../../doc/rfc/stovepipe/steps/build.md#why-separate-contracts)), yet both go through the same `Client.CreateBuild` call — the `Client` never inspects `CreateBuildRequest.Env` or picks a strategy, so there is nothing here that needs to "know" which pattern applies.
SubmitQueue's build materializes state that doesn't exist yet — the runner resolves a batch DAG into composite base/head commits by applying patches. Stovepipe's build checks out a commit that already exists on trunk and, for an incremental build, diffs it against a baseline. These are different problems at the CI-pipeline level (see [build.md's "Why separate contracts"](../../doc/rfc/stovepipe/steps/build.md#why-separate-contracts)), yet both go through the same `Client.CreateBuild` call — the `Client` never inspects `CreateBuildRequest.Env` or picks a strategy, so there is nothing here that needs to "know" which pattern applies.

The split happens entirely outside this package, at two layers below it:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ load("@rules_go//go:def.bzl", "go_library", "go_test")
go_library(
name = "go_default_library",
srcs = ["client.go"],
importpath = "github.com/uber/submitqueue/platform/extension/buildrunner/githubactions",
importpath = "github.com/uber/submitqueue/platform/githubactions",
visibility = ["//visibility:public"],
deps = ["//platform/http:go_default_library"],
)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# GitHub Actions client

Shared HTTP client and GitHub Actions-specific facts for every domain's GitHub Actions-backed `BuildRunner`. There is no `BuildRunner` interface here by design — each domain (`submitqueue`, `stovepipe`, ...) defines its own `BuildRunner` and its own `BuildStatus`, and adapts this package's `RunStatus` to it. Mirrors [`platform/extension/buildrunner/buildkite`](../buildkite/README.md)'s split; see that package's README and [`doc/rfc/stovepipe/steps/build.md`](../../../../doc/rfc/stovepipe/steps/build.md#alternatives-considered-for-sharing-the-contract) for the shared rationale.
Shared HTTP client and GitHub Actions-specific facts for every domain's GitHub Actions-backed `BuildRunner`. There is no `BuildRunner` interface here by design — each domain (`submitqueue`, `stovepipe`, ...) defines its own `BuildRunner` and its own `BuildStatus`, and adapts this package's `RunStatus` to it. Mirrors [`platform/buildkite`](../buildkite/README.md)'s split; see that package's README and [`doc/rfc/stovepipe/steps/build.md`](../../doc/rfc/stovepipe/steps/build.md#alternatives-considered-for-sharing-the-contract) for the shared rationale.

## What lives here

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
// BuildRunner interface or domain entity types — each domain (submitqueue,
// stovepipe, ...) defines its own BuildRunner and its own BuildStatus, and
// adapts this package's RunStatus to it. See
// platform/extension/buildrunner/buildkite's README for the analogous
// platform/buildkite's README for the analogous
// rationale applied to the Buildkite backend.
package githubactions

Expand Down
2 changes: 1 addition & 1 deletion stovepipe/extension/buildrunner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ Vendor-agnostic interface through which Stovepipe triggers and polls builds agai

Implementations return plain, unclassified errors — the calling controller decides retryable-vs-not and user-vs-infra, per `platform/errs`.

Real backends (`buildkite`, `githubactions`) are thin adapters over a shared platform client (`platform/extension/buildrunner/{backend}`) — see that package's README for the HTTP client and vendor-specific details.
Real backends (`buildkite`, `githubactions`) are thin adapters over a shared platform client (`platform/{backend}`) — see that package's README for the HTTP client and vendor-specific details.

See [doc/rfc/stovepipe/steps/build.md](../../../doc/rfc/stovepipe/steps/build.md#why-separate-contracts) for why this is a separate contract from SubmitQueue's own `buildrunner` rather than a shared one. To add a backend, create `buildrunner/{backend}/`, implement `BuildRunner`, and return it from a `New(...)` constructor.
4 changes: 2 additions & 2 deletions stovepipe/extension/buildrunner/buildkite/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ go_library(
importpath = "github.com/uber/submitqueue/stovepipe/extension/buildrunner/buildkite",
visibility = ["//visibility:public"],
deps = [
"//platform/buildkite:go_default_library",
"//platform/errs:go_default_library",
"//platform/extension/buildrunner/buildkite:go_default_library",
"//platform/http:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildrunner:go_default_library",
Expand All @@ -20,9 +20,9 @@ go_test(
srcs = ["buildkite_test.go"],
embed = [":go_default_library"],
deps = [
"//platform/buildkite:go_default_library",
"//platform/errs:go_default_library",
"//platform/errs/http:go_default_library",
"//platform/extension/buildrunner/buildkite:go_default_library",
"//platform/http:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildrunner:go_default_library",
Expand Down
2 changes: 1 addition & 1 deletion stovepipe/extension/buildrunner/buildkite/buildkite.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ import (

"go.uber.org/zap"

platformbuildkite "github.com/uber/submitqueue/platform/buildkite"
"github.com/uber/submitqueue/platform/errs"
platformbuildkite "github.com/uber/submitqueue/platform/extension/buildrunner/buildkite"
phttp "github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

platformbuildkite "github.com/uber/submitqueue/platform/buildkite"
"github.com/uber/submitqueue/platform/errs"
httperrs "github.com/uber/submitqueue/platform/errs/http"
platformbuildkite "github.com/uber/submitqueue/platform/extension/buildrunner/buildkite"
phttp "github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
Expand Down
4 changes: 2 additions & 2 deletions stovepipe/extension/buildrunner/githubactions/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ go_library(
visibility = ["//visibility:public"],
deps = [
"//platform/errs:go_default_library",
"//platform/extension/buildrunner/githubactions:go_default_library",
"//platform/githubactions:go_default_library",
"//platform/http:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildrunner:go_default_library",
Expand All @@ -22,7 +22,7 @@ go_test(
deps = [
"//platform/errs:go_default_library",
"//platform/errs/http:go_default_library",
"//platform/extension/buildrunner/githubactions:go_default_library",
"//platform/githubactions:go_default_library",
"//platform/http:go_default_library",
"//stovepipe/entity:go_default_library",
"//stovepipe/extension/buildrunner:go_default_library",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ import (
"go.uber.org/zap"

"github.com/uber/submitqueue/platform/errs"
platformgithubactions "github.com/uber/submitqueue/platform/extension/buildrunner/githubactions"
platformgithubactions "github.com/uber/submitqueue/platform/githubactions"
phttp "github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ import (

"github.com/uber/submitqueue/platform/errs"
httperrs "github.com/uber/submitqueue/platform/errs/http"
platformgithubactions "github.com/uber/submitqueue/platform/extension/buildrunner/githubactions"
platformgithubactions "github.com/uber/submitqueue/platform/githubactions"
phttp "github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/stovepipe/entity"
"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
Expand Down
2 changes: 1 addition & 1 deletion submitqueue/extension/buildrunner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ See [`doc/rfc/submitqueue/build-runner.md`](../../../doc/rfc/submitqueue/build-r
- `githubactions`: proof-of-architecture backend that dispatches a GitHub
Actions workflow. See [`githubactions/README.md`](githubactions/README.md)
for the workflow inputs and example orchestrator environment variables.
- `buildkite`: Buildkite-backed backend. Its HTTP client and Buildkite-specific facts (state vocabulary, metadata env-var round-trip) live in [`platform/extension/buildrunner/buildkite`](../../../platform/extension/buildrunner/buildkite/README.md), shared with `stovepipe`'s own Buildkite backend.
- `buildkite`: Buildkite-backed backend. Its HTTP client and Buildkite-specific facts (state vocabulary, metadata env-var round-trip) live in [`platform/buildkite`](../../../platform/buildkite/README.md), shared with `stovepipe`'s own Buildkite backend.
4 changes: 2 additions & 2 deletions submitqueue/extension/buildrunner/buildkite/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ go_library(
visibility = ["//visibility:public"],
deps = [
"//platform/base/change:go_default_library",
"//platform/extension/buildrunner/buildkite:go_default_library",
"//platform/buildkite:go_default_library",
"//submitqueue/core/changeset:go_default_library",
"//submitqueue/entity:go_default_library",
"//submitqueue/extension/buildrunner:go_default_library",
Expand All @@ -21,7 +21,7 @@ go_test(
embed = [":go_default_library"],
deps = [
"//platform/base/change:go_default_library",
"//platform/extension/buildrunner/buildkite:go_default_library",
"//platform/buildkite:go_default_library",
"//platform/http:go_default_library",
"//submitqueue/core/changeset:go_default_library",
"//submitqueue/core/changeset/fake:go_default_library",
Expand Down
2 changes: 1 addition & 1 deletion submitqueue/extension/buildrunner/buildkite/buildkite.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import (
"go.uber.org/zap"

"github.com/uber/submitqueue/platform/base/change"
platformbuildkite "github.com/uber/submitqueue/platform/extension/buildrunner/buildkite"
platformbuildkite "github.com/uber/submitqueue/platform/buildkite"
"github.com/uber/submitqueue/submitqueue/core/changeset"
"github.com/uber/submitqueue/submitqueue/entity"
"github.com/uber/submitqueue/submitqueue/extension/buildrunner"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/uber/submitqueue/platform/base/change"
platformbuildkite "github.com/uber/submitqueue/platform/extension/buildrunner/buildkite"
platformbuildkite "github.com/uber/submitqueue/platform/buildkite"
phttp "github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/submitqueue/core/changeset"
changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
Expand Down
4 changes: 2 additions & 2 deletions submitqueue/extension/buildrunner/githubactions/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ go_library(
visibility = ["//visibility:public"],
deps = [
"//platform/base/change:go_default_library",
"//platform/extension/buildrunner/githubactions:go_default_library",
"//platform/githubactions:go_default_library",
"//submitqueue/core/changeset:go_default_library",
"//submitqueue/entity:go_default_library",
"//submitqueue/extension/buildrunner:go_default_library",
Expand All @@ -21,7 +21,7 @@ go_test(
embed = [":go_default_library"],
deps = [
"//platform/base/change:go_default_library",
"//platform/extension/buildrunner/githubactions:go_default_library",
"//platform/githubactions:go_default_library",
"//platform/http:go_default_library",
"//submitqueue/core/changeset:go_default_library",
"//submitqueue/core/changeset/fake:go_default_library",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ architecture against a common CI system without adding local state.

Its HTTP client and GitHub Actions-specific facts (run status/conclusion
vocabulary, run id encoding) live in
[`platform/extension/buildrunner/githubactions`](../../../../platform/extension/buildrunner/githubactions/README.md),
[`platform/githubactions`](../../../../platform/githubactions/README.md),
shared with `stovepipe`'s own GitHub Actions backend.

## How it works
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import (
"go.uber.org/zap"

"github.com/uber/submitqueue/platform/base/change"
platformgithubactions "github.com/uber/submitqueue/platform/extension/buildrunner/githubactions"
platformgithubactions "github.com/uber/submitqueue/platform/githubactions"
"github.com/uber/submitqueue/submitqueue/core/changeset"
"github.com/uber/submitqueue/submitqueue/entity"
"github.com/uber/submitqueue/submitqueue/extension/buildrunner"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import (
"go.uber.org/zap"

"github.com/uber/submitqueue/platform/base/change"
platformgithubactions "github.com/uber/submitqueue/platform/extension/buildrunner/githubactions"
platformgithubactions "github.com/uber/submitqueue/platform/githubactions"
phttp "github.com/uber/submitqueue/platform/http"
"github.com/uber/submitqueue/submitqueue/core/changeset"
changesetfake "github.com/uber/submitqueue/submitqueue/core/changeset/fake"
Expand Down
Loading