Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ Start at the [docs index](docs/) for the full map. Quick links:
- [Configuration](docs/configuration.md) — Config options, env-var overrides, tier shortcuts
- [Architecture](docs/architecture.md) — System design, codegen pipeline, plugin model, multi-CSP vision
- [Services Matrix](docs/services-matrix.md) — services, coverage status, boto3 compatibility
- [Plugin API](docs/plugin-api.md) — the stable `ServicePlugin` contract and v1.x stability policy
- [Compatibility Policy](docs/compatibility-policy.md) — what v1.0 guarantees across 1.x, and what it explicitly does not
- [Fidelity Manifest](docs/fidelity-manifest.md) — per-operation tiers: how much to trust any given call
- [Plugin API](docs/plugin-api.md) — the in-tree `ServicePlugin` contract for contributors
- [Roadmap](docs/roadmap.md) — Phased plan toward multi-CSP support
- [FAQ](docs/faq.md) / [Troubleshooting](docs/troubleshooting.md) — Common questions and errors
- [Contributing](docs/contributing.md) — Development setup, adding new services
Expand Down
5 changes: 5 additions & 0 deletions changes/unreleased/Added-20260809-230000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Added
body: A published compatibility policy (docs/compatibility-policy.md) stating what v1.0 guarantees across the 1.x line — config keys, environment variables, the CLI, admin API response keys, fidelity tier names, and the response shape of hand-verified operations covered by the boto3 compatibility suite — what it explicitly does not guarantee, and the deprecation procedure that precedes any removal
time: 2026-08-09T23:00:00.000000+09:00
custom:
Issue: "129"
5 changes: 5 additions & 0 deletions changes/unreleased/Documentation-20260810-101500.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Documentation
body: Corrected what the fidelity manifest says an `unimplemented` operation returns. It claimed JSON and Query services answer `InvalidAction` (HTTP 400), but only the 46 providers that fall through to the CRUD engine do; 32 answer `NotImplemented` (HTTP 501) from their own dispatch default, several use their own vocabulary, and `sqs` differs by protocol. Only the failure itself is stable — the specific code and status are documented, not guaranteed
time: 2026-08-10T10:15:00.000000+09:00
custom:
Issue: "129"
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260810-093000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: Service selection now matches what the docs promise. An empty `services` block is still a block, so it starts nothing rather than every registered service; and `DEVCLOUD_SERVICES` names the running set outright rather than intersecting with the `services` block, so it starts a service the block omits and overrides a block entry the file disabled. Previously `DEVCLOUD_SERVICES=sqs` alongside a block listing only `s3` started nothing at all
time: 2026-08-10T09:30:00.000000+09:00
custom:
Issue: "129"
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260810-140000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: CloudFront no longer fabricates a success for operations it does not implement. Its dispatch fallback answered HTTP 200 with an empty XML document, which boto3 parses as a successful empty result, on all 122 CloudFront operations the fidelity manifest classifies as `unimplemented` — including `GetPublicKey`. It now returns `NotImplemented` (HTTP 501), matching the other 32 providers that decline from their own dispatch default. It was the only provider in the tree that answered an unimplemented call with a success
time: 2026-08-10T14:00:00.000000+09:00
custom:
Issue: "129"
5 changes: 5 additions & 0 deletions changes/unreleased/Fixed-20260811-103000.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
kind: Fixed
body: Generated routers now honour route patterns that constrain the query string, so operations distinguished only by a query parameter are reachable. `matchURI` split the whole pattern into path segments, which made `/2020-05-31/tagging?Operation=Tag` unable to match the path segment `tagging` — CloudFront `TagResource` and `UntagResource` were implemented but unroutable, and answered as unimplemented. A pattern's constraint must now be satisfied by the request query, and query-constrained routes are tried ahead of unconstrained ones so a bare path cannot shadow a more specific route
time: 2026-08-11T10:30:00.000000+09:00
custom:
Issue: "129"
10 changes: 7 additions & 3 deletions cmd/devcloud/fidelity_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,13 @@ func TestFidelityManifestCoversRegisteredServices(t *testing.T) {
}
}

// TestFidelityManifestCoversCRUDRegistry checks the other direction: every
// operation the CRUD engine would serve is declared, and never as unimplemented
// — that would promise an InvalidAction the runtime does not return.
// TestFidelityManifestCoversCRUDRegistry checks the other direction, against a
// source the manifest was not generated from: every operation the CRUD engine
// would serve at runtime is declared, and never as unimplemented — that would
// promise a failure the runtime does not deliver. This is also the only guard
// that can notice an engine-served operation going *missing* from the manifest;
// TestFidelityManifestCoverage can only inspect what the manifest already
// lists.
func TestFidelityManifestCoversCRUDRegistry(t *testing.T) {
for _, id := range plugin.DefaultRegistry.RegisteredServices() {
for op := range crud.RegisteredOps(id) {
Expand Down
9 changes: 8 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,14 @@ This directory holds DevCloud's technical documentation. Use the map below to ju
- **[Configuration](configuration.md)** — YAML options, env-var overrides, tier shortcuts
- **[Architecture](architecture.md)** — system design, plugin model, codegen pipeline, multi-CSP vision
- **[Roadmap](roadmap.md)** — phased plan toward multi-CSP support
- **[Services Matrix](services-matrix.md)** — 101 services, coverage status, boto3 pass rate
- **[Services Matrix](services-matrix.md)** — 104 services, coverage status, boto3 pass rate

## What you can rely on

- **[Compatibility Policy](compatibility-policy.md)** — what v1.0 guarantees across 1.x, what it explicitly does not, and how deprecation works
- **[Fidelity Manifest](fidelity-manifest.md)** — per-operation tiers: how much to trust any given call
- **[CRUD Engine](crud-engine.md)** — how engine-served operations behave, and where they stop
- **[Plugin API](plugin-api.md)** — the in-tree `ServicePlugin` contract for contributors

## Per-service references

Expand Down
167 changes: 167 additions & 0 deletions docs/compatibility-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
# Compatibility Policy

What DevCloud **v1.0** promises, and what it deliberately does not.

This document covers the surfaces you touch as a *user* of DevCloud — the config file, the
environment, the CLI, the admin API, and the AWS wire protocol. For the in-tree Go contract
that service implementations are written against, see
[plugin-api.md](plugin-api.md#api-stability); that surface lives under `internal/` and is not
importable from another module.

Versions follow [Semantic Versioning](https://semver.org). "Across 1.x" below means every
release from v1.0.0 up to but not including v2.0.0.

## Guaranteed across 1.x

### Configuration file

These keys keep their name, type and meaning. New keys may be added; existing ones are not
removed or repurposed. Defined in [`internal/config/config.go`](../internal/config/config.go).

| Key | Type | Meaning |
|---|---|---|
| `server.port` | int | Listen port. Default `4747` when absent or `0`. |
| `services` | map | Presence of the block is authoritative — only the services it lists run, and an empty block runs nothing. Absent means every registered service runs. |
| `services.<id>.enabled` | bool | Whether that service starts. |
| `services.<id>.data_dir` | string | Where that service stores data. |
| `admin.enabled` | bool | Whether the admin API is served. Default `false`. |
| `logging.level` | string | Log level. |
| `logging.format` | string | Log format. |

### Environment variables

| Variable | Meaning |
|---|---|
| `DEVCLOUD_PORT` | Overrides `server.port`. |
| `DEVCLOUD_SERVICES` | Names the running service set. `all`, a comma-separated list of service ids, or the `tier1` / `tier2` / `tier3` shortcuts. Unknown tokens are treated as literal service names. |
Comment thread
skyoo2003 marked this conversation as resolved.
| `DEVCLOUD_DATA_DIR` | Base directory; each service stores under `<base>/<id>`. Overrides `data_dir`. |

Environment overrides config file, and that precedence is guaranteed.
Comment thread
skyoo2003 marked this conversation as resolved.

When `DEVCLOUD_SERVICES` names services, it decides membership on its own: it
starts a service the `services` block omits and stops one the block enables. It
replaces the block's selection rather than intersecting with it. The literal
`all` is the exception — it switches the filter off and hands the decision back
to the `services` block, which is what "all" has always meant here
([configuration.md](configuration.md#devcloud_services)).

### Command line

`-config <path>` keeps its meaning. With no flag, DevCloud uses `./devcloud.yaml` if present
and the embedded defaults otherwise — zero-config startup keeps working.

### Admin API

Served at `/devcloud/api/` when `admin.enabled: true`. These routes keep responding, and their
JSON responses **only gain fields** — no documented key is removed or repurposed.

| Route | Guaranteed response keys |
|---|---|
| `GET /devcloud/api/services` | array of `id`, `name`, `status`, `resourceCount` |
| `GET /devcloud/api/services/{id}/resources` | array of `type`, `id`, `name` |
| `GET /devcloud/api/logs` | array of `method`, `path`, `status`, `duration`, `timestamp`, `service`; newest first; `?limit=` honoured |
| `GET /devcloud/api/fidelity` | object keyed by service id, each with `modelBacked` and `counts`; `?service=<id>` adds `operations` |

### Fidelity tier names

`hand-verified`, `auto-crud` and `unimplemented` keep the meanings given in
[fidelity-manifest.md](fidelity-manifest.md). The set does not shrink, and a name is never
reused for a different meaning.

Every operation the manifest lists carries a tier from that set, every registered service
appears, and every operation the CRUD engine serves is present and not filed as
`unimplemented` — all three fail the build, in
[`cmd/devcloud/fidelity_test.go`](../cmd/devcloud/fidelity_test.go).

What no test can catch is an operation that never reaches the manifest at all, and that is
bounded rather than eliminated: for the 93 services with an in-tree Smithy model the operation
universe comes from the model, so an operation losing its implementation reclassifies to
`unimplemented` instead of disappearing. For the 11 without one, the universe *is* what the
providers serve, so the manifest lists no unimplemented tail for them — `modelBacked` on
`GET /devcloud/api/fidelity` reports which is which.

### Wire behaviour — scoped to the compatibility suite

**Whatever a test in [`tests/compatibility/`](../tests/compatibility/) asserts about a response
keeps holding across 1.x — that property, and nothing wider.**

The promise is as wide as each individual assertion: not as wide as the field, and not as wide as
the operation. `CreateFunction` is covered by `test_lambda.py`, and its two assertions are worth
reading closely:

- `FunctionName` is asserted **equal** to the name that was sent, so its key and its value are
both promised.
- `FunctionArn` is asserted only to be **present**, so its presence is promised and its type,
format and meaning are not. If it stopped being ARN-shaped the suite would stay green — so this
policy does not promise it stays ARN-shaped.
- `Runtime`, `Handler` and `MemorySize` are not asserted at all, so they carry no promise even
though today's response includes them.

That narrowness is the point: it is the promise the repo can actually keep. The suite — 775 tests
driving real boto3 clients — runs in CI on every push and again against the tagged commit before
a release publishes, so breaking an assertion fails the build rather than depending on review
discipline. Anything the suite does not assert rests on nothing but intent. Widening the promise
means adding or strengthening assertions, and such contributions are welcome.

## Not guaranteed

Depending on any of the following will break, and breaking it is **not** a major-version event.

- **`auto-crud` response content.** 948 operations are served by the
[generic CRUD engine](crud-engine.md) at fidelity that is deliberately *plausible, not
faithful*: store-backed responses echoing your input plus synthesized ids and ARNs, with no
validation, no cross-resource integrity, no pagination correctness and no business logic.
Their shape and content may change in any release. Use them to wire an SDK up, nothing more.
- **Hand-verified operations with no compatibility test.** Of 4,496 hand-verified operations,
only what the suite covers is promised. The rest are best-effort.
- **Data durability.** Stores are local development stores. Several are in-memory and
per-process; on-disk layouts under `data_dir` may change format between releases without a
migration. Do not treat DevCloud as a database.
- **`unimplemented` → served transitions.** An operation that returns an error today may start
returning a response. This is additive, and ships in a minor release.
- **Service coverage.** New services may be added in a minor release. The 104 services present
at v1.0 are a floor, not a ceiling.
- **Error codes, HTTP status and message wording.** What *is* guaranteed for an `unimplemented`
operation is that it **fails** — an AWS-shaped error, never a fabricated success. Which error
Comment on lines +124 to +125

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fix Auto Scaling before guaranteeing unimplemented failures

This guarantee is already false for Auto Scaling: internal/generated/fidelity/manifest_gen.go:528 classifies DeleteNotificationConfiguration as TierUnimplemented, but an invocation reaches the default branch in internal/services/autoscaling/provider.go:176-180, which returns HTTP 200 with a generic XML response. Fresh evidence beyond the previously identified CloudFront case is that Auto Scaling independently still fabricates success, so either make this provider reject its 18 manifest-unimplemented operations or narrow the guarantee.

Useful? React with 👍 / 👎.

is not: it comes from whichever provider handles the request, so it is `InvalidAction` (400)
for services that fall through to the CRUD engine, `NotImplemented` (501) for the 33 providers
with their own dispatch default, and each path-routed provider's own vocabulary otherwise
(`s3` `MethodNotAllowed` 405, `bedrock` `UnsupportedOperation` 400). `sqs` even differs by
protocol. [fidelity-manifest.md](fidelity-manifest.md) records the current behaviour;
normalizing it is a minor release, not a major one.
- **Log output.** Format, levels and wording of server logs are operational, not an API.
- **Everything under `internal/`.** Go forbids importing it from another module, and DevCloud
reserves the right to restructure it freely across 1.x — explicitly including the planned
intermediate representation and `ModelSource` work on the [roadmap](roadmap.md). Internal
churn is not a compatibility event. The in-tree `ServicePlugin` contract in
[plugin-api.md](plugin-api.md#api-stability) is not an exception to this: it is a convention
that keeps in-tree plugins compiling, and it does not gate release versioning.
- **Behavioural parity with AWS.** No release of DevCloud promises AWS's validation, business
logic, eventual-consistency timing, rate limits, or IAM enforcement. Credentials are accepted
without signature verification.

## Deprecation procedure

Removing anything from the guaranteed list is a **major** version bump. Before that can happen:

1. **Deprecate in a minor release.** The old form keeps working and emits a runtime warning
naming its replacement. The precedent is the `dashboard` → `admin` config rename: the old key
still enables the admin API, warns, and yields to an explicit `admin` block
([`config.go`](../internal/config/config.go)).
2. **Document it** — in the release notes for that version, and here.
3. **Remove no earlier than the next major.** At least one released version must have shipped
the warning.

Silence is not deprecation. A removed key that YAML would otherwise drop without comment is
kept in the parser purely to warn — that is why `auth` still produces a message telling you
SigV4 is not enforced rather than being ignored.

The pre-flight checklist in [release.md](release.md#pre-flight-checklist) makes this a step in
cutting a release, not a thing to remember.

## Reporting a break

If a 1.x release breaks something on the guaranteed list, that is a bug — please
[open an issue](https://github.com/skyoo2003/devcloud/issues) with the DevCloud version and a
reproducing snippet. If it breaks something on the not-guaranteed list, an issue is still
useful: it is evidence for tightening the policy in a future major.
4 changes: 3 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ Environment variables override YAML values for selected keys (see [Environment V
The `services` block is **optional and authoritative**: omit it (as the embedded
default does) and every registered service starts with `data_dir
./data/<service>`; list any service and *only* the services you list start.
Writing `services: {}` lists nothing, so nothing starts — a block is a block
even when it is empty.

| Key | Default | Description |
|-----|---------|-------------|
Expand Down Expand Up @@ -112,7 +114,7 @@ docker run -p 8080:8080 -e DEVCLOUD_PORT=8080 ghcr.io/skyoo2003/devcloud:latest

### `DEVCLOUD_SERVICES`

Comma-separated list of services to enable. When set, **only** the listed services are enabledall others are disabled regardless of their `enabled` setting in YAML. When not set, each service uses its YAML `enabled` value (or the embedded default of `true`). An unknown `tierN` token is treated as a literal service name and logged as a warning.
Comma-separated list of services to enable. When set, it names the running set outright: **only** the listed services are enabled, all others are disabled regardless of their `enabled` setting in YAML, and a service the `services` block omits entirely still starts if you name it here. The block still supplies that service's `data_dir`. When not set, each service uses its YAML `enabled` value (or the embedded default of `true`). An unknown `tierN` token is treated as a literal service name and logged as a warning.

**Tier shortcuts** (expand to predefined service groups — see [`internal/config/config.go`](https://github.com/skyoo2003/devcloud/blob/main/internal/config/config.go) for the exact list):

Expand Down
21 changes: 20 additions & 1 deletion docs/fidelity-manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,30 @@ Smithy models, the CRUD registry, and each provider's dispatch code.
|------|---------|--------------|
| `hand-verified` | The service's provider implements the operation explicitly. | Behaviour, not just shape. Covered by the boto3 compatibility suite where tests exist. |
| `auto-crud` | Served by the [generic CRUD engine](crud-engine.md) with plausible, store-backed responses. No validation, no business logic, no cross-resource integrity. | Wiring your SDK calls and round-tripping create → get → list → delete. Nothing else. |
| `unimplemented` | Not served — the call fails instead of inventing a success. The error is the provider's own: JSON and Query services return `InvalidAction` (HTTP 400), while the path-routed providers answer in their own vocabulary (`s3` `MethodNotAllowed` 405, `lambda` `ResourceNotFoundException` 404, `bedrock` `UnsupportedOperation` 400). | Knowing early that DevCloud will not serve this call. |
| `unimplemented` | Not served — the call fails instead of inventing a success. The error is whichever the handling provider emits, and that varies (see below). | Knowing early that DevCloud will not serve this call. |
Comment thread
skyoo2003 marked this conversation as resolved.

`hand-verified` always wins: the CRUD engine is reached only when a provider's
dispatch falls through, so a hand-written implementation is never shadowed.

### What an `unimplemented` call actually returns

There is no single error. Which one you get depends on how the owning provider
declines the operation:

| How the provider declines | Error | Status | Providers |
|---|---|---|---|
| Returns `ErrUnhandledOp`, and the CRUD engine cannot classify the operation either — [`gateway/router.go`](../internal/gateway/router.go) emits the fallback | `InvalidAction` | 400 | 46 |
| Its dispatch `default:` answers directly | `NotImplemented` | 501 | 33 |
| Its dispatch `default:` answers in its own vocabulary | `UnsupportedOperation` / `MethodNotAllowed` | 400 / 405 | `iot`, `iotwireless`, `apigatewayv2`, `backup`, `bedrock`, `s3` |

A service can even answer differently per protocol: `sqs` returns
`NotImplemented` (501) on the Query protocol and `InvalidAction` (400) on JSON.

Only the *failure* is stable, and that is all
[compatibility-policy.md](compatibility-policy.md) promises — an `unimplemented`
operation never fabricates a success. The specific code and status are not
guaranteed across 1.x; normalizing them is a minor release.

## Current coverage

| Tier | Operations |
Expand Down
Loading