diff --git a/CHANGELOG.md b/CHANGELOG.md index d31ca4a..215b1e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,6 +70,30 @@ Notable changes to Cloudfall. The format follows ### Changed +- **Breaking:** the CLI follows one name per meaning, and its output is + typed: + - Decision records have their own group: `decisions list` (was + `operations decisions`), `decisions approve` (was `operations + approve`), and a new `decisions show`. `operations` keeps the catalog: + `list`, `show`, `propose` + - `data migrate` is `data copy`, no longer read as `migrate`, and + `services inspect` is `services observe`, as `observe` collects server + evidence. Each old path answers 13 `REDIRECTED` with the new command in + `error.redirect.command` + - `migrate --release` and `--env-file` are `--component-release` and + `--component-env-file`: they take `COMPONENT=VALUE` pairs, where + `deploy`'s take one value + - `--observed` defaults to `tmp/observed` on every command, where + `observe` writes: `audit`, `services status` and `dashboard` required + it, and the operator's drift checks used `tmp/operator/observed` + - Field names in `data` are snake_case at every depth, as treaty's own + envelope keys are (`openProposals` is `open_proposals`, `wouldRun` is + `would_run`). Map keys that are data keep their spelling (resource + kinds, environment variable names, ids), and a record or resource in + `data`, anything with an `apiVersion`, keeps its stored form + - Every command's output schema gives each `data` key its JSON type, and + an optional list or object is written empty rather than `null` + (`inventory show` writes `ansible` as `{}` for a project) - **Breaking:** argparse is gone: the last three `cloudfall` commands move to treaty, and so do root `--help`, `--version` and `--schema`. - `operator run` streams one envelope line per pass as it ends, with diff --git a/README.md b/README.md index 3b5671b..b46dbec 100644 --- a/README.md +++ b/README.md @@ -287,7 +287,7 @@ routes. Progress persists after every step, so a failed step or the DNS pause resumes exactly where it stopped. Managed-database contents follow the same guided path: save the source -connection URL into a local file and either run `cloudfall data migrate` +connection URL into a local file and either run `cloudfall data copy` directly (it shows the plan until you add `--yes`) or add `--data =` to the migration plan. The engine dumps on the target host, restores over the peer-authenticated socket, refuses non-empty target databases, verifies per-table row counts, diff --git a/ROADMAP.md b/ROADMAP.md index b329c0a..72e6c9b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -126,7 +126,7 @@ records an explicit origin skip when origin and proxy share a host. unmappable ✔ (blueprint files; the Render API and cron jobs are pending) - Data migration: managed-Postgres dump and restore with verification ✔ - (`cloudfall data migrate`, the `--data` step of `cloudfall migrate`, and + (`cloudfall data copy`, the `--data` step of `cloudfall migrate`, and the confirm-gated `migrate_database` MCP tool: per-table row-count verification, a non-empty-target refusal guard, secret-file hygiene, and a `DataMigrationReceipt`; proven live in the diff --git a/Taskfile.yml b/Taskfile.yml index 6fdb112..73a42dd 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -65,7 +65,7 @@ tasks: - validate cmds: - >- - uv run cloudfall services inspect --project {{ .CONFIG_DIR }} + uv run cloudfall services observe --project {{ .CONFIG_DIR }} --schemas {{ .SCHEMA_DIR }} --service-observed {{ .SERVICE_OBSERVATION_DIR }} services:status: diff --git a/docs/fleet-goals.md b/docs/fleet-goals.md index 7e03d4e..6d36b0d 100644 --- a/docs/fleet-goals.md +++ b/docs/fleet-goals.md @@ -112,7 +112,7 @@ acceptable if the following hold: | 6 | Failover on demand | `availability` block, formation mode | Designed, not implemented (demand-gated) | | 7 | One-command node setup | Baseline over SSH from config | In place for installed Debian; provisioning (installimage, RAID) out of band | | 8 | Resource limits | systemd resource control from declared envelopes | Missing | -| 9 | Data movement verb | `data migrate` + file-storage move + receipts | Partial: databases yes, file storage no | +| 9 | Data movement verb | `data copy` + file-storage move + receipts | Partial: databases yes, file storage no | | 10 | Routing follows placement | Domain re-render on convergence | Partial: rendering exists, not wired to placement change | | 11 | Drain / decommission | Evacuation procedure over 9 + 10 | Missing | | 12 | Declared operations with verify | `Operation` catalog, MCP annotations, verify playbook | Partial: MCP tools are annotated, server changes preview until confirmed, and approval stays a person's command; deploys, backups and migrations verify; no catalog over arbitrary playbooks, no declared verify step | diff --git a/docs/neon-migration-guide.md b/docs/neon-migration-guide.md index f25ce9c..3138ec3 100644 --- a/docs/neon-migration-guide.md +++ b/docs/neon-migration-guide.md @@ -14,7 +14,7 @@ restorable, and the always-on operator watches the database with the evidence attached to every decision. Managed PostgreSQL is mostly insurance; this keeps the coverage and makes the proof yours. -> **Status:** the guided data migration (`cloudfall data migrate`) is +> **Status:** the guided data migration (`cloudfall data copy`) is > proven live against an external managed PostgreSQL — dump, restore, > per-table row-count verification, refusal guards, and a > `DataMigrationReceipt` (see the @@ -152,9 +152,9 @@ after the dump starts are lost. Put the application into maintenance mode or stop its writers, then: ```console -uv run cloudfall data migrate postgresql-main \ +uv run cloudfall data copy postgresql-main \ --database app --source-url-file tmp/neon-app-url -uv run cloudfall data migrate postgresql-main \ +uv run cloudfall data copy postgresql-main \ --database app --source-url-file tmp/neon-app-url --yes ``` diff --git a/docs/render-migration-guide.md b/docs/render-migration-guide.md index bc39e8d..e7ac78f 100644 --- a/docs/render-migration-guide.md +++ b/docs/render-migration-guide.md @@ -141,7 +141,7 @@ Save the Render database's connection URL alone into a local file, then either run the data migration directly or add it to the plan: ```console -uv run cloudfall data migrate acme-postgresql --database acme --source-url-file ~/.render/acme-db.url +uv run cloudfall data copy acme-postgresql --database acme --source-url-file ~/.render/acme-db.url uv run cloudfall migrate --build acme-api=main --data acme=~/.render/acme-db.url --yes ``` diff --git a/docs/secrets-guide.md b/docs/secrets-guide.md index 702e9bf..ee09a25 100644 --- a/docs/secrets-guide.md +++ b/docs/secrets-guide.md @@ -100,7 +100,7 @@ uv run cloudfall secrets render crm-backend \ This writes `tmp/env/crm-backend.env` with mode `0600` and prints an envelope carrying the key **names**, the file hash, and the resolved references — never a value. Pass that file to `cloudfall deploy --yes ---env-file` or map it in `cloudfall migrate`; the deploy role installs it +--env-file` or map it with `cloudfall migrate --component-env-file`; the deploy role installs it into the release owned by the application user and the systemd unit loads it with `EnvironmentFile=`. diff --git a/sdk/README.md b/sdk/README.md index 738e17b..a212bd2 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -57,7 +57,7 @@ uv run cloudfall restart --project config/examples crm-backend --yes uv run cloudfall health --project config/examples crm-backend ``` -`deploy`, `rollback`, `restart`, and `data migrate` change servers only with +`deploy`, `rollback`, `restart`, and `data copy` change servers only with `--yes`. Without it they run every controller-side check (declared component, verified artifact, declared database, source URL file) and print a `plan` naming the target servers, exit `0`, so a request can be reviewed before it @@ -80,8 +80,8 @@ fleet over stdio, and its tools are Cloudfall's own commands. The fleet, the schema and engine directories, and every directory a tool writes are fixed when the server starts, so a tool call cannot point at another project, schema directory or engine, or write outside the project. Read-only commands (`config_validate`, `inventory_show`, `audit`, -`services_status`, `health`, `observe`, `services_inspect`) carry read-only -annotations. `deploy`, `rollback`, `restart`, `migrate` and `data_migrate` +`services_status`, `health`, `observe`, `services_observe`) carry read-only +annotations. `deploy`, `rollback`, `restart`, `migrate` and `data_copy` return their plan until called again with `yes: true`. Four tools have no command of their own: `build_artifact`, and `converge_baseline`, `converge_services` and `converge_domains`, which are destructive and run @@ -90,7 +90,7 @@ only with `confirm_destructive: true`. The fleet-declaring tools `add_ssh-key`, `add_server-type` and `add_server` are `cloudfall add`: they write schema-validated resource files into the project and never overwrite an existing resource. `cloudfall init`, -`operations approve`, `operator run` and `dashboard serve` are never tools: +`decisions approve`, `operator run` and `dashboard serve` are never tools: the server starts inside an existing project, approving a recorded operation is a person's command, and the loops run until stopped. diff --git a/sdk/src/cloudfall/agent_server.py b/sdk/src/cloudfall/agent_server.py index 38bd53c..3c713ad 100644 --- a/sdk/src/cloudfall/agent_server.py +++ b/sdk/src/cloudfall/agent_server.py @@ -9,7 +9,7 @@ server-changing ones previewing until called with ``yes``. On a repository, they are the read-only fleet commands and one tool per declared operation, whose call runs check mode and records a proposal; approving it is -``cloudfall operations approve``, which no tool can stand in for. +``cloudfall decisions approve``, which no tool can stand in for. Nothing here imports ``cloudfall.app``: ``App(mcp=...)`` needs these at construction. The tools Cloudfall provides beside its commands, which use @@ -45,7 +45,7 @@ if TYPE_CHECKING: from cloudfall.catalog import Operation -APPROVAL_COMMAND = "cloudfall operations approve" +APPROVAL_COMMAND = "cloudfall decisions approve" """The command a person runs to approve a proposal; no tool stands in for it.""" ERROR_INVENTORY_UNDECLARED = "fleet_inventory_undeclared" @@ -287,7 +287,7 @@ def operation_tool_description(operation: Operation) -> str: "inventory.show", "audit", "services.status", - "services.inspect", + "services.observe", "health", "observe", "add.ssh-key", @@ -299,7 +299,7 @@ def operation_tool_description(operation: Operation) -> str: "deploy", "rollback", "restart", - "data.migrate", + "data.copy", "migrate", "backup.run", "backup.verify", @@ -317,7 +317,8 @@ def operation_tool_description(operation: Operation) -> str: "audit", "operations.list", "operations.show", - "operations.decisions", + "decisions.list", + "decisions.show", "why", } ) @@ -427,7 +428,7 @@ def bound_arguments(args: McpServeArgs) -> dict[str, object]: Cloudfall manages declarative infrastructure for dedicated Debian servers, in the one project this server was started for. Read-only tools validate the project and derive evidence. deploy, rollback, restart, migrate and -data_migrate change servers: called without yes they only return the plan; +data_copy change servers: called without yes they only return the plan; review it, then call again with yes: true. The converge_* tools converge every declared server and are destructive: they run only with confirm_destructive: true. Every change writes receipts; nothing reports diff --git a/sdk/src/cloudfall/app.py b/sdk/src/cloudfall/app.py index f2e1f95..07fcb17 100644 --- a/sdk/src/cloudfall/app.py +++ b/sdk/src/cloudfall/app.py @@ -87,7 +87,7 @@ RiskLevel, load_catalog, ) -from cloudfall.commands import CLI_COMMANDS +from cloudfall.commands import CLI_COMMANDS, snake_case, snake_case_keys from cloudfall.dashboard import RefreshInterval, build_dashboard from cloudfall.dashboard_server import ( DashboardHTTPServer, @@ -351,8 +351,8 @@ def __init__(self, body: Mapping[str, object]) -> None: self.body = dict(body) -def _shape_keys(command: str) -> tuple[tuple[str, bool], ...]: - """Return each data key of the command's output shape and if it is optional.""" +def _shape_keys(command: str) -> tuple[tuple[str, str, bool], ...]: + """Return each data key of the command's output: name, JSON type, optional.""" contract = next( entry for entry in CLI_COMMANDS @@ -360,49 +360,56 @@ def _shape_keys(command: str) -> tuple[tuple[str, bool], ...]: ) # A command with two shapes (a plan without --yes, a result with it) # writes one object: a key not in every shape is optional, so null. - names: list[str] = [] + keys: dict[str, tuple[str, bool]] = {} for shape in contract.output: - names.extend( - key.name - for key in shape.keys - if key.name != "error" and key.name not in names - ) - return tuple( - ( - name, - any( - all(key.name != name for key in shape.keys) - or any(key.name == name and key.optional for key in shape.keys) - for shape in contract.output - ), - ) - for name in names - ) + for key in shape.keys: + name = snake_case(key.name) + if key.name == "error" or name in keys: + continue + optional = any( + all(other.name != key.name for other in candidate.keys) + or any( + other.name == key.name and other.optional + for other in candidate.keys + ) + for candidate in contract.output + ) + keys[name] = (key.type, optional) + return tuple((name, kind, optional) for name, (kind, optional) in keys.items()) def _payload_schema(cls: type[Payload]) -> dict[str, object]: - names = [name for name, _ in _shape_keys(cls.command)] + properties: dict[str, object] = {} + for name, kind, optional in _shape_keys(cls.command): + schema: dict[str, object] = {"type": [kind, "null"] if optional else kind} + if kind in {"array", "object"}: + # The domain writes each array in the order it means: a ranking, + # a timeline, the order checks ran in, and a stored record keeps + # its own. treaty would sort them, nested ones included. + schema["x-ordered"] = True + properties[name] = schema if cls.writes: - names.append("effect") + properties["effect"] = {"type": "string"} return { "type": "object", - # The domain writes each array in the order it means: a ranking, a - # timeline, the order checks ran in. treaty would sort them. - "properties": {name: {"x-ordered": True} for name in names}, - "required": names, + "properties": properties, + "required": list(properties), "additionalProperties": False, } def _payload_document(payload: Payload) -> dict[str, object]: - document = dict(payload.body) - for name, optional in _shape_keys(payload.command): + # Field names are snake_case, as treaty's own envelope keys are. + document = cast("dict[str, object]", snake_case_keys(dict(payload.body))) + for name, _kind, optional in _shape_keys(payload.command): if optional and name not in document: document[name] = None return document -app.output_adapter(Payload, schema=_payload_schema, dump=_payload_document) +app.output_adapter( + Payload, schema=_payload_schema, dump=_payload_document, none_as_empty=True +) @dataclass(frozen=True, slots=True, kw_only=True) @@ -625,6 +632,10 @@ def inventory_show( operations = app.group( "operations", description="Read the catalog of operations an agent may run" ) +decisions = app.group( + "decisions", + description="Read the decision records and approve the proposed ones", +) @dataclass(frozen=True, slots=True, kw_only=True) @@ -678,7 +689,7 @@ class ShowOperationArgs(CatalogArgs): @dataclass(frozen=True, slots=True, kw_only=True) class DecisionsArgs(RepositoryArgs): - """Arguments of ``operations decisions``.""" + """Options of a command that reads the decision records.""" decisions: Path = Flag( default=Path(DECISION_DIRECTORY), @@ -691,6 +702,12 @@ def __post_init__(self) -> None: """Keep a relative record directory inside the repository.""" _inside_project(self.decisions, "decisions") + def store(self) -> DecisionStore: + """Open the decision records under the repository.""" + return DecisionStore( + directory=self.root / self.decisions, catalog=SchemaCatalog(self.schemas) + ) + class CatalogPayload(Payload): """``operations list``: every declared operation and the risk counts.""" @@ -705,9 +722,22 @@ class OperationPayload(Payload): class DecisionsPayload(Payload): - """``operations decisions``: every decision record, oldest first.""" + """``decisions list``: every decision record, oldest first.""" - command = "operations decisions" + command = "decisions list" + + +class DecisionPayload(Payload): + """``decisions show``: one decision record.""" + + command = "decisions show" + + +@dataclass(frozen=True, slots=True, kw_only=True) +class ShowDecisionArgs(DecisionsArgs): + """Arguments of ``decisions show``.""" + + decision: ResourceId = Arg(description="Decision id") @operations.command( @@ -742,27 +772,43 @@ def operations_show(args: ShowOperationArgs, _ctx: Ctx) -> OperationPayload: return OperationPayload({"status": "ok", "operation": operation.as_dict()}) -@operations.command( - "decisions", - description="List the decision records this repository holds", +@decisions.command( + "list", + description="List the decision records this repository holds, oldest first", danger_level="safe", exit_codes=["RECORD_INVALID"], - examples=[("List the decision records", "cloudfall operations decisions")], + examples=[("List the decision records", "cloudfall decisions list")], ) -def operations_decisions(args: DecisionsArgs, _ctx: Ctx) -> DecisionsPayload: +def decisions_list(args: DecisionsArgs, _ctx: Ctx) -> DecisionsPayload: """Answer every decision record: the record outlives the catalog.""" - store = DecisionStore( - directory=args.root / args.decisions, catalog=SchemaCatalog(args.schemas) - ) + store = args.store() try: - decisions = [decision.as_document() for decision in store.list()] + documents = [decision.as_document() for decision in store.list()] except DecisionError as error: raise _record_invalid(error) from error return DecisionsPayload( - {"status": "ok", "directory": str(store.directory), "decisions": decisions} + {"status": "ok", "directory": str(store.directory), "decisions": documents} ) +@decisions.command( + "show", + description="Show one decision record", + danger_level="safe", + exit_codes=["NOT_FOUND", "RECORD_INVALID"], + examples=[("Show one decision", "cloudfall decisions show restart-nginx-20260101")], +) +def decisions_show(args: ShowDecisionArgs, _ctx: Ctx) -> DecisionPayload: + """Answer one decision record, or say there is none.""" + try: + decision = args.store().load(args.decision) + except DecisionError as error: + if error.code == ERROR_DECISION_MISSING: + raise Exit.NOT_FOUND(error.detail, context={"code": error.code}) from error + raise _record_invalid(error) from error + return DecisionPayload({"status": "ok", "decision": decision.as_document()}) + + # operator list, show @@ -851,7 +897,10 @@ class AuditArgs(FleetArgs): """Arguments of ``audit``.""" observed: Path = Flag( - description="Directory containing observed-server JSON snapshots" + default=Path("tmp/observed"), + description=( + "Server snapshot directory, as observe writes it (default: tmp/observed)" + ), ) env_receipts: Path = Flag( default=Path("tmp/env-receipts"), @@ -925,7 +974,10 @@ class EvidenceArgs(ProjectArgs): """Options of a command that reads the evidence a fleet's view is built from.""" observed: Path = Flag( - description="Directory containing observed-server JSON snapshots" + default=Path("tmp/observed"), + description=( + "Server snapshot directory, as observe writes it (default: tmp/observed)" + ), ) service_observed: Path = Flag( default=Path("tmp/observed-services"), @@ -1098,7 +1150,7 @@ def init(args: InitArgs, ctx: Ctx) -> Scaffolded: return Scaffolded( effect="created", status="ok", - project=dict(cast("Mapping[str, object]", body["project"])), + project=cast("dict[str, object]", snake_case_keys(body["project"])), files=list(scaffold.files), next=list(cast("list[str]", body["next"])), ) @@ -1232,7 +1284,7 @@ def _added(result: AddResult) -> Added: effect="created", status="ok", project=str(result.project), - added=list(cast("list[dict[str, object]]", body["added"])), + added=cast("list[dict[str, object]]", snake_case_keys(body["added"])), ) @@ -1337,7 +1389,7 @@ def add_server_command( raise _config_invalid(error) from error -# dashboard build, services inspect +# dashboard build, services observe dashboard = app.group("dashboard", description="Build operations dashboard artifacts") @@ -1511,7 +1563,7 @@ def dashboard_serve( @dataclass(frozen=True, slots=True, kw_only=True) class ServicesInspectArgs(ProjectArgs): - """Arguments of ``services inspect``.""" + """Arguments of ``services observe``.""" service_observed: Path = Flag( default=Path("tmp/observed-services"), @@ -1524,13 +1576,13 @@ def __post_init__(self) -> None: class InspectedPayload(Payload): - """``services inspect``: the domain observation files written.""" + """``services observe``: the domain observation files written.""" - command = "services inspect" + command = "services observe" @services.command( - "inspect", + "observe", description="Collect DNS, TLS, origin, and public route evidence", danger_level="safe", # DNS, TLS, and HTTP probes of every declared domain, one after another. @@ -1540,9 +1592,9 @@ class InspectedPayload(Payload): SideEffect("{project_root}/tmp/observed-services/", "output") ], exit_codes=["PROJECT_INVALID", "CONFIG_INVALID", "PERMISSION_DENIED"], - examples=[("Probe every declared domain", "cloudfall services inspect")], + examples=[("Probe every declared domain", "cloudfall services observe")], ) -def services_inspect( +def services_observe( args: ServicesInspectArgs, _ctx: Ctx, fleet: Fleet ) -> InspectedPayload: """Probe each declared domain and write one observation per domain.""" @@ -1876,7 +1928,7 @@ def inputs(self) -> dict[str, object]: @dataclass(frozen=True, slots=True, kw_only=True) class ApproveDecisionArgs(DecisionsArgs): - """Arguments of ``operations approve``.""" + """Arguments of ``decisions approve``.""" decision: ResourceId = Arg(description="Decision id from `operations propose`") approver: str | None = Flag( @@ -1963,7 +2015,7 @@ def operations_propose(args: ProposeArgs, ctx: Ctx) -> Decided: return decided -@operations.command( +@decisions.command( "approve", # A person approves; no agent tool stands in for it. mcp=False, @@ -1978,11 +2030,11 @@ def operations_propose(args: ProposeArgs, ctx: Ctx) -> Decided: subprocess=Subprocess("ansible-playbook"), required_tools={"ansible-playbook": "2.21.0"}, examples=[ - ("Review a proposal", "cloudfall operations approve restart-nginx-20260101"), - ("Run it", "cloudfall operations approve restart-nginx-20260101 --yes"), + ("Review a proposal", "cloudfall decisions approve restart-nginx-20260101"), + ("Run it", "cloudfall decisions approve restart-nginx-20260101 --yes"), ], ) -def operations_approve(args: ApproveDecisionArgs, ctx: Ctx) -> Decided: +def decisions_approve(args: ApproveDecisionArgs, ctx: Ctx) -> Decided: """Run what was proposed, as recorded, and record how it ended.""" store = DecisionStore( directory=args.root / args.decisions, catalog=SchemaCatalog(args.schemas) @@ -2038,10 +2090,10 @@ class OperatorEngineArgs(ProposalArgs): description="Rendered inventory path (default: tmp/ansible-inventory.json)", ) observed: Path = Flag( - default=Path("tmp/operator/observed"), + default=Path("tmp/observed"), description=( - "Observation directory for drift checks and their verification " - "(default: tmp/operator/observed)" + "Server snapshot directory drift checks refresh and verify against " + "(default: tmp/observed)" ), ) gateway_url: str | None = Flag( @@ -2376,11 +2428,11 @@ class MigrateArgs(EngineArgs): default=(), description="Build a component from a git ref, as COMPONENT=REF (repeatable)", ) - release: tuple[str, ...] = Flag( + component_release: tuple[str, ...] = Flag( default=(), description="Deploy an existing release, as COMPONENT=RELEASE (repeatable)", ) - env_file: tuple[str, ...] = Flag( + component_env_file: tuple[str, ...] = Flag( default=(), description="Environment file for a component, as COMPONENT=PATH (repeatable)", ) @@ -2417,8 +2469,8 @@ def __post_init__(self) -> None: _inside_project(getattr(self, flag), flag.replace("_", "-")) for option, entries in ( ("build", self.build), - ("release", self.release), - ("env-file", self.env_file), + ("component-release", self.component_release), + ("component-env-file", self.component_env_file), ("data", self.data), ): _pairs(entries, option) @@ -2428,11 +2480,13 @@ def options(self, fleet: Fleet) -> MigrateOptions: return MigrateOptions( plan_file=fleet.path(self.plan_file), builds=_pairs(self.build, "build"), - releases=_pairs(self.release, "release"), + releases=_pairs(self.component_release, "component-release"), # Relative to the project, as every other path here is. environment_files={ component: fleet.path(Path(value)) - for component, value in _pairs(self.env_file, "env-file").items() + for component, value in _pairs( + self.component_env_file, "component-env-file" + ).items() }, data_migrations={ database: fleet.path(Path(value)) @@ -2517,7 +2571,7 @@ def migrate(args: MigrateArgs, ctx: Ctx, fleet: Fleet) -> Migration: # A plan is the dry run --yes confirms. effect="would_update" if status == "plan" else "updated", status=status, - steps=list(cast("list[dict[str, object]]", result["steps"])), + steps=cast("list[dict[str, object]]", snake_case_keys(result["steps"])), completed=int(cast("int", result["completed"])), next=cast("str | None", result.get("next")), step=cast("str | None", result.get("step")), @@ -3006,7 +3060,7 @@ def import_render_api_command( return ImportedApiPayload({**result.as_dict(), "effect": "created"}) -# deploy, rollback, restart, data migrate: --yes runs them; without it they +# deploy, rollback, restart, data copy: --yes runs them; without it they # validate the request and answer the plan. @@ -3070,7 +3124,7 @@ class RollbackArgs(ComponentArgs): @dataclass(frozen=True, slots=True, kw_only=True) class DataMigrateArgs(EngineArgs): - """Arguments of ``data migrate``.""" + """Arguments of ``data copy``.""" service: ResourceId = Arg(description="Declared service to restore into") database: str = Flag(description="Declared database name inside the service") @@ -3123,9 +3177,9 @@ class RestartPayload(Payload): class DataMigratePayload(Payload): - """``data migrate``: the plan without --yes, the verified restore with it.""" + """``data copy``: the plan without --yes, the verified restore with it.""" - command = "data migrate" + command = "data copy" writes = True @@ -3261,7 +3315,7 @@ def restart_command(args: ComponentArgs, ctx: Ctx, fleet: Fleet) -> RestartPaylo @data.command( - "migrate", + "copy", description=( "Dump an external PostgreSQL database and restore it into a declared " "service with row-count verification; without --yes it shows the plan" @@ -3275,12 +3329,12 @@ def restart_command(args: ComponentArgs, ctx: Ctx, fleet: Fleet) -> RestartPaylo examples=[ ( "Show the plan", - "cloudfall data migrate postgresql-main --database crm " + "cloudfall data copy postgresql-main --database crm " "--source-url-file source.url", ), ], ) -def data_migrate( +def data_copy( args: DataMigrateArgs, ctx: Ctx, fleet: Fleet ) -> DataMigratePayload: """Restore the source database into the service, or show the plan.""" @@ -3469,7 +3523,9 @@ def build_artifact(arguments: Mapping[str, object], call: Ctx) -> ArtifactBuilt: ) except LifecycleError as error: raise _lifecycle_failed(error) from error - return ArtifactBuilt(status="ok", artifact=built) + return ArtifactBuilt( + status="ok", artifact=cast("dict[str, object]", snake_case_keys(built)) + ) def converge( action: str, @@ -3561,3 +3617,10 @@ def domain_vars(arguments: Mapping[str, object]) -> dict[str, object]: exit_codes=_ENGINE_EXIT_CODES, ), ] + + +# Renamed commands keep answering at their old paths, naming the new one. +app.redirect("operations.decisions", to="decisions.list") +app.redirect("operations.approve", to="decisions.approve") +app.redirect("services.inspect", to="services.observe") +app.redirect("data.migrate", to="data.copy") diff --git a/sdk/src/cloudfall/commands.py b/sdk/src/cloudfall/commands.py index 49a7859..231df40 100644 --- a/sdk/src/cloudfall/commands.py +++ b/sdk/src/cloudfall/commands.py @@ -10,9 +10,43 @@ from __future__ import annotations +import re from dataclasses import dataclass from enum import Enum +_LOWER_CAMEL = re.compile(r"[a-z][a-z0-9]*(?:[A-Z][a-z0-9]*)+") +_HUMP = re.compile(r"(?<=[a-z0-9])(?=[A-Z])") + + +def snake_case(name: str) -> str: + """Return a lowerCamel output key in snake_case; any other name as it is. + + Only lowerCamel names are field names. A resource kind (``AlertRule``), + an environment variable (``DATABASE_URL``) or an id (``crm-backend``) + used as a map key is data, and keeps its spelling. + """ + if not _LOWER_CAMEL.fullmatch(name): + return name + return _HUMP.sub("_", name).lower() + + +def snake_case_keys(value: object) -> object: + """Return ``value`` with its field names in snake_case, at every depth. + + A dict carrying ``apiVersion`` is a schema-validated document, a record + or a resource as it is stored, and is returned as it is. + """ + if isinstance(value, dict): + if "apiVersion" in value: + return value + return { + snake_case(key) if isinstance(key, str) else key: snake_case_keys(item) + for key, item in value.items() + } + if isinstance(value, list | tuple): + return [snake_case_keys(item) for item in value] + return value + class CommandEffect(Enum): """The furthest a command's side effects reach.""" @@ -33,10 +67,49 @@ class OutputKey: """One key of a command's ``data``, as the command writes it.""" name: str + type: str + """The JSON type of its value: string, integer, boolean, array or object.""" optional: bool = False """Written only under some conditions, such as a flag or a result.""" +KEY_TYPES: dict[str, str] = { + **dict.fromkeys( + ( + "action", "application", "component", "database", "detail", + "directory", "environmentFile", "health", "instruction", "output", + "pass", "path", "project", "receipt", "release", "report", + "service", "sha256", "status", "step", "verification", "wouldRun", + ), + "string", + ), + **dict.fromkeys( + ("completed", "count", "exitCode", "openProposals", "resources", "tasks"), + "integer", + ), + "healthy": "boolean", + **dict.fromkeys( + ( + "added", "answers", "components", "decisions", "declaredKeys", + "domains", "environmentFiles", "executed", "files", "gaps", "keys", + "missing", "next", "observations", "observed", "operations", + "proposals", "proposed", "references", "requested", "servers", + "services", "skipped", "steps", "unmatchedObservations", "withheld", + "written", + ), + "array", + ), + **dict.fromkeys( + ( + "ansible", "byKind", "byRisk", "dashboard", "decision", "error", + "inventory", "operation", "proposal", "query", "summary", + ), + "object", + ), +} +"""The JSON type of each output key, by name; a shape overrides one as ``name:type``.""" + + @dataclass(frozen=True, slots=True) class OutputShape: """One shape of a command's ``data``, and when it is written.""" @@ -46,12 +119,14 @@ class OutputShape: def _shape(*keys: str, when: str | None = None) -> OutputShape: - """Declare a shape from key names: ``name?`` is optional.""" - return OutputShape( - keys=tuple( - OutputKey(key.rstrip("?"), optional=key.endswith("?")) for key in keys - ), - when=when, + """Declare a shape from keys: ``name?`` is optional, ``name:type`` typed here.""" + return OutputShape(keys=tuple(_key(key) for key in keys), when=when) + + +def _key(declared: str) -> OutputKey: + name, _, json_type = declared.rstrip("?").partition(":") + return OutputKey( + name, json_type or KEY_TYPES[name], optional=declared.endswith("?") ) @@ -63,7 +138,7 @@ class CommandContract: """Entry point: ``cloudfall`` or ``cloudfall-engine``.""" name: str - """Space-joined subcommand path, such as ``data migrate``.""" + """Space-joined subcommand path, such as ``data copy``.""" effect: CommandEffect @@ -99,7 +174,7 @@ def invocation(self) -> str: ), ) _PROPOSAL = (_shape("status", "proposal"),) -_BACKUP = (_shape("status", "receipt", "path"),) +_BACKUP = (_shape("status", "receipt:object", "path"),) _EXECUTED = ("status", "action", "component", "servers", "healthy") @@ -122,7 +197,7 @@ def _plan(subject: str, *extra: str) -> OutputShape: "init", CommandEffect.PROJECT, "lay out a new project directory", - output=(_shape("status", "project", "files", "next"),), + output=(_shape("status", "project:object", "files", "next"),), ), CommandContract( "cloudfall", @@ -189,7 +264,7 @@ def _plan(subject: str, *extra: str) -> OutputShape: ), CommandContract( "cloudfall", - "operations approve", + "decisions approve", CommandEffect.SERVERS, "approve one recorded proposal, run it for real, and verify it", gate=_YES, @@ -200,11 +275,18 @@ def _plan(subject: str, *extra: str) -> OutputShape: ), CommandContract( "cloudfall", - "operations decisions", + "decisions list", CommandEffect.READ, "list what was proposed, what check mode showed, and who approved", output=(_shape("status", "directory", "decisions"),), ), + CommandContract( + "cloudfall", + "decisions show", + CommandEffect.READ, + "show one decision record: the proposal, its check-mode diff, and its outcome", + output=(_shape("status", "decision"),), + ), CommandContract( "cloudfall", "why", @@ -248,7 +330,7 @@ def _plan(subject: str, *extra: str) -> OutputShape: ), CommandContract( "cloudfall", - "services inspect", + "services observe", CommandEffect.READ, "probe DNS, TLS, origin, and public routes into evidence", output=(_shape("status", "observations"),), @@ -399,7 +481,7 @@ def _plan(subject: str, *extra: str) -> OutputShape: ), CommandContract( "cloudfall", - "data migrate", + "data copy", CommandEffect.SERVERS, "dump an external database and restore it into a declared service", gate=_YES, @@ -428,14 +510,14 @@ def _plan(subject: str, *extra: str) -> OutputShape: "status", "steps", "completed", - "next?", + "next:string?", when="`plan` without `--yes`; `ok` when every step is done", ), _shape( "status", "steps", "completed", - "next?", + "next:string?", "step", "error", when="`paused` (exit 3) or `error` (exit 1) at `step`", diff --git a/sdk/src/cloudfall/importer.py b/sdk/src/cloudfall/importer.py index cbdb4a0..1b2c32d 100644 --- a/sdk/src/cloudfall/importer.py +++ b/sdk/src/cloudfall/importer.py @@ -590,7 +590,7 @@ def _import_databases( _CATEGORY_ACTION, _POSTGRES_SERVICE_ID, "before cutover, save the Render database connection URL into a " - "local file and migrate the data with: cloudfall data migrate " + "local file and migrate the data with: cloudfall data copy " f"{_POSTGRES_SERVICE_ID} --database --source-url-file " "--yes (or pass --data = to cloudfall migrate)", ) diff --git a/sdk/src/cloudfall/project.py b/sdk/src/cloudfall/project.py index 5f56870..21c7cfe 100644 --- a/sdk/src/cloudfall/project.py +++ b/sdk/src/cloudfall/project.py @@ -710,8 +710,10 @@ def _agent_contract(options: InitOptions) -> str: names the command and the request. Branch on the exit code first, then on the snake_case code in `error.context.code`, never on the message. A negative verdict such as drift exits non-zero and keeps its report in -`data`. `operator run` and `dashboard serve` stream: one envelope line per -event, then a closing line with `meta.end`. +`data`. Field names are snake_case; a record or resource inside `data` +(anything with an `apiVersion`) keeps the form it is stored in. `operator +run` and `dashboard serve` stream: one envelope line per event, then a +closing line with `meta.end`. `uv run cloudfall --schema` prints every command with its flags, output schema, and exit codes; `uv run cloudfall --schema` prints one. diff --git a/sdk/tests/test_agent_server.py b/sdk/tests/test_agent_server.py index 0f631dd..4a9f04f 100644 --- a/sdk/tests/test_agent_server.py +++ b/sdk/tests/test_agent_server.py @@ -172,7 +172,8 @@ def test_a_repository_serves_its_catalog_and_no_approval(tmp_path: Path) -> None "audit", "inventory_show", "observe", - "operations_decisions", + "decisions_list", + "decisions_show", "operations_list", "operations_show", "why", @@ -183,7 +184,7 @@ def test_a_repository_serves_its_catalog_and_no_approval(tmp_path: Path) -> None purge = tools["operation_purge"].annotations assert (marker.read_only_hint, marker.destructive_hint) == (True, False) assert (purge.read_only_hint, purge.destructive_hint) == (True, True) - assert "cloudfall operations approve" in tools["operation_write_marker"].description + assert "cloudfall decisions approve" in tools["operation_write_marker"].description def test_an_operation_tool_records_a_proposal_and_changes_nothing( @@ -197,7 +198,7 @@ def test_an_operation_tool_records_a_proposal_and_changes_nothing( "operation_write_marker", {"target": "web-1", "inputs": {"version": "2.0.0"}}, ), - ("operations_decisions", {}), + ("decisions_list", {}), ("why", {"operation": "write-marker"}), ) @@ -206,7 +207,7 @@ def test_an_operation_tool_records_a_proposal_and_changes_nothing( assert decision["spec"]["status"] == "proposed" assert decision["spec"]["inputs"] == {"version": "2.0.0"} assert proposed["data"]["next"] == [ - f"cloudfall operations approve {decision['metadata']['id']} --yes" + f"cloudfall decisions approve {decision['metadata']['id']} --yes" ] assert not (repository / "marker.txt").exists() recorded = decisions["data"]["decisions"] @@ -221,7 +222,7 @@ def test_a_call_cannot_reach_another_fleet_or_approve(tmp_path: Path) -> None: _, (redirected, approved) = _talk( _serve("--repository", str(repository), cwd=tmp_path), ("operations_list", {"repository": str(other)}), - ("operations_approve", {"decision": "anything", "yes": True}), + ("decisions_approve", {"decision": "anything", "yes": True}), ) assert redirected["ok"] is False @@ -238,7 +239,7 @@ def test_a_project_serves_its_commands_with_the_project_bound(tmp_path: Path) -> assert {"deploy", "audit", "operator_approve", "build_artifact"} <= set(tools) assert not { "init", - "operations_approve", + "decisions_approve", "operator_run", "dashboard_serve", "cleanup", diff --git a/sdk/tests/test_ansible_reader.py b/sdk/tests/test_ansible_reader.py index 2ae304c..9afc438 100644 --- a/sdk/tests/test_ansible_reader.py +++ b/sdk/tests/test_ansible_reader.py @@ -328,7 +328,7 @@ def test_the_cli_finds_the_inventory_an_ansible_cfg_names( payload = json.loads(capsys.readouterr().out) assert exit_code == 0 - assert payload["data"]["ansible"]["inProcess"] is True + assert payload["data"]["ansible"]["in_process"] is True assert [server["id"] for server in payload["data"]["inventory"]["servers"]] == [ "web-1", "web-2", diff --git a/sdk/tests/test_app.py b/sdk/tests/test_app.py index 9374c3a..00225c6 100644 --- a/sdk/tests/test_app.py +++ b/sdk/tests/test_app.py @@ -11,17 +11,14 @@ import sys import urllib.request from pathlib import Path -from typing import TYPE_CHECKING +import pytest from cloudfall.app import Fleet, MigrateArgs from cloudfall.cli import main from cloudfall.commands import CLI_COMMANDS from cloudfall.resources import default_schema_directory from cloudfall.validation import validate_config -if TYPE_CHECKING: - import pytest - ROOT = Path(__file__).parents[2] EXAMPLES = ROOT / "config" / "examples" COMPLIANT = ROOT / "config" / "tests" / "observed" / "compliant" @@ -53,7 +50,7 @@ def test_root_schema_lists_every_cataloged_command( def test_a_command_keeps_its_keys_with_status_under_data( capsys: pytest.CaptureFixture[str], ) -> None: - code = main(["operations", "decisions", "--repository", str(EXAMPLES)]) + code = main(["decisions", "list", "--repository", str(EXAMPLES)]) document = json.loads(capsys.readouterr().out) assert code == 0 @@ -87,7 +84,7 @@ def test_migrate_resolves_env_and_data_files_against_the_project( """Relative env and data paths name files inside the project.""" fleet = Fleet(EXAMPLES, validate_config(EXAMPLES, SCHEMAS)) args = MigrateArgs( - env_file=("crm-backend=tmp/env/crm.env",), + component_env_file=("crm-backend=tmp/env/crm.env",), data=(f"crm={tmp_path / 'source.url'}",), ) @@ -153,3 +150,31 @@ def test_dashboard_serve_names_a_port_it_cannot_listen_on( closing = json.loads(capsys.readouterr().out.splitlines()[-1]) assert code == 4 assert closing["error"]["context"]["code"] == "dashboard_listen_failed" + + +def test_decisions_show_says_when_there_is_no_such_record( + capsys: pytest.CaptureFixture[str], +) -> None: + code = main(["decisions", "show", "ghost", "--repository", str(EXAMPLES)]) + + document = json.loads(capsys.readouterr().out) + assert code == 5 + assert document["error"]["context"]["code"] == "decision_missing" + + +@pytest.mark.parametrize( + ("old", "new"), + [ + (["operations", "decisions"], "cloudfall decisions list"), + (["services", "inspect"], "cloudfall services observe"), + (["data", "migrate", "postgresql-main"], "cloudfall data copy postgresql-main"), + ], +) +def test_a_renamed_command_names_its_new_path( + old: list[str], new: str, capsys: pytest.CaptureFixture[str] +) -> None: + code = main(old) + + document = json.loads(capsys.readouterr().out) + assert code == 13 + assert document["error"]["redirect"]["command"] == new diff --git a/sdk/tests/test_arguments.py b/sdk/tests/test_arguments.py index 5d66a40..a56fef7 100644 --- a/sdk/tests/test_arguments.py +++ b/sdk/tests/test_arguments.py @@ -20,7 +20,7 @@ def test_abbreviated_secret_file_flag_is_rejected_without_echo( payload = _usage_error( [ "data", - "migrate", + "copy", "postgresql-main", "--project", str(EXAMPLES), diff --git a/sdk/tests/test_audit.py b/sdk/tests/test_audit.py index 10ee1db..fdad182 100644 --- a/sdk/tests/test_audit.py +++ b/sdk/tests/test_audit.py @@ -550,3 +550,17 @@ def test_no_environment_receipts_means_no_environment_checks() -> None: check.check.startswith("environment.file") for check in server.checks ) + + +def test_audit_reads_the_snapshots_observe_writes_by_default( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + project = tmp_path / "project" + shutil.copytree(EXAMPLES, project) + shutil.copytree(COMPLIANT, project / "tmp" / "observed") + + code = main(["audit", "--project", str(project)]) + + report = json.loads(capsys.readouterr().out)["data"] + assert code == 0 + assert report["status"] == "compliant" diff --git a/sdk/tests/test_commands.py b/sdk/tests/test_commands.py index 5936d77..c91f7d5 100644 --- a/sdk/tests/test_commands.py +++ b/sdk/tests/test_commands.py @@ -11,6 +11,8 @@ ENGINE_COMMANDS, CommandEffect, commands_with_effect, + snake_case, + snake_case_keys, ) from cloudfall_engine.cli import app as engine_app @@ -61,9 +63,9 @@ def test_every_yes_gated_command_is_classified_as_changing_servers() -> None: "deploy", "rollback", "restart", - "data migrate", + "data copy", "migrate", - "operations approve", + "decisions approve", } assert gated <= changing for command in commands_with_effect(CommandEffect.SERVERS): @@ -82,5 +84,29 @@ def test_only_server_changing_commands_carry_a_gate() -> None: def test_invocation_is_the_canonical_project_command() -> None: by_name = {command.name: command for command in COMMANDS} - assert by_name["data migrate"].invocation == "uv run cloudfall data migrate" + assert by_name["data copy"].invocation == "uv run cloudfall data copy" assert by_name["playbook run"].invocation == "uv run cloudfall-engine playbook run" + + +def test_output_field_names_are_snake_case_and_data_keys_keep_their_spelling() -> None: + record = {"apiVersion": "cloudfall/v1", "spec": {"proposedAt": "now"}} + + converted = snake_case_keys( + { + "openProposals": 1, + "byKind": {"AlertRule": 1}, + "environment": {"DATABASE_URL": "x"}, + "servers": [{"serverType": "web", "crm-backend": True}], + "decision": record, + } + ) + + assert converted == { + "open_proposals": 1, + "by_kind": {"AlertRule": 1}, + "environment": {"DATABASE_URL": "x"}, + "servers": [{"server_type": "web", "crm-backend": True}], + "decision": record, + } + assert snake_case("unmatchedObservations") == "unmatched_observations" + assert snake_case("status") == "status" diff --git a/sdk/tests/test_decision.py b/sdk/tests/test_decision.py index 512f138..ceae79f 100644 --- a/sdk/tests/test_decision.py +++ b/sdk/tests/test_decision.py @@ -386,7 +386,7 @@ def test_the_cli_proposes_and_then_lists_the_record( "check-then-approve" ) - listed = main(["operations", "decisions", "--repository", str(repository)]) + listed = main(["decisions", "list", "--repository", str(repository)]) payload = json.loads(capsys.readouterr().out) assert listed == 0 @@ -427,7 +427,7 @@ def test_the_record_is_readable_without_a_catalog( path.unlink() (repository / "operations").rmdir() - exit_code = main(["operations", "decisions", "--repository", str(repository)]) + exit_code = main(["decisions", "list", "--repository", str(repository)]) payload = json.loads(capsys.readouterr().out) assert exit_code == 0 @@ -615,7 +615,7 @@ def test_the_cli_shows_the_proposal_before_it_runs_anything( exit_code = main( [ - "operations", + "decisions", "approve", proposed.decision_id.value, "--repository", diff --git a/sdk/tests/test_inventory.py b/sdk/tests/test_inventory.py index 9d4cdc7..40c2fa3 100644 --- a/sdk/tests/test_inventory.py +++ b/sdk/tests/test_inventory.py @@ -218,5 +218,5 @@ def test_inventory_cli_emits_structured_output( assert exit_code == 0 assert payload["data"]["status"] == "ok" assert payload["data"]["inventory"]["components"][0]["id"] == "crm-backend" - assert payload["data"]["ansible"] is None + assert payload["data"]["ansible"] == {} assert captured.err == "" diff --git a/sdk/tests/test_lifecycle.py b/sdk/tests/test_lifecycle.py index 547e7c7..1a46b34 100644 --- a/sdk/tests/test_lifecycle.py +++ b/sdk/tests/test_lifecycle.py @@ -258,7 +258,7 @@ def test_cli_server_changes_show_a_plan_without_yes( assert payload["data"]["action"] == action assert payload["data"]["component"] == "crm-backend" assert payload["data"]["servers"] - assert "Id(" not in str(payload["data"]["wouldRun"]) + assert "Id(" not in str(payload["data"]["would_run"]) assert "--yes" in str(payload["data"]["instruction"]) assert not (tmp_path / "inventory.json").exists() @@ -286,7 +286,7 @@ def test_cli_deploy_plan_verifies_the_artifact( assert payload["data"]["effect"] == "would_update" assert payload["meta"]["dry_run"] is True assert payload["data"]["release"] == RELEASE - would_run = str(payload["data"]["wouldRun"]) + would_run = str(payload["data"]["would_run"]) assert f"release {RELEASE} of component crm-backend" in would_run assert not (tmp_path / "inventory.json").exists() @@ -308,7 +308,7 @@ def test_cli_data_migration_plan_checks_the_source_url_file( source_url_file = tmp_path / "source.url" argv = [ "data", - "migrate", + "copy", "postgresql-main", "--database", "crm", diff --git a/sdk/tests/test_project.py b/sdk/tests/test_project.py index 2152b58..0e46676 100644 --- a/sdk/tests/test_project.py +++ b/sdk/tests/test_project.py @@ -481,7 +481,7 @@ def test_agent_contract_classifies_every_command_by_effect(tmp_path: Path) -> No "deploy", "rollback", "restart", - "data migrate", + "data copy", "migrate", "backup run", "backup verify", diff --git a/sdk/tests/test_validation.py b/sdk/tests/test_validation.py index 00103c7..fb14c50 100644 --- a/sdk/tests/test_validation.py +++ b/sdk/tests/test_validation.py @@ -278,7 +278,7 @@ def test_cli_emits_structured_success( assert document["data"] == { "status": "ok", "resources": 12, - "byKind": { + "by_kind": { "AlertRule": 1, "OperatorPolicy": 1, "Component": 1,