[the-gigi-pplx]

Summary

Merging this lets a test cluster upgrade a stopped sandbox's runtime without starting it, through a test-only admin trigger, and adds a Kind bench and a typed space_smoke scenario that drive the stay-stopped and start-time runtime upgrades end to end. It also carries the long-form documentation for the whole stack.

Terms

• Stay-stopped upgrade: SubstrateAdapter::upgrade_stopped_runtime, the operation the PRs below add. It boots a stopped sandbox's disk under a new execution environment in a runtime-upgrade pod, records the result, and leaves the sandbox stopped with its disk backup owed.
• Test hook: a setting read through pplx_space_common_core::env::test_hook_var, which returns nothing unless the binary was built with the test-hooks feature. Kind's default images have it; release images never do. Test hooks stay out of docs/configuration.md.
• E0, E1, E2: the scenario's names for three compiled execution environments: Firecracker 1.15.1 / Linux 6.1.155 (the bench's default), Firecracker 1.17.0 / Linux 6.1.155 (the upgrade target, with a certified edge from E0), and Firecracker 1.17.0 / Linux 6.18.44 (no certified edge from E0).
• Stranded: a sandbox whose environment is blocked and has no certified edge to the cluster default. The api refuses its upgrade with HTTP 412 before claiming anything.

The problem

The PRs below add the stay-stopped upgrade, its claim, its owed backup, and its handling in the drain and the cleanup loops, but nothing in SPACE calls the operation yet. Unit fixtures exercise each piece against fakes. Nothing proves the pieces work together on a real cluster: a pod that boots the disk under the new environment, a node that records the outcome, an api that settles it and leaves the sandbox stopped and placed, and a later start that runs under the target.

A stock Kind bench can't run that check either. Its workers allow only E0 and the legacy environment and default to E0, and internal-api serves neither the cluster's execution policy nor the provisioning recipe, so no worker ever installs or enforces the target.

The stack's behavior is also spread over its PRs, with no single description of how the pieces fit together.

How this fixes it

• Expose the operation to a test-only trigger (crates/l40_api/api/src/routes/debug/runtime_upgrade_hook.rs). POST /debug/sandbox/{sandbox_id}/upgrade-stopped-runtime on the admin surface runs upgrade_stopped_runtime for a managing-org admin. It's mounted only while the SPACE_ADMIN_RUNTIME_UPGRADE_TEST_HOOK_ENABLED test hook is on. The API reads it through test_hook_var, so a binary built without the test-hooks feature, as release images are, never mounts the route; off, the route answers 404. The scenario declares the hook, so a release-build bench skips it visibly.
• Let a Kind bench install and enforce the target, off by default (Tiltfile, infra/space/kind/scripts/up.sh, the space-node-manager-local chart). Five settings, unset by default: SPACE_KIND_ALLOWED_EXECUTION_ENVIRONMENTS and SPACE_KIND_DEFAULT_EXECUTION_ENVIRONMENT render node-manager's startup policy, and SPACE_KIND_RUNTIME_DESIRED_STATE_ENABLED, SPACE_KIND_RUNTIME_PROVISIONING_RECIPE_ENABLED and SPACE_KIND_RUNTIME_PROVISIONING_PERCENT turn on internal-api's runtime desired state and provisioning recipe. infra/space/kind/README.md gains a "Runtime-upgrade bench" section.
• Drive both upgrades end to end in an explicit-only scenario, native.runtime_upgrade_stopped_and_start (space_smoke/scenarios/native_runtime_upgrade_stopped_and_start.py, space_smoke/runtime_upgrade.py, space_smoke/adapters.py, space_smoke/capabilities.py). It's a manual-tier scenario, so --all and --category native list it as MANUAL; it runs only when named, and on a non-Kind backend it ends with a configuration error before it changes anything. It scopes the settings to the API and controller Deployments, seeds the catalog, sets the policy through the admin API, and restores what it captured. Arms: a stranded stay-stopped upgrade is refused with 412 and places nothing (stranded by a default, E2, that no node certifies an edge to from E0, so the controller's edge rebuild can't undo the setup); a stay-stopped upgrade leaves the sandbox stopped, published under the target, backed up and podless, and a later start runs it under the target with its file intact; a plain start of a stopped sandbox boots it under the target. A bench whose workers don't enforce Firecracker 1.17.0 / Linux 6.1.155 is an infrastructure error, never a pass.
• Document the stack in one place. docs/execution-environments.md gains the long form described under Docs below.

What this leaves alone: the operation itself, its claim, owed backup, drain and cleanup-loop handling, and the node and wait rules all land in the PRs below. This PR only exposes the operation to a test trigger and adds the bench and scenario that exercise it.

Validation

• cargo clippy --workspace --all-features --all-targets -- -D warnings is clean at this head, stacked on its bases.
• routes::debug::runtime_upgrade_hook::tests (the_trigger_is_mounted_only_when_enabled_and_only_for_operators, which pins the substrate to Direct, and only_a_truthy_setting_enables_the_trigger) pass, with the api's stay-stopped suites and archcheck's repository test (136 tests).
• space_smoke's unit tests, run as CI runs them (uv run --group test python -m pytest space_smoke/tests), pass on Linux: 683 passed, 1 skipped; the generated scenario index (python -m space_smoke.inventory --check) lists the scenario; ruff check and ruff format --check pass on the new modules. The harness itself is validated by the Kind run below, as the repository's test policy asks for smoke definitions.
• Kind (a Linux/KVM bench with two Firecracker sandbox workers, test-hooks Rust build), at this PR head. Before rolling, the bench's earlier per-pod sandbox-manager migrations were reversed with their own down scripts, after removing synthetic refused rows whose sandboxes were gone, so the renumbered per-attempt migrations applied fresh. The controller's node_drain.stopped_evacuation counter reported only released and awaiting_durable_disk outcomes, none upgrade_abandon_refused. Every SPACE image built from the stack was certified clean at that commit; the worker DaemonSets (sandbox-manager, node-api, runtime-manager) and the api, cluster-controller and internal-api rolled without restarts, and every sandbox-manager migration applied with checksums matching their files on all four sandbox-manager nodes. With the test hook off, the trigger answered 404.
  • native.runtime_upgrade_stopped_and_start passed on its first run. Negative arm: with E0 blocked and a default it has no certified edge to, the trigger answered HTTP 412 with the stranded message and no pod was created. Stopped arm: the trigger answered upgraded under a fresh UUIDv7 operation carried by the runtime-upgrade pod, the node verified and retired the upgrade, the disk backup was recorded, and the sandbox stayed stopped and placed on its worker (active_node_id and target_node_id unchanged throughout) until a user start ran it under the new environment. Start arm: starting a stopped sandbox booted it under the new environment in a runtime-upgrade pod targeting started. Afterwards the catalog, the cluster policy and the api and cluster-controller env were back to their standing values.
  • Node refusal: with the control-plane catalog claiming an edge the node doesn't certify, the trigger answered HTTP 412 ("refused before it began") on both calls. Each call ran under its own fresh operation, carried by its own pod, with one boot attempt per pod; the node recorded one refused per-attempt outcome for each pod, keyed by the operation and the pod's UID, and no refused attempt, the api deleted each pod, and the sandbox stayed stopped and placed on its worker with nothing published.
  • native.forced_drain_re_drains_re_placed_stopped_sandbox and native.forced_drain_re_drains_restarted_re_placed_stopped_sandbox passed. Each starts with a forced drain that re-places the stopped sandbox onto the other worker as a stopped materialization without starting it; then a re-drain released the destination on the re-placement's backup, or on the newer backup a start and stop published, and the next pod stayed off both drained workers.
  • Not exercised on the bench: the drain's takeover and settling of an expired upgrade, the owed backup's reservation and abandon, a verified boot settled by an abandon, and the stale-pod and abandoned-claim paths that keep a recorded execution. Unit fixtures cover each.

Not covered

• The scenario doesn't exercise the drain's takeover and settling of an expired upgrade, the owed backup's reservation and abandon, a verified boot settled by an abandon, or the stale-pod and abandoned-claim paths that keep a recorded execution. Unit fixtures in the PRs below cover each.
• Nothing outside the test trigger calls the stay-stopped upgrade, and no PR in this stack adds a production caller.

Dependencies

This PR is the top of a stacked chain. Merge each PR below first, in order from the bottom, then retarget this PR to main:

• cluster-controller: evacuate a stopped sandbox an unfinished stay-stopped runtime upgrade left, and test the sweepers against it #1733 (this PR's base: the cluster-controller paths)
• [api] upgrade a stopped sandbox's runtime in place without starting it, and owe its disk backup #1714 (the api's stay-stopped upgrade)
• [api] boot a start's or resume's runtime upgrade in its pod and read its outcome from the node #1708 (booting a start's or resume's upgrade in its pod)
• [sbx-mgr] keep a pod's runtime upgrade finality when its sandbox row is deleted #1707, which is itself the top of a chain: [sbx-mgr] add AbandonRuntimeUpgrade, so no late call for an abandoned operation fences or boots #1756, [sbx-mgr] make a runtime upgrade rollback or refusal final for its attempt #1755, [sbx-mgr] record runtime upgrade outcomes and refused attempts per attempt id in the store #1754, [sbx-mgr] add the per-attempt runtime upgrade fields and AbandonRuntimeUpgrade to the wire #1753, [sbx-mgr] publish the execution a rollback restores while a runtime upgrade fences a sandbox #1752, [sbx-mgr] hold suspends while a runtime upgrade that ends stopped runs #1751 and [sbx-mgr] answer a backup of a sandbox not yet at rest as retryable #1750 (the bottom, based on main)

Severity and risk

Severity: low. Bench and test support, plus documentation.

Risk: low. The trigger is a test hook: release binaries, built without the test-hooks feature, never read SPACE_ADMIN_RUNTIME_UPGRADE_TEST_HOOK_ENABLED and never mount the route, and a test-hook build mounts it only while the hook is on. The Kind settings are unset by default and change nothing for an existing bench. The scenario is explicit-only and refuses to run anywhere but Kind. Roll back by reverting this PR; nothing persists.

Docs

docs/execution-environments.md gains the stack's long form: "Booting an upgrade in its pod" (the node's per-pod records and AbandonRuntimeUpgrade), "Upgrading at resume and start", and "Upgrading a stopped sandbox without starting it" (the claim, the owed backup, the drain's exclusion and the failure paths). Like every test hook, the trigger's setting stays out of docs/configuration.md.

AI assistance: Claude Code drafted the code, tests, docs and this description.

[ac-pplx]

One design problem worth settling before the rebase. A fuller review will follow on the rebased head. Line references are at 6c2085473.

On Kubernetes, the upgrade's boot can't run from the Wake, and the pod boots the old environment instead.

• The pod starts the sandbox. The operation places the restore pod a start places, with checkpoint stopped (crates/libs/common/core/src/k8s.rs:769-772). For a sandbox that is stopped on the node, the shim clears that checkpoint and calls an ordinary Start at container create (shim/space.go:536-557, :1065-1066). That Start carries no upgrade (crates/libs/sbx/mgr/server/src/service/grpc_core.rs:570), so the sandbox boots under the environment it's moving off. #826 adds a stopped-materialization checkpoint for the same reason.
• The Wake can't launch. It names the pod's route identity (kubernetes/support.rs:27-41) and sends netns_path: None (stopped_runtime_upgrade.rs:453). Sandbox-manager refuses a launch for a Kubernetes Wake without a network namespace, with MISSING_RUNTIME_CONTEXT_MARKER (crates/libs/sbx/mgr/server/src/service/activation_support.rs:806-838). Main pins this in direct_wake_resources_preserve_manager_owned_network_context. wake_stopped_runtime_upgrade builds its boot with boot.start_request(...)? (service/runtime_upgrade.rs:1084), so that refusal comes back before the upgrade runs, and before mark_unstarted can see it.
• What follows. The api treats the missing-namespace refusal as a readiness poll and re-sends the Wake (kubernetes/support.rs:853-870). If the shim has booted the sandbox by then, the re-send is refused as "not started ... needs a stopped sandbox" (service/runtime_upgrade.rs:1062-1066). The api reads that as "nothing runs" and deletes the pod under the running guest, without a stop or a backup. If the old environment no longer executes, the shim's Start is refused instead, and nothing boots under the target either. Either way nothing is upgraded.
• Why the tests pass. The api tests run against a scripted node and pod server. The sandbox-manager tests' upgrade_wake helper, "shaped as the control plane sends one" (runtime_upgrade/node_effects/tests.rs:713-726), builds a Direct-kind identity with no pod UID. That takes the launch-allowed branch.

Suggested direction: run the upgrade's boot where Kubernetes launches VMs, in the shim at container create, with the pod's network namespace. That means a pod materialization mode that carries the upgrade and never falls back to an ordinary Start, along the lines of #826's checkpoint. Then show one upgrade on Kind, with the test inserting the catalog row and policy it needs.

Question: #1015's start and resume upgrade goes through the same start.start_request(...)? (crates/libs/sbx/mgr/server/src/service/activation.rs:724-760). Is it affected the same way? I haven't run it.

[the-gigi-pplx]

You're right. On Kubernetes the upgrade's boot can't run from the Wake, and this PR as written never upgrades anything there. I traced each step at the PR head. None of this is confirmed on Kind yet: nothing calls the operation, and #1015's path needs Firecracker 1.17.0 approved on the worker.

• The pod starts the sandbox. The operation's pod is a disk restore with checkpoint stopped (pod_ops/create_request.rs:219-226, common/core/src/k8s.rs:769-772). For a sandbox stopped on the node, stoppedRestartCheckpoint returns nil (shim/space.go:1065-1066), so the shim sends an ordinary Start (space.go:554-558). Start has no upgrade field and goes through start_with_runtime_upgrade(request, None) (grpc_core.rs:566-571), so it boots the row's saved environment.
• The Wake can't launch. It sends netns_path: None with the pod's identity, so netns_path_for_launch returns MISSING_RUNTIME_CONTEXT_MARKER (activation_support.rs:806-838). That ? sits in the match scrutinee (sbx/mgr/server/src/service/runtime_upgrade.rs:1081-1086), so the error returns before the upgrade runs and before mark_unstarted sees it.
• What follows. The api polls on that marker (kubernetes/support.rs:853-860, pod_ops/wake.rs:523-553). Once the shim's Start is admitted, the re-send gets the not-started marker, the operation reads it as "nothing runs", and it deletes the pod with no backup (stopped_runtime_upgrade.rs:642-649, :296-304). One correction: deleting the pod doesn't kill the guest outright. The shim's Kill holds a generation-fenced task until PrepareTeardownSuspend answers (space.go:840-890), so it ends in a teardown suspend under the old environment. Either way, nothing is upgraded. The Wake and the shim's Start also race (the redirect install at space.go:511 runs concurrently with the Start at :560), and neither ordering upgrades.
• Why the tests passed. That matches your read. The sandbox-manager helper upgrade_wake uses a Direct identity with no pod UID (node_effects/tests.rs:714-734), which is the launch-allowed branch.

#1015: yes, it's affected. Its Kubernetes start and resume send netns_path: None with runtime_upgrade (adapter/substrate_adapter.rs:1091, kubernetes/execution.rs:2003). The shim launches the VM at container create without the upgrade, and once the row is started, the Wake's Started arm never reads runtime_upgrade (activation.rs:670-690, :725). With SPACE_RUNTIME_UPGRADE_AT_ACTIVATION_ENABLED on, it would boot the old environment and report success. It's dormant while that setting is off, which is the default. It moves onto the same mode in this PR.

Plan: we're splitting this along your suggestion, and the node half lands first as a separate PR on top of #826:

• a runtime-upgrade pod checkpoint that carries the upgrade;
• a shim mode that runs UpgradeRuntime at container create with the pod's network namespace and never falls back to an ordinary Start;
• a sandbox-manager RPC that reports the operation's outcome;
• Kubernetes Wakes refuse runtime_upgrade.

This PR becomes the api half, stacked on it:

• render that pod mode for stopped upgrades;
• move [api] decide the runtime upgrade at resume and start, and stage pod-less sandboxes on the node #1015's Kubernetes start and resume onto it, and drop runtime_upgrade from those Wakes;
• wait on the node's outcome instead of the Wake;
• keep the claim, record, backup and retire steps.

A Kind scenario proves it end to end: a stopped upgrade, a #1015 start upgrade, and a negative arm. The scenario seeds the catalog rows and policy it needs, and it uses a test-only caller for the operation. The Wake-based boot comes out of this PR, and I'll ask for the fuller review once it's rebased onto the node half.

[coderabbitai]

Pre-merge checks: 5 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Title check		This assessment has not completed. The check will be retried on the next pre-merge check run.

Passed checks: 5 passed

Check name	Status	Explanation
Description Follows The Pr Template		The PR description contains all four required items with real content. 1. "## Summary" section: Present. It states that merging the change lets a test cluster trigger a stopped sandbox's runtime u…
Bug Fixes Name A Regression Test		The check applies only when the pull request fixes a defect in application behavior under crates/, shim/, or ui/. This pull request is not a bug fix. Its title is "[api] trigger a stopped sandbox's ru…
Weakened Safeguards Are Justified		The check passes. I compared HEAD (8de562d) against its parent c4964a9, which is the PR's actual base. The diff touches 19 files. Nothing in it removes, disables, or loosens a test, assertion, lint, C…
Agent Tooling Runs No New Commands		Summary: No agent-tooling content that runs commands or starts servers exists at this PR head. The check passes. Scope of evidence: - The checkout is a single root commit (8de562d, no parent…
Public-Safe Content		Public-safe content check: no failing content found in the PR title, description, or the changed files named in the description. Scope and limits: the checkout is a single root commit with no merge ba…

Full details: Description Follows The Pr Template

Explanation

The PR description contains all four required items with real content. 1. "## Summary" section: Present. It states that merging the change lets a test cluster trigger a stopped sandbox's runtime upgrade through a test-only admin hook. It also adds a Kind bench and a space_smoke scenario. 2. "## Validation" section: Present. It names the commands and tests run, such as cargo clippy, the routes::debug::runtime_upgrade_hook::tests suites, and space_smoke pytest (683 passed, 1 skipped). It also reports the Kind run results. 3. "## Severity and risk" section: Present. It states "Severity: low" and gives a risk statement ("Risk: low. The trigger is a test hook..."). 4. "AI assistance:" line: Present. It reads "AI assistance: Claude Code drafted the code, tests, docs and this description." This names the tool and the parts it drafted. The check judges presence only, so I did not inspect the code. No tool calls were needed.

Full details: Bug Fixes Name A Regression Test

Explanation

The check applies only when the pull request fixes a defect in application behavior under crates/, shim/, or ui/. This pull request is not a bug fix. Its title is "[api] trigger a stopped sandbox's runtime upgrade from a test hook, and smoke it on Kind". The description says it adds a test-only admin trigger (POST /debug/sandbox/{sandbox_id}/upgrade-stopped-runtime, mounted only when a test hook is on), Kind bench settings that are unset by default, a manual-tier space_smoke scenario, and documentation. It also states "Severity: low. Bench and test support, plus documentation." It does not claim to correct a defect in existing behavior. Per the check, a pull request that is not an application bug fix passes. I did not inspect the diff, because the description states the scope clearly and the check's applicability depends only on whether the change is a bug fix.

Full details: Weakened Safeguards Are Justified

Explanation

The check passes. I compared HEAD (8de562d) against its parent c4964a9, which is the PR's actual base. The diff touches 19 files. Nothing in it removes, disables, or loosens a test, assertion, lint, CI step, or repository guard, and it adds no allowlist, baseline, or exception entry. Evidence: - Test and lint suppressions: No added #[allow], #[expect], #[ignore], pytest.mark.skip/xfail, or cfg(not(test)) lines. No changes to .config/nextest.toml, clippy.toml, deny.toml, Cargo.toml lint tables, pyproject.toml, ci/, .github/, or ARCHITECTURE.toml/architecture-baseline.toml. No continue-on-error or SPACE_TEST_DEPS_OPTIONAL additions. - Test deletions: None. The one test file is added (runtime_upgrade_hook/tests.rs), and its assertions are new. - Guard in sandbox.rs: require_managing_org_admin still runs first in the new hook. The only change is widening api_error and require_managing_org_admin from private to pub(super). - Removed lines: The - lines are a Tiltfile set= list refactor, a debug/mod.rs router signature change (the route is now mounted only when the hook is enabled), an admin/mod.rs merge call, and a doc sentence. None of them weakens a guard. - CodeRabbit guidance files: No changes to AGENTS.md, CLAUDE.md, CONTRIBUTING.md, .agents/, or docs/rust/. docs/execution-environments.md changes but is not one of the guidance files, and its single removed line is prose about resume. - Only new suppression: space_smoke/runtime_upgrade.py:235 adds # noqa: BLE001 - attached to the body's error on a new line. It is a lint suppression in new Python code, not a #[allow] in Rust. It carries an inline reason, and the code re-raises the original error with a note attached, so it does not weaken a guard. The main branch is not the PR base, so git diff main shows 1460 unrelated files. I used the parent commit for all findings.

Full details: Agent Tooling Runs No New Commands

Explanation

Summary: No agent-tooling content that runs commands or starts servers exists at this PR head. The check passes. Scope of evidence: - The checkout is a single root commit (8de562d, no parents). git diff main HEAD has no merge base, and it shows 1,460 files of stacked-base drift (for example, plugin versions 0.2.55 on main vs 0.2.53 at HEAD). That drift is not this PR's patch, so I did not use it to decide changed-line causality. - The PR description lists no agent files among its changes. Its files are the debug route runtime_upgrade_hook.rs, the Tiltfile, Kind scripts and chart, the space_smoke scenario modules, and docs/execution-environments.md. - Because the check fails on content that is present, I checked the state of HEAD directly. Any hooks, MCP, LSP, or shell-substitution content the PR added would still be present there. Findings at HEAD: - Inline shell substitutions: none. git grep for !`...` across .agents, .claude, *.md, and docs/execution-environments.md, AGENTS.md, CLAUDE.md found no matches. - Plugin manifests: .agents/plugins/pplx-space/.claude-plugin/plugin.json, .codex-plugin/plugin.json, and plugin.yaml contain only name, description, version, skills path, and interface metadata. None has hooks, mcpServers, lspServers, or any command entry. - SKILL.md frontmatter: every SKILL.md (16 files) has only name and description keys. No allowed-tools, hooks, context, agent, or model keys appear in .agents or .claude. - Marketplace: .agents/plugins/marketplace.json only registers a local plugin path and policy. - Hook matches elsewhere: the hooks and mcp matches in the repo are Cargo test-hooks features, .pre-commit-config.yaml, and .coderabbit.yaml. None is an agent plugin entry. Remaining risk: Scripts under skill directories, such as skills/space-validation/scripts/ensure-docker.sh, are runnable files. The PR description does not touch them. Their presence is pre-existing, and I did not find them in this PR's described changes. Result: No hooks, MCP or LSP servers, frontmatter keys beyond name and description, or inline shell substitutions were added or present in agent tooling files at HEAD.

Full details: Public-Safe Content

Explanation

Public-safe content check: no failing content found in the PR title, description, or the changed files named in the description. Scope and limits: the checkout is a single root commit with no merge base against main, so the PR diff could not be isolated mechanically. I inspected the files the description names as changed: crates/l40_api/api/src/routes/debug/runtime_upgrade_hook.rs, Tiltfile, infra/space/kind/scripts/up.sh, infra/space/kind/README.md, docs/execution-environments.md, and space_smoke/{scenarios/native_runtime_upgrade_stopped_and_start.py, runtime_upgrade.py, adapters.py, capabilities.py}. Findings checked against the criteria: - Credentials, tokens, keys: the hits are variable names with empty defaults (up.sh ~131–167, Tiltfile ~843), a printf template that writes values from environment variables, and token-handling code. No literal secret values were found. The token=secrets.token_hex(16) call in adapters.py generates a value at runtime. - ARNs and cloud account IDs: arn:aws:iam::000000000000:role/kind-storage-vend in infra/space/kind/README.md (~2734) uses an all-zero account ID, which is an obvious placeholder. The ARN in capabilities.py (~2877) is built from an f-string. - Internal hostnames and URLs: README.md references in-cluster Service DNS names (*.svc.cluster.local), RFC1918 ranges, and localhost. These are generic Kind and Kubernetes examples, not internal hosts. The description links only to public perplexityai/SPACE PRs. - Personal-workstation paths: space_smoke/scenarios/native_runtime_upgrade_stopped_and_start.py:17 uses /home/user/data/..., a path inside the sandbox guest. The README.md (~1411) /home/dev is the default Kind home directory. Neither is a personal workstation path. - Production logs, measurements, incidents, or unfixed vulnerabilities: the description reports test counts and pass/fail outcomes from a Kind bench. It contains no raw production output and no incident details. - Private trackers, chats, or documents: none found in the description or the inspected files. Values from the criteria were not quoted in this explanation.