Skip to content

feat(driver-kubernetes): support ConfigMap and Secret volume attachments in sandbox driver config #5708

Open

Description

@laik

User Story

I run OpenShell on Kubernetes and operate agent sandboxes (coding agents such as opencode against a local OpenAI-compatible inference endpoint) that need operator-managed configuration files and secret material projected into the workload. I directly encountered the missing ConfigMap/Secret volume support: I verified that the only way to project a Secret into a sandbox today is to run an extra sidecar pod sharing a PVC with the sandbox and mount the secret through it. That workaround is not what I want to operate.

Problem Statement

The typed Kubernetes driver config (volumes[] in --driver-config-json / SandboxTemplate.driver_config) only supports persistent_volume_claim. The resource-admission inventory explicitly rejects configMap and secret volume sources. There is no supported way to mount an existing namespace-local ConfigMap or Secret into the agent container, even though these are the standard Kubernetes primitives for configuration and secret-file projection.

Impact / Why This Matters

Users who need config files or secret files in a sandbox must today:

  • operate a sidecar pod plus a shared PVC to copy or mount the material (extra moving parts, secret data transiting shared storage, broader RBAC surface), or
  • bake configuration into the sandbox image (image rebuild for every config change), or
  • use --upload / --env, which puts secret material in the caller's hands instead of keeping it operator-managed.

All of these are insufficient for the common pattern "mount this ConfigMap as my agent's config directory and this Secret as read-only files, while API keys stay in the credential store." It blocks standard Kubernetes configuration workflows for agent workloads and pushes users toward precisely the sidecar topology OpenShell's in-process driver model is meant to avoid.

Proposed Design

Extend the Kubernetes driver-config volume attachment surface so a caller can select a namespace-local ConfigMap or Secret as a volume source, with the same admission guarantees as PVCs:

  • Each volume selects exactly one source: PVC, ConfigMap, or Secret.
  • ConfigMap and Secret volumes are always read-only, resolve only from the sandbox namespace, and support the usual items (key/path/mode), default_mode, and optional projections.
  • Resource admission inventories the new sources and resolves object metadata only; secret values never pass through the gateway. Operator approval labels apply exactly as for PVCs.
  • Gateway RBAC grants metadata get on ConfigMaps/Secrets under the same allow_driver_config conditions as PVCs (sandbox Role in shared mode, workspace ClusterRole otherwise).
  • Stored attachment mounts are revalidated on restart/reattach, excluding driver-managed volumes.

Suggested UX

openshell sandbox create \
  --from registry.example.com/team/agent:1.0 \
  --driver-config-json '{
    "kubernetes": {
      "volumes": [
        {"name": "agent-config", "config_map": {"name": "agent-config"}},
        {"name": "agent-secrets", "secret": {"secret_name": "agent-extra"}}
      ],
      "containers": {
        "agent": {
          "volume_mounts": [
            {"name": "agent-config", "mount_path": "/etc/agent"},
            {"name": "agent-secrets", "mount_path": "/etc/agent-secrets"}
          ]
        }
      }
    }
  }' \
  -- agent

Both objects carry the usual admission labels (openshell.ai/sandbox-attachable, openshell.ai/sandbox-attachable-workspace). API keys and tokens that should be resolved and injected without caller-visible values continue to use credential providers.

Acceptance Criteria

  • --driver-config-json and sandbox templates accept config_map and secret volume sources; a volume with zero or multiple sources is rejected with a clear error.
  • ConfigMap/Secret mounts are read-only; read_only=false mounts are rejected.
  • Objects resolve only from the sandbox namespace; admission requires operator labels and reads metadata only.
  • Key-to-path projections, octal modes (including protobuf Struct integral floats), and optional behave like the Kubernetes API; invalid paths/modes are rejected.
  • Chart RBAC grants metadata get on configmaps/secrets only when caller driver config is enabled, in the correct Role/ClusterRole per workspace mode; unittests cover both.
  • Restart/reattach revalidates caller attachment mounts without rejecting driver-managed volumes.
  • Docs (runtimes.mdx, configuration.mdx) describe the new sources and admission behavior.
  • Unit tests cover parsing, validation, rendering, and admission for the new sources.

Alternatives Considered

  • Sidecar pod + shared PVC (current workaround): works but adds a pod, copies secret material through shared storage, and widens RBAC; operationally heavy for simple file projection.
  • --upload / --env: exposes secret material to the caller and CLI process; wrong boundary for operator-managed secrets.
  • Baking config into the image: rebuilds per change; no runtime operator control.
  • Credential providers: the right tool for injectable API keys (and complementary to this proposal), but they do not project arbitrary config/secret files.
  • PVC-only status quo: forces every config/secret projection through one of the workarounds above.

Agent Investigation

Static review at dc236df1 (main):

  • crates/openshell-driver-kubernetes/src/driver.rs: KubernetesDriverVolumeConfig carries only persistent_volume_claim; validate_kubernetes_driver_volumes requires a claim name; From<&KubernetesDriverVolumeConfig> for Volume emits only a PVC source.
  • crates/openshell-driver-kubernetes/src/resource_admission.rs: the closed-set inventory comment states Secret/ConfigMap volumes are rejected "instead of expanding gateway RBAC for attachments callers cannot request."
  • deploy/helm/openshell/templates/role.yaml / clusterrole.yaml: metadata-only PVC get gated on allow_driver_config (and non-shared mode for the ClusterRole) is the existing pattern to extend.

A prototype implementation with unit tests (driver parsing/validation/rendering/admission) and Helm unittest updates exists on a fork branch and can be submitted as a PR once this issue is accepted. It was validated end-to-end on a kind cluster: an opencode sandbox received its config via ConfigMap mount, extra material via Secret mount, and its inference API key via a Vault-backed provider with L7 egress injection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions