diff --git a/docs/keycloak.md b/docs/keycloak.md new file mode 100644 index 00000000..d71170d9 --- /dev/null +++ b/docs/keycloak.md @@ -0,0 +1,285 @@ +# Red Hat Build of Keycloak (RHBK) Deployment + +## Overview + +Red Hat Build of Keycloak provides OIDC-based identity services for all pattern +components. It deploys a dedicated `ztvp` realm with pre-configured clients, +users, roles, and a SPIFFE Identity Provider that enables workload identity +federation — allowing applications to authenticate using SPIFFE SVIDs instead +of static client secrets. + +**Chart:** `charts/rh-keycloak` (wrapper) + `rhbk` subchart from `oci://quay.io/validatedpatterns` +**Namespace:** `keycloak-system` +**Sync wave:** 35 +**Operator:** `rhbk-operator` (channel: `stable-v26.4`, source: `redhat-marketplace`) + +## Architecture + +```text +┌──────────────────────────────────────────────────────────────┐ +│ keycloak-system │ +│ │ +│ ┌───────────────┐ ┌────────────────┐ ┌───────────────┐ │ +│ │ Keycloak │ │ PostgreSQL │ │ Realm Import │ │ +│ │ Server │◀─▶│ Database │ │ Job │ │ +│ │ (port 8443) │ │ (port 5432) │ │ (one-time) │ │ +│ └──────┬────────┘ └────────────────┘ └───────────────┘ │ +│ │ │ +│ ▼ │ +│ SPIFFE Identity Provider ◀──── SPIRE OIDC Discovery │ +│ (federated JWT auth) Provider (JWKS) │ +└──────────────────────────────────────────────────────────────┘ + │ + ▼ OIDC Clients + ┌─────────┼──────┬────────┬──────────────┐ + │ │ │ │ │ + qtodo-app ACS RHTAS RHTPA-CLI RHTPA-frontend +``` + +### Chart Structure + +The `rh-keycloak` chart is a thin wrapper around the generic `rhbk` Helm chart +(pulled from `oci://quay.io/validatedpatterns`). The wrapper adds: + +- The full `ztvp` realm definition (clients, users, roles, scopes) +- SPIFFE Identity Provider configuration +- ExternalSecret definitions for Vault-sourced credentials +- A PostSync cleanup Job that removes temporary secrets after realm import + +## The ZTVP Realm + +The `ztvp` realm is the central identity configuration. User self-registration +is disabled — all users are pre-provisioned via the realm import. + +### OIDC Clients + +| Client ID | Type | Auth Method | Purpose | +| --- | --- | --- | --- | +| `qtodo-app` | Confidential | `federated-jwt` (SPIFFE) | qtodo application — authenticates via SPIFFE SVID | +| `trusted-artifact-signer` | Public | Direct access grants (ROPC) | RHTAS — automated pipeline signing | +| `acs-central` | Confidential | Client secret | ACS Central — SSO for the security console | +| `rhtpa-cli` | Confidential | Client secret | RHTPA — machine-to-machine SBOM operations | +| `rhtpa-frontend` | Public | Standard + implicit flow | RHTPA — browser-based UI access | + +#### qtodo-app (Federated JWT Authentication) + +This client uses a unique authentication flow — instead of a client secret, it +authenticates using a SPIFFE JWT assertion through the SPIFFE Identity Provider: + +1. The qtodo application obtains a SPIFFE SVID from the SPIRE agent +2. It presents the SVID as a JWT client assertion to Keycloak's token endpoint +3. Keycloak validates the JWT against the SPIFFE Identity Provider's JWKS endpoint +4. Keycloak issues an access token to the application + +Key attributes: + +- `clientAuthenticatorType: federated-jwt` +- `jwt.credential.issuer: spiffe` (links to the SPIFFE IdP alias) +- `serviceAccountsEnabled: true` +- `standardFlowEnabled: true` (also supports browser-based login) + +#### trusted-artifact-signer (RHTAS) + +A public client used for automated pipeline signing via Direct Access Grants. The pipeline fetches credentials from Vault and exchanges them for a token non-interactively. + +- `directAccessGrantsEnabled: true` — used by the Tekton pipeline (non-interactive, credentials from Vault) +- `oauth2.device.authorization.grant.enabled: true` — available for interactive use (e.g., manual `cosign` invocation) +- Protocol mappers: + - `audience-mapper` — adds `trusted-artifact-signer` to the `aud` claim + - `email-mapper` — hardcodes `email_verified: true` for Fulcio identity + +#### acs-central (ACS) + +A confidential client with a secret sourced from Vault: + +- Client secret: `${ACS_CLIENT_SECRET}` (placeholder resolved from Vault) +- Protocol mappers: + - `groups` — maps group membership to the `groups` claim + - `roles` — maps realm roles to the `roles` claim + +#### rhtpa-cli (RHTPA Service Account) + +Machine-to-machine client for the RHTPA CLI and pipeline operations: + +- `serviceAccountsEnabled: true` (no user flows) +- `access.token.lifespan: 300` (5 minutes) +- Custom scopes: `create:document`, `read:document`, `update:document`, `delete:document` + +#### rhtpa-frontend (RHTPA UI) + +Public client for the RHTPA web console: + +- `standardFlowEnabled: true` and `implicitFlowEnabled: true` +- Same document permission scopes as `rhtpa-cli` + +### Custom Client Scopes + +Beyond standard OIDC scopes (`openid`, `basic`, `email`, `profile`, `roles`, +`web-origins`), the realm defines four RHTPA-specific document permission scopes: + +| Scope | Purpose | +| --- | --- | +| `create:document` | Permission to create SBOM documents | +| `read:document` | Permission to read SBOM documents | +| `update:document` | Permission to update SBOM documents | +| `delete:document` | Permission to delete SBOM documents | + +### Realm Roles + +| Role | Description | Assigned To | +| --- | --- | --- | +| `qtodo-admin` | qtodo App Administrator | qtodo-admin user | +| `viewer` | Read-only access | qtodo-user1, rhtas-user, rhtpa-user | +| `create:sbom` | RHTPA SBOM Creator | rhtpa-user | +| `create:document` | RHTPA Document Creator | rhtpa-user | +| `acs-admin` | ACS Administrator | acs-admin user | + +### Pre-Provisioned Users + +| Username | Email | Roles | Password Behavior | +| --- | --- | --- | --- | +| `qtodo-admin` | `qtodo-admin@example.com` | `qtodo-admin` | Temporary — must change on first login | +| `qtodo-user1` | `qtodo-user1@example.com` | `viewer` | Temporary — must change on first login | +| `rhtas-user` | `rhtas-user@example.com` | `viewer` | Permanent | +| `rhtpa-user` | `rhtpa-user@example.com` | `viewer`, `create:sbom`, `create:document` | Permanent | +| `acs-admin` | `acs-admin@example.com` | `acs-admin`, `offline_access` | Permanent | + +All passwords are sourced from Vault via ExternalSecrets (see [Secrets +Integration](#secrets-integration) below). + +## SPIFFE Identity Provider + +The SPIFFE Identity Provider enables workload identity federation — allowing +applications with SPIFFE SVIDs to authenticate to Keycloak without static +credentials. It is configured as a hidden OIDC identity provider (not shown on +the login page) used only for programmatic federated client authentication. + +### Configuration + +| Setting | Value | Purpose | +| --- | --- | --- | +| `alias` | `spiffe` | Internal identifier referenced by clients | +| `providerId` | `oidc` | Standard OIDC provider type | +| `hideOnLogin` | `true` | Not visible to end users on the login page | +| `supportsClientAssertions` | `true` | Accepts JWT client assertions | +| `supportsClientAssertionReuse` | `true` | Same assertion can be reused within its lifetime | +| `validateSignature` | `true` | Validates JWT against JWKS endpoint | +| `useJwksUrl` | `true` | Fetches signing keys from the SPIRE OIDC Discovery Provider | + +The `issuer`, `authorizationUrl`, `tokenUrl`, and `jwksUrl` fields are +auto-generated by the `rhbk` subchart from the cluster domain, pointing to the +SPIRE OIDC Discovery Provider route. These can be overridden in `values-hub.yaml` +if the SPIRE OIDC endpoint uses a non-standard URL. + +### How Federated JWT Authentication Works + +```text + Application Pod Keycloak SPIRE OIDC + (e.g., qtodo) (ztvp realm) Discovery Provider + │ │ │ + │ 1. Obtain SVID from │ │ + │ SPIRE agent │ │ + │ │ │ + │ 2. POST /token │ │ + │ grant_type= │ │ + │ client_credentials │ │ + │ client_assertion= │ │ + │ │ │ + │ ────────────────────────────▶│ │ + │ │ 3. Fetch JWKS │ + │ │ ───────────────────────▶│ + │ │◀────────────────────────│ + │ │ │ + │ │ 4. Validate JWT │ + │ │ signature │ + │ │ │ + │◀─────────────────────────────│ │ + │ 5. Keycloak access token │ │ +``` + +## Secrets Integration + +### Vault-to-Keycloak Secret Flow + +All sensitive values in the realm definition use `${PLACEHOLDER}` syntax. +The `rhbk` subchart resolves these placeholders from Kubernetes Secrets that are +populated by ExternalSecrets from Vault. + +| Placeholder | Source Secret | Source Key | Vault Path | +| --- | --- | --- | --- | +| `QTODO_ADMIN_PASSWORD` | `keycloak-users` | `qtodo-admin-password` | `secret/data/hub/infra/users/keycloak-users` | +| `QTODO_USER1_PASSWORD` | `keycloak-users` | `qtodo-user1-password` | `secret/data/hub/infra/users/keycloak-users` | +| `RHTAS_USER_PASSWORD` | `keycloak-users` | `rhtas-user-password` | `secret/data/hub/infra/users/keycloak-users` | +| `RHTPA_USER_PASSWORD` | `keycloak-users` | `rhtpa-user-password` | `secret/data/hub/infra/users/keycloak-users` | +| `ACS_ADMIN_PASSWORD` | `keycloak-users` | `acs-admin-password` | `secret/data/hub/infra/acs/acs-central` | +| `RHTPA_CLI_SECRET` | `rhtpa-oidc-cli-secret` | `client-secret` | `secret/data/hub/infra/rhtpa/rhtpa-oidc-cli` | +| `ACS_CLIENT_SECRET` | `acs-oidc-client-secret` | `client-secret` | `secret/data/hub/infra/acs/acs-central` | + +### PostSync Secret Cleanup + +The `keycloak-users` ExternalSecret is created as an Argo CD Sync hook and +labeled with `validatedpatterns.io/cleanup: delete`. After the realm import +completes, a PostSync Job deletes these temporary secrets to reduce the +credential exposure window — the passwords are stored in Vault and in the +Keycloak database, not persisted as Kubernetes Secrets. + +## Network Policies + +When enabled via the `values-keycloak-network-policy.yaml` overlay (loaded by +default), comprehensive network segmentation is applied: + +| Policy | Pod | Allowed Traffic | +| --- | --- | --- | +| `keycloak` | Keycloak server | Egress to: DNS, PostgreSQL, JGroups (clustering), K8s API, SPIRE OIDC (443) | +| `postgresql` | PostgreSQL database | Ingress from: Keycloak, realm import. Egress to: DNS only | +| `realmImport` | Realm import Job | Egress to: DNS, PostgreSQL, K8s API, Keycloak API (8443) | +| `operator` | Keycloak operator | Egress to: DNS, K8s API, Keycloak management (9000), Keycloak API (8443) | + +## Component Integration + +| Component | Client Used | Auth Flow | +| --- | --- | --- | +| [qtodo](multi-tier.md) | `qtodo-app` | Federated JWT via SPIFFE SVID | +| [ACS Central](acs-deployment.md) | `acs-central` | OIDC SSO (standard flow + direct access) | +| [RHTAS](supply-chain.md) | `trusted-artifact-signer` | Direct Access Grants | +| [RHTPA](supply-chain.md) | `rhtpa-cli` + `rhtpa-frontend` | Service account (CLI) + standard flow (UI) | + +## Azure Entra ID Alternative + +The pattern supports replacing Keycloak with Azure Entra ID as the OIDC +provider for downstream components. Two feature variants are available: + +- `entra-id` — replaces OIDC for all components (qtodo, RHTAS, RHTPA, supply chain) +- `entra-id-qtodo` — replaces OIDC for qtodo only + +When Entra ID features are enabled, the Keycloak deployment remains active but +is no longer the OIDC provider for the affected components. See the +[Entra ID integration guide](oidc/entraid.md) for configuration details. + +## Accessing Keycloak + +### Admin Console + +The Keycloak admin console is available at: + +```text +https://keycloak.apps./admin/ +``` + +Admin credentials: + +```bash +# Username is 'admin'. Retrieve the password from Vault: +oc extract -n imperative secret/vaultkeys --to=- --keys=vault_data_json 2>/dev/null | \ + jq -r ".root_token" +``` + +### ZTVP Realm Endpoints + +| Endpoint | URL | +| --- | --- | +| Realm metadata | `https://keycloak.apps./realms/ztvp/.well-known/openid-configuration` | +| Token endpoint | `https://keycloak.apps./realms/ztvp/protocol/openid-connect/token` | +| Authorization | `https://keycloak.apps./realms/ztvp/protocol/openid-connect/auth` | +| JWKS | `https://keycloak.apps./realms/ztvp/protocol/openid-connect/certs` | +| ztvp realm console | `https://keycloak.apps./realms/ztvp/account/` | diff --git a/docs/spire.md b/docs/spire.md new file mode 100644 index 00000000..4b003abb --- /dev/null +++ b/docs/spire.md @@ -0,0 +1,292 @@ +# SPIRE / Zero Trust Workload Identity Manager (ZTWIM) + +## Overview + +The Zero Trust Workload Identity Manager (ZTWIM) deploys SPIRE on OpenShift, +providing every workload with a cryptographically verifiable SPIFFE identity. +SPIRE is the foundation of the pattern's zero-trust architecture — it enables +workloads to authenticate to Vault, Keycloak, and RHTAS without static secrets. + +**Chart:** `ztwim` (external, from Validated Patterns Helm repository) +**Namespace:** `zero-trust-workload-identity-manager` +**Sync wave:** 30 +**Operator:** `openshift-zero-trust-workload-identity-manager` (channel: `stable-v1`, source: `redhat-marketplace`) + +## Architecture + +```text + ┌──────────────────────────────────────┐ + │ SPIRE Server │ + │ (StatefulSet) │ + │ │ + │ ┌────────────┐ ┌────────────────┐ │ + │ │ Workload │ │ OIDC Discovery │ │ + │ │ Registrar │ │ Provider │ │ + │ └────────────┘ └───────┬────────┘ │ + └──────────┬───────────────┼───────────┘ + │ │ + Node attestation │ │ JWKS / OIDC + │ │ metadata + │ │ + ┌──────────▼──────┐ ┌─────▼──────────┐ + │ SPIRE Agent │ │ OpenShift │ + │ (DaemonSet) │ │ Route │ + │ │ │ (reencrypt) │ + │ CSI Driver │ └────────────────┘ + │ (csi.spiffe.io)│ │ + └────────┬────────┘ │ + │ │ + ┌────────▼────────┐ │ + │ Workload Pod │ Consumed by: + │ ┌────────────┐ │ - Vault (JWT auth) + │ │ SVID mount │ │ - Keycloak (SPIFFE IdP) + │ │ via CSI │ │ - RHTAS (issuer) + │ └────────────┘ │ - Tekton Chains + └─────────────────┘ +``` + +### Components + +| Component | Type | Purpose | +| --- | --- | --- | +| SPIRE Server | StatefulSet | Certificate authority, workload registration, SVID issuance | +| SPIRE Agent | DaemonSet | Per-node agent, node attestation, workload attestation | +| CSI Driver | DaemonSet | Mounts SPIFFE workload API socket into pods via `csi.spiffe.io` | +| OIDC Discovery Provider | Deployment | Exposes SPIRE as an OIDC-compatible identity provider | +| Controller Manager | Deployment | Processes `ClusterSPIFFEID` CRDs for automatic workload registration | + +## Trust Domain + +The SPIRE trust domain is set to `apps.` (e.g., +`apps.cluster.example.com`). All SPIFFE IDs in the pattern are scoped under +this trust domain. + +## Workload Registration + +### ClusterSPIFFEID (Automatic) + +Most workloads are registered automatically via `ClusterSPIFFEID` custom +resources. The ZTWIM controller manager watches for pods matching the CRD +selectors and creates SPIRE registration entries. SPIFFE IDs follow the +convention: + +```text +spiffe://apps./ns//sa/ +``` + +### ClusterStaticEntry (Manual) + +Some workloads in operator-managed namespaces cannot use `ClusterSPIFFEID` and +require `ClusterStaticEntry` resources — one per cluster node, because each +node runs its own SPIRE agent. The Tekton Chains controller uses this approach +(see [Tekton Chains](tekton-chains.md#spire-workload-registration)). + +## Registered Workload Identities + +| Workload | SPIFFE ID | Registration | Vault Role | +| --- | --- | --- | --- | +| qtodo | `spiffe://apps./ns/qtodo/sa/qtodo` | ClusterSPIFFEID | `qtodo` | +| RHTPA | `spiffe://apps./ns/trusted-profile-analyzer/sa/rhtpa` | ClusterSPIFFEID | `rhtpa` | +| Supply chain pipeline | `spiffe://apps./ns/-hub/sa/pipeline` | ClusterSPIFFEID | `supply-chain` | +| Tekton Chains controller | `spiffe://apps./ns/openshift-pipelines/sa/tekton-chains-controller` | ClusterStaticEntry | (used with Fulcio) | + +## OIDC Discovery Provider + +The SPIRE OIDC Discovery Provider makes SPIRE act as an OIDC-compliant identity +provider. This is critical for integrating SPIFFE identities with systems that +understand OIDC but not SPIFFE natively. + +### Ingress Configuration + +The OIDC Discovery Provider is exposed via an OpenShift reencrypt Route +configured in `values-hub.yaml`: + +```yaml +spire: + oidcDiscoveryProvider: + ingress: + enabled: true + annotations: + route.openshift.io/termination: reencrypt + route.openshift.io/destination-ca-certificate-secret: spire-bundle +``` + +The `spire-bundle` secret provides the destination CA certificate for reencrypt +TLS termination. + +### Endpoints + +| Endpoint | URL | Consumers | +| --- | --- | --- | +| External route | `https://spire-spiffe-oidc-discovery-provider.apps.` | RHTAS Fulcio, Tekton Chains, supply-chain tasks | +| Internal service | `https://spire-spiffe-oidc-discovery-provider.zero-trust-workload-identity-manager.svc.cluster.local` | Vault JWT auth | +| JWKS endpoint | `https://spire-spiffe-oidc-discovery-provider.apps./keys` | Keycloak SPIFFE Identity Provider | + +### Consumers + +The OIDC Discovery Provider is consumed by four different systems: + +1. **Vault** — uses the internal service URL as `oidc_discovery_url` to validate + SPIFFE JWT SVIDs during JWT auth (see [Vault](vault.md#jwt-auth-spiffe)) +2. **Keycloak** — the SPIFFE Identity Provider fetches JWKS from the external + route to validate federated JWT client assertions + (see [Keycloak](keycloak.md#spiffe-identity-provider)) +3. **RHTAS Fulcio** — accepts the OIDC Discovery Provider as a trusted issuer + for keyless signing certificates (SPIFFE type) +4. **Tekton Chains** — the Chains controller presents its SVID to Fulcio, which + validates it against the OIDC Discovery Provider + +## SPIFFE Socket and CSI Driver + +### Standard Workloads + +All standard workloads access SPIFFE SVIDs through the CSI driver: + +```yaml +volumes: + - name: spiffe-workload-api + csi: + driver: csi.spiffe.io + readOnly: true +``` + +| Setting | Value | +| --- | --- | +| CSI driver | `csi.spiffe.io` | +| Mount path | `/spiffe-workload-api` | +| Socket file | `spire-agent.sock` | +| Full socket path | `/spiffe-workload-api/spire-agent.sock` | +| Environment variable | `SPIFFE_ENDPOINT_SOCKET=/spiffe-workload-api/spire-agent.sock` | + +### Confidential Container Workloads (CoCo) + +CoCo peer-pods cannot use the CSI driver because they run in isolated TEE VMs. +Instead, they embed a full SPIRE agent as a sidecar container: + +| Setting | Value | +| --- | --- | +| Agent type | Sidecar container (not DaemonSet) | +| Socket path | `/tmp/spire-agent/public/spire-agent.sock` | +| Node attestor | `x509pop` (instead of `k8s_psat`) | +| Workload attestor | `unix` | + +See [Confidential Containers](CONFIDENTIAL-CONTAINERS.md) for the CoCo-specific +SPIRE architecture. + +## SVID Output Paths + +The `spiffe-helper` sidecar writes SVIDs to a shared `emptyDir` volume: + +| Component | Cert | Key | Bundle | JWT Token | JWT Audience | +| --- | --- | --- | --- | --- | --- | +| qtodo | `/svids/svid.pem` | `/svids/svid_key.pem` | `/svids/svid_bundle.pem` | `/svids/jwt.token` | `https://keycloak.apps./realms/ztvp` | +| RHTPA | `/run/secrets/spiffe/svid.pem` | `/run/secrets/spiffe/svid_key.pem` | `/run/secrets/spiffe/svid_bundle.pem` | `/run/secrets/spiffe/jwt.token` | `rhtpa` | +| Supply chain token refresher | `/svids/svid.pem` | `/svids/svid_key.pem` | `/svids/svid_bundle.pem` | `/svids/jwt.token` | `supply-chain` | +| hello-coco | `/svids/svid.pem` | `/svids/svid_key.pem` | `/svids/svid_bundle.pem` | — | — | + +## spiffe-helper Configuration + +Each component that uses SPIFFE includes a `spiffe-helper` init container +(one-shot mode) and optionally a sidecar (daemon mode) for continuous renewal. +The configuration is stored in a ConfigMap: + +```ini +agent_address = "/spiffe-workload-api/spire-agent.sock" +cert_dir = "/svids" +svid_file_name = "svid.pem" +svid_key_file_name = "svid_key.pem" +svid_bundle_file_name = "svid_bundle.pem" +jwt_svids = [{jwt_audience="", jwt_svid_file_name="jwt.token"}] +jwt_bundle_file_name = "jwt_bundle.json" +``` + +## Integration Points + +### Vault Integration + +SPIRE provides the identity layer for Vault's JWT auth method. Workloads +authenticate to Vault by presenting a SPIFFE JWT SVID. Vault validates the JWT +against the SPIRE OIDC Discovery Provider's JWKS endpoint. + +See [Vault — JWT Auth](vault.md#jwt-auth-spiffe) for role and policy details. + +### Keycloak Integration + +The SPIFFE Identity Provider in Keycloak enables federated JWT client +assertions — workloads can authenticate to Keycloak using SPIFFE SVIDs instead +of client secrets. Keycloak validates the JWT against the SPIRE OIDC Discovery +Provider's JWKS endpoint. + +See [Keycloak — SPIFFE Identity Provider](keycloak.md#spiffe-identity-provider). + +### RHTAS Integration + +RHTAS Fulcio accepts the SPIRE OIDC Discovery Provider as an issuer for keyless +signing. When a workload (e.g., a supply-chain pipeline task or the Tekton +Chains controller) requests a signing certificate, Fulcio verifies the SPIFFE +JWT SVID and issues a short-lived x509 certificate containing the SPIFFE ID. + +The Securesign CR includes the SPIRE issuer: + +```yaml +fulcio: + oidcIssuers: + - ClientID: "sigstore" + Issuer: "https://spire-spiffe-oidc-discovery-provider.apps." + Type: "spiffe" + SPIFFETrustDomain: "apps." +``` + +### Supply Chain Integration + +Five Tekton tasks mount the SPIFFE CSI volume for artifact and image signing: + +- `qtodo-sign-artifact` — signs build artifacts +- `qtodo-verify-artifact` — verifies artifact signatures +- `qtodo-sign-image` — signs container images +- `qtodo-verify-image` — verifies image signatures +- `qtodo-sbom-attestation` — attests SBOMs + +Additionally, the registry token refresher CronJob uses SPIFFE to authenticate +to Vault for writing refreshed OpenShift registry tokens. + +## Network Policies + +When enabled (default via `values-ztwim-network-policy.yaml`): + +| Component | Ingress | Egress | +| --- | --- | --- | +| SPIRE Server | gRPC (8081) from agents, federation (8443) from router, webhook (9443) from K8s API, metrics (9402) | DNS (5353), K8s API (6443) | +| OIDC Discovery Provider | HTTPS (8443) from router | DNS (5353) | +| CSI Driver | — | DNS (5353) | +| Operator | Metrics (8443) | DNS (5353), K8s API (6443) | + +## CoCo-Specific Configuration + +For Confidential Containers, additional SPIRE configuration is required: + +1. **x509pop Node Attestor** — the SPIRE server must be patched to accept + `x509pop` attestation (via `ansible/configure-spire-server-x509pop.yaml`) +2. **Certificate Generation** — CA and agent certificates are generated for each + CoCo workload (via `ansible/generate-certs.yaml`) +3. **CREATE_ONLY_MODE** — the ZTWIM operator is set to `CREATE_ONLY_MODE=true` + to prevent it from reverting the manual x509pop patches + +See [Confidential Containers](CONFIDENTIAL-CONTAINERS.md) for the full CoCo +deployment workflow. + +## Operational Considerations + +### CA Expiration on Cluster Restart + +When using SPIRE's default self-signed CA (24h TTL), a cluster shutdown +exceeding the TTL causes SPIRE agent attestation to break on restart. See +[SPIRE CA Expiration Recovery](spire-ca-expiration-cluster-restart.md) for the +recovery procedure. + +### OIDC Discovery Provider Route + +The OIDC Discovery Provider route must be reachable from both inside the cluster +(Vault, Keycloak) and externally (for verification tooling). The reencrypt route +uses the `spire-bundle` secret as the destination CA — if this secret is missing +or the route is misconfigured, all SPIFFE-based authentication chains will fail. diff --git a/docs/tekton-chains.md b/docs/tekton-chains.md new file mode 100644 index 00000000..15743d10 --- /dev/null +++ b/docs/tekton-chains.md @@ -0,0 +1,154 @@ +# Tekton Chains Configuration + +Tekton Chains adds supply chain security to OpenShift Pipelines by automatically +generating signed provenance attestations for every pipeline run. In the ZTVP, +Chains is configured for **keyless signing via SPIFFE** — the Chains controller +obtains a short-lived x509 certificate from Fulcio using its SPIFFE SVID, +signs the attestation, and logs the signature to Rekor for transparency. + +**Chart:** `charts/tekton-chains` +**Namespace:** `openshift-pipelines` (via `spec.targetNamespace` in the TektonConfig CR) +**Sync wave:** 47 +**Feature flag:** `tekton-chains` (depends on `pipelines` + `rhtas`) +**Status:** Disabled by default — enabled via `gen-feature-variants.py` or by uncommenting in `values-hub.yaml` + +## Architecture + +```text +┌─────────────────────────────────────────────────────────────────┐ +│ OpenShift Pipelines │ +│ │ +│ Pipeline Run ──▶ Chains Controller ──▶ Signed Attestation │ +│ (builds, │ │ (in-toto SLSA) │ +│ signs, │ │ │ +│ deploys) │ ▼ │ +│ │ OCI Registry │ +│ │ (attestation + signature stored │ +│ │ alongside the container image) │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Fulcio │ │ Rekor │ │ TUF │ │ +│ │ (signing │ │ (trans- │ │ (trust │ │ +│ │ cert) │ │ parency │ │ root) │ │ +│ └─────┬────┘ │ log) │ └──────────┘ │ +│ │ └──────────┘ │ +│ ▼ │ +│ SPIRE OIDC │ +│ Discovery Provider │ +│ (SVID identity) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### How Keyless Signing Works + +1. The Chains controller holds a SPIFFE SVID obtained via the CSI driver (`csi.spiffe.io`) +2. When a pipeline run completes, Chains generates an in-toto provenance attestation +3. Chains presents its SVID to Fulcio, which verifies it against the SPIRE OIDC Discovery Provider +4. Fulcio issues a short-lived x509 signing certificate tied to the SPIFFE identity +5. Chains signs the attestation with this certificate +6. The signature is logged to the Rekor transparency log +7. The signed attestation is pushed to the OCI registry alongside the container image + +## How It Works: TektonConfig Patching + +The OpenShift Pipelines operator manages a singleton `TektonConfig` CR named +`config`. The tekton-chains chart patches this CR using **Argo CD +ServerSideApply** rather than creating a separate resource, because the operator +owns the CR lifecycle. + +The Argo CD sync options on the TektonConfig resource are: + +- `ServerSideApply=true` — merge the Chains fields into the operator-managed CR +- `Delete=false` — Argo CD will never delete the CR (the operator owns it) +- `SkipDryRunOnMissingResource=true` — handles the case where the CRD does not yet exist + +### Chains Configuration Applied + +The patch sets the following in `spec.chain`: + +| Parameter | Value | Purpose | +| --- | --- | --- | +| `artifacts.taskrun.format` | `in-toto` | SLSA provenance format for task runs | +| `artifacts.taskrun.storage` | `oci` | Store attestations in the OCI registry | +| `artifacts.pipelinerun.format` | `in-toto` | SLSA provenance format for pipeline runs | +| `artifacts.pipelinerun.storage` | `oci` | Store attestations in the OCI registry | +| `artifacts.pipelinerun.enable-deep-inspection` | `true` | Inspect individual task results within the pipeline | +| `artifacts.oci.storage` | `oci` | OCI artifact storage | +| `transparency.enabled` | `true` | Log signatures to Rekor | +| `transparency.url` | `https://rekor-server-trusted-artifact-signer.` | RHTAS Rekor instance | +| `signers.x509.fulcio.enabled` | `true` | Enable Fulcio-based keyless signing | +| `signers.x509.fulcio.address` | `https://fulcio-server-trusted-artifact-signer.` | RHTAS Fulcio instance | +| `signers.x509.fulcio.issuer` | `https://spire-spiffe-oidc-discovery-provider.` | SPIRE OIDC Discovery Provider | +| `signers.x509.fulcio.provider` | `spiffe` | Use SPIFFE SVIDs (not OIDC tokens) | +| `signers.x509.tuf.mirror.url` | `https://tuf-trusted-artifact-signer.` | RHTAS TUF mirror for trust root | + +### Controller Deployment Overrides + +The patch also uses the TektonConfig `options.deployments` mechanism to inject +volumes into the `tekton-chains-controller` pod without directly modifying the +operator-managed deployment: + +| Volume | Type | Mount Path | Purpose | +| --- | --- | --- | --- | +| `spiffe-workload-api` | CSI (`csi.spiffe.io`, readOnly) | `/spiffe-workload-api` | SPIFFE SVID socket | +| `tuf` | emptyDir | `/tuf` | TUF trust root (populated at runtime) | + +Environment variables set on the controller: + +| Variable | Value | +| --- | --- | +| `SPIFFE_ENDPOINT_SOCKET` | `/spiffe-workload-api/spire-agent.sock` | +| `TUF_ROOT` | `/tuf` | + +## SPIRE Workload Registration + +The Chains controller needs a SPIFFE identity to authenticate with Fulcio. +Unlike application workloads that use `ClusterSPIFFEID` CRDs, the Chains +controller requires `ClusterStaticEntry` resources — one per cluster node — +because the standard CRD-based registration does not cover operator-managed +deployments in system namespaces. + +A **PostSync Job** (`tekton-chains-spire-entries`) handles this automatically: + +1. Reads the SPIRE trust domain, cluster name, and class name from the + `spire-controller-manager` ConfigMap in `zero-trust-workload-identity-manager` +2. Enumerates all cluster nodes +3. Creates a `ClusterStaticEntry` per node: + - **SPIFFE ID:** `spiffe:///ns/openshift-pipelines/sa/tekton-chains-controller` + - **Parent ID:** `spiffe:///spire/agent/k8s_psat//` + - **Selectors:** `k8s:ns:openshift-pipelines`, `k8s:sa:tekton-chains-controller` +4. Cleans up stale entries for nodes that no longer exist + +The Job runs at sync wave 48 with `argocd.argoproj.io/hook: PostSync`, ensuring +the TektonConfig CR has been applied before registration begins. The +`hook-delete-policy: BeforeHookCreation` annotation ensures old Job resources +are cleaned up on re-sync. + +## Pipeline Integration + +The [supply-chain pipeline](supply-chain.md) emits four well-known results that +Chains uses to construct the provenance attestation: + +| Result Name | Source Task | Purpose | +| --- | --- | --- | +| `CHAINS-GIT_URL` | `qtodo-clone-repository` | Source repository URL (materials) | +| `CHAINS-GIT_COMMIT` | `qtodo-clone-repository` | Source commit SHA (materials) | +| `IMAGE_URL` | `qtodo-build-image` | Built image URL (subject) | +| `IMAGE_DIGEST` | `qtodo-build-image` | Built image digest (subject) | + +The `CHAINS-` prefix is a Tekton Chains convention — Chains uses these results +to populate the `materials` section of the in-toto SLSA provenance attestation. +`IMAGE_URL` and `IMAGE_DIGEST` identify the artifact the attestation describes, +so Chains pushes the attestation and signature to the same OCI repository. + +## Deployment Sequence + +| Order | Component | Sync Wave | What Happens | +| --- | --- | --- | --- | +| 1 | SPIRE / ZTWIM | 30 | SPIRE server, agents, and CSI driver deployed | +| 2 | RHTAS | 46 | Fulcio, Rekor, and TUF services deployed | +| 3 | OpenShift Pipelines | (subscription) | Operator installs, creates `TektonConfig` CR | +| 4 | tekton-chains | 47 | ServerSideApply patches `TektonConfig` with Chains config | +| 5 | SPIRE static entries | 48 (PostSync) | Registers the Chains controller SPIFFE identity | +| 6 | supply-chain | 48 | Pipeline, tasks, and triggers deployed | diff --git a/docs/vault.md b/docs/vault.md new file mode 100644 index 00000000..ca7c3f29 --- /dev/null +++ b/docs/vault.md @@ -0,0 +1,333 @@ +# HashiCorp Vault + +## Overview + +HashiCorp Vault is the centralized secrets management engine for the entire +pattern. Every component retrieves credentials from Vault — either through the +External Secrets Operator (ESO) for Kubernetes-native secret sync, or through +direct SPIFFE JWT authentication for workloads that need dynamic credentials. + +**Chart:** `hashicorp-vault` (external, from Validated Patterns Helm repository) +**Namespace:** `vault` +**Sync wave:** 25 +**Service endpoint:** `https://vault.vault.svc.cluster.local:8200` + +## Architecture + +```text + ┌─────────────────────────┐ + │ Validated Patterns │ + │ Framework │ + │ - Initialize Vault │ + │ - Unseal │ + │ - Configure k8s auth │ + │ - Seed secrets │ + └───────────┬─────────────┘ + │ + ▼ + ┌──────────────┐ + │ Vault │ + │ (port 8200) │ + └──┬───────┬───┘ + │ │ + k8s Auth │ │ JWT Auth + (ServiceAccount) │ │ (SPIFFE SVID) + │ │ + ┌───────────────┘ └────────────────┐ + ▼ ▼ + ┌─────────────────────┐ ┌──────────────────────┐ + │ External Secrets │ │ SPIFFE Vault Client │ + │ Operator (ESO) │ │ (sidecar pattern) │ + │ │ │ │ + │ ClusterSecretStore │ │ SPIRE CSI ──▶ JWT │ + │ "vault-backend" │ │ ──▶ Vault login │ + │ │ │ ──▶ read secret │ + │ Syncs to k8s │ │ ──▶ write to file │ + │ Secrets │ │ ──▶ renew lease │ + └────────┬────────────┘ └──────────┬───────────┘ + │ │ + ▼ ▼ + All charts via qtodo, RHTPA, supply-chain + ExternalSecret CRs +``` + +## Authentication Methods + +Vault uses two authentication methods, each serving a different purpose: + +### Kubernetes Auth (ESO) + +Configured automatically by the Validated Patterns framework Ansible playbooks. +Used by the External Secrets Operator's `ClusterSecretStore` named +`vault-backend`. Every `ExternalSecret` CR across all charts references this +store to sync Vault secrets into Kubernetes Secrets. + +This is the default path for secrets that need to exist as Kubernetes Secrets +(operator configurations, database passwords, OIDC client secrets). + +### JWT Auth (SPIFFE) + +Configured via `overrides/values-vault-jwt.yaml`. This method validates SPIFFE +JWT SVIDs against the SPIRE OIDC Discovery Provider, enabling workloads to +authenticate directly to Vault using their workload identity. + +The OIDC discovery URL points to the cluster-internal SPIRE endpoint: +`https://spire-spiffe-oidc-discovery-provider.zero-trust-workload-identity-manager.svc.cluster.local` + +#### JWT Roles + +Each workload that authenticates directly to Vault has a dedicated role that +binds a specific SPIFFE ID to a set of policies: + +| Role | SPIFFE Subject | Audience | Policies | +| --- | --- | --- | --- | +| `qtodo` | `spiffe://apps./ns/qtodo/sa/qtodo` | `https://keycloak.apps./realms/ztvp` | `apps-qtodo-jwt-secret` | +| `rhtpa` | `spiffe://apps./ns/trusted-profile-analyzer/sa/rhtpa` | `rhtpa` | `hub-infra-rhtpa-jwt-secret` | +| `supply-chain` | `spiffe://apps./ns/-hub/sa/pipeline` | `supply-chain` | `hub-supply-chain-jwt-secret` | + +The `qtodo` role is active by default. The `rhtpa` and `supply-chain` roles are +activated when their respective features are enabled. + +#### JWT Policies + +| Policy | Vault Path | Capabilities | Used By | +| --- | --- | --- | --- | +| `apps-qtodo-jwt-secret` | `secret/data/apps/qtodo/*` | read | qtodo | +| `hub-infra-rhtpa-jwt-secret` | `secret/data/hub/infra/rhtpa/*` | read | RHTPA | +| `hub-supply-chain-jwt-secret` | `secret/data/hub/infra/quay/*` | read | supply-chain | +| | `secret/data/hub/infra/registry/*` | read, create, update | supply-chain (token refresh) | +| | `secret/data/hub/infra/rhtpa/rhtpa-oidc-cli` | read | supply-chain | +| | `secret/data/hub/supply-chain/*` | read | supply-chain | + +## Secret Path Hierarchy + +Secrets are organized in a least-privilege, path-based hierarchy: + +```text +secret/data/ +├── global/ # Shared secrets +│ └── config-demo # Demo secret +├── apps/ +│ └── qtodo/ # Application-scoped secrets +│ ├── qtodo-db # DB credentials (admin-password, db-password) +│ ├── qtodo-truststore # Java truststore password +│ ├── qtodo-oidc-client # OIDC client secret (optional) +│ └── qtodo-oidc-entraid # Entra ID client secret (optional) +├── hub/ +│ ├── infra/ # Infrastructure secrets (components) +│ │ ├── keycloak/ +│ │ │ └── keycloak # Keycloak admin + DB passwords +│ │ ├── acs/ +│ │ │ └── acs-central # ACS admin password + OIDC client secret +│ │ ├── rhtpa/ +│ │ │ ├── rhtpa-db # RHTPA PostgreSQL password +│ │ │ └── rhtpa-oidc-cli # RHTPA CLI client secret +│ │ ├── rhtas/ +│ │ │ └── rhtas-oidc-client-secret # RHTAS OIDC client secret +│ │ ├── quay/ +│ │ │ └── quay-users # Built-in Quay user password +│ │ ├── registry/ +│ │ │ └── registry-user # BYO/embedded registry credentials +│ │ └── users/ +│ │ └── keycloak-users # All user passwords for realm import +│ ├── supply-chain/ +│ │ └── git-credentials # Git HTTPS/SSH credentials +│ └── kubeconfig-spoke-* # Spoke cluster kubeconfigs +└── pushsecrets/ + └── # CoCo agent certificates (via PushSecret) +``` + +### Path Convention + +- `apps/` — application-scoped secrets, accessed by workloads directly +- `hub/infra/` — infrastructure secrets, accessed by operators and shared services +- `hub/supply-chain/` — pipeline-specific secrets +- `hub/` (top-level) — Hub cluster level secrets (kubeconfigs, attestation) + +## SPIFFE Vault Client Sidecar Pattern + +The pattern implements a sidecar-based authentication flow for workloads that +need dynamic secrets from Vault. This avoids storing secrets as Kubernetes +Secrets, keeping them in-memory only. + +### How It Works + +```text +Pod Lifecycle: + + 1. init-spiffe-helper Obtains initial SPIFFE X.509 + JWT SVIDs + │ from SPIRE agent via CSI driver + ▼ + 2. init-spiffe-vault Authenticates to Vault with JWT SVID, + │ fetches DB password, writes credentials file + ▼ + 3. Application starts Reads credentials from shared volume + │ + ▼ + 4. spiffe-helper sidecar Continuously renews SVIDs + │ + ▼ + 5. vault-client sidecar Renews Vault lease at 50% of duration, + re-authenticates on failure +``` + +### Implementation + +The sidecar is a Python script (`spiffe-vault-client.py`) that: + +1. Reads the SPIFFE JWT from `/svids/jwt.token` +2. Authenticates to Vault via `POST /v1/auth/jwt/login` with the JWT and role name +3. Reads the target secret from the configured Vault path +4. Writes the credentials as a properties file to `/run/secrets/db-credentials/` +5. In daemon mode, renews the Vault token at 50% of its lease duration + +This pattern is used by: + +- **qtodo** — reads DB password from `secret/data/apps/qtodo/qtodo-db` +- **RHTPA** — reads DB password from `secret/data/hub/infra/rhtpa/rhtpa-db` +- **Registry token refresher** — reads and **writes** registry tokens to `secret/data/hub/infra/registry/registry-user` + +### Volumes + +| Volume | Type | Purpose | +| --- | --- | --- | +| `spiffe-workload-api` | CSI (`csi.spiffe.io`) | SPIRE agent socket | +| `svids` | emptyDir | Shared SVID storage between spiffe-helper and vault-client | +| `db-credentials` | emptyDir | Credentials file consumed by the application | +| `spiffe-vault-client` | ConfigMap | The Python script | +| `ztvp-trusted-ca` | ConfigMap | CA bundle for TLS verification | + +## ExternalSecret Pattern + +Every chart that needs Vault secrets uses `ExternalSecret` resources referencing +the `vault-backend` `ClusterSecretStore`: + +```yaml +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +spec: + secretStoreRef: + name: vault-backend + kind: ClusterSecretStore + target: + name: + data: + - secretKey: + remoteRef: + key: secret/data/ + property: +``` + +### ExternalSecrets by Chart + +| Chart | ExternalSecret | Vault Path | Purpose | +| --- | --- | --- | --- | +| qtodo | `qtodo-db-secret` | `apps/qtodo/qtodo-db` | PostgreSQL credentials | +| qtodo | `qtodo-truststore-secret` | `apps/qtodo/qtodo-truststore` | Java truststore password | +| rh-keycloak | `keycloak-users` | `hub/infra/users/keycloak-users` + `hub/infra/acs/acs-central` | Realm user passwords | +| rh-keycloak | `acs-oidc-client-secret` | `hub/infra/acs/acs-central` | ACS OIDC client secret | +| rh-keycloak | `rhtpa-oidc-cli-secret` | `hub/infra/rhtpa/rhtpa-oidc-cli` | RHTPA CLI client secret | +| acs-central | `central-htpasswd` | `hub/infra/acs/acs-central` | ACS admin password | +| acs-central | `keycloak-acs-client` | `hub/infra/acs/acs-central` | ACS Keycloak client secret | +| rhtpa-operator | `rhtpa-db-secret` | `hub/infra/rhtpa/rhtpa-db` | RHTPA DB password | +| rhtpa-operator | `rhtpa-oidc-cli-secret` | `hub/infra/rhtpa/rhtpa-oidc-cli` | RHTPA CLI secret | +| supply-chain | `qtodo-registry-auth` | `hub/infra/quay/*` or `hub/infra/registry/*` | Registry credentials | +| supply-chain | `qtodo-rhtpa-cli-password` | `hub/infra/rhtpa/rhtpa-oidc-cli` | RHTPA CLI secret | +| supply-chain | `qtodo-git-credentials` | `hub/supply-chain/git-credentials` | Git credentials | +| acm-managed-clusters | `kubeconfig-` | `hub/kubeconfig-` | Spoke kubeconfigs | + +## Initialization and Seeding + +Vault initialization, unsealing, and secret seeding are handled automatically +by the Validated Patterns framework: + +1. The VP operator detects Vault pods are ready +2. Ansible playbooks run as imperative Jobs to: + - Initialize Vault (generate unseal keys and root token) + - Store unseal keys in a Kubernetes Secret (`vaultkeys` in `imperative` namespace) + - Unseal Vault + - Configure Kubernetes auth method + - Seed secrets from `~/values-secret-layered-zero-trust.yaml` +3. The JWT auth method is configured via `overrides/values-vault-jwt.yaml` + +The root token and unseal keys are stored in: + +```bash +oc extract -n imperative secret/vaultkeys --to=- --keys=vault_data_json 2>/dev/null | jq -r ".root_token" +``` + +## Network Policies + +When enabled (default via `values-vault-network-policy.yaml`), Vault has +fine-grained network segmentation: + +**Ingress to Vault (port 8200):** + +| Source | Purpose | +| --- | --- | +| OpenShift router | Vault UI/API route | +| `qtodo` namespace | SPIFFE vault client sidecar | +| `registry-token-refresher` pods | Supply chain token refresh | +| Vault pods (port 8201) | HA replication | + +**Egress from Vault:** + +| Destination | Purpose | +| --- | --- | +| `openshift-dns` (port 5353) | DNS resolution | +| SPIRE OIDC Discovery Provider (ports 443/8443) | JWKS fetch for JWT auth validation | +| Vault pods (port 8201) | HA replication | +| Kubernetes API (ports 443/6443) | TokenReview for ESO service account validation | + +## Accessing Vault + +### CLI Access + +```bash +# Get the root token +VAULT_TOKEN=$(oc extract -n imperative secret/vaultkeys \ + --to=- --keys=vault_data_json 2>/dev/null | jq -r ".root_token") + +# Read a secret +oc exec -n vault vault-0 -- env VAULT_TOKEN="$VAULT_TOKEN" \ + vault kv get secret/hub/infra/users/keycloak-users + +# List secrets at a path +oc exec -n vault vault-0 -- env VAULT_TOKEN="$VAULT_TOKEN" \ + vault kv list secret/hub/infra +``` + +### Web UI + +```text +https://vault-vault.apps. +``` + +Login with the root token retrieved above. + +### Reading ESO-Synced Secrets + +For secrets already synced to Kubernetes by ESO: + +```bash +oc get secret -n -o jsonpath='{.data.}' | base64 -d +``` + +## Secrets Template + +The `values-secret.yaml.template` file defines all secrets that need to be +seeded into Vault. Copy it to `~/values-secret-layered-zero-trust.yaml` and +customize before deploying: + +```bash +cp values-secret.yaml.template ~/values-secret-layered-zero-trust.yaml +# Edit the file to set any manual values (most are auto-generated) +``` + +Password generation policies: + +| Policy | Length | Used For | +| --- | --- | --- | +| `basicPolicy` | 10 chars | Standard passwords | +| `advancedPolicy` | 20 chars | Admin passwords | +| `alphaNumericPolicy` | 32 chars | Passwords where the use of special characters can cause errors (e.g., RHBK) |