DEVOPS-3406 - add separate ingress templates for API and web services

docs/deploy/gcp/prompt.md

@@ -39,7 +39,7 @@ Agent-created resource names and defaults (change if required):
 - Cloud SQL instance name: GCP Cloud SQL resource to create, default `openwork-ee-mysql`.
 - Cloud SQL database name: MySQL database to create inside Cloud SQL, default `openwork_den`.
 - Cloud SQL user: MySQL user to create for OpenWork, default `openwork`.
-- Reserved global address name: GCP resource name for the static global IPv4 address used by the HTTPS load balancer, default `openwork-ee-ip`. This is not the IP address; the agent creates the address and reports the allocated IP.
+- Reserved global address names: GCP resource names for the two static global IPv4 addresses used by the web and API HTTPS load balancers (GKE gives each Ingress its own load balancer, so they cannot share one address). Defaults `openwork-ee-web-ip` and `openwork-ee-api-ip`. These are not the IP addresses; the agent creates the addresses and reports both allocated IPs.
 
 Operating rules:
 
@@ -67,7 +67,7 @@ Provision and deploy according to the GCP runbook:
 - Create or reuse private services access for the target VPC.
 - Create a regional GKE Autopilot cluster using the requested/default cluster name.
 - Create Cloud SQL for MySQL 8 with private IP, backups enabled, and public IP disabled unless the docs require otherwise.
-- Reserve a global IPv4 address.
+- Reserve two global IPv4 addresses, one for the web hostname and one for the API hostname.
 - Apply the documented GKE `BackendConfig` and `ManagedCertificate` resources.
 - Create the namespace and runtime Secret.
 - Install the Helm release using GCP ingress values and the requested/default release name.
@@ -76,7 +76,7 @@ Provision and deploy according to the GCP runbook:
 
 DNS handoff:
 
-- After the reserved global IP exists, pause and tell me the exact DNS records to create for `{{WEB_HOSTNAME}}` and `{{API_HOSTNAME}}`.
+- After both reserved global IPs exist, pause and tell me the exact DNS records to create for `{{WEB_HOSTNAME}}` (pointing at the web address) and `{{API_HOSTNAME}}` (pointing at the API address).
 - Wait for my confirmation that DNS is updated.
 - Verify public DNS resolution yourself before continuing.
 - Do not claim HTTPS is ready until the managed certificate is active and `openssl` or equivalent external checks show trusted certificates for both hostnames.
@@ -102,5 +102,5 @@ Track any reusable documentation gaps, failed commands, unclear values, required
 
 Final report:
 
-Report `Passed`, `Incomplete`, or `Failed`, plus chart/app version, resources created, reserved IP, DNS records, Cloud SQL tier/region/private-IP/backup state, GKE type/region, Helm revision/status, migration result, pod readiness, backend health, certificate status, external readiness checks, administrator setup result, replay/signup rejection result, sign-out/sign-in result, browser handoffs completed by the operator, non-secret commands run, deviations from docs, documentation PRs, ongoing cost items, and exact cleanup commands.
+Report `Passed`, `Incomplete`, or `Failed`, plus chart/app version, resources created, reserved IPs (web and API), DNS records, Cloud SQL tier/region/private-IP/backup state, GKE type/region, Helm revision/status, migration result, pod readiness, backend health, certificate status, external readiness checks, administrator setup result, replay/signup rejection result, sign-out/sign-in result, browser handoffs completed by the operator, non-secret commands run, deviations from docs, documentation PRs, ongoing cost items, and exact cleanup commands.
 ```

docs/gcp-gke-helm.md

@@ -5,8 +5,9 @@ Related: `packaging/helm/openwork-ee`, `packaging/helm/openwork-ee/examples/valu
 
 This is the recommended Google Cloud path for a first production-like OpenWork
 EE self-host install. Use Helm on GKE Autopilot with Cloud SQL for MySQL. For
-web/API exposure, use GKE Ingress with Google-managed certificates, a reserved
-global IP address, and explicit backend health checks.
+web/API exposure, use two GKE Ingresses (one per host) with a shared
+Google-managed certificate, one reserved global IP address per Ingress, and
+explicit backend health checks.
 
 Google recommends Gateway API for new L7 traffic management, and GKE Ingress is
 in maintenance mode. The current OpenWork chart emits Ingress resources, so GKE
@@ -25,13 +26,14 @@ balancing, host routing, managed certificates, and backend health checks.
 - optional inference service, disabled by default
 - one Cloud SQL for MySQL database
 - one single-org OpenWork deployment
-- one external GKE Ingress backed by a Google Cloud Application Load Balancer
+- one external GKE Ingress per host (web and API), each backed by its own
+  Google Cloud Application Load Balancer
 - one Google-managed certificate covering web and API hosts
 
 Google Cloud owns the GKE cluster, Autopilot compute lifecycle, VPC networking,
 Cloud Load Balancing, managed certificates, Cloud SQL, IAM, and firewall rules.
 The OpenWork Helm chart owns OpenWork Deployments, Services, ConfigMaps,
-Secrets, health probes, the optional Ingress, and the database migration Job.
+Secrets, health probes, the optional Ingresses, and the database migration Job.
 The `BackendConfig` and `ManagedCertificate` resources in this guide are
 GKE-specific platform resources applied alongside the chart.
 
@@ -197,15 +199,23 @@ kubectl run mysql-client \
     --execute "select 1"
 ```
 
-## 3. Reserve a global IP and create GKE resources
+## 3. Reserve global IPs and create GKE resources
 
-Reserve a global IP address for the HTTPS load balancer:
+GKE gives each Ingress its own load balancer, so the web and API Ingresses
+each need their own reserved global IP; reusing one static IP name for both
+means only one Ingress can bind its forwarding rule and the other never
+reconciles. Reserve one address per host:
 
 ```bash
-gcloud compute addresses create openwork-ee-ip \
+gcloud compute addresses create openwork-ee-web-ip \
+  --global
+gcloud compute addresses create openwork-ee-api-ip \
   --global
 
-gcloud compute addresses describe openwork-ee-ip \
+gcloud compute addresses describe openwork-ee-web-ip \
+  --global \
+  --format='value(address)'
+gcloud compute addresses describe openwork-ee-api-ip \
   --global \
   --format='value(address)'
 ```
@@ -407,24 +417,27 @@ migrations:
   backoffLimit: 2
 ```
 
-## 7. Point DNS at the global load balancer IP
+## 7. Point DNS at the global load balancer IPs
 
-Get the reserved IP address:
+Get the reserved IP addresses:
 
 ```bash
-gcloud compute addresses describe openwork-ee-ip \
+gcloud compute addresses describe openwork-ee-web-ip \
+  --global \
+  --format='value(address)'
+gcloud compute addresses describe openwork-ee-api-ip \
   --global \
   --format='value(address)'
 ```
 
 Create DNS records:
 
-- `openwork.example.com` -> the reserved global IP address.
-- `api.openwork.example.com` -> the reserved global IP address.
+- `openwork.example.com` -> the reserved `openwork-ee-web-ip` address.
+- `api.openwork.example.com` -> the reserved `openwork-ee-api-ip` address.
 
-GKE can take several minutes to provision the load balancer. Google-managed
-certificates can take up to an hour to become active after DNS points at the
-load balancer.
+GKE can take several minutes to provision each load balancer. Google-managed
+certificates can take up to an hour to become active after DNS points at both
+load balancers.
 
 Check status:
 
@@ -542,7 +555,7 @@ single organization. Password sign-in for that organization is rejected.
 | Ingress does not reconcile | HTTP load balancing add-on is disabled or Ingress annotation is wrong | Keep HTTP load balancing enabled and use `kubernetes.io/ingress.class: gce` |
 | Backends are unhealthy | GKE load balancer health checks do not match OpenWork readiness endpoints | Apply the `BackendConfig` resources and keep the service annotations from the starter values |
 | Ingress events report `TimeoutSec should be less than checkIntervalSec` | The backend health-check timeout is greater than or equal to its effective interval | Set `checkIntervalSec: 15` and `timeoutSec: 5` on both `BackendConfig` resources |
-| Managed certificate is not `Active` | DNS does not point at the load balancer or provisioning is still running | Point both hosts at the reserved global IP and wait; check `kubectl describe managedcertificate` |
+| Managed certificate is not `Active` | DNS does not point at the corresponding load balancer or provisioning is still running | Point each host at its own reserved global IP and wait; check `kubectl describe managedcertificate` |
 | Migration Job fails to connect to MySQL | Private services access, VPC, credentials, IP, or TLS mode are wrong | Test from `mysql-client`, confirm the private IP, and confirm GKE and Cloud SQL share VPC reachability |
 | Migration Job logs show `self-signed certificate in certificate chain` | Strict certificate verification is being used without the cloud MySQL CA bundle | Use `?sslaccept=accept` for the smoke path or mount/configure the CA bundle before strict verification |
 | `ImagePullBackOff` from GHCR | Private image or missing pull token | Add `imagePullSecrets` |
@@ -556,7 +569,8 @@ For a disposable test:
 
 ```bash
 helm uninstall openwork-ee -n openwork-ee
-gcloud compute addresses delete openwork-ee-ip --global
+gcloud compute addresses delete openwork-ee-web-ip --global
+gcloud compute addresses delete openwork-ee-api-ip --global
 gcloud container clusters delete "$GKE_CLUSTER" --location "$GCP_REGION"
 gcloud sql instances delete "$SQL_INSTANCE"
 ```

packages/docs/self-host/deploy-to-your-cloud/google-cloud.mdx

@@ -3,7 +3,7 @@ title: "Deploy on Google Cloud"
 description: "Run OpenWork on GKE Autopilot with Cloud SQL for MySQL."
 ---
 
-The recommended Google Cloud path is Helm on a regional GKE Autopilot cluster with Cloud SQL for MySQL. Use GKE Ingress, a reserved global IP address, Google-managed certificates, and explicit backend health checks for Den web and Den API.
+The recommended Google Cloud path is Helm on a regional GKE Autopilot cluster with Cloud SQL for MySQL. Use two GKE Ingresses (one for Den web, one for Den API), each with its own reserved global IP address, a shared Google-managed certificate, and explicit backend health checks.
 
 ## What Google Cloud manages
 
@@ -12,17 +12,17 @@ The recommended Google Cloud path is Helm on a regional GKE Autopilot cluster wi
 - Cloud SQL for MySQL, encryption, backups, and failover
 - Cloud DNS and Google-managed certificates when you use those services
 
-The OpenWork chart manages Deployments, Services, ConfigMaps, Secret references, probes, the Ingress, and the database migration Job. GKE `BackendConfig` and `ManagedCertificate` resources are applied alongside the chart.
+The OpenWork chart manages Deployments, Services, ConfigMaps, Secret references, probes, the two Ingresses, and the database migration Job. GKE `BackendConfig` and `ManagedCertificate` resources are applied alongside the chart.
 
 ## Google Cloud checklist
 
 1. Enable the Kubernetes Engine, Compute Engine, Cloud SQL Admin, and Service Networking APIs.
 2. Create a regional GKE Autopilot cluster and install `gke-gcloud-auth-plugin` if your Cloud SDK does not include it.
 3. Configure private services access and create Cloud SQL for MySQL with a private IP.
-4. Reserve a global IPv4 address and create the GKE health-check and certificate resources.
+4. Reserve two global IPv4 addresses (one for each hostname) and create the GKE health-check and certificate resources.
 5. Store `DATABASE_URL`, `BETTER_AUTH_SECRET`, and `DEN_DB_ENCRYPTION_KEY` in a Kubernetes Secret.
 6. Start from the chart's `values.gcp-ingress.yaml` example, install the chart, and verify migrations and readiness.
-7. Point both hostnames at the reserved address and wait for the managed certificate to become active.
+7. Point each hostname at its own reserved address and wait for the managed certificate to become active.
 
 For exact `gcloud` commands, values, health checks, migration troubleshooting, verification, and cleanup, follow the [GKE operator runbook](https://github.com/different-ai/openwork/blob/dev/docs/gcp-gke-helm.md).
 

packaging/helm/openwork-ee/README.md

@@ -405,8 +405,9 @@ Provider-specific starter guides:
 - Google Cloud GKE:
   [guide](../../../docs/gcp-gke-helm.md),
   [`examples/values.gcp-ingress.yaml`](examples/values.gcp-ingress.yaml).
-  The recommended first GCP path is GKE Ingress with a reserved global IP,
-  Google-managed certificate, and BackendConfig health checks.
+  The recommended first GCP path is two GKE Ingresses (one per host) each
+  with its own reserved global IP, a shared Google-managed certificate, and
+  BackendConfig health checks.
 
 `ingress.enabled=true` only emits Kubernetes `Ingress` resources; it does not
 install an ingress controller. Use it only when the cluster already has a

packaging/helm/openwork-ee/examples/values.gcp-ingress.yaml

@@ -94,12 +94,20 @@ migrations:
 ingress:
   enabled: true
   className: ""
+  # Shared across both Ingress objects: the managed certificate covers both
+  # hosts, so it is safe to reuse here. The reserved static IP is NOT — GKE
+  # gives each Ingress its own GCLB, so two Ingresses referencing the same
+  # global-static-ip-name would fight over one forwarding rule and only one
+  # would reconcile. Each host therefore gets its own reserved IP below.
   annotations:
     kubernetes.io/ingress.class: gce
-    kubernetes.io/ingress.global-static-ip-name: REPLACE_GLOBAL_STATIC_IP_NAME
     networking.gke.io/managed-certificates: REPLACE_MANAGED_CERTIFICATE_NAME
   web:
     host: REPLACE_WEB_HOST
+    annotations:
+      kubernetes.io/ingress.global-static-ip-name: REPLACE_WEB_GLOBAL_STATIC_IP_NAME
   api:
     enabled: true
     host: REPLACE_API_HOST
+    annotations:
+      kubernetes.io/ingress.global-static-ip-name: REPLACE_API_GLOBAL_STATIC_IP_NAME

packaging/helm/openwork-ee/templates/ingress-api.yaml

@@ -0,0 +1,33 @@
+{{- if and .Values.ingress.enabled .Values.ingress.api.enabled }}
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: {{ include "openwork-ee.fullname" . }}-api
+  namespace: {{ include "openwork-ee.namespace" . }}
+  labels:
+    {{- include "openwork-ee.labels" . | nindent 4 }}
+  {{- $apiAnnotations := mergeOverwrite (deepCopy .Values.ingress.annotations) .Values.ingress.api.annotations }}
+  {{- with $apiAnnotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  {{- if .Values.ingress.className }}
+  ingressClassName: {{ .Values.ingress.className }}
+  {{- end }}
+  {{- with .Values.ingress.tls }}
+  tls:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+  rules:
+    - host: {{ .Values.ingress.api.host | quote }}
+      http:
+        paths:
+          - path: {{ .Values.ingress.api.path }}
+            pathType: {{ .Values.ingress.api.pathType }}
+            backend:
+              service:
+                name: {{ include "openwork-ee.denApiServiceName" . }}
+                port:
+                  name: http
+{{- end }}

packaging/helm/openwork-ee/templates/ingress.yaml → packaging/helm/openwork-ee/templates/ingress-den.yaml

@@ -6,7 +6,8 @@ metadata:
   namespace: {{ include "openwork-ee.namespace" . }}
   labels:
     {{- include "openwork-ee.labels" . | nindent 4 }}
-  {{- with .Values.ingress.annotations }}
+  {{- $webAnnotations := mergeOverwrite (deepCopy .Values.ingress.annotations) .Values.ingress.web.annotations }}
+  {{- with $webAnnotations }}
   annotations:
     {{- toYaml . | nindent 4 }}
   {{- end }}
@@ -29,16 +30,4 @@ spec:
                 name: {{ include "openwork-ee.denWebServiceName" . }}
                 port:
                   name: http
-    {{- if .Values.ingress.api.enabled }}
-    - host: {{ .Values.ingress.api.host | quote }}
-      http:
-        paths:
-          - path: {{ .Values.ingress.api.path }}
-            pathType: {{ .Values.ingress.api.pathType }}
-            backend:
-              service:
-                name: {{ include "openwork-ee.denApiServiceName" . }}
-                port:
-                  name: http
-    {{- end }}
 {{- end }}