Skip to content

[federation] Remove manual Keycloak Route creation to fix SSL race condition#4074

Open
mauricioharley wants to merge 1 commit into
openstack-k8s-operators:mainfrom
mauricioharley:remove-create-keycloak-route
Open

[federation] Remove manual Keycloak Route creation to fix SSL race condition#4074
mauricioharley wants to merge 1 commit into
openstack-k8s-operators:mainfrom
mauricioharley:remove-create-keycloak-route

Conversation

@mauricioharley

Copy link
Copy Markdown

Problem

The crc-federation CI job started failing intermittently with:

SSL exception connecting to https://keycloak-openstack.apps-crc.testing/...
SSLCertVerificationError: certificate verify failed: self-signed certificate in certificate chain

No code changes in the federation role between passing and failing builds.

Root cause

The task "Create Route for Keycloak" in run_keycloak_setup.yml creates a
passthrough TLS Route right after the Keycloak Service appears. The RHSSO
operator also creates/reconciles a Route for Keycloak (because
externalAccess.enabled: true in the Keycloak CR), but as reencrypt.

This is a race condition:

  • When the operator wins: Route becomes reencrypt, ingress CA is the correct
    trust anchor, SSL works.
  • When Ansible wins: Route stays passthrough, the Keycloak pod serves its own
    self-signed cert, full-ca-list.crt (which only has the ingress CA) cannot
    verify it, SSL fails.

Fix

Remove the manual Route creation entirely. The RHSSO operator handles it
correctly with reencrypt and the proper destinationCACertificate.

Testing

Verified on a CRC environment:

  1. Deployed RHSSO operator + Keycloak CR (without manual Route)
  2. Confirmed the operator created the Route as reencrypt
  3. Deployed OpenStack control plane with federation configuration
  4. Successfully obtained a federated Keystone token via openstack token issue
    using OIDC password grant through Keycloak

@openshift-ci

openshift-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mauricioharley for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The "Create Route for Keycloak" task creates a passthrough TLS Route
that races with the RHSSO operator's own Route reconciliation. When
the operator wins the race, it overwrites the Route with reencrypt
and everything works. When Ansible wins, the Route stays as
passthrough and OIDC authentication fails with SSL verification
errors because the trust bundle only contains the ingress operator
CA, which does not sign the Keycloak pod's self-signed certificate.

The Keycloak CR already sets externalAccess.enabled=true, which
tells the RHSSO operator to create and manage the Route itself as
reencrypt with the correct destinationCACertificate. Removing the
manual Route creation eliminates the race condition entirely.

Signed-off-by: Mauricio Harley <mharley@redhat.com>
@mauricioharley
mauricioharley force-pushed the remove-create-keycloak-route branch from 5835c3a to cbfb76f Compare July 23, 2026 16:17
@mauricioharley mauricioharley changed the title Federation: Remove manual Keycloak Route creation to fix SSL race condition [federation] Remove manual Keycloak Route creation to fix SSL race condition Jul 23, 2026
@mauricioharley
mauricioharley requested review from afaranha and jagee July 23, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant