From 9d5d5730775a33320a02c276c39a8e246b92d216 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Thu, 1 Oct 2026 16:41:22 +0300 Subject: [PATCH 1/2] RDI: Document Google Cloud SQL for MySQL as a source Add a preparation page for Cloud SQL for MySQL with self-managed RDI: binary logging, the lowercase binlog_row_image flag, authorized networks, creating the CDC user with an explicit host, the database.ssl.mode setting that enforced encryption needs, and what to expect during a Regional HA failover. Link to it from the MySQL/MariaDB preparation page. Co-Authored-By: Claude Opus 5.5 --- .../prepare-dbs/gcp-cloud-sql-mysql.md | 238 ++++++++++++++++++ .../prepare-dbs/my-sql-mariadb.md | 7 + 2 files changed, 245 insertions(+) create mode 100644 content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md new file mode 100644 index 0000000000..89016c4fb8 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md @@ -0,0 +1,238 @@ +--- +Title: Prepare Google Cloud SQL for MySQL for RDI +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Prepare a Google Cloud SQL for MySQL database to work with RDI +group: di +linkTitle: Prepare Cloud SQL for MySQL +summary: Configure a Google Cloud SQL for MySQL instance for snapshot and change data capture with Redis Data Integration. +type: integration +weight: 12 +--- + +[Google Cloud SQL for MySQL](https://cloud.google.com/sql/docs/mysql) is a +managed MySQL service on Google Cloud. RDI captures changes from a Cloud SQL +for MySQL instance through its binary log (binlog), in the same way as for a +self-managed MySQL server. + +> [!NOTE] +> This page describes Cloud SQL for MySQL setup for a self-managed RDI +> deployment (VM or Kubernetes). The integration was validated with RDI 2.0.0 +> and Cloud SQL for MySQL 8.0 on a Regional (high availability) instance, using +> a public IP connection with encrypted connections enforced. See +> [RDI in Redis Cloud](/content/integrate/redis-data-integration/_index.md#rdi-in-redis-cloud) +> for the source databases that the managed service supports. + +Cloud SQL differs from a self-managed MySQL source in the following ways: + +- You can't edit the MySQL configuration file. You enable binary logging as an + instance setting and set other server variables as Cloud SQL + [database flags](https://cloud.google.com/sql/docs/mysql/flags). +- Cloud SQL already sets `binlog_format` to `ROW` and enables GTIDs, so you don't + need to change them. +- Cloud SQL can enforce encrypted connections. If it does, you must configure RDI + to connect over TLS (Transport Layer Security), or the connection fails with an + error that looks like a credentials problem. +- On a Regional instance, a failover moves the primary to another zone. RDI + reconnects automatically, as described in + [Failover on Regional instances](#failover-on-regional-instances). + +The following checklist summarizes the setup: + +```checklist {id="cloudsqlmysqllist"} +- [ ] [Enable binary logging](#1-enable-binary-logging) +- [ ] [Set the binlog row image](#2-set-the-binlog-row-image) +- [ ] [Check the binlog settings](#3-check-the-binlog-settings) +- [ ] [Allow network access from RDI](#4-allow-network-access-from-rdi) +- [ ] [Create a CDC user](#5-create-a-cdc-user) +- [ ] [Configure RDI](#6-configure-rdi) +``` + +## 1. Enable binary logging + +RDI reads changes from the MySQL binlog, which Cloud SQL only writes when binary +logging is enabled. Binary logging requires automated backups. + +To enable binary logging when you create an instance, pass `--enable-bin-log` to +[`gcloud sql instances create`](https://cloud.google.com/sdk/gcloud/reference/sql/instances/create). +The following example also sets the binlog row image flag from +[step 2](#2-set-the-binlog-row-image) and enforces encrypted connections: + +```bash +gcloud sql instances create \ + --database-version=MYSQL_8_0 \ + --region= \ + --tier= \ + --availability-type=REGIONAL \ + --enable-bin-log \ + --database-flags=binlog_row_image=full \ + --ssl-mode=ENCRYPTED_ONLY +``` + +To enable binary logging on an existing instance, run: + +```bash +gcloud sql instances patch --enable-bin-log +``` + +In the Google Cloud console, binary logging is part of +[point-in-time recovery](https://cloud.google.com/sql/docs/mysql/backup-recovery/pitr). +Enable point-in-time recovery for the instance to enable binary logging. + +## 2. Set the binlog row image + +RDI requires the full row image. Set the `binlog_row_image` database flag to +`full`: + +```bash +gcloud sql instances patch --database-flags=binlog_row_image=full +``` + +The flag value must be lowercase. Cloud SQL rejects `binlog_row_image=FULL` +with the error `FULL was not an expected string`. MySQL reports the value as +`FULL` after you set it. + +> [!WARNING] +> The `--database-flags` option replaces all the database flags that are +> currently set on the instance. If the instance already has other flags, include +> them in the same command. Changing database flags can restart the instance. See +> [Configure database flags](https://cloud.google.com/sql/docs/mysql/flags) for +> details. + +You can also set the flag in the Google Cloud console, in the **Flags** section +when you edit the instance. + +## 3. Check the binlog settings + +Connect to the instance with the +[MySQL CLI client](https://dev.mysql.com/doc/refman/8.0/en/mysql.html) and run +the following query: + +```sql +SHOW GLOBAL VARIABLES WHERE Variable_name IN + ('log_bin', 'binlog_format', 'binlog_row_image', 'gtid_mode', + 'binlog_row_value_options'); +``` + +Check that the result has the following values: + +| Variable | Value | +| :-- | :-- | +| `log_bin` | `ON` | +| `binlog_format` | `ROW` | +| `binlog_row_image` | `FULL` | +| `gtid_mode` | `ON` | +| `binlog_row_value_options` | Empty (not `PARTIAL_JSON`) | + +See +[Check `binlog_row_value_options`](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md#6-check-binlog_row_value_options) +to learn why `PARTIAL_JSON` is a problem. + +## 4. Allow network access from RDI + +If RDI connects to the instance's public IP address, add the public egress IP +address of the RDI host or Kubernetes cluster to the instance's +[authorized networks](https://cloud.google.com/sql/docs/mysql/authorize-networks). +Use a `/32` CIDR for an individual IPv4 address: + +```bash +gcloud sql instances patch --authorized-networks=/32 +``` + +The `--authorized-networks` option replaces the existing list, so include any +networks that are already authorized. + +## 5. Create a CDC user + +The Debezium connector needs a MySQL user account to connect to the instance. +Create the user with +[`gcloud sql users create`](https://cloud.google.com/sdk/gcloud/reference/sql/users/create) +and set `--host` explicitly: + +```bash +gcloud sql users create \ + --instance= \ + --host=% \ + --password= +``` + +The `%` host lets the user connect from any client address. To restrict the user +to the RDI host, use that host's IP address instead. + +Then connect to the instance as an administrator user, such as `root`, and grant +the user the permissions that RDI needs. Use the same host value that you used +to create the user: + +```sql +GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO ''@'%'; +FLUSH PRIVILEGES; +``` + +If the `GRANT` statement fails with +`ERROR 1410 (42000): You are not allowed to create a user with GRANT`, the user +doesn't exist with that host. Cloud SQL's administrator user can't create a user +implicitly with `GRANT`. Check the user's host with +`gcloud sql users list --instance=`, and either grant to the host +shown there or recreate the user with the `--host` value you want. + +## 6. Configure RDI + +Store the CDC user's credentials as RDI secrets. Pass the source name with +`--db` (the source in the following example is named `mysql`): + +```bash +redis-di set-secret USERNAME --db mysql +redis-di set-secret PASSWORD --db mysql +``` + +Add a MySQL source to `config.yaml`. If the instance enforces encrypted +connections, set `database.ssl.mode` to `required` (or a stricter mode) in the +source's `advanced.source` section: + +```yaml +sources: + mysql: + type: cdc + connection: + type: mysql + host: + port: 3306 + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} + databases: + - + advanced: + source: + database.ssl.mode: required +``` + +If you don't set `database.ssl.mode` and the instance only accepts encrypted +connections, the pipeline deployment fails with an error like +`Access denied for user ''@'' (using password: YES)`. This +error looks like a credentials problem, but Cloud SQL is rejecting the +unencrypted connection. + +The `required` mode encrypts the connection but doesn't verify the server +certificate. To verify the certificate as well, set the source's `CACERT` secret +to the instance's server CA certificate and choose a verifying mode, as described in +[Connect over TLS or mTLS](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md#7-connect-over-tls-or-mtls). + +## Failover on Regional instances + +A Regional (high availability) Cloud SQL instance has a standby in a second +zone. During a failover, Cloud SQL promotes the standby to primary and keeps the +same IP address. Expect the following behavior: + +- Clients, including your application, can't connect to the instance for up to + about a minute while the failover runs. Use connection retry logic in your + application. +- RDI doesn't need a restart or any manual step. The RDI collector detects that + its binlog connection is no longer working, reconnects to the new primary, and + then captures the changes that were committed while it was disconnected. +- Expect about one minute of extra change data capture (CDC) lag after the + instance becomes available again, while the collector detects the lost + connection and catches up. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index f07ccc8dfc..88522a7862 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -125,6 +125,13 @@ You can run the query above again to check that `log-bin` is now `ON`. > If you don't enable automated backups first then the settings above will have no > effect. +> [!NOTE] +> If you are using [Google Cloud SQL for MySQL](https://cloud.google.com/sql/docs/mysql), +> you enable binary logging and set `binlog_row_image` through Cloud SQL instead of the +> server configuration file. See +> [Prepare Google Cloud SQL for MySQL for RDI](/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md) +> for the full setup. + ## 3. Enable GTIDs *Global transaction identifiers (GTIDs)* uniquely identify the transactions that occur From f940ca53ab3d5f0bc9d6494e98e9bcabeb6345a4 Mon Sep 17 00:00:00 2001 From: Zdravko Donev Date: Sat, 3 Oct 2026 20:44:49 +0300 Subject: [PATCH 2/2] Address Cloud SQL MySQL documentation review comments --- .../prepare-dbs/gcp-cloud-sql-mysql.md | 28 +++++++++---------- .../prepare-dbs/my-sql-mariadb.md | 2 ++ 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md index 89016c4fb8..ef1e1dfda8 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/gcp-cloud-sql-mysql.md @@ -15,12 +15,12 @@ weight: 12 --- [Google Cloud SQL for MySQL](https://cloud.google.com/sql/docs/mysql) is a -managed MySQL service on Google Cloud. RDI captures changes from a Cloud SQL -for MySQL instance through its binary log (binlog), in the same way as for a +managed MySQL service on Google Cloud. RDI captures changes from an instance of Cloud SQL +for MySQL through its binary log (binlog), in the same way it would for a self-managed MySQL server. > [!NOTE] -> This page describes Cloud SQL for MySQL setup for a self-managed RDI +> This page describes Cloud SQL for MySQL configured for a self-managed RDI > deployment (VM or Kubernetes). The integration was validated with RDI 2.0.0 > and Cloud SQL for MySQL 8.0 on a Regional (high availability) instance, using > a public IP connection with encrypted connections enforced. See @@ -41,7 +41,7 @@ Cloud SQL differs from a self-managed MySQL source in the following ways: reconnects automatically, as described in [Failover on Regional instances](#failover-on-regional-instances). -The following checklist summarizes the setup: +The following checklist summarizes the steps to prepare Cloud SQL for MySQL to work with RDI: ```checklist {id="cloudsqlmysqllist"} - [ ] [Enable binary logging](#1-enable-binary-logging) @@ -59,7 +59,7 @@ logging is enabled. Binary logging requires automated backups. To enable binary logging when you create an instance, pass `--enable-bin-log` to [`gcloud sql instances create`](https://cloud.google.com/sdk/gcloud/reference/sql/instances/create). -The following example also sets the binlog row image flag from +The following example shows how to do this and also sets the binlog row image flag from [step 2](#2-set-the-binlog-row-image) and enforces encrypted connections: ```bash @@ -81,7 +81,7 @@ gcloud sql instances patch --enable-bin-log In the Google Cloud console, binary logging is part of [point-in-time recovery](https://cloud.google.com/sql/docs/mysql/backup-recovery/pitr). -Enable point-in-time recovery for the instance to enable binary logging. +You must therefore enable point-in-time recovery for the instance to enable binary logging. ## 2. Set the binlog row image @@ -92,7 +92,7 @@ RDI requires the full row image. Set the `binlog_row_image` database flag to gcloud sql instances patch --database-flags=binlog_row_image=full ``` -The flag value must be lowercase. Cloud SQL rejects `binlog_row_image=FULL` +The flag value must be lowercase. Cloud SQL rejects `binlog_row_image=FULL` (that is with `FULL` in uppercase) with the error `FULL was not an expected string`. MySQL reports the value as `FULL` after you set it. @@ -160,7 +160,7 @@ gcloud sql users create \ --password= ``` -The `%` host lets the user connect from any client address. To restrict the user +The value of `%` for `host` lets the user connect from any client address. To restrict the user to the RDI host, use that host's IP address instead. Then connect to the instance as an administrator user, such as `root`, and grant @@ -172,9 +172,9 @@ GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON * FLUSH PRIVILEGES; ``` -If the `GRANT` statement fails with -`ERROR 1410 (42000): You are not allowed to create a user with GRANT`, the user -doesn't exist with that host. Cloud SQL's administrator user can't create a user +Note that the `GRANT` statement might fail with +`ERROR 1410 (42000): You are not allowed to create a user with GRANT`. This happens when the user +doesn't exist with that host, since Cloud SQL's administrator user can't create a user implicitly with `GRANT`. Check the user's host with `gcloud sql users list --instance=`, and either grant to the host shown there or recreate the user with the `--host` value you want. @@ -213,7 +213,7 @@ sources: If you don't set `database.ssl.mode` and the instance only accepts encrypted connections, the pipeline deployment fails with an error like `Access denied for user ''@'' (using password: YES)`. This -error looks like a credentials problem, but Cloud SQL is rejecting the +error looks like a credentials problem, but Cloud SQL is actually rejecting the unencrypted connection. The `required` mode encrypts the connection but doesn't verify the server @@ -225,11 +225,11 @@ to the instance's server CA certificate and choose a verifying mode, as describe A Regional (high availability) Cloud SQL instance has a standby in a second zone. During a failover, Cloud SQL promotes the standby to primary and keeps the -same IP address. Expect the following behavior: +same IP address. Expect the following behavior when this happens: - Clients, including your application, can't connect to the instance for up to about a minute while the failover runs. Use connection retry logic in your - application. + application to handle this. - RDI doesn't need a restart or any manual step. The RDI collector detects that its binlog connection is no longer working, reconnects to the new primary, and then captures the changes that were committed while it was disconnected. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index 88522a7862..2a52612416 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -125,6 +125,8 @@ You can run the query above again to check that `log-bin` is now `ON`. > If you don't enable automated backups first then the settings above will have no > effect. +  + > [!NOTE] > If you are using [Google Cloud SQL for MySQL](https://cloud.google.com/sql/docs/mysql), > you enable binary logging and set `binlog_row_image` through Cloud SQL instead of the