Skip to content

Commit 668028a

Browse files
cmilesbcmilesb
authored
Merge pull request #1544 from redis/DOC-5222
RC RDI: Add more setup databases pages
2 parents 14f2d28 + fca0769 commit 668028a

File tree

6 files changed

+291
-64
lines changed

6 files changed

+291
-64
lines changed

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md

Lines changed: 0 additions & 59 deletions
This file was deleted.

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
Title: Prepare AWS RDS and Aurora databases for RDI
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Learn how to prepare AWS RDS and Aurora databases for RDI.
11+
group: di
12+
linkTitle: Prepare AWS RDS and Aurora
13+
summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration.
14+
hideListLinks: false
15+
type: integration
16+
weight: 5
17+
---
18+
19+
You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html).
20+
21+
The pages in this section give detailed instructions to get your source
22+
database ready for Debezium to use:

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Enable CDC features in your source databases
11+
group: di
12+
hideListLinks: false
13+
linkTitle: Prepare AWS Aurora/RDS MySQL
14+
summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration.
15+
type: integration
16+
weight: 2
17+
---
18+
19+
Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database.
20+
database to work with RDI.
21+
22+
## Create and apply parameter group
23+
24+
RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.
25+
26+
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
27+
28+
| Name | Value |
29+
| :-- | :-- |
30+
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` |
31+
| **Description** | (Optional) Enter a description for the parameter group |
32+
| **Engine Type** | Choose **Aurora MySQL** for Aurora MySQL or **MySQL Community** for AWS RDS MySQL. |
33+
| **Parameter group family** | Choose **aurora-mysql8.0** for Aurora MySQL or **mysql8.0** for AWS RDS MySQL. |
34+
35+
Select **Create** to create the parameter group.
36+
37+
1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
38+
39+
| Name | Value |
40+
| :-- | :-- |
41+
| `binlog_format` | `ROW` |
42+
| `binlog_row_image` | `FULL` |
43+
44+
Select **Save Changes** to apply the changes to the parameter group.
45+
46+
1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
47+
48+
Select **Save changes** to apply the parameter group to the new database.
49+
50+
## Create Debezium user
51+
52+
The Debezium connector needs a user account to connect to MySQL. This
53+
user must have appropriate permissions on all databases where you want Debezium
54+
to capture changes.
55+
56+
1. Connect to your database as an admin user and create a new user for the connector:
57+
58+
```sql
59+
CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
60+
```
61+
62+
Replace `<username>` and `<password>` with a username and password for the new user.
63+
64+
The `%` means that the user can connect from any client. If you want to restrict the user to connect only from the RDI host, replace `%` with the IP address of the RDI host.
65+
66+
1. Grant the user the necessary permissions:
67+
68+
```sql
69+
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO '<username>'@'%';
70+
```
71+
72+
Replace `<username>` with the username of the Debezium user.
73+
74+
1. Finalize the user's permissions:
75+
76+
```sql
77+
FLUSH PRIVILEGES;
78+
```

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
Title: Prepare AWS Aurora PostgreSQL/AWS RDS PostgreSQL for RDI
3+
aliases:
4+
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aur-pgsql/
5+
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql/
6+
- /integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql/
7+
alwaysopen: false
8+
categories:
9+
- docs
10+
- integrate
11+
- rs
12+
- rc
13+
- rdi
14+
description: Prepare AWS Aurora PostgreSQL databases to work with RDI
15+
group: di
16+
linkTitle: Prepare AWS Aurora PostgreSQL
17+
summary: Prepare AWS Aurora PostgreSQL databases to work with Redis Data Integration.
18+
type: integration
19+
weight: 1
20+
---
21+
22+
Follow the steps in the sections below to prepare an
23+
[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html)
24+
database to work with RDI.
25+
26+
## Create and apply parameter group
27+
28+
RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.
29+
30+
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
31+
32+
| Name | Value |
33+
| :-- | :-- |
34+
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-aurora-pg` or `rdi-rds-pg` |
35+
| **Description** | (Optional) Enter a description for the parameter group |
36+
| **Engine Type** | Choose **Aurora PostgreSQL** for Aurora PostgreSQL or **PostgreSQL** for AWS RDS PostgreSQL. |
37+
| **Parameter group family** | Choose **aurora-postgresql15** for Aurora PostgreSQL or **postgresql13** for AWS RDS PostgreSQL. |
38+
39+
Select **Create** to create the parameter group.
40+
41+
1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters:
42+
43+
| Name | Value |
44+
| :-- | :-- |
45+
| `rds.logical_replication` | `1` |
46+
47+
Select **Save Changes** to apply the changes to the parameter group.
48+
49+
1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
50+
51+
Select **Save changes** to apply the parameter group to your database.
52+
53+
## Create Debezium user
54+
55+
The Debezium connector needs a user account to connect to PostgreSQL. This
56+
user must have appropriate permissions on all databases where you want Debezium
57+
to capture changes.
58+
59+
1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector:
60+
61+
```sql
62+
CREATE ROLE <username> WITH LOGIN PASSWORD '<password>' VALID UNTIL 'infinity';
63+
```
64+
65+
Replace `<username>` and `<password>` with a username and password for the new user.
66+
67+
1. Grant the user the necessary replication permissions:
68+
69+
```sql
70+
GRANT rds_replication TO <username>;
71+
```
72+
73+
Replace `<username>` with the username of the Debezium user.
74+
75+
1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database:
76+
77+
```sql
78+
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <username>;
79+
```
80+
81+
Replace `<username>` with the username of the Debezium user and `<schema>` with the schema name.
82+

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
Title: Prepare Microsoft SQL Server on AWS RDS for RDI
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Enable CDC features in your source databases
11+
group: di
12+
hideListLinks: false
13+
linkTitle: Prepare Microsoft SQL Server on AWS RDS
14+
summary: Prepare Microsoft SQL Server on AWS RDS databases to work with Redis Data Integration.
15+
type: integration
16+
weight: 3
17+
---
18+
19+
Follow the steps in the sections below to prepare a [Microsoft SQL Server on AWS RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.SQLServer.html) database to work with RDI.
20+
21+
## Create the Debezium user
22+
23+
The Debezium connector needs a user account to connect to SQL Server. This
24+
user must have appropriate permissions on all databases where you want Debezium
25+
to capture changes.
26+
27+
1. Connect to your database as an admin user and create a new user for the connector:
28+
29+
```sql
30+
USE master
31+
GO
32+
CREATE LOGIN <username> WITH PASSWORD = '<password>'
33+
GO
34+
USE <database>
35+
GO
36+
CREATE USER <username> FOR LOGIN <username>
37+
GO
38+
```
39+
40+
Replace `<username>` and `<password>` with a username and password for the new user and replace `<database>` with the name of your database.
41+
42+
1. Grant the user the necessary permissions:
43+
44+
```sql
45+
USE master
46+
GO
47+
GRANT VIEW SERVER STATE TO <username>
48+
GO
49+
USE <database>
50+
GO
51+
EXEC sp_addrolemember N'db_datareader', N'<username>'
52+
GO
53+
```
54+
55+
Replace `<username>` with the username of the Debezium user and replace `<database>` with the name of your database.
56+
57+
## Enable CDC on the database
58+
59+
Change Data Capture (CDC) must be enabled for the database and for each table you want to capture.
60+
61+
1. Enable CDC for the database by running the following command:
62+
63+
```sql
64+
EXEC msdb.dbo.rds_cdc_enable_db '<database>'
65+
GO
66+
```
67+
68+
Replace `<database>` with the name of your database.
69+
70+
1. Enable CDC for each table you want to capture by running the following commands:
71+
72+
```sql
73+
USE <database>
74+
GO
75+
EXEC sys.sp_cdc_enable_table
76+
@source_schema = N'<schema>',
77+
@source_name = N'<table>',
78+
@role_name = N'<role>',
79+
@supports_net_changes = 0
80+
GO
81+
```
82+
83+
Replace `<database>` with the name of your database, `<schema>` with the name of the schema containing the table, `<table>` with the name of the table, and `<role>` with the name of a new role that will be created to manage access to the CDC data.
84+
85+
{{< note >}}
86+
The value for `@role_name` can’t be a fixed database role, such as `db_datareader`.
87+
Specifying a new name will create a corresponding database role that has full access to the
88+
captured change data.
89+
{{< /note >}}
90+
91+
1. Add the Debezium user to the CDC role:
92+
93+
```sql
94+
USE <database>
95+
GO
96+
EXEC sp_addrolemember N'<role>', N'<username>'
97+
GO
98+
```
99+
100+
Replace `<role>` with the name of the role you created in the previous step and replace `<username>` with the username of the Debezium user.

‎content/operate/rc/databases/rdi/setup.md

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,15 @@ weight: 1
3535
Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See [Prerequisites]({{<relref "/operate/rc/databases/rdi#prerequisites">}}) to find a list of supported source databases and database versions.
3636

3737
See [Prepare source databases]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/">}}) to find steps for your database type:
38-
- [MySQL and mariaDB]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb">}})
39-
- [Oracle]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle">}})
40-
- [SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server">}})
41-
- [AWS Aurora PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql">}})
42-
- [AWS RDS PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql">}})
38+
- Hosted on an AWS EC2 instance:
39+
- [MySQL and mariaDB]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb">}})
40+
- [Oracle]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle">}})
41+
- [SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server">}})
42+
- [PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql">}})
43+
- Hosted on AWS RDS or AWS Aurora:
44+
- [AWS Aurora PostgreSQL and AWS RDS PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql">}})
45+
- [AWS Aurora MySQL and AWS RDS MySQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql">}})
46+
- [AWS RDS SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver">}})
4347

4448
See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC.
4549

0 commit comments

Comments
 (0)