|
| 1 | +--- |
| 2 | +Title: Prepare source database |
| 3 | +alwaysopen: false |
| 4 | +categories: |
| 5 | +- docs |
| 6 | +- operate |
| 7 | +- rc |
| 8 | +description: Prepare your source database and database credentials for Data integration. |
| 9 | +hideListLinks: true |
| 10 | +weight: 1 |
| 11 | +--- |
| 12 | + |
| 13 | +## Create new data pipeline |
| 14 | + |
| 15 | +1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. |
| 16 | +1. Select **Create data pipeline**. |
| 17 | + {{<image filename="images/rc/rdi/rdi-create-data-pipeline.png" alt="The create data pipeline button." width=200px >}} |
| 18 | +1. Select your source database type. The following database types are supported: |
| 19 | + - MySQL |
| 20 | + - mariaDB |
| 21 | + - Oracle |
| 22 | + - SQL Server |
| 23 | + - PostgreSQL |
| 24 | + {{<image filename="images/rc/rdi/rdi-select-source-db.png" alt="The select source database type list." width=500px >}} |
| 25 | +1. If you know the size of your source database, enter it into the **Source dataset size** field. |
| 26 | + {{<image filename="images/rc/rdi/rdi-source-dataset-size.png" alt="Enter the amount of source data you plan to ingest." width=400px >}} |
| 27 | +1. Under **Setup connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it. |
| 28 | + |
| 29 | + {{<image filename="images/rc/rdi/rdi-setup-connectivity-arn.png" alt="The select source database type list." width=80% >}} |
| 30 | + |
| 31 | + The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`. |
| 32 | + |
| 33 | +## Prepare source database |
| 34 | + |
| 35 | +Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). |
| 36 | + |
| 37 | +See [Prepare source databases]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/">}}) to find steps for your database type: |
| 38 | +- [MySQL and mariaDB]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb">}}) |
| 39 | +- [Oracle]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle">}}) |
| 40 | +- [SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server">}}) |
| 41 | +- [PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql">}}) |
| 42 | + |
| 43 | +See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC. |
| 44 | + |
| 45 | +## Share source database credentials |
| 46 | + |
| 47 | +You need to share your source database credentials and certificates in an Amazon secret with Redis Cloud so that the pipeline can connect to your database. |
| 48 | + |
| 49 | +To do this, you need to: |
| 50 | +1. [Create an encryption key](#create-encryption-key) using AWS Key Management Service with the right permissions. |
| 51 | +1. [Create a secret](#create-database-credentials-secret) containing the source database credentials encrypted using that key. |
| 52 | + |
| 53 | +### Create encryption key |
| 54 | + |
| 55 | +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Key Management Service**. [Create an encryption key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) with the following settings: |
| 56 | + |
| 57 | +1. In **Step 1 - Configure key**: |
| 58 | + - **Key type**: Select **Symmetric**. |
| 59 | + - **Key usage**: Select **Encrypt and decrypt**. |
| 60 | + - Under **Advanced options**, set the following: |
| 61 | + - **Key material origin**: Select **KMS - recommended**. |
| 62 | + - **Regionality**: Select **Single-Region key**. |
| 63 | +1. In **Step 2 - Add labels**, add an alias and description for the key. |
| 64 | +1. In **Step 3 - Define key administrative permissions**, under **Key deletion**, select **Allow key administrators to delete this key**. |
| 65 | +1. In **Step 4 - Define key usage permissions**, under **Other AWS accounts**, select **Add another AWS account**. Enter the AWS account ID for the Redis Cloud cluster that you saved earlier. |
| 66 | + |
| 67 | +### Create database credentials secret |
| 68 | + |
| 69 | +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings: |
| 70 | + |
| 71 | +- **Key/value pairs**: Enter the following key/value pairs. |
| 72 | + |
| 73 | + - `username`: Database username |
| 74 | + - `password`: Database password |
| 75 | + - `trust_certificate`: Server certificate in PEM format *(TLS only)* |
| 76 | + - `client_public_key`: [X.509 client certificate](https://en.wikipedia.org/wiki/X.509) or chain in PEM format *(mTLS only)* |
| 77 | + - `client_private_key`: Key for the client certificate or chain in PEM format *(mTLS only)* |
| 78 | + - `client_private_key_passphrase`: Passphrase or password for the client certificate or chain in PEM format *(mTLS only)* |
| 79 | + |
| 80 | + {{<note>}} |
| 81 | +If your source database has TLS or mTLS enabled, we recommend that you enter the `trust_certificate`, `client_public_key`, and `client_private_key` into the secret editor using the **Key/Value** input method instead of the **JSON** input method. Pasting directly into the JSON editor may cause an error. |
| 82 | + {{</note>}} |
| 83 | + |
| 84 | +- **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier. |
| 85 | + |
| 86 | +- **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `<AWS ACCOUNT ID>` with the AWS account ID for the Redis Cloud cluster that you saved earlier. |
| 87 | + |
| 88 | + ```json |
| 89 | + { |
| 90 | + "Version" : "2012-10-17", |
| 91 | + "Statement" : [ { |
| 92 | + "Sid" : "RedisDataIntegrationRoleAccess", |
| 93 | + "Effect" : "Allow", |
| 94 | + "Principal" : "*", |
| 95 | + "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], |
| 96 | + "Resource" : "*", |
| 97 | + "Condition" : { |
| 98 | + "StringLike" : { |
| 99 | + "aws:PrincipalArn" : "arn:aws:iam::<AWS ACCOUNT ID>:role/redis-data-pipeline-secrets-role" |
| 100 | + } |
| 101 | + } |
| 102 | + } ] |
| 103 | + } |
| 104 | + ``` |
| 105 | + |
| 106 | +After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. |
| 107 | + |
| 108 | +## Set up connectivity |
| 109 | + |
| 110 | +To expose your source database to Redis, you need to add Redis Cloud as an Allowed Principal on the [AWS PrivateLink VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions) for the PrivateLink connected to your source database. |
| 111 | + |
| 112 | +1. Copy the Amazon Resource Name (ARN) provided in the **Setup connectivity** section. |
| 113 | +1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and select **Endpoint services**. |
| 114 | +1. Navigate to **Allow principals** tab. |
| 115 | +1. Add the Redis Cloud ARN and choose **Allow principals**. |
| 116 | +1. Copy your PrivateLink service name for later. |
| 117 | + |
| 118 | +For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). |
| 119 | + |
| 120 | + |
| 121 | +## Next steps |
| 122 | + |
| 123 | +After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{<relref "/operate/rc/databases/rdi/define">}}). |
| 124 | + |
| 125 | +{{<image filename="images/rc/rdi/rdi-define-source-database.png" alt="ADD ALT TEXT" width=200px >}} |
0 commit comments