Enable TLS for DM

This document describes how to enable TLS between components of the DM cluster in Kubernetes and how to use DM to migrate data between MySQL/TiDB databases that enable TLS for the MySQL client.

Enable TLS between DM components

Starting from v1.2, TiDB Operator supports enabling TLS between components of the DM cluster in Kubernetes.

To enable TLS between components of the DM cluster, perform the following steps:

Generate certificates for each component of the DM cluster to be created:
- A set of server-side certificates for the DM-master/DM-worker component, saved as the Kubernetes Secret objects: ${cluster_name}-${component_name}-cluster-secret
- A set of shared client-side certificates for the various clients of each component, saved as the Kubernetes Secret objects: ${cluster_name}-dm-client-secret.
Note:
The Secret objects you created must follow the above naming convention. Otherwise, the deployment of the DM cluster will fail.
Deploy the cluster, and set .spec.tlsCluster.enabled to true.
Note:
After the cluster is created, do not modify this field; otherwise, the cluster will fail to upgrade.
Configure dmctl to connect to the cluster.

Certificates can be issued in multiple methods. This document describes two methods. You can choose either of them to issue certificates for the DM cluster:

Using the cfssl system
Using the cert-manager system

If you need to renew the existing TLS certificate, refer to Renew and Replace the TLS Certificate.

Generate certificates for components of the DM cluster

This section describes how to issue certificates using two methods: cfssl and cert-manager.

Using `cfssl`

Download cfssl and initialize the certificate issuer:

mkdir -p ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin

mkdir -p cfssl
cd cfssl

Generate the ca-config.json configuration file:

cat << EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "internal": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            }
        }
    }
}
EOF

Generate the ca-csr.json configuration file:

cat << EOF > ca-csr.json
{
    "CN": "TiDB",
    "CA": {
        "expiry": "87600h"
    },
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "CA",
            "O": "PingCAP",
            "ST": "Beijing",
            "OU": "TiDB"
        }
    ]
}
EOF

Generate CA by the configured option:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

Generate the server-side certificates:

In this step, a set of server-side certificate is created for each component of the DM cluster.

DM-master

First, generate the default dm-master-server.json file:

cfssl print-defaults csr > dm-master-server.json

Then, edit this file to change the CN and hosts attributes:

...
    "CN": "TiDB",
    "hosts": [
      "127.0.0.1",
      "::1",
      "${cluster_name}-dm-master",
      "${cluster_name}-dm-master.${namespace}",
      "${cluster_name}-dm-master.${namespace}.svc",
      "${cluster_name}-dm-master-peer",
      "${cluster_name}-dm-master-peer.${namespace}",
      "${cluster_name}-dm-master-peer.${namespace}.svc",
      "*.${cluster_name}-dm-master-peer",
      "*.${cluster_name}-dm-master-peer.${namespace}",
      "*.${cluster_name}-dm-master-peer.${namespace}.svc"
    ],
...

${cluster_name} is the name of the DM cluster. ${namespace} is the namespace in which the DM cluster is deployed. You can also add your customized hosts.

Finally, generate the DM-master server-side certificate:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=internal dm-master-server.json | cfssljson -bare dm-master-server

DM-worker

First, generate the default dm-worker-server.json file:

cfssl print-defaults csr > dm-worker-server.json

Then, edit this file to change the CN and hosts attributes:

...
    "CN": "TiDB",
    "hosts": [
      "127.0.0.1",
      "::1",
      "${cluster_name}-dm-worker",
      "${cluster_name}-dm-worker.${namespace}",
      "${cluster_name}-dm-worker.${namespace}.svc",
      "${cluster_name}-dm-worker-peer",
      "${cluster_name}-dm-worker-peer.${namespace}",
      "${cluster_name}-dm-worker-peer.${namespace}.svc",
      "*.${cluster_name}-dm-worker-peer",
      "*.${cluster_name}-dm-worker-peer.${namespace}",
      "*.${cluster_name}-dm-worker-peer.${namespace}.svc"
    ],
...

${cluster_name} is the name of the cluster. ${namespace} is the namespace in which the DM cluster is deployed. You can also add your customized hosts.

Finally, generate the DM-worker server-side certificate:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=internal dm-worker-server.json | cfssljson -bare dm-worker-server

Generate the client-side certificates:
First, generate the default client.json file:
```
cfssl print-defaults csr > client.json
```
Then, edit this file to change the CN, hosts attributes. You can leave the hosts empty:
```
...
    "CN": "TiDB",
    "hosts": [],
...
```
Finally, generate the client-side certificate:
```
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client
```

Create the Kubernetes Secret object:

If you have already generated a set of certificates for each component and a set of client-side certificate for each client as described in the above steps, create the Secret objects for the DM cluster by executing the following command:

The DM-master cluster certificate Secret:

kubectl create secret generic ${cluster_name}-dm-master-cluster-secret --namespace=${namespace} --from-file=tls.crt=dm-master-server.pem --from-file=tls.key=dm-master-server-key.pem --from-file=ca.crt=ca.pem

The DM-worker cluster certificate Secret：

kubectl create secret generic ${cluster_name}-dm-worker-cluster-secret --namespace=${namespace} --from-file=tls.crt=dm-worker-server.pem --from-file=tls.key=dm-worker-server-key.pem --from-file=ca.crt=ca.pem

Client certificate Secret：

kubectl create secret generic ${cluster_name}-dm-client-secret --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem

You have created two Secret objects:

One Secret object for each DM-master/DM-worker server-side certificate to load when the server is started;
One Secret object for their clients to connect.

Using `cert-manager`

Install cert-manager.
Refer to cert-manager installation in Kubernetes for details.

Create an Issuer to issue certificates to the DM cluster.

To configure cert-manager, create the Issuer resources.

First, create a directory which saves the files that cert-manager needs to create certificates:

mkdir -p cert-manager
cd cert-manager

Then, create a dm-cluster-issuer.yaml file with the following content:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ${cluster_name}-selfsigned-ca-issuer
  namespace: ${namespace}
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${cluster_name}-ca
  namespace: ${namespace}
spec:
  secretName: ${cluster_name}-ca-secret
  commonName: "TiDB"
  isCA: true
  duration: 87600h # 10yrs
  renewBefore: 720h # 30d
  issuerRef:
    name: ${cluster_name}-selfsigned-ca-issuer
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ${cluster_name}-dm-issuer
  namespace: ${namespace}
spec:
  ca:
    secretName: ${cluster_name}-ca-secret

${cluster_name} is the name of the cluster. The above YAML file creates three objects:

An Issuer object of the SelfSigned type, used to generate the CA certificate needed by Issuer of the CA type;
A Certificate object, whose isCa is set to true.
An Issuer, used to issue TLS certificates between components of the DM cluster.

Finally, execute the following command to create an Issuer:

kubectl apply -f dm-cluster-issuer.yaml

Generate the server-side certificate.
In cert-manager, the Certificate resource represents the certificate interface. This certificate is issued and updated by the Issuer created in Step 2.
Each component needs a server-side certificate, and all components need a shared client-side certificate for their clients.
- The DM-master server-side certificate
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${cluster_name}-dm-master-cluster-secret
  namespace: ${namespace}
spec:
  secretName: ${cluster_name}-dm-master-cluster-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - PingCAP
  commonName: "TiDB"
  usages:
    - server auth
    - client auth
  dnsNames:
  - "${cluster_name}-dm-master"
  - "${cluster_name}-dm-master.${namespace}"
  - "${cluster_name}-dm-master.${namespace}.svc"
  - "${cluster_name}-dm-master-peer"
  - "${cluster_name}-dm-master-peer.${namespace}"
  - "${cluster_name}-dm-master-peer.${namespace}.svc"
  - "*.${cluster_name}-dm-master-peer"
  - "*.${cluster_name}-dm-master-peer.${namespace}"
  - "*.${cluster_name}-dm-master-peer.${namespace}.svc"
  ipAddresses:
  - 127.0.0.1
  - ::1
  issuerRef:
    name: ${cluster_name}-dm-issuer
    kind: Issuer
    group: cert-manager.io
```
  ${cluster_name} is the name of the cluster. Configure the items as follows:
  - Set spec.secretName to ${cluster_name}-dm-master-cluster-secret.
  - Add server auth and client auth in usages.
  - Add the following DNSs in dnsNames. You can also add other DNSs according to your needs:
    - "${cluster_name}-dm-master"
    - "${cluster_name}-dm-master.${namespace}"
    - "${cluster_name}-dm-master.${namespace}.svc"
    - "${cluster_name}-dm-master-peer"
    - "${cluster_name}-dm-master-peer.${namespace}"
    - "${cluster_name}-dm-master-peer.${namespace}.svc"
    - "*.${cluster_name}-dm-master-peer"
    - "*.${cluster_name}-dm-master-peer.${namespace}"
    - "*.${cluster_name}-dm-master-peer.${namespace}.svc"
  - Add the following two IPs in ipAddresses. You can also add other IPs according to your needs:
    - 127.0.0.1
    - ::1
  - Add the Issuer created above in issuerRef.
  - For other attributes, refer to cert-manager API.
    After the object is created, cert-manager generates a ${cluster_name}-dm-master-cluster-secret Secret object to be used by the DM-master component of the DM cluster.
- The DM-worker server-side certificate
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${cluster_name}-dm-worker-cluster-secret
  namespace: ${namespace}
spec:
  secretName: ${cluster_name}-dm-worker-cluster-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - PingCAP
  commonName: "TiDB"
  usages:
    - server auth
    - client auth
  dnsNames:
  - "${cluster_name}-dm-worker"
  - "${cluster_name}-dm-worker.${namespace}"
  - "${cluster_name}-dm-worker.${namespace}.svc"
  - "${cluster_name}-dm-worker-peer"
  - "${cluster_name}-dm-worker-peer.${namespace}"
  - "${cluster_name}-dm-worker-peer.${namespace}.svc"
  - "*.${cluster_name}-dm-worker-peer"
  - "*.${cluster_name}-dm-worker-peer.${namespace}"
  - "*.${cluster_name}-dm-worker-peer.${namespace}.svc"
  ipAddresses:
  - 127.0.0.1
  - ::1
  issuerRef:
    name: ${cluster_name}-dm-issuer
    kind: Issuer
    group: cert-manager.io
```
  ${cluster_name} is the name of the cluster. Configure the items as follows:
  - Set spec.secretName to ${cluster_name}-dm-worker-cluster-secret.
  - Add server auth and client auth in usages.
  - Add the following DNSs in dnsNames. You can also add other DNSs according to your needs:
    - "${cluster_name}-dm-worker"
    - "${cluster_name}-dm-worker.${namespace}"
    - "${cluster_name}-dm-worker.${namespace}.svc"
    - "${cluster_name}-dm-worker-peer"
    - "${cluster_name}-dm-worker-peer.${namespace}"
    - "${cluster_name}-dm-worker-peer.${namespace}.svc"
    - "*.${cluster_name}-dm-worker-peer"
    - "*.${cluster_name}-dm-worker-peer.${namespace}"
    - "*.${cluster_name}-dm-worker-peer.${namespace}.svc"
  - Add the following two IPs in ipAddresses. You can also add other IPs according to your needs:
    - 127.0.0.1
    - ::1
  - Add the Issuer created above in issuerRef.
  - For other attributes, refer to cert-manager API.
    After the object is created, cert-manager generates a ${cluster_name}-dm-cluster-secret Secret object to be used by the DM-worker component of the DM cluster.
- A set of client-side certificates of DM cluster components.
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${cluster_name}-dm-client-secret
  namespace: ${namespace}
spec:
  secretName: ${cluster_name}-dm-client-secret
  duration: 8760h # 365d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - PingCAP
  commonName: "TiDB"
  usages:
    - client auth
  issuerRef:
    name: ${cluster_name}-dm-issuer
    kind: Issuer
    group: cert-manager.io
```
  ${cluster_name} is the name of the cluster. The above YAML file creates three objects:
  - Set spec.secretName to ${cluster_name}-dm-master-cluster-secret.
  - Add server auth and client auth in usages.
  - dnsNames and ipAddresses are not required.
  - Add the Issuer created above in the issuerRef
  - For other attributes, refer to cert-manager API
    After the object is created, cert-manager generates a ${cluster_name}-cluster-client-secret Secret object to be used by the clients of the DM components.

Deploy the DM cluster

When you deploy a DM cluster, you can enable TLS between DM components, and set the cert-allowed-cn configuration item to verify the CN (Common Name) of each component's certificate.

note

Currently, you can set only one value for the cert-allowed-cn configuration item of DM-master. Therefore, the commonName of all Certificate objects must be the same.

Create the dm-cluster.yaml file:

apiVersion: pingcap.com/v1alpha1
kind: DMCluster
metadata:
  name: ${cluster_name}
  namespace: ${namespace}
spec:
  tlsCluster:
    enabled: true
  version: v5.3.0
  pvReclaimPolicy: Retain
  discovery: {}
  master:
    baseImage: pingcap/dm
    maxFailoverCount: 0
    replicas: 1
    storageSize: "1Gi"
    config:
      cert-allowed-cn:
        - TiDB
  worker:
    baseImage: pingcap/dm
    maxFailoverCount: 0
    replicas: 1
    storageSize: "1Gi"
    config:
      cert-allowed-cn:
        - TiDB

Use the kubectl apply -f dm-cluster.yaml file to create a DM cluster.

Configure `dmctl` and connect to the cluster

Get into the DM-master Pod:

kubectl exec -it ${cluster_name}-dm-master-0 -n ${namespace} sh

Use dmctl:

cd /var/lib/dm-master-tls
/dmctl --ssl-ca=ca.crt --ssl-cert=tls.crt --ssl-key=tls.key --master-addr 127.0.0.1:8261 list-member

Use DM to migrate data between MySQL/TiDB databases that enable TLS for the MySQL client

This section describes how to configure DM to migrate data between MySQL/TiDB databases that enable TLS for the MySQL client.

To learn how to enable TLS for the MySQL client of TiDB, refer to Enable TLS for the MySQL Client.

Step 1: Create the Kubernetes Secret object for each TLS-enabled MySQL

Suppose you have deployed a MySQL/TiDB database with TLS-enabled for the MySQL client. To create Secret objects for the TiDB cluster, execute the following command:

kubectl create secret generic ${mysql_secret_name1} --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem
kubectl create secret generic ${tidb_secret_name} --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem

Step 2: Mount the Secret objects to the DM cluster

After creating the Kubernetes Secret objects for the upstream and downstream databases, you need to set spec.tlsClientSecretNames so that you can mount the Secret objects to the Pod of DM-master/DM-worker.

apiVersion: pingcap.com/v1alpha1
kind: DMCluster
metadata:
  name: ${cluster_name}
  namespace: ${namespace}
spec:
  version: v5.3.0
  pvReclaimPolicy: Retain
  discovery: {}
  tlsClientSecretNames:
    - ${mysql_secret_name1}
    - ${tidb_secret_name}
  master:
    ...

Step 3: Modify the data source and migration task configuration

After configuring spec.tlsClientSecretNames, TiDB Operator will mount the Secret objects ${secret_name} to the path /var/lib/source-tls/${secret_name}.

Configure from.security in the source1.yaml file as described in the data source configuration:

source-id: mysql-replica-01
relay-dir: /var/lib/dm-worker/relay
from:
  host: ${mysql_host1}
  user: dm
  password: ""
  port: 3306
  security:
    ssl-ca: /var/lib/source-tls/${mysql_secret_name1}/ca.crt
    ssl-cert: /var/lib/source-tls/${mysql_secret_name1}/tls.crt
    ssl-key: /var/lib/source-tls/${mysql_secret_name1}/tls.key

Configure target-database.security in the task.yaml file as described in the Configure Migration Tasks:

name: test
task-mode: all
is-sharding: false

target-database:
  host: ${tidb_host}
  port: 4000
  user: "root"
  password: ""
  security:
    ssl-ca: /var/lib/source-tls/${tidb_secret_name}/ca.crt
    ssl-cert: /var/lib/source-tls/${tidb_secret_name}/tls.crt
    ssl-key: /var/lib/source-tls/${tidb_secret_name}/tls.key

mysql-instances:
- source-id: "replica-01"
  loader-config-name: "global"

loaders:
  global:
    dir: "/var/lib/dm-worker/dumped_data"

Step 4: Start the migration tasks

Refer to Start the migration tasks.

Enable TLS for DM

Enable TLS between DM components​

Generate certificates for components of the DM cluster​

Using cfssl​

Using cert-manager​

Deploy the DM cluster​

note

Configure dmctl and connect to the cluster​

Use DM to migrate data between MySQL/TiDB databases that enable TLS for the MySQL client​

Step 1: Create the Kubernetes Secret object for each TLS-enabled MySQL​

Step 2: Mount the Secret objects to the DM cluster​

Step 3: Modify the data source and migration task configuration​

Step 4: Start the migration tasks​

Enable TLS between DM components

Generate certificates for components of the DM cluster

Using `cfssl`

Using `cert-manager`

Deploy the DM cluster

Configure `dmctl` and connect to the cluster

Use DM to migrate data between MySQL/TiDB databases that enable TLS for the MySQL client

Step 1: Create the Kubernetes Secret object for each TLS-enabled MySQL

Step 2: Mount the Secret objects to the DM cluster

Step 3: Modify the data source and migration task configuration

Step 4: Start the migration tasks