在K8S集群中一步步构建一个复杂的MySQL数据库

服务器

浏览数:227

2019-3-18

AD:资源代下载服务

文档说明

​ 本文面向容器初学者,作者先简单的用MySQL官方镜像搭建一个可运行的单实例数据库,而后考虑生产或现实需求,一步一步完善并揉合K8S多个技术,从而构建一个复杂且可供生产用的MySQL单实例库。

简单部署

​ 如下所示,我们仅需设置root用户密码(环境变量MYSQL_ROOT_PASSWORD), 便可轻松的使用MySQL官方镜像构建一个MySQL数据库。

# kubectl create -f - <<EOF
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: mysql
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql
        name: mysql
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: Changeme
EOF

注意:若你的K8S集群是minishiftopenshiftorigin,因其为安全考虑,不允许容器以root用户运行,而官方MySQL镜像却需root权限,故为使其能顺利运行,我们需将anyuid scc赋予default serviceaccount

# oc adm policy add-scc-to-user anyuid -z default

​ 创建一Service以便集群内外均可访问数据库,其中集群外需通过nodePort设置的30006端口访问。

# kubectl create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    app: mysql
  name: mysql
spec:
  type: NodePort
  ports:
  - port: 3306
    nodePort: 30006
    protocol: TCP
    targetPort: 3306
  selector:
    app: mysql
EOF

​ 接着,访问数据库并验证其运行正常:

# kubectl get pod                                   # 当前Pod名称
NAME                     READY     STATUS    RESTARTS   AGE
mysql-5b5668c448-t44ml   1/1       Running   0          3h

# 通过本机访问
# kubectl exec -it mysql-5b5668c448-t44ml -- mysql -uroot -pChangeme

mysql> select 1;
+---+
| 1 |
+---+
| 1 |
+---+

# 集群内部通过mysql service访问:
# kubectl exec -it mysql-5b5668c448-t44ml -- mysql -uroot -pChangeme -hmysql

mysql> select now();
+---------------------+
| now()               |
+---------------------+
| 2018-05-21 07:19:14 |
+---------------------+

# 集群外部,可通过任何一个K8S节点访问数据库:
# mysql -uroot -pChangeme -horigin-lb-01 -P30006

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+

扩展部署

持久化存储

​ 若要确保MySQL重启后数据仍然存在,我们需为其配置可持久化存储,作者的实验环境配置了GlusterFS分布式存储,其支持K8S动态提供特性,故可执行如下命令创建PVC

# kubectl create -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: glusterfs-raid0
EOF

​ 而后,调整Deploy并挂载卷:

    spec:
      containers:
      - image: mysql
...
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-data
        persistentVolumeClaim:
          claimName: mysql

自定义配置文件

​ 通过创建cm并挂载到容器中,我们可自定义MySQL配置文件。如下所示,名为mysql-configcm包含一个custom.cnf文件:

apiVersion: v1
metadata:
  name: mysql-config
data:
  custom.cnf: |
        [mysqld]
        default_storage_engine=innodb
        skip_external_locking
        lower_case_table_names=1
        skip_host_cache
        skip_name_resolve
kind: ConfigMap

​ 将cm挂载到容器内:

    spec:
...
      containers:
      - image: mysql
...
        volumeMounts:
        - name: mysql-config
          mountPath: /etc/mysql/conf.d/
...
      volumes:
      - name: mysql-config
        configMap:
          name: mysql-config
...

加密铭感数据

​ 用户密码等铭感数据以Secret加密保存,而后被Deployment通过volume挂载或环境变量引用。如本例,我们创建rootapptest用户,将3个用户的密码加密保存:

# echo -n Changeme | base64
Q2hhbmdlbWU=
# kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: mysql-user-pwd
data:
  mysql-root-pwd: Q2hhbmdlbWU=
  mysql-app-user-pwd: Q2hhbmdlbWU=
  mysql-test-user-pwd: Q2hhbmdlbWU=
EOF

Secret创建完成后,我们将用户明文密码从Deployment去除,采用环境变量方式引用Secret数据,参见如下Yaml修改,做了3个调整:

  • 镜像初始化时自动创建MYSQL_DATABASE环境变量1设置的数据库;
  • 镜像初始化时将MYSQL_DATABASE数据库赋予MYSQL_USER用户;
  • root用户及MYSQL_USER用户,其密码均通过secretKeyRefsecret获取;
    spec:
...
      containers:
      - image: mysql
        name: mysql
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pwd
              key: mysql-root-pwd
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pwd
              key: mysql-app-user-pwd
        - name: MYSQL_USER
          value: app
        - name: MYSQL_DATABASE
          value: appdb

容器健康检查

K8S镜像控制器可通过livenessProbe判断容器是否异常,进而决定是否重建容器;而Service服务可通过readinessProbe判断容器服务是否正常,从而确保服务可用性。

​ 本例,作者配置的livenessProbereadinessProbe是一样的,即连续3次查询数据库失败,则定义为异常。对livenessProbereadinessProbe详细用法,不在本文的讨论范围内,可参考K8S官方文档:

    spec:
      containers:
      - image: mysql
...
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - "-c"
            - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
            - mysql -h 127.0.0.1 -u root -e "SELECT 1"
          initialDelaySeconds: 30
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - "-c"
            - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
            - mysql -h 127.0.0.1 -u root -e "SELECT 1"
          initialDelaySeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
...

容器初始化

假设,我们有这样的需求:“初始部署MySQL时,其已包应用所需的数据库、用户、权限、表结构与数据”。研究MySQL官方镜像的Dockerfile可知,数据库初始化时将自动执行目录/docker-entrypoint-initdb.d内的.sh.sql.sql.gz文件,鉴于此,我们可有如下两种方法:

  1. 基于官方镜像重新编写Dockerfile,将脚本copy到新镜像/docker-entrypoint-initdb.d目录,因需编译新镜像,故此方法不灵活;
  2. 初始化容器(initContainers)在常规容器(containers)前运行,故在初始化容器中可将脚本拷贝到共享目录,而后MySQL镜像挂载此目录到/docker-entrypoint-initdb.d,此方法灵活。

本例,作者采用初始化容器方案,功能如下:

  • 初始化与常规容器共享名为mysql-initdbemptyDir,均被挂载到/docker-entrypoint-initdb.d目录;
  • 初始化容器将.sql文件置于共享的/docker-entrypoint-initdb.d目录,其含初始化testdbappdb数据库;
  • 为了避免MySQL数据库目录内的lost+found目录被误认为是数据库,初始化容器中将其删除;
    spec:
      initContainers:
      - name: mysql-init
        image: busybox
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_TEST_USER_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pwd
              key: mysql-test-user-pwd
        command:  
          - sh
          - "-c"
          - |
            set -ex
            rm -fr /var/lib/mysql/lost+found
            cat > /docker-entrypoint-initdb.d/mysql-testdb-init.sql <<EOF
            create database testdb default character set utf8;
            grant all on testdb.* to 'test'@'%' identified by '$MYSQL_TEST_USER_PASSWORD';
            flush privileges;
            EOF
            cat > /docker-entrypoint-initdb.d/mysql-appdb-init.sql <<EOF
            create table app(id int);
            insert into app values(1);
            commit;
            EOF
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
        - name: mysql-initdb
          mountPath: /docker-entrypoint-initdb.d
      containers:
      - image: mysql
        volumeMounts:
        - name: mysql-initdb
          mountPath: /docker-entrypoint-initdb.d
...
      volumes:
      - name: mysql-data
        persistentVolumeClaim:
          claimName: mysql
      - name: mysql-initdb
        emptyDir: {}
...

完整Deployment

​ 通过如上多步调整,MySQL数据库的Deplyment如下所示:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: mysql
  name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      initContainers:
      - name: mysql-init
        image: busybox
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_TEST_USER_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pwd
              key: mysql-test-user-pwd
        command:  
          - sh
          - "-c"
          - |
            set -ex
            rm -fr /var/lib/mysql/lost+found
            cat > /docker-entrypoint-initdb.d/mysql-testdb-init.sql <<EOF
            create database testdb default character set utf8;
            grant all on testdb.* to 'test'@'%' identified by '$MYSQL_TEST_USER_PASSWORD';
            flush privileges;
            EOF
            cat > /docker-entrypoint-initdb.d/mysql-appdb-init.sql <<EOF
            create table app(id int);
            insert into app values(1);
            commit;
            EOF
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
        - name: mysql-initdb
          mountPath: /docker-entrypoint-initdb.d
      containers:
      - image: mysql
        name: mysql
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pwd
              key: mysql-root-pwd
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pwd
              key: mysql-app-user-pwd
        - name: MYSQL_USER
          value: app
        - name: MYSQL_DATABASE
          value: appdb
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
        - name: mysql-initdb
          mountPath: /docker-entrypoint-initdb.d
        - name: mysql-config
          mountPath: /etc/mysql/conf.d/
        ports:
        - name: mysql
          containerPort: 3306
            command:
            - /bin/sh
            - "-c"
            - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
            - mysql -h 127.0.0.1 -u root -e "SELECT 1"
          initialDelaySeconds: 30
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - "-c"
            - MYSQL_PWD="${MYSQL_ROOT_PASSWORD}"
            - mysql -h 127.0.0.1 -u root -e "SELECT 1"
          initialDelaySeconds: 10
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
      volumes:
      - name: mysql-data
        persistentVolumeClaim:
          claimName: mysql
      - name: mysql-initdb
        emptyDir: {}
      - name: mysql-config
        configMap:
          name: mysql-config

创建此Deployment后,我们有如下组件:

# kubectl get all,pvc,cm,secret
# MySQL Deployment:
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/mysql   1         1         1            1           1m

# RS被Deployment调用,其是自动生成的
NAME                 DESIRED   CURRENT   READY     AGE
rs/mysql-998977cdd   1         1         1         1m

# Pod:
NAME                       READY     STATUS    RESTARTS   AGE
po/mysql-998977cdd-v2ks2   1/1       Running   1          1m

# Service:
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
svc/mysql         NodePort    172.30.3.200    <none>        3306:30006/TCP   8h

# Pvc:
NAME            STATUS    VOLUME     CAPACITY   ACCESS MODES STORAGECLASS      AGE
pvc/mysql       Bound     pvc-fe..   1Gi        ROX          glusterfs-raid0   2m

# Configmap:
NAME              DATA      AGE
cm/mysql-config   1         6h

# Secret:
NAME                        TYPE                                  DATA      AGE
secrets/mysql-user-pwd      Opaque                                3         1h

定期自动备份

​ 考虑到数据安全性,我们定期备份数据库,在K8S集群中,我们可配置CronJob实现自动备份作业。首先,创建一个持久化存储供备份用:

# kubectl create -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-backup
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: glusterfs-raid0
EOF

​ 继而,配置实际的自动化作业任务,如下所示,每天凌晨0点将使用mysqldump备份appdb数据库。

# kubectl create -f - <<EOF
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: mysql-backup
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: mysql-backup
            imagePullPolicy: IfNotPresent
            image: mysql
            env:
            - name: MYSQL_BACKUP_USER
              value: root
            - name: MYSQL_BACKUP_USER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-user-pwd
                  key: mysql-root-pwd
            - name: MYSQL_HOST
              value: mysql
            command:
            - /bin/sh
            - -c
            - |
              set -ex
              mysqldump --host=$MYSQL_HOST --user=$MYSQL_BACKUP_USER \
                        --password=$MYSQL_BACKUP_USER_PASSWORD \
                        --routines --databases appdb --single-transaction \
                        > /mysql-backup/mysql-`date +"%Y%m%d"`.sql
            volumeMounts:
            - name: mysql-backup
              mountPath: /mysql-backup
          restartPolicy: OnFailure
          volumes:
          - name: mysql-backup
            persistentVolumeClaim:
              claimName: mysql-backup
EOF

结束语

​ 本文揉合K8S多项技术,构建了一个复杂且可做生产使用的范例,当然,此库是单实例数据库,倘若需构建数据库高可用方案,需部署如MySQL HAPXC集群,其中自动作业备份范例仅使用mysqldump备份,在生产环境不是很实用,我们需要考虑使用xtrabackup备份以及mysqlbinlog备份日志。

  1. 参见docker-entrypoint.sh