一、部署etcd集群
1、集群规划
10.4.7.12 leader
10.4.7.21 follower
10.4.7.22 follower
本例部署以10.4.7.12为例，另外两台安装类似
2、签发证书
在10.4.7.200上操作
创建基于根证书的config配置文件
[root@hdss7-200 ~]# vim /opt/certs/ca-config.json
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
注释：
Server certificate：服务端使用，客户端以此验证服务端身份，例如docker服务端，kube-apiserver服务端；
Client certificate：客户端使用，用于服务端验证客户端身份，etcdctl、etcd-proxy、fleetctl、docker客户端；
Peer certificate：双向证书，用于etcd集群之间相互通信。
3、创建生成自签证书签名请求（csr）的JSON配置文件
[root@hdss7-200 ~]# vim /opt/certs/etcd-peer-csr.json
{
"CN": "k8s-etcd",
"hosts": [
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
注意：重点在hosts上，将所有可能的etcd服务器添加到host列表，不能使用网段，新增etcd服务器需要重新签发证书
4、签发证书
[root@hdss7-200 harbor]# cd /opt/certs/
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
[root@hdss7-200 certs]# ls etcd*
etcd-peer.csr etcd-peer-csr.json etcd-peer-key.pem etcd-peer.pem
5、安装etcd服务
在10.4.7.12/21/22上部署，以12为例
[root@hdss7-12 ~]# mkdir /opt/src
[root@hdss7-12 ~]# cd /opt/src/
[root@hdss7-12 src]# curl -L https://github.com/coreos/etcd/releases/download/v3.3.1/etcd-v3.3.1-linux-amd64.tar.gz -o etcd-v3.3.1-linux-amd64.tar.gz
[root@hdss7-12 src]# tar -zxvf etcd-v3.3.1-linux-amd64.tar.gz -C /opt/etcd-v3.3.1
[root@hdss7-12 src]# ln -s /opt/etcd-v3.3.1 /opt/etcd
[root@hdss7-12 src]# ll /opt/
总用量 8
lrwxrwxrwx 1 root root 16 6月 8 20:36 etcd -> /opt/etcd-v3.3.1
drwxr-xr-x 5 etcd etcd 4096 7月 9 20:40 etcd-v3.3.1
drwxr-xr-x 2 root root 4096 7月 9 20:36 src
创建目录
[root@hdss7-12 src]# mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
6、下发证书
将10.4.7.200上生成的证书分配到etcd集群服务器上
[root@hdss7-200 certs]# for i in 12 21 22;do scp ca.pem etcd-peer.pem etcd-peer-key.pem hdss7-${i}:/opt/etcd/certs/ ;done
7、创建etcd启动脚本
[root@hdss7-12 src]# vim /opt/etcd/etcd-server-startup.sh
#!/bin/sh
# listen-peer-urls etcd节点之间通信端口
# listen-client-urls 客户端与etcd通信端口
# quota-backend-bytes 配额大小
# 需要修改的参数：name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls,advertise-client-urls
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/etcd/etcd --name etcd-server-7-12 --data-dir /data/etcd/etcd-server --listen-peer-urls https://10.4.7.12:2380 --listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 --quota-backend-bytes 8000000000 --initial-advertise-peer-urls https://10.4.7.12:2380 --advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 --initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 --ca-file ./certs/ca.pem --cert-file ./certs/etcd-peer.pem --key-file ./certs/etcd-peer-key.pem --client-cert-auth --trusted-ca-file ./certs/ca.pem --peer-ca-file ./certs/ca.pem --peer-cert-file ./certs/etcd-peer.pem --peer-key-file ./certs/etcd-peer-key.pem --peer-client-cert-auth --peer-trusted-ca-file ./certs/ca.pem --log-output stdout
[root@hdss7-12 src]# chmod +x /opt/etcd/etcd-server-startup.sh
[root@hdss7-12 src]# chown -R etcd.etcd /opt/etcd/ /data/etcd /data/logs/etcd-server
8、安装后台管理工具supervisor
[root@hdss7-12 src]# yum -y install supervisor
[root@hdss7-12 src]# systemctl start supervisord ; systemctl enable supervisord
[root@hdss7-12 src]# vim /etc/supervisord.d/etcd-server.ini
[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=30 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; ‘expected‘ exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=5 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in ‘capturemode‘ (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
[root@hdss7-12 src]# supervisorctl update
etcd-server-7-12: added process group/
[root@hdss7-12 src]# supervisorctl start etcd-server-7-12
etcd-server-7-12: started
9、查看集群服务状态
[root@hdss7-12 src]# supervisorctl status
etcd-server-7-12 RUNNING pid 2270, uptime 0:00:38
查看etcd后台端口2379与2380
查看器群状态，如下是集群未全部启动
[root@hdss7-12 src] /opt/etcd/etcdctl member list
client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://127.0.0.1:2379 exceeded header timeout
; error #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused
如下是起来两个节点的时候
[root@hdss7-12 src] /opt/etcd/etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs= isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
全部起来的情况
[root@hdss7-12 src] /opt/etcd/etcdctl member list
988139385f78284: name=etcd-server-7-22 peerURLs=https://10.4.7.22:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
5a0ef2a004fc4349: name=etcd-server-7-21 peerURLs=https://10.4.7.21:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=false
f4a0cb0a765574a8: name=etcd-server-7-12 peerURLs=https://10.4.7.12:2380 clientURLs=http://127.0.0.1:2379,https://10.4.7.12:2379 isLeader=true
随着etcd服务的重启会发生变化
查看健康状态
[root@hdss7-12 src] /opt/etcd/etcdctl cluster-health
member 988139385f78284 is healthy: got healthy result from http://127.0.0.1:2379
member 5a0ef2a004fc4349 is healthy: got healthy result from http://127.0.0.1:2379
member f4a0cb0a765574a8 is healthy: got healthy result from http://127.0.0.1:2379
cluster is healthy
启停方式
src]# supervisorctl stop etcd-server-7-12
src]# supervisorctl start etcd-server-7-12
src]# supervisorctl restart etcd-server-7-12
src]# supervisorctl status etcd-server-7-12