部署etcd集群
etcd 的leader选举机制,要求至少3台或以上的奇数台主机。本次安装涉及:hdss7-12,hdss7-21,hdss7-22。这里以hdss7-12 主机为例,另外两台主机安装部署方式相同。
集群规划
主机名 | 角色 | ip |
---|---|---|
hdss7-12.host.com | etcd lead | 10.4.7.12 |
hdss7-21.host.com | etcd follow | 10.4.7.21 |
hdss7-22.host.com | etcd follow | 10.4.7.22 |
签发etcd证书
证书签发在 hdss7-200 主机上操作
创建 ca-config.json
文件,
vi /opt/certs/ca-config.json
配置如下
{
"signing": {
"default": {
"expiry": "175200h"
},
"profiles": {
"server": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "175200h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
server 表示服务端连接客户端时携带的证书,用于客户端验证服务端身份
-
client 表示客户端连接服务端时携带的证书,用于服务端验证客户端身份
-
peer 表示相互之间连接时使用的证书,如etcd节点之间验证
-
"expiry": "175200h"
证书有效期10年 ,证书过期后集群会立刻宕机
创建etcd证书配置
vi /opt/certs/etcd-peer-csr.json
内容如下,将所有需要的etcd服务器添加到host列表,不能使用网段,新增etcd服务器需要重新签发证书。
{
"CN": "k8s-etcd",
"hosts": [
"10.4.7.11",
"10.4.7.12",
"10.4.7.21",
"10.4.7.22"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
签发证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssljson -bare etcd-peer
# 查看证书
ll etcd-peer*
安装etcd
操作 hdss7-12 主机
etcd地址:https://github.com/etcd-io/etcd/
本次安装涉及:hdss7-12,hdss7-21,hdss7-22
下载etcd
mkdir /opt/src/
cd /opt/src/
wget --no-check-certificate https://github.com/etcd-io/etcd/releases/download/v3.1.20/etcd-v3.1.20-linux-amd64.tar.gz
tar -xf etcd-v3.1.20-linux-amd64.tar.gz
mv etcd-v3.1.20-linux-amd64 /opt/etcd-v3.1.20
ln -s /opt/etcd-v3.1.20 /opt/etcd
ll /opt/etcd
mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server
创建etcd用户
useradd -s /sbin/nologin -M etcd
# 查看
id etcd
复制证书、私钥
cd /opt/etcd/certs
scp hdss7-200:/opt/certs/ca.pem .
scp hdss7-200:/opt/certs/etcd-peer.pem .
scp hdss7-200:/opt/certs/etcd-peer-key.pem .
# for i in 12 21 22;do scp ca.pem etcd-peer.pem etcd-peer-key.pem hdss7-${i}:/opt/etcd/certs/ ;done
创建etcd启动文件
vim /opt/etcd/etcd-server-startup.sh
脚本内容如下
# listen-peer-urls etcd节点之间通信端口
# listen-client-urls 客户端与etcd通信端口
# quota-backend-bytes 配额大小
# 需要修改的参数:name,listen-peer-urls,listen-client-urls,initial-advertise-peer-urls
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/etcd/etcd --name etcd-server-7-12 \
--data-dir /data/etcd/etcd-server \
--listen-peer-urls https://10.4.7.12:2380 \
--listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--quota-backend-bytes 8000000000 \
--initial-advertise-peer-urls https://10.4.7.12:2380 \
--advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
--initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
--ca-file ./certs/ca.pem \
--cert-file ./certs/etcd-peer.pem \
--key-file ./certs/etcd-peer-key.pem \
--client-cert-auth \
--trusted-ca-file ./certs/ca.pem \
--peer-ca-file ./certs/ca.pem \
--peer-cert-file ./certs/etcd-peer.pem \
--peer-key-file ./certs/etcd-peer-key.pem \
--peer-client-cert-auth \
--peer-trusted-ca-file ./certs/ca.pem \
--log-output stdout
修改权限
chmod u+x /opt/etcd/etcd-server-startup.sh
chown -R etcd.etcd /opt/etcd/ /data/etcd /data/logs/etcd-server
启动etcd
让etcd进程在后台运行,此处采用supervisor进程管理工具
安装 supervisor
yum install -y supervisor
systemctl start supervisord ; systemctl enable supervisord
vim /etc/supervisord.d/etcd-server.ini
配置内容如下
[program:etcd-server-7-12] command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/etcd ; directory to cwd to before exec (def no cwd) autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) startretries=3 ; max # of serial start failures (default 3) exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) stopsignal=QUIT ; signal used to kill process (default TERM) stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=etcd ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB) stdout_logfile_backups=5 ; # of stdout logfile backups (default 10) stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0) stdout_events_enabled=false ; emit events on stdout writes (default false)
启动守护进程
supervisorctl update
如果启动失败,重启当前进程
ps -ef | grep supervisord kill -s SIGTERM 2802 supervisord -c /etc/supervisord.conf
验证
# 状态为RUNING
supervisorctl status
# 查看端口 存在2379和2380 启动成功
netstat -luntp|grep etcd
# 查看日志
tail -f /data/logs/etcd-server/etcd.stdout.log
安装其他2台主机
-
hdss7-21
-
hdss7-22
mkdir /opt/src/
cd /opt/src/
# 从hdss7-12拷贝安装包,或者直接从远程下载
scp hdss7-12:/opt/src/etcd-v3.1.20-linux-amd64.tar.gz .
检查状态
etcd节点安装完成后,用etcdctl 检查健康状态,显示 cluster is healthy 。
cd /opt/etcd
./etcdctl cluster-health
# 或者
./etcdctl member list
etcd 启停方式
supervisorctl start etcd-server-7-12 supervisorctl stop etcd-server-7-12 supervisorctl restart etcd-server-7-12 supervisorctl status etcd-server-7-12