连续花了两天时间在搭建Zookeeper集群这件事上了,碰到了N多坑,终于是搭建成功了。
准备工作
- Zookeeper镜像,经过我的测试,在Kubernetes上搭建Zookeeper需要用Kubernetes-Zookeeper这个镜像,而不是用官方的Zookeeper镜像(后面有时间再试一试),Kubernetes-Zookeeper这个镜像在国内不好拿,我是采用docker hub构建出来的,Dockerfile如下:
FROM k8s.gcr.io/kubernetes-zookeeper:1.0-3.4.10
MAINTAINER leo.lee <lis85@163.com>
当然你也可以直接拉取我构建成功的镜像
docker pull leolee32/kubernetes-library:kubernetes-zookeeper1.0-3.4.10
- Zookeeper集群需要用到存储,这里需要准备持久卷(PersistentVolume,简称PV),我这里以yaml文件创建3个PV,供待会儿3个Zookeeper节点创建出来的持久卷声明(PersistentVolumeClaim,简称PVC)绑定。
persistent-volume.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: k8s-pv-zk1
annotations:
volume.beta.kubernetes.io/storage-class: "anything"
labels:
type: local
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/lib/zookeeper"
persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: k8s-pv-zk2
annotations:
volume.beta.kubernetes.io/storage-class: "anything"
labels:
type: local
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/lib/zookeeper"
persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: k8s-pv-zk3
annotations:
volume.beta.kubernetes.io/storage-class: "anything"
labels:
type: local
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/var/lib/zookeeper"
persistentVolumeReclaimPolicy: Recycle
使用如下命令创建
kubectl create -f persistent-volume.yaml
查看PV
kubectl get pv -o wide
部署Zookeeper集群
zookeeper.yaml
apiVersion: v1
kind: Service
metadata:
name: zk-hs
labels:
app: zk
spec:
ports:
- port: 2888
name: server
- port: 3888
name: leader-election
clusterIP: None
selector:
app: zk
---
apiVersion: v1
kind: Service
metadata:
name: zk-cs
labels:
app: zk
spec:
ports:
- port: 2181
name: client
selector:
app: zk
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
selector:
matchLabels:
app: zk
maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zk
spec:
selector:
matchLabels:
app: zk
serviceName: zk-hs
replicas: 3
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
template:
metadata:
labels:
app: zk
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- zk
topologyKey: "kubernetes.io/hostname"
containers:
- name: kubernetes-zookeeper
imagePullPolicy: Always
image: "192.168.242.132/library/kubernetes-zookeeper:1.0-3.4.10"
resources:
requests:
memory: "1Gi"
cpu: "0.5"
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: server
- containerPort: 3888
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers=3 \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port=2181 \
--election_port=3888 \
--server_port=2888 \
--tick_time=2000 \
--init_limit=10 \
--sync_limit=5 \
--heap=512M \
--max_client_cnxns=60 \
--snap_retain_count=3 \
--purge_interval=12 \
--max_session_timeout=40000 \
--min_session_timeout=4000 \
--log_level=INFO"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
securityContext:
runAsUser: 1000
fsGroup: 1000
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.beta.kubernetes.io/storage-class: "anything"
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 3Gi
使用如下命令创建
kubectl create -f zookeeper.yaml
创建完后会出现一个问题,就是所有的Zookeeper pod都启动不起来,查看日志发现是用户对文件夹【/var/lib/zookeeper】没有权限引起的,文件夹的权限是root用户。
这里如何通过安装解决还有待研究,目前可以先手动把文件夹【/var/lib/zookeeper】的权限修改为普通用户,Zookeeper pod就可以正常启动了。
通过命令查看pod
kubectl get pod -o wide
查看PV,发现持久卷声明已经绑定上了。
kubectl get pv -o wide
查看PVC
kubectl get pvc -o wide
最后来验证Zookeeper集群是否正常,查看集群节点状态
for i in 0 1 2; do kubectl exec zk-$i zkServer.sh status; done
一个leader,两个follower,成功!!!