背景

随着elk版本不断升级，特别是4月份es出了7.0大版本，到最近的7.2.0小版本，有了很多实用新功能，这边准备进行升级。

由于我司全面推广k8s项目，所有新上项目都会考虑部署至k8s集群，因此本次升级也是在k8s环境下进行的。

升级步骤

• 官网上分别拉取elasticsearch/kibana/logstash最新7.2.0版本镜像，修改tag，并上传至公司私有镜像。
• 修改yml文件，删除原先容器，开启新容器，完成升级。

遇到的坑

当然，升级过程没这么简单，遇到了几个问题点在这里列下。

集群探测配置

新版废弃了原先discovery.zen.ping.unicast.hosts及discovery.zen.minimum_master_nodes的探测方式，而是改为了discovery.seed_hosts及cluster.initial_master_nodes，其中第一项换汤不换药，值还是一样的，我们还是填内部服务域名，而后一项就有点恶心了，这项在实体服务器上配置没任何问题，把符合master节点的ip或主机名配进去就可以了，但在k8s环境下，master节点的ip是pod ip，随即分配的，主机名由于是deployment下部署的，主机名的尾巴也是随机字符，同样无法指定，不配的话各节点服务器无法选举master，无法组成集群：

{“type”: “server”, “timestamp”: “2019-06-26T11:03:02,636+0000”, “level”: “WARN”, “component”: “o.e.c.c.ClusterFormationFailureHelper”, “cluster.name”: “elasticsearch-cluster-v7”, “node.name”: “elasticsearch-master-8b5bc975b-vdb94”, “message”: “master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [HOSTNAME] to bootstrap a cluster: have discovered []; discovery will continue using [192.153.20.100:9300] from hosts providers and [{elasticsearch-master-8b5bc975b-vdb94}{HMo1b8jfSXSNtl1VSTlMsg}{yCQqrg3xTcqO7eXQr3sEnQ}{192.152.13.233}{192.152.13.233:9300}{ml.machine_memory=33728757760, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0” }

- name: cluster.initial_master_nodes
      value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2"

master成功探测对方的日志如下：

{“type”: “server”, “timestamp”: “2019-06-27T10:30:40,996+0000”, “level”: “WARN”, “component”: “o.e.c.c.ClusterFormationFailureHelper”, “cluster.name”: “elasticsearch-cluster-v7”, “node.name”: “elasticsearch-master-2”, “message”: “master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-service] to bootstrap a cluster: have discovered [{elasticsearch-master-1}{EnGhzi7JRLmDkaYuZKuADg}{t0kR5vjaSsKpOfbptBbZqQ}{192.152.203.168}{192.152.203.168:9300}{ml.machine_memory=33728757760, ml.max_open_jobs=20, xpack.installed=true}, {elasticsearch-master-0}{CZ60XOP3QwSJth9Fp2gWzA}{7QM1yDB_SKCHv5wEPlaXBw}{192.152.13.201}{192.152.13.201:9300}{ml.machine_memory=33728757760, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [192.153.144.253:9300] from hosts providers and [{elasticsearch-master-2}{GxnZZFVUSB2rJwAkA9sGfw}{rX8ha6MfQaq4GjF80MOTeA}{192.152.53.241}{192.152.53.241:9300}{ml.machine_memory=33728757760, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0” }

master及data集群uuid冲突问题

上个问题解决后，又遇到了新的问题，master集群节点可正常启动，但是data节点启动后，无法加入master集群，查看日志显示uuid不匹配：

“Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid bb79xFABQAq_gkKEc1TTpw than local cluster uuid DekASqJ2TIOtB12Q6KeZEw, rejecting”

 

 

解决了这两个问题后，新版es集群也即部署好了，kibana及logstash则较为简单，配置不用便，修改下镜像版本就可以了。

参考文档

• 通过环境变量将Pod信息呈现给容器
• Master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster
• elasticsearch-kubed

转载自：https://stephenzhou.net/2019/06/28/k8s-elastic-upgrade/