关于 Kibana 、Elasticsearch 奇怪的知识点,一般人可能永远遇不到

最近在公司折腾 Kibana 的时候遇到了个还挺有意思的问题,估计正常情况下几乎没人遇到。

先简单说说 Kibana 和 Elasticsearch ,Elasticsearch 是个分布式、RESTful 风格、非常强大的搜索引擎,被广泛地使用于各个IT公司。与 Logstash 及 Kibana 开源项目组合在一起,形成了 ELK 软件栈。

Elasticsearch 常年霸占 DB-Engines 搜索引擎排名的榜首,且与第二名有不小的差距。

关于 Kibana 、Elasticsearch 奇怪的知识点,一般人可能永远遇不到

Elasticsearch 在各大厂商有广泛的应用,一般有拿来做日志实时分析,搜索服务,比如搜索商品、文档,时序数据分析等等。

这次是在做 Amazon ES 与 自建 ES 服务对比时遇到的问题。问题现象是在AWS 的 EC2 主机上装了 官网下载的 kibana 后,连接不上 Amazon ES 服务。

https://www.elastic.co/downloads/kibana

下面是详细经过

首先在AWS的管理页面新建了 Amazon ES(7.10版本),确认同内网的EC2主机能正常访问endpoint后,打算在EC2上安装本地 kibana 服务器,连接到 Amazon ES。

根据AWS官方文档对 config/kibana.yml 配置文件进行修改

https://docs.amazonaws.cn/elasticsearch-service/latest/developerguide/es-kibana.html#es-kibana-local

kibana.index: ".kibanalocal"# Use elasticsearch.url for versions older than 6.6# elasticsearch.url: "https://domain-endpoint:443"# Use elasticsearch.hosts for versions 6.6 and laterelasticsearch.hosts: "https://domain-endpoint:443"

然后使用 systemctl 启动服务,发现一直是启动异常。

ubuntu@11:/usr/share/kibana/config$ sudo systemctl status kibana.service ● kibana.service - Kibana   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-04-07 10:12:15 UTC; 1s ago  Process: 19550 ExecStart=/usr/share/kibana/bin/kibana (code=exited, status=64) Main PID: 19550 (code=exited, status=64)

既然 systemctl 无法启动,那就直接使用 kibana 命令行启动,进一步查看下报错信息去解决。

sudo -u kibana -EH ../bin/kibana  log   [10:06:16.366] [info][plugins-service] Plugin "auditTrail" is disabled.  log   [10:06:16.372] [info][plugins-service] Plugin "visTypeXy" is disabled.  log   [10:06:16.446] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."  ...  log   [10:06:16.776] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml  log   [10:06:16.776] [warning][config][plugins][security] Session cookies will be transmitted over insecure connections. This is not recommended.  log   [10:06:16.808] [warning][config][encryptedSavedObjects][plugins] Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml  log   [10:06:16.813] [warning][ingestManager][plugins] Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.  log   [10:06:16.924] [warning][config][plugins][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml  ...  log   [10:06:16.937] [warning][actions][actions][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.  log   [10:06:16.955] [warning][alerting][alerts][plugins][plugins] APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml.  log   [10:06:17.072] [info][monitoring][monitoring][plugins] config sourced from: production cluster  log   [10:06:17.299] [info][savedobjects-service] Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations...  log   [10:06:17.354] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to Authentication Exception :: {"path":"/_xpack?accept_enterprise=true","statusCode":401,"response":"{\"Message\":\"Your request: '/_xpack' is not allowed.\"}"} error  log   [10:06:17.356] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: Authentication Exception  log   [10:06:17.389] [info][savedobjects-service] Starting saved objects migrations  log   [10:06:17.425] [info][savedobjects-service] Creating index .kibanalocal_1.  log   [10:06:17.518] [error][data][elasticsearch] [mapper_parsing_exception]: Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [config]  log   [10:06:17.519] [warning][savedobjects-service] Unable to connect to Elasticsearch. Error: mapper_parsing_exception  log   [10:06:17.520] [fatal][root] { ResponseError: mapper_parsing_exception    at IncomingMessage.response.on (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)    at IncomingMessage.emit (events.js:203:15)    at endReadableNT (_stream_readable.js:1145:12)    at process._tickCallback (internal/process/next_tick.js:63:19)  name: 'ResponseError',  meta:   { body: { error: [Object], status: 400 },     statusCode: 400,     headers:      { date: 'Fri, 07 May 2021 10:06:17 GMT',        'content-type': 'application/json; charset=UTF-8',        'content-length': '399',        connection: 'keep-alive',        'access-control-allow-origin': '*',        server: 'Jetty(8.1.12.v20130726)' },     meta:      { context: null,        request: [Object],        name: 'elasticsearch-js',        connection: [Object],        attempts: 0,        aborted: false } } }  log   [10:06:17.523] [info][plugins-system] Stopping all plugins. FATAL  ResponseError: mapper_parsing_exception

启动日志中 warning 都可以先不看,比如提示设置 xpack.security.encryptionKey 这些警告都不是重点,服务无法启动先看 FATAL ResponseError: mapper_parsing_exception

直接去百度上搜索对应的报错解决办法,会发现与实际遇到的情况不一样,没什么参考价值。

关于 Kibana 、Elasticsearch 奇怪的知识点,一般人可能永远遇不到

经过一番仔细查看官方说明和查找资料,发现原来是 kibana 版本和 ES版本不匹配导致的。

Amazon ES 因为开源许可协议原因只能只用OSS版本,而默认情况下在官网下载的 kibana 是basic版本

https://www.elastic.co/cn/downloads/kibana

需要在选择版本页面选择OSS版本的 kibana 才能和 Amazon ES 连接上

关于 Kibana 、Elasticsearch 奇怪的知识点,一般人可能永远遇不到

直接访问ES的endpoint也可以查看到当前ES的信息,其中也能查看到当前ES是oss版本还是basic版本

# OSS 版本{  "name" : "5960c6706c6fafd1488e0a82062xxxxx",  "cluster_name" : "072146811111:testes",  "cluster_uuid" : "Acbla-GOQbmZIQltrEUxxA",  "version" : {    "number" : "7.10.2",    "build_flavor" : "oss",    "build_type" : "tar",    "build_hash" : "unknown",    "build_date" : "2021-03-10T22:29:10.311605Z",    "build_snapshot" : false,    "lucene_version" : "8.7.0",    "minimum_wire_compatibility_version" : "6.8.0",    "minimum_index_compatibility_version" : "6.0.0-beta1"  },  "tagline" : "You Know, for Search"}# Basic 版本ubuntu@test:/usr/share/kibana$ curl http://172.31.20.84:9200{  "name" : "node-2",  "cluster_name" : "log",  "cluster_uuid" : "l3EHQCP-SQG-PCK6CDxxxx",  "version" : {    "number" : "7.7.1",    "build_flavor" : "default",    "build_type" : "deb",    "build_hash" : "ad56dce891c901a492bb1ee393f12dfffxxxaxx3",    "build_date" : "2020-05-28T16:30:01.040088Z",    "build_snapshot" : false,    "lucene_version" : "8.5.1",    "minimum_wire_compatibility_version" : "6.8.0",    "minimum_index_compatibility_version" : "6.0.0-beta1"  },  "tagline" : "You Know, for Search"}

关于OSS版本和Basic版本差异可以查看官方介绍。

差异还是很大的,不少功能都被阉割了,比如常用的生命周期管理和数据流。

各版本差异对比:

https://www.elastic.co/cn/subscriptions

关于 Kibana 、Elasticsearch 奇怪的知识点,一般人可能永远遇不到

至于为什么Amazon ES 只能使用阉割版本,是因为在 2021年1月20日elastic对 Elasticsearch 和 Kibana 的许可做了变更,不再允许云服务商“白嫖”。

详细经过可以查看官方说明:「对 Elasticsearch 和 Kibana 的许可变更 」

https://www.elastic.co/cn/blog/licensing-change

以及补充说明的两篇博客「 许可协议变更澄清 」和「 为什么我们必须变更许可协议」

如果你们也在开始考虑使用云厂商的ES服务,或许也需要关注下OSS版本的问题。

【推荐阅读】

1、京东云无线宝怎么设置收益高?一个不需要光猫后台超级账号密码的方法

2、两台主机、多台主机怎么共用键鼠?Windows、Mac OS、Linux 多系统怎么共用键鼠?

3、玩玩带ESP32的电子墨水屏模块

上一篇:使用Cmder的几个问题


下一篇:每天进步一点点---------kibana/Grafana场景3 时间曲线