26.1 ELK入门
在日常运维工作中，对于系统和业务日志的处理尤为重要。作为运维工程师，假如管理的服务器数量并不是很多，那么不需要借助任何工具也能管理过来。但如果服务器数量非常多，而因为业务的关系，运维和开发人员常常需要通过查看日志来定位问题，很明显不借助任何工具来帮助我们管理日志是不现实的，不仅让人感觉繁琐，效率也会低下。
ELK介绍
需求背景：
1. 业务发展越来越庞大，服务器越来越多； 2. 各种访问日志、应用日志及错误日志的量也越来越多； 3. 开发人员排查问题，需要到服务器上查看日志分析解决问题，不方便； 4. 运营人员需要一些数据，需要我们运维到服务器上分析日志，不方便。
概念：
ELK 是 elastic 公司旗下三款产品ElasticSearch、Logstash、Kibana的首字母组合，也即Elastic Stack包含ElasticSearch、Logstash、Kibana、Beats ElasticSearch是一个搜索引擎，用来搜索、分析、存储日志。它是分布式的，可以横向扩容，可以自动发现，索引自动分片 Logstash用来采集日志，把日志解析为json格式交给ElasticSearch Kibana是一个数据可视化组件，把处理后的结果通过web界面展示 Beats是一个轻量型日志采集器 X-Pack对Elastic Stack提供了安全、警报、监控、报表、图表于一身的扩展包，是收费的
为什么要使用ELK：
一般我们需要进行日志分析场景：直接在日志文件中 grep、awk 就可以获得自己想要的信息。但在规模较大也就是日志量多而复杂的场景中，
此方法效率低下，面临问题包括日志量太大如何归档、文本搜索太慢怎么办、如何多维度查询。需要集中化的日志管理，所有服务器上的日志
收集汇总。常见解决思路是建立集中式日志收集系统，将所有节点上的日志统一收集，管理，访问。
大型系统通常都是一个分布式部署的架构，不同的服务模块部署在不同的服务器上，问题出现时，大部分情况需要根据问题暴露的关键信息，定位到具体的服务器和服务模块，构建一套集中式日志系统，可以提高定位问题的效率。
一个完整的集中式日志系统，需要包含以下几个主要特点：
收集－能够采集多种来源的日志数据 传输－能够稳定的把日志数据传输到*系统 存储－如何存储日志数据 分析－可以支持 UI 分析 警告－能够提供错误报告，监控机制
而ELK则提供了一整套解决方案，并且都是开源软件，之间互相配合使用，完美衔接，高效的满足了很多场合的应用。是目前主流的一种日志系统。
ELK架构：
上面是 ELK 技术栈的一个架构图，从图中可以清楚的看到数据流向： Beats是单一用途的数据传输平台，它可以将多台机器的数据发送到 Logstash 或 ElasticSearch。但 Beats 并不是不可或缺的一环，所以本文中暂不介绍 Logstash是一个动态数据收集管道，支持以 TCP/UDP/HTTP 多种方式收集数据（也可以接受 Beats 传输来的数据），并对数据做进一步丰富或提取字段处理 ElasticSearch是一个基于 JSON 的分布式的搜索和分析引擎，作为 ELK 的核心，它集中存储数据 Kibana是 ELK 的用户界面，它将收集的数据进行可视化展示（各种报表、图形化数据），并提供配置、管理 ELK 的界面。
ELK安装准备
官网：https://www.elastic.co/cn/ ，中文文档：https://elkguide.elasticsearch.cn/
环境准备：
3台机器：lzx：192.168.100.150 ，lzx1：192.168.100.160 ，lzx2:192.168.100.170 角色划分： 3台机器都安装elasticSearch（简称es），1个主节点为lzx，2个数据节点分别是lzx1和lzx2 es主节点lzx上安装kibana，其中1台es数据节点lzx1上安装logstash，另外一台lzx2上安装beats 3台机器都要安装jdk8（openjdk也可以），关闭3防火墙和selinux
- 三台机器都编辑hosts：
# vim /etc/hosts192.168.100.150 lzx 192.168.100.160 lzx1 192.168.100.170 lzx2
- 三台机器都安装openjdk：
# yum install -y java-1.8.0-openjdk# which java/usr/bin/java
es安装
- 三台机器都安装es：
# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch #--import，导入密钥# vim /etc/yum.repos.d/elastic.repo[elasticsearch-6.x]name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md# yum install -y elasticsearch
也可以使用下载rpm包安装
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.0.0.rpm rpm -ivh elasticsearch-6.0.0.rpm
es配置
elasticsearch有两个配置文件：/etc/elasticsearch和/etc/sysconfig/elasticsearch，配置集群我们需要配置/etc/elasticsearch这个配置文件。
- 三台机器都要配置：
lzx
# vim /etc/elasticsearch/elasticsearch.ymlcluster.name: lzxlinux #Cluster里面添加该行，定义集群名node.name: lzx #指定节点主机名，在Node中添加该行node.master: true #表示是否为主节点，在Node中添加该行node.data: false #表示是否是数据节点，在Node中添加该行network.host: 192.168.100.150 #在Network中添加该行，监听ipdiscovery.zen.ping.unicast.hosts: ["192.168.100.150","192.168.100.160","192.168.100.170"]# 在Discovery中添加该行，定义集群中那些角色，可以写ip地址，也可以写主机名
lzx1
# vim /etc/elasticsearch/elasticsearch.ymlcluster.name: lzxlinux node.name: lzx1 node.master: false #表示不是主节点node.data: true #表示是数据节点network.host: 192.168.100.160 discovery.zen.ping.unicast.hosts: ["192.168.100.150","192.168.100.160","192.168.100.170"]
lzx2
# vim /etc/elasticsearch/elasticsearch.ymlcluster.name: lzxlinux node.name: lzx2 node.master: false #表示不是主节点node.data: true #表示是数据节点network.host: 192.168.100.170 discovery.zen.ping.unicast.hosts: ["192.168.100.150","192.168.100.160","192.168.100.170"]
- 全部启动es：
# systemctl start elasticsearch# netstat -lntp |grep javaActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 192.168.100.150:9200 :::* LISTEN 1305/java tcp6 0 0 192.168.100.150:9300 :::* LISTEN 1305/java # ls /var/log/elasticsearch/ #已经生成日志gc.log.0.current lzxlinux_deprecation.log lzxlinux_index_search_slowlog.log lzxlinux_access.log lzxlinux_index_indexing_slowlog.log lzxlinux.log
已经在监听9200和9300端口。
- curl查看es：
lzx上执行
# curl '192.168.100.150:9200/_cluster/health?pretty' #集群健康检查{
"cluster_name" : "lzxlinux", "status" : "green", #status是green就说明集群没问题，如果是yellow或red都说明有问题
"timed_out" : false, "number_of_nodes" : 3, #3个节点
"number_of_data_nodes" : 2, #2个数据节点
"active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0}
还可以查看集群详细信息：
# curl '192.168.100.150:9200/_cluster/state?pretty' #查看集群详细信息
安装kibana
前面讲到过，kibana是一个数据可视化组件，把处理后的结果通过web界面展示，我们需要在主节点机器上安装它。
- lzx上下载安装：
# yum install -y kibana
- 修改配置文件：
# vim /etc/kibana/kibana.ymlserver.port: 5601 server.host: 192.168.100.150 elasticsearch.url: "http://192.168.100.150:9200"logging.dest: /var/log/kibana.log #指定kibana日志路径
- 启动服务：
# touch /var/log/kibana.log; chmod 777 /var/log/kibana.log# systemctl start kibana# ps aux |grep kibanakibana 1062 18.9 18.2 1198292 182080 ? Dsl 21:37 0:08 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml# netstat -lntp |grep nodetcp 0 0 192.168.100.150:5601 0.0.0.0:* LISTEN 1062/node
已经在监听5601端口。
- 访问web界面：
在浏览器输入192.168.150:5601，访问web界面，因为没有安装x-pack，所以没有用户验证。
安装logstash
除了kibana之外，我们还需要安装logstash，按照之前的角色划分，这次在lzx1上操作。
- lzx1上下载安装：
# yum install -y logstash
- 编辑配置文件：
# vim /etc/logstash/conf.d/syslog.conf #写入下面内容input {
syslog {
type => "system-syslog" #定义日志类型
port => 10514 #定义监听端口
}} #input部分定义日志源output {
stdout {
codec => rubydebug #表示将输出在当前屏幕显示出来
}} #output部分定义输出位置
- 检查配置文件是否有错：
# cd /usr/share/logstash/bin/# lsbenchmark.sh logstash logstash.lib.sh pqrepair
cpdump logstash.bat logstash-plugin ruby
dependencies-report logstash-keystore logstash-plugin.bat setup.bat
ingest-convert.sh logstash-keystore.bat pqcheck system-install# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit# 检查配置文件是否出错。--path.settings指定配置文件所在目录；-f指定具体要检查的配置文件；--config.test_and_exit表示检查配置文件且检查完退出OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties[2018-09-30T22:32:30,002][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}[2018-09-30T22:32:30,030][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}[2018-09-30T22:32:34,201][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Configuration OK #显示配置OK就说明刚刚的配置文件没问题[2018-09-30T22:32:56,973][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
将系统日志输出到10514端口：
# vim /etc/rsyslog.conf #配置系统日志文件，在RULES下面增加一行*.* @@192.168.100.160:10514 #*.* 表示所有类型的日志；将所有日志都输出到192.168.100.160的10514端口
- 启动服务：
# systemctl restart rsyslog #重启rsyslog服务，使配置文件生效 # ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/syslog.conf #启动logstash服务OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties[2018-09-30T22:49:54,963][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified[2018-09-30T22:49:55,111][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"4923c0c7-3e8c-47d1-a484-e66a164e0d3d", :path=>"/var/lib/logstash/uuid"}[2018-09-30T22:50:04,030][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.4.1"}[2018-09-30T22:50:21,698][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}[2018-09-30T22:50:26,644][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}[2018-09-30T22:50:26,718][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}[2018-09-30T22:50:26,752][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}[2018-09-30T22:50:26,887][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}[2018-09-30T22:50:30,553][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}[2018-09-30T22:57:42,621][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.100.160:57566"}{
"facility_label" => "syslogd", "message" => "[origin software=\"rsyslogd\" swVersion=\"8.24.0\" x-pid=\"1296\" x-info=\"http://www.rsyslog.com\"] exiting on signal 15.\n", "severity_label" => "Informational", "program" => "rsyslogd", "timestamp" => "Sep 30 22:57:40", "@version" => "1", "facility" => 5, "@timestamp" => 2018-10-01T02:57:40.000Z, "priority" => 46, "host" => "192.168.100.160", "logsource" => "lzx1", "type" => "system-syslog", "severity" => 6}{
"facility_label" => "system", "message" => "Stopping System Logging Service...\n", "severity_label" => "Informational", "program" => "systemd", "timestamp" => "Sep 30 22:57:40", "@version" => "1", "facility" => 3, "@timestamp" => 2018-10-01T02:57:40.000Z, "priority" => 30, "host" => "192.168.100.160", "logsource" => "lzx1", "type" => "system-syslog", "severity" => 6}{
"facility_label" => "system", "message" => "Starting System Logging Service...\n", "severity_label" => "Informational", "program" => "systemd", "timestamp" => "Sep 30 22:57:41", "@version" => "1", "facility" => 3, "@timestamp" => 2018-10-01T02:57:41.000Z, "priority" => 30, "host" => "192.168.100.160", "logsource" => "lzx1", "type" => "system-syslog", "severity" => 6}{
"facility_label" => "syslogd", "message" => "[origin software=\"rsyslogd\" swVersion=\"8.24.0\" x-pid=\"1329\" x-info=\"http://www.rsyslog.com\"] start\n", "severity_label" => "Informational", "program" => "rsyslogd", "timestamp" => "Sep 30 22:57:42", "@version" => "1", "facility" => 5, "@timestamp" => 2018-10-01T02:57:42.000Z, "priority" => 46, "host" => "192.168.100.160", "logsource" => "lzx1", "type" => "system-syslog", "severity" => 6}{
"facility_label" => "security/authorization", "message" => "Unregistered Authentication Agent for unix-process:1321:674979 (system bus name :1.24, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)\n", "severity_label" => "Notice", "program" => "polkitd", "timestamp" => "Sep 30 22:57:42", "@version" => "1", "facility" => 10, "pid" => "498", "@timestamp" => 2018-10-01T02:57:42.000Z, "priority" => 85, "host" => "192.168.100.160", "logsource" => "lzx1", "type" => "system-syslog", "severity" => 5}{
"facility_label" => "system", "message" => "Started System Logging Service.\n", "severity_label" => "Informational", "program" => "systemd", "timestamp" => "Sep 30 22:57:42", "@version" => "1", "facility" => 3, "@timestamp" => 2018-10-01T02:57:42.000Z, "priority" => 30, "host" => "192.168.100.160", "logsource" => "lzx1", "type" => "system-syslog", "severity" => 6}
上面就是收集的系统日志，输出在当前屏幕上。复制一个ssh渠道，查看端口是否启动。
# netstat -lntpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.100.160:27019 0.0.0.0:* LISTEN 884/mongod tcp 0 0 127.0.0.1:27019 0.0.0.0:* LISTEN 884/mongod tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 797/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 756/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 871/master tcp6 0 0 192.168.100.160:9200 :::* LISTEN 754/java tcp6 0 0 :::10514 :::* LISTEN 1243/java //已经在监听10514端口 tcp6 0 0 192.168.100.160:9300 :::* LISTEN 754/java tcp6 0 0 :::22 :::* LISTEN 756/sshd tcp6 0 0 ::1:25 :::* LISTEN 871/master tcp6 0 0 127.0.0.1:9600 :::* LISTEN 1243/java
可以看到已经在监听9600端口。
配置logstahsh
上面只是将系统日志输出在当前屏幕上，还没有输出到es中，所以需要重新配置logstash。
- 修改配置文件：
# vim /etc/logstash/conf.d/syslog.confinput {
syslog {
type => "system-syslog"
port => 10514 }}output {
elasticsearch {
hosts => ["192.168.100.150:9200"] #指向主节点的9200端口，这里写lzx1或lzx2的IP+端口也可以，因为是分布式的
index => "system-syslog-%{+YYYY.MM}" #定义索引
} }
- 检查配置文件：
# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/syslog.conf --config.test_and_exitOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties[2018-09-30T23:23:53,156][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK #配置没问题[2018-09-30T23:23:56,897][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
- 后台启动服务：
# chown logstash /var/log/logstash/logstash-plain.log# chown -R logstash:logstash /var/lib/logstash/# 上面测试时用的root身份启动logstash服务，所以生成的配置文件的属主属组是root，需要改回来，否则下面启动服务会有问题# systemctl start logstash# tail /var/log/logstash/logstash-plain.log[2018-09-30T23:59:15,135][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash[2018-09-30T23:59:15,935][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}[2018-09-30T23:59:16,035][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}[2018-09-30T23:59:16,053][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:10514"}[2018-09-30T23:59:16,079][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}[2018-09-30T23:59:16,818][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}[2018-09-30T23:59:23,411][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>"main", :thread=>"#"}# netstat -lntpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.100.160:27019 0.0.0.0:* LISTEN 884/mongod
tcp 0 0 127.0.0.1:27019 0.0.0.0:* LISTEN 884/mongod
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 797/nginx: master p
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 756/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 871/master
tcp6 0 0 192.168.100.160:9200 :::* LISTEN 754/java
tcp6 0 0 :::10514 :::* LISTEN 2858/java
tcp6 0 0 192.168.100.160:9300 :::* LISTEN 754/java
tcp6 0 0 :::22 :::* LISTEN 756/sshd
tcp6 0 0 ::1:25 :::* LISTEN 871/master
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 2858/java
但这里的端口是127.0.0.1:9600，这是无法和远程机器通信的，需要修改配置文件。
- 修改logstash配置文件：
# vim /etc/logstash/logstash.yml http.host: "192.168.100.160"
- 重启服务：
# systemctl restart logstash # netstat -lntpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.100.160:27019 0.0.0.0:* LISTEN 885/mongod tcp 0 0 127.0.0.1:27019 0.0.0.0:* LISTEN 885/mongod tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 794/nginx: master p tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 750/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 867/master tcp6 0 0 192.168.100.160:9200 :::* LISTEN 755/java tcp6 0 0 :::10514 :::* LISTEN 1161/java tcp6 0 0 192.168.100.160:9300 :::* LISTEN 755/java tcp6 0 0 :::22 :::* LISTEN 750/sshd tcp6 0 0 ::1:25 :::* LISTEN 867/master tcp6 0 0 192.168.100.160:9600 :::* LISTEN 1161/java
- 到lzx上查看是否有索引：
# curl '192.168.100.150:9200/_cat/indices?v' #获取索引信息health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open system-syslog-2018.10 NUEz10LGT1uvVNhn6yUw3g 5 1 32 0 315.6kb 153.6kb #有该索引生成就说明logstash与es通信正常
- 获取指定索引详细信息：
# curl '192.168.100.150:9200/system-syslog-2018.10?pretty' #获取指定索引详细信息{
"system-syslog-2018.10" : {
"aliases" : { }, "mappings" : {
"doc" : {
"properties" : {
"@timestamp" : {
"type" : "date"
}, "@version" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "facility" : {
"type" : "long"
}, "facility_label" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "host" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "logsource" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "message" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "pid" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "priority" : {
"type" : "long"
}, "program" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "severity" : {
"type" : "long"
}, "severity_label" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "timestamp" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}, "type" : {
"type" : "text", "fields" : {
"keyword" : {
"type" : "keyword", "ignore_above" : 256 }
}
}
}
}
}, "settings" : {
"index" : {
"creation_date" : "1538370063177", "number_of_shards" : "5", "number_of_replicas" : "1", "uuid" : "NUEz10LGT1uvVNhn6yUw3g", "version" : {
"created" : "6040199"
}, "provided_name" : "system-syslog-2018.10"
}
}
}}
kibana上查看日志
- 到kibana浏览器界面配置索引：
Management → Kibana → Index Patterns，填入system-syslog-2018.10。
点击Next step，选择文件夹
点击Create index pattern
点击Discover，可以看到lzx1上的日志
在命令行下查看，时间也是对应的
# tail -f /var/log/messagesOct 1 22:25:43 lzx1 logstash: [2018-10-01T22:25:43,874][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:10514"}Oct 1 22:25:43 lzx1 logstash: [2018-10-01T22:25:43,886][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}Oct 1 22:25:44 lzx1 logstash: [2018-10-01T22:25:44,543][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}Oct 1 22:33:55 lzx1 chronyd[512]: Source 106.39.20.237 replaced with 193.228.143.14
Oct 1 22:33:55 lzx1 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]Oct 1 22:33:55 lzx1 rsyslogd: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]Oct 1 22:33:55 lzx1 logstash: [2018-10-01T22:33:55,836][INFO ][logstash.inputs.syslog ] new connection {:client=>"192.168.100.160:59018"}Oct 1 23:01:05 lzx1 systemd: Started Session 3 of user root.
Oct 1 23:01:05 lzx1 systemd: Starting Session 3 of user root.
Oct 1 23:04:16 lzx1 chronyd[512]: Source 193.228.143.14 replaced with 193.228.143.13
Oct 1 23:11:58 lzx1 systemd: Started Session 4 of user root.
Oct 1 23:11:58 lzx1 systemd: Starting Session 4 of user root.
Oct 1 23:11:58 lzx1 systemd-logind: New session 4 of user root.
Oct 1 23:12:04 lzx1 systemd-logind: Removed session 4.
再做一个测试，在lzx上登录lzx1
# ssh 192.168.100.160Enter passphrase for key '/root/.ssh/id_rsa': Last login: Mon Oct 1 23:11:58 2018 from 192.168.100.170# logoutConnection to 192.168.100.160 closed.
到lzx1上查看日志
Oct 1 23:23:58 lzx1 systemd: Started Session 5 of user root. Oct 1 23:23:58 lzx1 systemd-logind: New session 5 of user root. Oct 1 23:23:58 lzx1 systemd: Starting Session 5 of user root. Oct 1 23:24:03 lzx1 kernel: sched: RT throttling activated Oct 1 23:25:56 lzx1 systemd-logind: Removed session 5.
刷新浏览器界面查看
可以看到日志已经显示在kibana界面。
收集nginx日志
上面配置了收集系统日志，那么接下来我们配置收集nginx日志。
- lzx1编辑配置文件：
# vim /etc/logstash/conf.d/nginx.confinput {
file {
path => "/tmp/elk_access.log" #指定一个文件，把文件内容作为logstash日志输入
start_position => "beginning" #指定开始收集位置
type => "nginx"
}}filter {
grok {
match => { "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
} #匹配nginx输出格式
geoip {
source => "clientip"
}}output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["192.168.100.160:9200"]
index => "nginx-test-%{+YYYY.MM.dd}"
}}
- 检查配置文件：
# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exitOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties[2018-10-01T23:45:01,238][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK #配置没问题[2018-10-01T23:45:11,827][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
- 编辑nginx虚拟主机配置文件：
# cd /usr/local/nginx/conf/vhost/ # vim elk.confserver {
listen 80;
server_name elk.lzx.com;
location / {
proxy_pass http://192.168.100.150:5601; #指定代理目标
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
access_log /tmp/elk_access.log main2; #定义访问日志，前面有定义日志路径，日志格式为main2
}
- 编辑nginx配置文件：
# vim /usr/local/nginx/conf/nginx.conf log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$upstream_addr" $request_time'; include vhost/*.conf;
- 检查配置文件：
# /usr/local/nginx/sbin/nginx -tnginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful# /usr/local/nginx/sbin/nginx -s reload
- 浏览器访问：
在Windows的hosts文件中添加一行：
192.168.100.160 elk.lzx.com
保存后输入elk.lzx.com就可以访问kibana界面了。
- 查看访问日志：
# ls /tmp/elk_access.log /tmp/elk_access.log# wc -l !$wc -l /tmp/elk_access.log 157 /tmp/elk_access.log# cat !$cat /tmp/elk_access.log elk.lzx.com 192.168.100.1 - - [02/Oct/2018:22:05:09 -0400] "GET /favicon.ico HTTP/1.1" 404 80 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6756.400 QQBrowser/10.3.2473.400" "192.168.100.150:5601" 0.113 elk.lzx.com 192.168.100.1 - - [02/Oct/2018:22:05:15 -0400] "GET /favicon.ico HTTP/1.1" 404 80 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6756.400 QQBrowser/10.3.2473.400" "192.168.100.150:5601" 0.062 elk.lzx.com 192.168.100.1 - - [02/Oct/2018:22:09:16 -0400] "GET /favicon.ico HTTP/1.1" 404 80 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6756.400 QQBrowser/10.3.2473.400" "192.168.100.150:5601" 0.033 elk.lzx.com 192.168.100.1 - - [02/Oct/2018:22:09:16 -0400] "GET /favicon.ico HTTP/1.1" 404 80 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6756.400 QQBrowser/10.3.2473.400" "192.168.100.150:5601" 0.039 elk.lzx.com 192.168.100.1 - - [02/Oct/2018:22:09:17 -0400] "GET /favicon.ico HTTP/1.1" 404 80 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6756.400 QQBrowser/10.3.2473.400" "192.168.100.150:5601" 0.042 elk.lzx.com 192.168.100.1 - - [02/Oct/2018:22:09:17 -0400] "GET /favicon.ico HTTP/1.1" 404 80 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.6756.400 QQBrowser/10.3.2473.400" "192.168.100.150:5601" 0.033
- 到lzx上检查是否有索引生成：
# curl '192.168.100.150:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open nginx-test-2018.10.02 -BRyeAQmRueqkQmSRQC4bg 5 1 7553 0 2mb 1014.2kb green open nginx-test-2018.10.03 Fbh6ooCgQSqqwuVJKrw5FQ 5 1 17026 0 4.4mb 2.2mb green open .kibana ngh8OGEuRUS5EJ9R53Ycww 1 1 2 0 21.6kb 10.8kb green open system-syslog-2018.10 NUEz10LGT1uvVNhn6yUw3g 5 1 24811 0 6.3mb 3.2mb
如果没有日志生成，重启logstash服务。
- 到kibana界面配置：
和之前配置系统日志时步骤相同，左侧点击Managerment → Index Patterns → Create Index Pattern
填入nginx-test-*，不要带具体日期，点击Next step→Create index pattern，然后点击Discover，选择nginx-test-*，查看日志。
使用beats采集日志
前面有提到，beats是一个轻量的日志采集器，logstash相对来说比较占用资源。
beats成员有：Filebeat（日志文件）、Metricbeat（指标）、Packetbeat（网络数据）、Winlogbeat（Windows事件日志）、Auditbeat（审计数据）、Heartbeat（运行时间监控）。
同时它是可扩展的，支持自定义构建。
- lzx2下载安装：
# yum install -y filebeat
- 编辑配置文件：
# vim /etc/filebeat/filebeat.ymlfilebeat.inputs: - type: log enabled: true paths: - /var/log/messages #Filebeat inputs部分 #output.elasticsearch: #hosts: ["localhost:9200"]output.console: enable: true #增加这两行，注意空格
- 前台启动查看日志：
# /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml #前台启动查看日志
同时，lzx1 ssh登录 lzx2
# ssh lzx2The authenticity of host 'lzx2 (192.168.100.170)' can't be established. ECDSA key fingerprint is SHA256:teKu3atU+OByPeXXD2xXhyb30vg6nW8ETqqCr785Dbc. ECDSA key fingerprint is MD5:13:a4:f1:c0:1f:62:65:d4:f4:4e:42:ab:40:f1:36:60. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'lzx2,192.168.100.170' (ECDSA) to the list of known hosts. root@lzx2's password: Last login: Tue Oct 2 21:57:43 2018 from 192.168.100.1# logoutConnection to lzx2 closed.
再查看lzx2显示
{"@timestamp":"2018-10-03T04:53:20.529Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"input":{"type":"log"},"host":{"name":"lzx2"},"beat":{"name":"lzx2","hostname":"lzx2","version":"6.4.2"},"source":"/var/log/messages","offset":128846,"message":"Oct 3 00:37:22 lzx2 chronyd[509]: Selected source 106.187.100.179","prospector":{"type":"log"}}{"@timestamp":"2018-10-03T04:53:20.529Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"input":{"type":"log"},"beat":{"name":"lzx2","hostname":"lzx2","version":"6.4.2"},"host":{"name":"lzx2"},"source":"/var/log/messages","offset":128913,"message":"Oct 3 00:37:24 lzx2 chronyd[509]: Source 5.79.108.34 replaced with 85.199.214.100","prospector":{"type":"log"}}{"@timestamp":"2018-10-03T04:53:35.531Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"prospector":{"type":"log"},"input":{"type":"log"},"beat":{"name":"lzx2","hostname":"lzx2","version":"6.4.2"},"host":{"name":"lzx2"},"source":"/var/log/messages","offset":128996,"message":"Oct 3 00:53:28 lzx2 systemd-logind: New session 5 of user root."}{"@timestamp":"2018-10-03T04:53:35.531Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"beat":{"name":"lzx2","hostname":"lzx2","version":"6.4.2"},"host":{"name":"lzx2"},"offset":129061,"message":"Oct 3 00:53:29 lzx2 systemd: Started Session 5 of user root.","source":"/var/log/messages","prospector":{"type":"log"},"input":{"type":"log"}}{"@timestamp":"2018-10-03T04:53:35.531Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"source":"/var/log/messages","offset":129123,"message":"Oct 3 00:53:29 lzx2 systemd: Starting Session 5 of user root.","prospector":{"type":"log"},"input":{"type":"log"},"beat":{"hostname":"lzx2","version":"6.4.2","name":"lzx2"},"host":{"name":"lzx2"}}{"@timestamp":"2018-10-03T04:53:50.533Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.4.2"},"host":{"name":"lzx2"},"beat":{"name":"lzx2","hostname":"lzx2","version":"6.4.2"},"source":"/var/log/messages","offset":129186,"message":"Oct 3 00:53:48 lzx2 systemd-logind: Removed session 5.","prospector":{"type":"log"},"input":{"type":"log"}}
上面简单测试日志在前台显示没问题。
- 编辑配置文件：
# ls /var/log/elasticsearch/lzxlinux.log #elasticsearch日志文件/var/log/elasticsearch/lzxlinux.log# vim /etc/filebeat/filebeat.ymlfilebeat.inputs: - type: log enabled: true paths: - /var/log/elasticsearch/lzxlinux.log output.elasticsearch: hosts: ["192.168.100.150:9200"] #Elasticsearch output部分#output.console: #enable: true
- 启动服务：
# systemctl start filebeat# ps aux |grep filebeatroot 10206 0.2 1.0 377268 19124 ? Ssl 01:28 0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
- 到lzx上查看是否生成新的索引：
# !curlcurl '192.168.100.150:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open nginx-test-2018.10.02 -BRyeAQmRueqkQmSRQC4bg 5 1 7553 0 1mb 1mb yellow open nginx-test-2018.10.03 Fbh6ooCgQSqqwuVJKrw5FQ 5 1 52791 0 6.6mb 6.6mb yellow open filebeat-6.4.2-2018.10.03 dZIeTzJ8QBO6UCkmWchBZw 3 1 616 0 205.8kb 205.8kb //新的索引生成 green open .kibana ngh8OGEuRUS5EJ9R53Ycww 1 0 3 0 17.4kb 17.4kb yellow open system-syslog-2018.10 NUEz10LGT1uvVNhn6yUw3g 5 1 59232 0 7.4mb 7.4mb
- 到kibana界面配置：
和上面配置nginx一样的步骤，配置索引，到Discover中查看。
这就是elasticsearch的日志文件。
更多资料参考：