日志监控和分析在保障业务稳定运行时，起到了很重要的作用，不过一般情况下日志都分散在各个生产服务器，且开发人员无法登陆生产服务器，这时候就需要一个集中式的日志收集装置，对日志中的关键字进行监控，触发异常时进行报警，并且开发人员能够查看相关日志。logstash+elasticsearch+kibana3就是实现这样功能的一套系统，并且功能更强大。
Logstash：负责日志的收集，处理和储存
Elasticsearch：负责日志检索和分析
Kibana：负责日志的可视化
1、环境介绍
elkServer
IP:192.168.7.27
OS：Centos7.1
FQDN:elk.server.com
elkClient
IP:192.168.31.23
OS：Centos7.1
2、下载准备
官网下载最新的安装包：https://www.elastic.co/downloads（目前有些版本的包可能下载不到了，请到该地址下载——链接：http://pan.baidu.com/s/1gfohO2Z 密码：5s1f）
elasticsearch-1.7..noarch.rpm （server上安装）
kibana-4.1.-linux-x64.tar.gz （server上安装）
logstash-1.5.-.noarch.rpm （server上安装）
logstash-forwarder-0.4.-.x86_64.rpm （client上安装）
3、Server端安装
3.1安装jdk1.7
[root@localhost ~]# yum install java-1.7.-openjdk
Loaded plugins: fastestmirror, langpacks
base | 3.6 kB ::
extras | 3.4 kB ::
updates | 3.4 kB ::
Loading mirror speeds from cached hostfile
* base: mirrors.btte.net
* extras: mirrors..com
* updates: mirrors..com
Package :java-1.7.-openjdk-1.7.0.91-2.6.2.1.el7_1.x86_64 already installed and latest version
Nothing to do
3.2安装elasticsearch
[root@localhost elk]# yum localinstall elasticsearch-1.7..noarch.rpm (yum 本地安装elasticsearch）
Loaded plugins: fastestmirror, langpacks
Examining elasticsearch-1.7..noarch.rpm: elasticsearch-1.7.-.noarch
elasticsearch-1.7..noarch.rpm: does not update installed package.
Nothing to do
[root@localhost elk]# systemctl daemon-reload
[root@localhost elk]# systemctl enable elasticsearch.service （设置开机自启动）
ln -s '/usr/lib/systemd/system/elasticsearch.service' '/etc/systemd/system/multi-user.target.wants/elasticsearch.service'
[root@localhost elk]# systemctl start elasticsearch.service （开启服务）
[root@localhost elk]# systemctl status elasticsearch.service （查看服务状态）
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: active (running) since Sun -- :: CST; 28s ago
Docs: http://www.elastic.co
Main PID: (java)
CGroup: /system.slice/elasticsearch.service
?.. java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction= -XX:+UseCMSInitiatingOccupancyOnly -XX:+Heap... Nov :: localhost.localdomain systemd[]: Started Elasticsearch.
[root@localhost elk]# rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf
[root@localhost elk]# netstat -nltp （查看端口监听状况）
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0.0.0.0: 0.0.0.0:* LISTEN /rpcbind
tcp 0.0.0.0: 0.0.0.0:* LISTEN /sshd
tcp 127.0.0.1: 0.0.0.0:* LISTEN /cupsd
tcp 127.0.0.1: 0.0.0.0:* LISTEN /master
tcp 127.0.0.1: 0.0.0.0:* LISTEN /sshd: root@pt
tcp6 ::: :::* LISTEN /rpcbind
tcp6 0 0 :::9200 :::* LISTEN 15345/java
tcp6 0 0 :::9300 :::* LISTEN 15345/java
tcp6 ::: :::* LISTEN /sshd
tcp6 ::: :::* LISTEN /cupsd
tcp6 ::: :::* LISTEN /master
tcp6 ::: :::* LISTEN /sshd: root@pt
[root@localhost elk]# firewall-cmd --permanent --add-port={/tcp,/tcp} （防火墙添加两个端口）
success
[root@localhost elk]# firewall-cmd --reload （重载防火墙）
success
[root@localhost elk]# firewall-cmd --list-all （查看防火墙开发端口）
public (default, active)
interfaces: ens33
sources:
services: dhcpv6-client ssh
ports: /tcp /tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules:
3.3安装kibana
[root@localhost elk]# tar zxf kibana-4.1.-linux-x64.tar.gz -C /usr/local/ （解压缩安装包到指定目录中）
[root@localhost elk]# cd /usr/local/
[root@localhost local]# ls
bin etc games include kibana-4.1.-linux-x64 lib lib64 libexec sbin share src
[root@localhost local]# mv kibana-4.1.-linux-x64/ kibana （重命名）
[root@localhost local]# cd kibana/
[root@localhost kibana]# ls
bin config LICENSE.txt node plugins README.txt src
[root@localhost kibana]# cd bin/
[root@localhost bin]# ls （运行./kibana即可开启服务，但我们将其做到service）
kibana kibana.bat
[root@localhost bin]# cd /etc/systemd/system/
[root@localhost system]# vi kibana.service （编辑kibana服务）
[Service]
ExecStart=/usr/local/kibana/bin/kibana [Install]
WantedBy=multi-user.target [root@localhost system]# systemctl enable kibana.service （设置开机自启动）
ln -s '/etc/systemd/system/kibana.service' '/etc/systemd/system/multi-user.target.wants/kibana.service'
[root@localhost system]# systemctl start kibana.service （开启服务）
[root@localhost system]# systemctl status kibana.service （查看服务运行状态）
kibana.service
Loaded: loaded (/etc/systemd/system/kibana.service; enabled)
Active: active (running) since Sun -- :: CST; 10s ago
Main PID: (node)
CGroup: /system.slice/kibana.service
?.. /usr/local/kibana/bin/../node/bin/node /usr/local/kibana/bin/../src/bin/kibana.js Nov :: localhost.localdomain systemd[]: Started kibana.service.
Nov :: localhost.localdomain kibana[]: {"name":"Kibana","hostname":"localhost.localdomain","pid":,"level":,"msg":"No existing kibana index found","time":"20...43Z","v":}
Nov :: localhost.localdomain kibana[]: {"name":"Kibana","hostname":"localhost.localdomain","pid":,"level":,"msg":"Listening on 0.0.0.0:5601","time":"2015-11...93Z","v":}
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost system]# netstat -nltp （查看端口监听状态）
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 16131/node
tcp 0.0.0.0: 0.0.0.0:* LISTEN /rpcbind
tcp 0.0.0.0: 0.0.0.0:* LISTEN /sshd
tcp 127.0.0.1: 0.0.0.0:* LISTEN /cupsd
tcp 127.0.0.1: 0.0.0.0:* LISTEN /master
tcp 127.0.0.1: 0.0.0.0:* LISTEN /sshd: root@pt
tcp6 ::: :::* LISTEN /rpcbind
tcp6 ::: :::* LISTEN /java
tcp6 ::: :::* LISTEN /java
tcp6 ::: :::* LISTEN /sshd
tcp6 ::: :::* LISTEN /cupsd
tcp6 ::: :::* LISTEN /master
tcp6 ::: :::* LISTEN /sshd: root@pt
[root@localhost system]# firewall-cmd --permanent --add-port=/tcp （防火墙开启5601端口）
success
[root@localhost system]# firewall-cmd --reload （重载防火墙）
success
[root@localhost system]# firewall-cmd --list-all （查看防火墙开放端口）
public (default, active)
interfaces: ens33
sources:
services: dhcpv6-client ssh
ports: /tcp /tcp /tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules: [root@localhost system]# firewall-cmd --permanent --add-forward-port=port=:proto=tcp:toport= （为5601端口添加80端口的映射，这样在浏览器中就可以不用输入端口了）
success
[root@localhost system]# firewall-cmd --reload （重载防火墙）
success
[root@localhost system]# firewall-cmd --list-all （查看防火墙开放端口）
public (default, active)
interfaces: ens33
sources:
services: dhcpv6-client ssh
ports: /tcp /tcp /tcp
masquerade: no
forward-ports: port=:proto=tcp:toport=:toaddr=
icmp-blocks:
rich rules:
3.4安装logstash
[root@localhost system]# cd /home/elk/
[root@localhost elk]# ls
elasticsearch-1.7..noarch.rpm kibana-4.1.-linux-x64.tar.gz logstash-1.5.-.noarch.rpm logstash-forwarder-0.4.-.x86_64.rpm
[root@localhost elk]# yum localinstall logstash-1.5.-.noarch.rpm （yum本地安装logstash）
Loaded plugins: fastestmirror, langpacks
Examining logstash-1.5.-.noarch.rpm: :logstash-1.5.-.noarch
Marking logstash-1.5.-.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package logstash.noarch :1.5.- will be installed
--> Finished Dependency Resolution
base//x86_64 | 3.6 kB ::
extras//x86_64 | 3.4 kB ::
extras//x86_64/primary_db | kB ::
updates//x86_64 | 3.4 kB ::
updates//x86_64/primary_db | 4.7 MB :: Dependencies Resolved ===============================================================================================================================================================================================
Package Arch Version Repository Size
===============================================================================================================================================================================================
Installing:
logstash noarch :1.5.- /logstash-1.5.-.noarch M Transaction Summary
===============================================================================================================================================================================================
Install Package Total size: M
Installed size: M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : :logstash-1.5.-.noarch /
Verifying : :logstash-1.5.-.noarch / Installed:
logstash.noarch :1.5.- Complete!
[root@localhost tls]# hostname -f （查看当前FQDN，FQDN设置参见http://www.cnblogs.com/zhenyuyaodidiao/p/4947930.html）
elk.server.com
[root@localhost ~]# cd /etc/pki/tls/ （进入到/etc/pki/tls/文件夹）
[root@localhost tls]# ls
cert.pem certs misc openssl.cnf private
（以下生成openssl key用于客户端上传日志文件用，在客户端配置时会用到）
[root@localhost tls]# openssl req -subj '/CN=elk.server.com/' -x509 -days -batch -nodes -newkey rsa: -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
Generating a bit RSA private key
..............+++
.............+++
writing new private key to 'private/logstash-forwarder.key'
-----
[root@localhost tls]# ls
cert.pem certs misc openssl.cnf private
[root@localhost tls]# cd private/
[root@localhost private]# ll
total
-rw-r--r--. root root Nov : logstash-forwarder.key
[root@localhost private]# cd ../certs/
[root@localhost certs]# ll
total
lrwxrwxrwx. root root Apr ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. root root Apr ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-r--r--. root root Nov : logstash-forwarder.crt
-rwxr-xr-x. root root Mar make-dummy-cert
-rw-r--r--. root root Mar Makefile
-rwxr-xr-x. root root Mar renew-dummy-cert
[root@localhost ~]# cd /etc/logstash/conf.d/
[root@localhost conf.d]# vi -logstash-initial.conf （编辑logstash配置文件）
input {
lumberjack {
port =>
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
} filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
} output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
} [root@localhost conf.d]# systemctl enable logstash （设置开机自启动）
logstash.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig logstash on
The unit files have no [Install] section. They are not meant to be enabled
using systemctl.
Possible reasons for having this kind of units are:
) A unit may be statically enabled by being symlinked from another unit's
.wants/ or .requires/ directory.
) A unit's purpose may be to act as a helper for some other unit which has
a requirement dependency on it.
) A unit may be started when needed via activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...).
[root@localhost conf.d]# systemctl start logstash.service （开启logstash服务）
[root@localhost conf.d]# systemctl status logstash.service （查看服务运行状态）
logstash.service - LSB: Starts Logstash as a daemon.
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (running) since Sun -- :: CST; 14s ago
Process: ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=/SUCCESS)
CGroup: /system.slice/logstash.service
?.. java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction= -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib... Nov :: elk logstash[]: logstash started.
Nov :: elk systemd[]: Started LSB: Starts Logstash as a daemon..
[root@localhost conf.d]# netstat -nltp （查看端口占用）
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0.0.0.0: 0.0.0.0:* LISTEN /node
tcp 0.0.0.0: 0.0.0.0:* LISTEN /rpcbind
tcp 0.0.0.0: 0.0.0.0:* LISTEN /sshd
tcp 127.0.0.1: 0.0.0.0:* LISTEN /cupsd
tcp 127.0.0.1: 0.0.0.0:* LISTEN /master
tcp 127.0.0.1: 0.0.0.0:* LISTEN /sshd: root@pt
tcp 127.0.0.1: 0.0.0.0:* LISTEN /sshd: root@pt
tcp6 0 0 :::5000 :::* LISTEN 20805/java
tcp6 ::: :::* LISTEN /rpcbind
tcp6 ::: :::* LISTEN /java
tcp6 ::: :::* LISTEN /java
tcp6 ::: :::* LISTEN /java
tcp6 ::: :::* LISTEN /sshd
tcp6 ::: :::* LISTEN /cupsd
tcp6 ::: :::* LISTEN /master
tcp6 ::: :::* LISTEN /sshd: root@pt
tcp6 ::: :::* LISTEN /sshd: root@pt
[root@localhost conf.d]# cd /var/log/logstash/
[root@localhost logstash]# ls （日志文件）
logstash.err logstash.log logstash.stdout
[root@localhost logstash]# firewall-cmd --permanent --add-port=/tcp （防火墙开放5000端口）
success
[root@localhost logstash]# firewall-cmd --reload （重载防火墙）
success
[root@localhost logstash]# firewall-cmd --list-all （查看端口开放情况）
public (default, active)
interfaces: ens33
sources:
services: dhcpv6-client ssh
ports: /tcp /tcp /tcp /tcp
masquerade: no
forward-ports: port=:proto=tcp:toport=:toaddr=
icmp-blocks:
rich rules:
4、Client端安装
[root@localhost elk]# vi /etc/hosts （编辑hosts文件） 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.7.27 elk.server.com [root@localhost elk]# service network restart
Restarting network (via systemctl): [ OK ]
[root@localhost elk]# ping elk.server.com （测试连接）
PING elk.server.com (192.168.7.27) () bytes of data.
bytes from elk.server.com (192.168.7.27): icmp_seq= ttl= time=0.754 ms
bytes from elk.server.com (192.168.7.27): icmp_seq= ttl= time=0.477 ms
^C
--- elk.server.com ping statistics ---
packets transmitted, received, % packet loss, time 1000ms
rtt min/avg/max/mdev = 0.477/0.615/0.754/0.140 ms
[root@localhost laizy]# mkdir elk
[root@localhost laizy]# cd elk/
[root@localhost elk]# ls
[root@localhost elk]# scp root@192.168.7.27:/home/elk/logstash-forwarder-0.4.-.x86_64.rpm . （拷贝logstash-forwarder到本地）
The authenticity of host '192.168.7.27 (192.168.7.27)' can't be established.
ECDSA key fingerprint is :b9::::f2:::9b::bb::a5::f1:f9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.7.27' (ECDSA) to the list of known hosts.
root@192.168.7.27's password:
logstash-forwarder-0.4.-.x86_64.rpm % 1692KB .7MB/s :
[root@localhost elk]# ls
logstash-forwarder-0.4.-.x86_64.rpm
[root@localhost elk]# scp root@192.168.7.27:/etc/pki/tls/certs/logstash-forwarder.crt . （拷贝Server端的key到本地）
root@192.168.7.27's password:
logstash-forwarder.crt % .1KB/s :
[root@localhost elk]# ll
total
-rw-r--r--. root root Nov : logstash-forwarder-0.4.-.x86_64.rpm
-rw-r--r--. root root Nov : logstash-forwarder.crt
[root@localhost elk]# cp logstash-forwarder.crt /etc/pki/tls/certs/ （将key拷贝到/etc/pki/tls/certs/下）
[root@localhost elk]# cd /etc/pki/tls/certs/
[root@localhost certs]# ls
ca-bundle.crt ca-bundle.trust.crt logstash-forwarder.crt make-dummy-cert Makefile renew-dummy-cert
[root@localhost certs]# cd /home/laizy/elk/
[root@localhost elk]# ls
logstash-forwarder-0.4.-.x86_64.rpm logstash-forwarder.crt
[root@localhost elk]# yum localinstall logstash-forwarder-0.4.-.x86_64.rpm （yum本地安装logstash-forwarder）
Loaded plugins: fastestmirror, langpacks
Examining logstash-forwarder-0.4.-.x86_64.rpm: logstash-forwarder-0.4.-.x86_64
Marking logstash-forwarder-0.4.-.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package logstash-forwarder.x86_64 :0.4.- will be installed
--> Finished Dependency Resolution
base//x86_64 | 3.6 kB ::
extras//x86_64 | 3.4 kB ::
updates//x86_64 | 3.4 kB :: Dependencies Resolved ===============================================================================================================================================================================================
Package Arch Version Repository Size
===============================================================================================================================================================================================
Installing:
logstash-forwarder x86_64 0.4.- /logstash-forwarder-0.4.-.x86_64 5.7 M Transaction Summary
===============================================================================================================================================================================================
Install Package Total size: 5.7 M
Installed size: 5.7 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : logstash-forwarder-0.4.-.x86_64 /
Logs for logstash-forwarder will be in /var/log/logstash-forwarder/
Verifying : logstash-forwarder-0.4.-.x86_64 / Installed:
logstash-forwarder.x86_64 :0.4.- Complete!
[root@localhost elk]# systemctl enable logstash-forwarder （设置开机自启动）
logstash-forwarder.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig logstash-forwarder on
The unit files have no [Install] section. They are not meant to be enabled
using systemctl.
Possible reasons for having this kind of units are:
) A unit may be statically enabled by being symlinked from another unit's
.wants/ or .requires/ directory.
) A unit's purpose may be to act as a helper for some other unit which has
a requirement dependency on it.
) A unit may be started when needed via activation (socket, path, timer,
D-Bus, udev, scripted systemctl call, ...).
[root@localhost elk]# systemctl start logstash-forwarder.service （开启服务）
[root@localhost elk]# cd /var/log/logstash-forwarder/ （日志目录）
[root@localhost logstash-forwarder]# ls
logstash-forwarder.err logstash-forwarder.log
[root@localhost elk]# vi /etc/logstash-forwarder.conf （编辑配置文件）
{
"network": {
"servers": [ "elk.server.com:5000" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout":
}, "files": [
{
"paths": [
"/var/log/messages",
"/var/log/secure"
], "fields": { "type": "syslog" }
}
]
} [root@localhost elk]# systemctl restart logstash-forwarder.service （重启服务）
[root@localhost elk]# systemctl status logstash-forwarder.service （查看服务运行状态）
logstash-forwarder.service - LSB: no description given
Loaded: loaded (/etc/rc.d/init.d/logstash-forwarder)
Active: active (running) since Sun -- :: CST; 18s ago
Process: ExecStop=/etc/rc.d/init.d/logstash-forwarder stop (code=exited, status=/SUCCESS)
Process: ExecStart=/etc/rc.d/init.d/logstash-forwarder start (code=exited, status=/SUCCESS)
CGroup: /system.slice/logstash-forwarder.service
?.. /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder.conf Nov :: localhost.localdomain systemd[]: Starting LSB: no description given...
Nov :: localhost.localdomain /etc/init.d/logstash-forwarder[]: logstash-forwarder started
Nov :: localhost.localdomain logstash-forwarder[]: logstash-forwarder started
Nov :: localhost.localdomain systemd[]: Started LSB: no description given.
5、界面验证
首先在client中手动增加一条日志：
[root@localhost elk]# logger zhenyuLogtest
界面登录 http://192.168.7.27/ ，做如下操作
从图中可以看到，手动添加的日志已经在界面中被搜索到了。
本文主要参考了国外一个搭建ELK的视频，操作的很详细，附上视频的下载链接，仅供参考。
链接：http://pan.baidu.com/s/1jGuBWCQ 密码：h0pq