ELK监控系统nginx / mysql慢日志
ELK监控系统nginx日志
1.环境准备
centos6.8_64 mini
IP:192.168.10.78
tar包：
logstash-2.4.0.tar.gz
elasticsearch-2.4.0.tar.gz
kibana-4.6.1-linux-x86_64.tar.gz
JDK环境
| elasticsearch | logstach | kibana | java |
|---|---|---|---|
| V2.4 | V2.4 | V4.6.1 | V1.8.0_111 |
nginx采用yum安装即可，安装后启动此服务。
2.logstash简单配置编写
logstash目录下新建测试文件test.conf 其内动如下
- input {
- file {
- type => "syslog"
- tags => ["log"]
- path => ["/var/log/messages","/log/*.log"]
- start_position => beginning
- ignore_older => 0
- }
- file {
- type => "nginx_log"
- tags => ["nginx"]
- path => ["/var/log/nginx/access.log"]
- start_position => beginning
- ignore_older => 0
- }
- }
- output
- {
- elasticsearch {
- hosts => [ "192.168.10.78:9200" ]
- }
- }
编写好后测试运行：
- [root@localhost logstash]# ./bin/logstash -f test.conf
- Settings: Default pipeline workers: 2
- Pipeline main started
此时logstash已经运行，我们通过浏览器访问web：192.168.10.68
同时在终端下重启网卡服务，用以查看系统日志是否有输出至kibana
打开kibana查看日志输出
可以看见有网卡日志和web访问日志。
接下来监控mysql日志
yum install mysql mysql-server -y
安装完直接在kibana就看见安装mysql的日志输出结果了
Time type tags message
November 10th 2016, 10:18:13.193 syslog log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:13.057 syslog log Nov 10 10:18:12 localhost yum[11445]: Installed: mysql-server-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.187 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.185 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64
November 10th 2016, 10:18:10.183 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64
November 10th 2016, 10:18:10.051 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: mysql-5.1.73-7.el6.x86_64
November 10th 2016, 10:18:10.050 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBD-MySQL-4.013-3.el6.x86_64
November 10th 2016, 10:18:10.048 syslog log Nov 10 10:18:09 localhost yum[11445]: Installed: perl-DBI-1.609-4.el6.x86_64
接下来修改test.conf配置文件，添加对mysql满日志的监控
1.yum install mysql mysql-server
2.service network restart
3.mysql慢日志开启:
(此处参考http://blog.itpub.net/29500582/viewspace-1432985/)
Mysql 启动慢查询日志 （不用重启）
查看mysql系统参数
mysql> show variables like "%slow%";
+---------------------------+-------------------------------+
| Variable_name | Value |
+---------------------------+-------------------------------+
| log_slow_admin_statements | OFF |
| log_slow_slave_statements | OFF |
| slow_launch_time | 2 |
| slow_query_log | OFF |
| slow_query_log_file | /mysql/data/nagiosdb-slow.log |
+---------------------------+-------------------------------+
5 rows in set (0.00 sec)
slow_query_log： off关闭状态 on开启状态
slow_launch_time 默认超过2s为慢查询
slow_query_log_file 慢查询日志存放地点
这三个参数，在不同的mysql版本中，不太一样，不过都可以通过 show variables like "%slow%" 查看出来
运行如下命令即可运行慢查询日志
mysql> set global slow_query_log=ON;
Query OK, 0 rows affected (0.03 sec)
mysql> set global slow_launch_time=5;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like "%slow%";
+---------------------------+-------------------------------+
| Variable_name | Value |
+---------------------------+-------------------------------+
| log_slow_admin_statements | OFF |
| log_slow_slave_statements | OFF |
| slow_launch_time | 5 |
| slow_query_log | ON |
| slow_query_log_file | /mysql/data/nagiosdb-slow.log |
+---------------------------+-------------------------------+
5 rows in set (0.00 sec)
mysql 5.1.6版本起，slow_query_log 和 slow_launch_time 支持写文件或写数据库表两种方式，并且日志的开启，输出方式的修改，都可以在global级别动态修改。
只需简单通过set global slow_query_log=ON;即可开启慢查询，而不需要重启数据库！
可以直接写到配置文件中 my.cnf
slow_query_log_file=/mysql/log/nagiosdb-slow.log
slow_launch_time=5
可以完成配置！！
！！！根据上述配置开启慢日志后，查询本机日志名称及目录，不要搞错。
在test.conf配置文件添加如下：
- # this is mysql log
- file {
- type => "mysql_log"
- tags => ["mysql"]
- path => ["/var/log/mysql/mysqld.log"]
- start_position => beginning
- ignore_older => 0
- }
- # this is mysql-slow log
- file {
- type => "mysql_slow"
- tags => ["mysql-slow"]
- path => ["/var/run/mysqld/mysqld-slow.log"]
- start_position => beginning
- ignore_older => 0
- }
配置完后启动logstash
[root@localhost logstash]# ./bin/logstash -f test.conf
确认慢查询已开启
mysql> show variables like "%slow%";
+---------------------+---------------------------------+
| Variable_name | Value |
+---------------------+---------------------------------+
| log_slow_queries | ON |
| slow_launch_time | 2 |
| slow_query_log | ON |
| slow_query_log_file | /var/run/mysqld/mysqld-slow.log |
+---------------------+---------------------------------+
4 rows in set (0.00 sec)
mysql>
输入测试命令，并查看kibana是否有mysql-low数据输出；
mysql> select sleep(6);
+----------+
| sleep(6) |
+----------+
| 0 |
+----------+
1 row in set (5.99 sec)
mysql>
慢数据输出正常：
!!!目前所有日志数据均未作规范化输出处理，只是简单测试了elk的基本功能，后续将升入研究学习elk.后面会继续做好笔记供自己参考学习。