ELK日志分析平台2-Logstash数据采集

2023-12-31 22:01:58

文章目录

1、 logstash介绍

  • Logstash是一个开源的服务器端数据处理管道。
  • logstash拥有200多个插件，能够同时从多个来源采集数据，转换数据，然后将数据发送到您最喜欢的 “存储库” 中。（大多都是 Elasticsearch。）
  • Logstash管道有两个必需的元素，输入和输出，以及一个可选元素过滤器。

2、安装logstash

2.1 下载jdk-8u181-linux-x64.rpm和logstash-7.6.1.rpm

下载链接

[root@server13 ~]# ll  ##官网下载
total 334812
-rw-r--r-- 1 root root 170023183 Sep  5  2018 jdk-8u181-linux-x64.rpm
-rw-r--r-- 1 root root 172821011 Mar 12  2020 logstash-7.6.1.rpm

2.2 安装

[root@server13 ~]# rpm -ivh jdk-8u181-linux-x64.rpm
[root@server13 ~]# rpm -ivh logstash-7.6.1.rpm

2.3 配置环境变量

[root@server13 opt]# cd /usr/share/logstash/bin
[root@server13 bin]# ls
benchmark.sh         logstash               logstash.lib.sh      pqrepair
cpdump               logstash.bat           logstash-plugin      ruby
dependencies-report  logstash-keystore      logstash-plugin.bat  setup.bat
ingest-convert.sh    logstash-keystore.bat  pqcheck              system-install
[root@server13 bin]# pwd
/usr/share/logstash/bin
[root@server13 bin]# cd

[root@server13 ~]# vim .bash_profile
[root@server13 ~]# cat .bash_profile | grep bin
PATH=$PATH:$HOME/bin:/usr/share/logstash/bin
[root@server13 ~]# source .bash_profile
[root@server13 ~]# which logstash
/usr/share/logstash/bin/logstash

2.4 安装完成测试一下

标准输入输出

[root@server13 ~]# logstash -e 'input { stdin { }} output { stdout {} }'  
##标准输入输出，键盘输入，屏幕输出。ctrl+c退出

2.5 标准输入与标准输出

[root@server13 ~]# cd /etc/logstash/
[root@server13 logstash]# ls
conf.d       log4j2.properties     logstash.yml   startup.options
jvm.options  logstash-sample.conf  pipelines.yml
[root@server13 logstash]# cd conf.d/   ##目录下任意的conf文件都可以执行
[root@server13 conf.d]# ls
[root@server13 conf.d]# vim test.conf
[root@server13 conf.d]# cat test.conf 
input {
  stdin {}
}

output {
  stdout {}
}
[root@server13 conf.d]# logstash -f /etc/logstash/conf.d/test.conf   ##执行文件

2.6 标准输入到文件

[root@server16 conf.d]# vim test.conf 
[root@server16 conf.d]# cat test.conf   ##输入到文件
input {
        stdin { }
}
output {
     	stdout {}
     	file {
                path => "/tmp/testfile"
                codec => line { format => "custom format: %{message}"}
        }
}
[root@server16 conf.d]# logstash -f /etc/logstash/conf.d/test.conf 运行
[root@server16 conf.d]# cat /tmp/testfile    ##查看文件内容
custom format: hello

3、elasticsearch输出插件

插件官网

[root@server13 conf.d]# cd /etc/logstash/conf.d/
[root@server13 conf.d]# vim test.conf 
[root@server13 conf.d]# cat test.conf 
input {
        file {
 	path => "/var/log/messages"
	start_position => "beginning"
	}
}
output {
     	stdout {}
     	#file {
        #        path => "/tmp/testfile"
        #        codec => line { format => "custom format: %{message}"}
        #}
	elasticsearch {
	hosts => ["172.25.200.5:9200"]
	index => "syslog-%{+yyyy.MM.dd}"
	}
}

[root@server13 conf.d]# logstash -f /etc/logstash/conf.d/test.conf

ELK日志分析平台2-Logstash数据采集

4、file输入插件

5、Syslog输入插件

logstash可以伪装成日志服务器，直接接受远程日志

[root@server13 conf.d]# vim test.conf 
[root@server13 conf.d]# cat test.conf 
input {
        #file {
 	#path => "/var/log/messages"
	#start_position => "beginning"
	#}
        syslog{}  ##日志默认收集端口514
}
output {
     	stdout {}
     	#file {
        #        path => "/tmp/testfile"
        #        codec => line { format => "custom format: %{message}"}
        #}
	elasticsearch {
	hosts => ["172.25.200.5:9200"]
	index => "syslog-%{+yyyy.MM.dd}"
	}
}

[root@server13 conf.d]# logstash -f /etc/logstash/conf.d/test.conf #指定配置文件运行


[root@server13 conf.d]# netstat -antulp |grep :514
tcp6       0      0 172.25.200.13:514       172.25.200.6:44300      FIN_WAIT2   -                   
[root@server6 ~]# netstat -antulp |grep :514
tcp        1      0 172.25.200.6:44298      172.25.200.13:514       CLOSE_WAIT

配置客户端传入日志

[root@server6 ~]# vim /etc/rsyslog.conf
[root@server6 ~]# systemctl restart rsyslog.service

ELK日志分析平台2-Logstash数据采集

测试：

ELK日志分析平台2-Logstash数据采集
ELK日志分析平台2-Logstash数据采集
ELK日志分析平台2-Logstash数据采集

6、多行过滤插件

多行过滤可以把多行日志记录合并为一行事件

6.1 原理测试

[root@server13 conf.d]# vim demo.conf 
[root@server13 conf.d]# logstash -f /etc/logstash/conf.d/demo.conf

ELK日志分析平台2-Logstash数据采集
查看输出
ELK日志分析平台2-Logstash数据采集

6.2 java类型报错日志采集

java一条错误会现实很多行，这里将日志按照时间戳归类为一条

[root@server5 elasticsearch]# scp my-es.log server13:/var/log
The authenticity of host 'server13 (172.25.200.13)' can't be established.
ECDSA key fingerprint is SHA256:nLo7FniQXEU0jhR+TkuB2oUVT7uSgCm3zDHaDmDSuXU.
ECDSA key fingerprint is MD5:80:06:78:da:83:6d:80:59:b4:bb:37:0a:e8:db:f4:10.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'server13,172.25.200.13' (ECDSA) to the list of known hosts.
root@server13's password: 
my-es.log                                     100%   15KB   6.2MB/s   00:00   


[root@server13 conf.d]# cat test.conf 
input {
        file {
 	path => "/var/log/my-es.log"
	start_position => "beginning"
	codec => multiline {
          pattern => "^\["
          negate => "true"
          what => "previous" 
        }
      }

}
output {
     	#stdout {}
     	#file {
        #        path => "/tmp/testfile"
        #        codec => line { format => "custom format: %{message}"}
        #}
	elasticsearch {
	hosts => ["172.25.200.5:9200"]
	index => "my-ce-%{+yyyy.MM.dd}"
	}
}


[root@server13 conf.d]# logstash -f /etc/logstash/conf.d/test.conf

ELK日志分析平台2-Logstash数据采集

ELK日志分析平台2-Logstash数据采集

7、grok过滤插件(采集过滤apache日志)

7.1 先安装apache

[root@server13 conf.d]# yum install -y httpd
[root@server13 conf.d]# systemctl start httpd.service
[root@server13 conf.d]# systemctl status httpd.service
[root@server13 conf.d]# echo server13 > /var/www/html/index.html

[root@haojin ~]# ab -c1 -n 100 172.25.200.13/index.html  ##真机压测获得日志数据

ELK日志分析平台2-Logstash数据采集

ELK日志分析平台2-Logstash数据采集
ELK日志分析平台2-Logstash数据采集

7.2 采集apache日志

1.增加被采集目录权限，否则采集不到数据
[root@server13 httpd]# ll -d /var/log/httpd/
drwx------ 2 root root 41 3月  10 11:27 /var/log/httpd/
[root@server13 httpd]# chmod 755 /var/log/httpd/
[root@server13 httpd]# ll -d /var/log/httpd/
drwxr-xr-x 2 root root 41 3月  10 11:27 /var/log/httpd/

2. 编辑conf文件，运行采集程序
[root@server13 conf.d]# vim grok.conf 
[root@server13 conf.d]# cat grok.conf 
input {
	file {
        path => "/var/log/httpd/access_log"
        start_position => "beginning"
 
   }
}
filter {
	grok {
	match => { "message" => "%{HTTPD_COMBINEDLOG}" }
	}
}
output {
	stdout {}
        elasticsearch{
        hosts => ["172.25.200.5:9200"]    
        index => "apache-%{+yyyy.MM.dd}"
  }
}
[root@server13 conf.d]# logstash -f grok.conf

ELK日志分析平台2-Logstash数据采集
ELK日志分析平台2-Logstash数据采集
ELK日志分析平台2-Logstash数据采集

希望本文对你有帮助，请点个赞鼓励一下作者吧~ 谢谢！

上一篇：ELK之rpm包安装


下一篇：ELK-logstash