十一、logstash配置文件及插件
11.1 logstash的第一个配置文件
–/etc/logstash/logstash.conf
input{
stdin{}
}
filter{ }
output{
stdout{}
}
?启劢并验证
–logstash –f logstash.conf
?logstash插件
–上页的配置文件使用了logstash-input-stdin和logstash-output-stdout两个插件，logstash还有filter 和codec 类插件，查看插件的方式是
/opt/logstash/bin/logstash-plugin list
[root@logstash ~]# /opt/logstash/bin/logstash-plugin list 查看logstash插件
Ignoring ffi-1.9.13 because its extensions are not built. Try: gem pristine ffi --version 1.9.13
logstash-codec-collectd
logstash-codec-dots
。。。。。。。。
logstash-filter-anonymize
logstash-filter-checksum
。。。。。。。。
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
。。。。。。
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
。。。。。。
logstash-patterns-core
注：插件有区域段的划分codec是编码可以用在所有区域段
–插件及文档地址
?练习
–logstash配置从标准输入读取输入源，然后从标准输出输出到屏幕
指定logstash配置文件的位置
[root@logstash logstash]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
[root@logstash logstash]# vim /etc/logstash/logstash.conf
input{
stdin{}
}
filter{ }
output{
stdout{}
}
/opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
2021-08-19T02:19:13.006Z logstash ddqw
源码地址：https://github.com/logstash-plugins
插件帮助的查看（重点）https://www.elastic.co/cn/
选择版本
选择模块
选择插件
?codec类插件
–常用的插件：plain、json、json_lines、rubydebug、multiline等
–我们还使用刚刚的例子，丌过这次我们输入json数据
–我们设置输入源的codec 是json，在输入的时候选择rubydebug
[root@logstash logstash]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
}
filter{ }
output{
stdout{ codec => "rubydebug" }
}
[root@logstash logstash]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
aaaa
{
"message" => "aaaa",
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"@timestamp" => "2021-08-19T02:22:22.444Z",
"host" => "logstash"
}
{"a":1,"b":2,"c":3}
{
"a" => 1,
"b" => 2,
"c" => 3,
"@version" => "1",
"@timestamp" => "2021-08-19T02:27:36.122Z",
"host" => "logstash"
}
?练习input file 插件
file{
start_position=> "beginning"
sincedb_path=> "/var/lib/logstash/sincedb-access"
path => [“/tmp/alog”, “/tmp/blog”]
type => ‘filelog‘
}
–sincedb_path 记录读取文件的位置
–start_position 配置第一次读取文件从什么地方开始
[root@logstash logstash]# vim /etc/logstash/logstash.conf
input{
stdin{ codec => "json" }
file {
path => ["/tmp/a.log","/tmp/b.log"]
sincedb_path => "/var/lib/logstash/sincedb.log"
start_position => "beginning"
type => "filelog"
}
}
filter{ }
output{
stdout{ codec => "rubydebug" }
}
[root@logstash logstash]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
Settings: Default pipeline workers: 2
Pipeline main started
{"a":1,"b":2}
{
"a" => 1,
"b" => 2,
"@version" => "1",
"@timestamp" => "2021-08-19T03:01:01.306Z",
"host" => "logstash"
打开另一个终端输入文件查看：
[root@logstash ~]# echo a4 >>/tmp/a.log
[root@logstash ~]# echo a1 >>/tmp/a.log
{
"message" => "a4",
"@version" => "1",
"@timestamp" => "2021-08-19T03:02:45.849Z",
"path" => "/tmp/a.log",
"host" => "logstash",
"type" => "filelog"
}
{
"message" => "a1",
"@version" => "1",
"@timestamp" => "2021-08-19T03:02:54.869Z",
"path" => "/tmp/a.log",
"host" => "logstash",
"type" => "filelog"