ELK学习笔记之logstash安装logstash-filter-multiline(在线离线安装)

0x00 概述

ELK-logstash在搬运日志的时候会出现多行日志,普通的搬运会造成保存到ES中单条单条,很丑,而且不方便读取,logstash-filter-multiline可以解决该问题

github地址:https://github.com/logstash-plugins/logstash-filter-multiline

其他插件的地址:https://github.com/logstash-plugins

官网地址:https://www.elastic.co/cn/products/logstash

普通日志如下:

ELK学习笔记之logstash安装logstash-filter-multiline(在线离线安装)

记录到es会的记录则是:

ELK学习笔记之logstash安装logstash-filter-multiline(在线离线安装)

我们希望的结果肯定是这样的

ELK学习笔记之logstash安装logstash-filter-multiline(在线离线安装)

0x01 原生安装

# /usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline
Validating logstash-filter-multiline
Installing logstash-filter-multiline
WARNING: can not set Session#timeout=(0) no session context

0x02 改镜像源安装

给 Ruby 加上国内的镜像站:https://gems.ruby-china.com/,替代https://rubygems.org。

注意原https://gems.ruby-china.org/目前已经可用,需要使用https://gems.ruby-china.com/

1. 安装Gem并更新

# yum install -y gem
# gem -v
2.0.14.1
# gem update --system
# gem -v
2.6.13

2. 检查并修改镜像源

# gem sources -l
*** CURRENT SOURCES *** https://rubygems.org/
# gem sources --add https://gems.ruby-china.com/ --remove https://rubygems.org/
https://gems.ruby-china.com/ added to sources
https://rubygems.org/ removed from sources
# cat ~/.gemrc
---
:backtrace: false
:bulk_threshold: 1000
:sources:
- https://gems.ruby-china.com/
:update_sources: true
:verbose: true

3. 安装 bundle 并更改镜像源

# gem install bundler
# bundle config mirror.https://rubygems.org https://gems.ruby-china.com

4. 修改 logstash的 gem 镜像源

# vim /usr/share/logstash/Gemfile

# This is a Logstash generated Gemfile.
# If you modify this file manually all comments and formatting will be lost. source "https://rubygems.org"
gem "logstash-core", :path => "./logstash-core"
......
# 更改默认的 https://rubygems.org 为https://gems.ruby-china.com

Gemfile.jruby-1.9.lock文件不用改,会自动更新。

5. 安装 logstash-filter-multiline

# /usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline
Validating logstash-filter-multiline
Installing logstash-filter-multiline
Installation successful

我们看看logstash都安装了哪些插件

# /usr/share/logstash/bin/logstash-plugin list
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-filter-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-jdbc_static
logstash-filter-jdbc_streaming
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-xml
logstash-input-beats
logstash-input-dead_letter_queue
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jdbc
logstash-input-kafka
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-kafka
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core

6. 离线打包

这里安装好了,那么就可以打个离线的 zip 包,方便生产环境直接安装。离线包将包含所有依赖的包。

# cd /usr/share/logstash/bin
# ./logstash-plugin prepare-offline-pack --overwrite --output logstash-filter-multiline.zip logstash-filter-multiline
Offline package created at: logstash-filter-multiline.zip You can install it with this command `bin/logstash-plugin install file:///usr/share/logstash/bin/logstash-filter-multiline.zip`

7. logstash-filter-multiline使用方法

codec =>multiline {
charset=>... #可选 字符编码
max_bytes=>... #可选 bytes类型 设置最大的字节数
max_lines=>... #可选 number类型 设置最大的行数,默认是500行
multiline_tag... #可选 string类型 设置一个事件标签,默认是multiline
pattern=>... #必选 string类型 设置匹配的正则表达式
patterns_dir=>... #可选 array类型 可以设置多个正则表达式
negate=>... #可选 boolean类型 默认false不显示,可设置ture
what=>... #必选 向前previous , 向后 next
}
## negate  只支持布尔值,true 或者false,默认为false。
如果设置为true,表示与正则表达式(pattern)不匹配的内容都需要整合,
具体整合在前还是在后,看what参数。如果设置为false,即与pattern匹配的内容 ## what 前一行 或者后一行,指出上面对应的规则与前一行内容收集为一行,还是与后一行整合在一起 negate默认是 false,不显示与patten匹配的行
由what决定 向前或向后 匹配 negate 设置为true
则与patten 不匹配的行
由what决定 向前或向后 匹配
上一篇:ELK学习总结(4-1)elasticsearch更改mapping(不停服务重建索引)


下一篇:ELK 学习笔记之 Kibana安装