在使用ELK做日志分析的时候,有时需要一个filebeat采集多个日志,送给ES,或者给logstash做解析。下面举例演示以下filebeat采集messages日志,secure日志,以及nginx日志送给ES或者送给logstash做解析的正确配置方法。
一、filebeat采集日志发送给ES:
1.1、filebeat.yml 配置如下:
filebeat.inputs:
- type: log
tail_files: true
scan_frequency: 5s
backoff: 1s
max_backoff: 10s
paths:
- /usr/local/apache-tomcat-8.0.39/logs/catalina.out
- /usr/local/apache-tomcat-8.0.39/logs/*.txt
fields:
type: tomcat
ip: 172.16.3.226
fields_under_root: true
- type: log
tail_files: true
scan_frequency: 5s
backoff: 1s
max_backoff: 10s
paths:
- /var/log/nginx/access.log
fields:
type: nginx
ip: 172.16.3.226
fields_under_root: true
output.elasticsearch:
hosts: ["172.16.3.227:9200"]
二、filebeat采集日志发送给logstash(不解析):
2.1、filebeat.yml 配置如下:
filebeat.inputs:
- type: log
tail_files: true
scan_frequency: 5s
backoff: 1s
max_backoff: 10s
paths:
- /usr/local/apache-tomcat-8.0.39/logs/catalina.out
- /usr/local/apache-tomcat-8.0.39/logs/*.txt
fields:
type: tomcat
ip: 172.16.3.226
fields_under_root: true
- type: log
tail_files: true
scan_frequency: 5s
backoff: 1s
max_backoff: 10s
paths:
- /var/log/nginx/access.log
fields:
type: nginx
ip: 172.16.3.226
fields_under_root: true
output.logstash:
hosts: ["172.16.3.227:5044"]
2.2、logstash.conf 配置如下(不解析):
input {
beats {
host => '0.0.0.0'
port => 5044
}
}
output{
if [type] == "tomcat" {
elasticsearch {
hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
index => "tomcat_log-%{+YYYY.MM.dd}"
# user => xxx # 这里需要注意的是如果es配置了X-pack那么就需要在这里加上用户密码
# password => xxx
}
stdout{
codec=>rubydebug
}
}
else if [type] == "nginx" {
elasticsearch {
hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
index => "nginx_log_-%{+YYYY.MM.dd}"
}
stdout{
codec=>rubydebug
}
}
三、filebeat采集日志发送给logstash(解析):
3.1、filebeat.yml 配置如下:
filebeat.inputs:
- type: log
tail_files: true
scan_frequency: 5s
backoff: 1s
max_backoff: 10s
paths:
- /usr/local/apache-tomcat-8.0.39/logs/catalina.out
- /usr/local/apache-tomcat-8.0.39/logs/*.txt
fields:
type: tomcat
ip: 172.16.3.226
fields_under_root: true
- type: log
tail_files: true
scan_frequency: 5s
backoff: 1s
max_backoff: 10s
paths:
- /var/log/nginx/access.log
fields:
type: nginx
ip: 172.16.3.226
fields_under_root: true
output.logstash:
hosts: ["172.16.3.227:5044"]
3.2、logstash.conf 配置如下(解析):
input {
beats {
host => '0.0.0.0'
port => 5044
}
}
filter {
if [type] == "access" {
grok {
match => {
"message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - (?<user>\S+) \[(?<timestamp>[^ ]+ \+[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^
]+) HTTP/\d.\d" (?<status>\d+) (?<bodysize>\d+) "(?<url>\S+)" "[^"]+"'}
#移除不需要的字段
remove_field => ["message","@version","path"]
}
date {
match => ["requesttime", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
}
}
output{
if [type] == "nginx" {
elasticsearch {
hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
index => "nginx_log-%{+YYYY.MM.dd}"
}
}
else if [type] == "tomcat" {
elasticsearch {
hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
index => "tomcat_log-%{+YYYY.MM.dd}"
}
}
else if [type] == "access" {
elasticsearch {
hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
index => "access-%{+YYYY.MM.dd}"
}
}
stdout{
codec=>rubydebug
}
}