filebeat采集多个日志(推送给ES或者logstash)

filebeat采集多个日志

在使用ELK做日志分析的时候,有时需要一个filebeat采集多个日志,送给ES,或者给logstash做解析。下面举例演示以下filebeat采集messages日志,secure日志,以及nginx日志送给ES或者送给logstash做解析的正确配置方法。

一、filebeat采集日志发送给ES:

1.1、filebeat.yml 配置如下:

filebeat.inputs:
- type: log
  tail_files: true
  scan_frequency: 5s
  backoff: 1s
  max_backoff: 10s
  paths:
      - /usr/local/apache-tomcat-8.0.39/logs/catalina.out
      - /usr/local/apache-tomcat-8.0.39/logs/*.txt
  fields:
    type: tomcat
    ip: 172.16.3.226
  fields_under_root: true

- type: log
  tail_files: true
  scan_frequency: 5s
  backoff: 1s
  max_backoff: 10s
  paths:
      - /var/log/nginx/access.log
  fields:
    type: nginx
    ip: 172.16.3.226
  fields_under_root: true

output.elasticsearch:
  hosts: ["172.16.3.227:9200"]

二、filebeat采集日志发送给logstash(不解析):

2.1、filebeat.yml 配置如下:

filebeat.inputs:
- type: log
  tail_files: true
  scan_frequency: 5s
  backoff: 1s
  max_backoff: 10s
  paths:
      - /usr/local/apache-tomcat-8.0.39/logs/catalina.out
      - /usr/local/apache-tomcat-8.0.39/logs/*.txt
  fields:
    type: tomcat
    ip: 172.16.3.226
  fields_under_root: true

- type: log
  tail_files: true
  scan_frequency: 5s
  backoff: 1s
  max_backoff: 10s
  paths:
      - /var/log/nginx/access.log
  fields:
    type: nginx
    ip: 172.16.3.226
  fields_under_root: true

output.logstash:
  hosts: ["172.16.3.227:5044"]

2.2、logstash.conf 配置如下(不解析):

input {
        beats {
                host => '0.0.0.0'
                port => 5044 
        }
}

output{
  if [type] == "tomcat" {
    elasticsearch {
      hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
      index => "tomcat_log-%{+YYYY.MM.dd}"
#      user =>	xxx		# 这里需要注意的是如果es配置了X-pack那么就需要在这里加上用户密码
#	   password =>  xxx	
    }
    stdout{
      codec=>rubydebug
    }
  }
  else if [type] == "nginx" {
    elasticsearch {
      hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
      index => "nginx_log_-%{+YYYY.MM.dd}"
    }
    stdout{
      codec=>rubydebug
    }
  }

三、filebeat采集日志发送给logstash(解析):

3.1、filebeat.yml 配置如下:

filebeat.inputs:
- type: log
  tail_files: true
  scan_frequency: 5s
  backoff: 1s
  max_backoff: 10s
  paths:
    - /usr/local/apache-tomcat-8.0.39/logs/catalina.out
    - /usr/local/apache-tomcat-8.0.39/logs/*.txt
  fields:
    type: tomcat
    ip: 172.16.3.226
  fields_under_root: true

- type: log
  tail_files: true
  scan_frequency: 5s
  backoff: 1s
  max_backoff: 10s
  paths:
      - /var/log/nginx/access.log
  fields:
    type: nginx
    ip: 172.16.3.226
  fields_under_root: true

output.logstash:
  hosts: ["172.16.3.227:5044"]

3.2、logstash.conf 配置如下(解析):

input {
        beats {
                host => '0.0.0.0'
                port => 5044 
        }
}

filter {
 if [type] == "access" {
    grok {
        match => {
            "message" => '(?<clientip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) - (?<user>\S+) \[(?<timestamp>[^ ]+ \+[0-9]+)\] "(?<requesttype>[A-Z]+) (?<requesturl>[^
 ]+) HTTP/\d.\d" (?<status>\d+) (?<bodysize>\d+) "(?<url>\S+)" "[^"]+"'}
#移除不需要的字段
       remove_field => ["message","@version","path"] 
    }
    date {
        match => ["requesttime", "dd/MMM/yyyy:HH:mm:ss Z"]
        target => "@timestamp"
    }
  }
}

output{
  if [type] == "nginx" {
    elasticsearch {
      hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
      index => "nginx_log-%{+YYYY.MM.dd}"
    }
  }
  else if [type] == "tomcat" {
    elasticsearch {
      hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
      index => "tomcat_log-%{+YYYY.MM.dd}"
    }
  }
  else if [type] == "access" {
    elasticsearch {
      hosts => ["http://172.16.3.225:9200","http://172.16.3.226:9200","http://172.16.3.227:9200"]
      index => "access-%{+YYYY.MM.dd}"
    }
  }
    stdout{
      codec=>rubydebug
    }
}
上一篇:日志收集-Elk6


下一篇:使用ElasticSearch服务从MySQL同步数据实现搜索即时提示与全文搜索功能