蜂巢APP对接ELK统一日志

docker-compose的方式对接ELK

(1)一键部署docker和docker-compose环境

docker容器简介 (一键部署脚本)

#!/bin/bash

install docker shell

#step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

2: 添加软件源信息

sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3: 更新并安装 Docker-CE

sudo yum makecache fast
sudo yum -y install docker-ce
#4: 开启Docker服务
sudo service docker start

4.添加镜像加速器

echo “4.添加镜像加速器…”
sudo tee /etc/docker/daemon.json <<-‘EOF’
{
“registry-mirrors”: [“https://xcweb0za.mirror.aliyuncs.com”]
}
EOF

5.重新启动服务

echo “5.重新启动服务…”
sudo systemctl daemon-reload
sudo systemctl restart docker

6.检测

echo “6.检测…”
docker info

7.安装docker-compose

echo “7.安装docker-compose”
curl -L https://get.daocloud.io/docker/compose/releases/download/1.22.0/docker-compose-uname -s-uname -m > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

8.验证是否安装成功

echo “8.验证是否安装成功…”
docker-compose -v

(2)编写docker-compose文件

version: ‘2’
services:
filebeat-logcenter:
image: harbor.qilu-dev.com/elk_public/filebeat:6.8.2
container_name: filebeat-logcenter
network_mode: host
restart: always
volumes:

  • /root/log/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
  • /usr/java/logs/:/usr/java/logs/:ro
  • /usr/java/apache-tomcat-8.0.46/logs/:/usr/java/apache-tomcat-8.0.46/logs/:ro
    environment:
  • TZ=Asia/Shanghai
  • KAFKA_HOSTS=10.29.131.62:9092,10.29.131.63:9092,10.29.131.64:9092,10.29.131.65:9092

(3)filebeat模板文件

filebeat.inputs:

  • type: log
    enabled: true
    paths:

  • /usr/java/logs/ztapi.log
    fields:
    headers.appName: ztapi
    headers.key: ztapi
    fields_under_root: true
    multiline.pattern: ‘^[’
    multiline.negate: ‘true’
    multiline.match: ‘after’

  • type: log
    enabled: true
    paths:

  • /usr/java/apache-tomcat-8.0.46/logs/catalina.out
    fields:
    headers.appName: ztapitomcat
    headers.key: ztapitomcat
    fields_under_root: true
    multiline.pattern: ‘^\d{2}-\w{3}-\d{4}’
    multiline.negate: ‘true’
    multiline.match: ‘after’

  • type: log
    enabled: true
    paths:

  • /usr/java/apache-tomcat-8.0.46/logs/localhost_access_log*
    fields:
    headers.appName: ztapilocalhostaccess
    headers.key: ztapilocalhostaccess
    fields_under_root: true
    multiline.pattern: ‘^\d+.\d+.\d+.\d+’
    multiline.negate: ‘true’
    multiline.match: ‘after’
    filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

output.kafka:
enabled: true
hosts: “${KAFKA_HOSTS}”
topic: logCenter-other
key: appname
partition.hash:
reachable_only: false
keep_alive: 60
required_acks: 1

processors:

  • add_host_metadata:
    cache.ttl: 5m
  • rename:
    fields:
  • from: “host.name”
    to: “headers.host”
  • drop_fields:
    fields: [“metadata”, “log”, “input”, “ecs”, “agent”, “host”, “prospector”, “source”, “beat”, “offset”]

(3)一键启动对接

设置好相应目录文件

一键启动

docker-compose up -d

上一篇:fields.E304 Reverse accessor *es in Django


下一篇:Structured Streaming编程练习-日志分析