利用docker部署elk交换机日志分析

今天我们来聊一下利用docker部署elk日志分析系统,这里解析一下elk是啥东西。elk分别是Elasticsearch,Logstash和Kibana的首字母缩写。

  • Elasticsearch是一个基于JSON的分布式搜索和分析引擎,专为水平可扩展性,最高可靠性和易管理性而设计。
  • Logstash是一个动态数据收集管道,具有可扩展的插件生态系统和强大的Elasticsearch协同作用。
  • Kibana通过UI 提供数据可视化

架构简述

日志系统首先面临几个问题:

不同厂商设备的不同日志格式的处理,如何调用微信来发送报警信息。采用的解决办法是不同厂商的设备发送日志的时候采用不同端口,日志先发送到logstash,

logstash会先解析日志成标准格式,然后logstash会做2件事情,一个是存放日志到es里面,通过kibana做出展示。

环境搭建:

我们这里采用docker容器技术来部署elk。先从docker hub 下载 Elasticsearch,Logstash和Kibana这三个镜像。执行如下命令

[root@node1 ~]# docker pull elasticsearch
[root@node1 ~]# docker pull kibana
[root@node1 ~]# docker pull logstash:7.2.0

为了后续管理方便,这里编写yml文件,然后使用docker-compose来启动。

version: '3.3'

services:

  elasticsearch:
image: elasticsearch:latest
container_name: elasticsearch
hostname: elasticsearch
ports:
- '9200:9200'
environment:
ES_JAVA_OPTS: "-Xms256m -Xmx256m"
discovery.type: "single-node"
volumes:
- "/data/elk/elasticsearch/data:/usr/elasticsearch/data" logstash:
image: logstash:7.2.0
container_name: logstash
hostname: logstash
ports:
- "514:514/udp"
user: 'root'
command: "logstash -f /etc/logstash.conf --config.reload.automatic"
volumes:
- "/data/elk/logstash/logstash.conf:/etc/logstash.conf"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
links:
- "elasticsearch:elasticsearch" kibana:
image: kibana:latest
container_name: kibana
hostname: kibana
ports:
- '5601:5601'
environment:
ELASTICSEARCH_URL: "http://elasticsearch:9200"
links:
- "elasticsearch:elasticsearch"

注意:/data/elk/logstash/logstash.conf这个文件需要我们提前准备,后面会提到。

交换机配置

我们需要把交换机的日志指定到elk服务器上。

cisco:
logging host 10.100.18.18 transport udp port 5002 H3C
info-center enable
info-center source default channel 2 trap state off
// 必要,不然日志会出现 不符合级别的 alert 日志
info-center loghost 10.100.18.18 port 5003 huawei
info-center enable
info-center loghost 10.100.18.18
info-center timestamp log short-date
info-center timestamp trap short-date

Logstash 的配置

不同厂商的日志 gork我都写好了,复制过去就能用

input{
tcp {port => 5002 type => "Cisco"}
udp {port => 514 type => "HUAWEI"}
udp {port => 5002 type => "Cisco"}
udp {port => 5003 type => "H3C"}
}
filter {
if [type] == "Cisco"{
grok{
match => { "message" => "<%{BASE10NUM:syslog_pri}>%{NUMBER:log_sequence}: .%{SYSLOGTIMESTAMP:timestamp}: %%{DATA:facility}-%{POSINT:severity}-%{CISCO_REASON:mnemonic}: %{GREEDYDATA:message}" }
match => { "message" => "<%{BASE10NUM:syslog_pri}>%{NUMBER:log_sequence}: %{SYSLOGTIMESTAMP:timestamp}: %%{DATA:facility}-%{POSINT:severity}-%{CISCO_REASON:mnemonic}: %{GREEDYDATA:message}" }
add_field => {"severity_code" => "%{severity}"}
overwrite => ["message"]
}
}
else if [type] == "H3C"{
grok {
match => { "message" => "<%{BASE10NUM:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{YEAR:year} %{DATA:hostname} %%%{DATA:vvmodule}/%{POSINT:severity}/%{DATA:digest}: %{GREEDYDATA:message}" }
remove_field => [ "year" ]
add_field => {"severity_code" => "%{severity}"}
overwrite => ["message"]
}
}
else if [type] == "HUAWEI"{
grok {
match => { "message" => "<%{BASE10NUM:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{DATA:hostname} %%%{DATA:ddModuleName}/%{POSINT:severity}/%{DATA:Brief}:%{GREEDYDATA:message}"}
match => { "message" => "<%{BASE10NUM:syslog_pri}>%{SYSLOGTIMESTAMP:timestamp} %{DATA:hostname} %{DATA:ddModuleName}/%{POSINT:severity}/%{DATA:Brief}:%{GREEDYDATA:message}"}
remove_field => [ "timestamp" ]
add_field => {"severity_code" => "%{severity}"}
overwrite => ["message"]
}
} mutate {
gsub => [
"severity", "0", "Emergency",
"severity", "1", "Alert",
"severity", "2", "Critical",
"severity", "3", "Error",
"severity", "4", "Warning",
"severity", "5", "Notice",
"severity", "6", "Informational",
"severity", "7", "Debug"
]
}
}
output{
elasticsearch {
index => "syslog-%{+YYYY.MM.dd}"
hosts => ["your_ipaddress:9200"]
}
}

Normal
0

7.8 磅
0
2

false
false
false

EN-US
ZH-CN
X-NONE

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:普通表格;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
mso-bidi-font-size:11.0pt;
font-family:等线;
mso-ascii-font-family:等线;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:微软雅黑;
mso-hansi-font-family:等线;
mso-hansi-theme-font:minor-latin;
mso-font-kerning:1.0pt;}

上一篇:卷积相关公式的matlab代码


下一篇:Linux实战(20):非docker部署ELK