部署环境：

Elasticsearch：7.5.2

Kibana：7.5.2

Logstash：7.5.2

filebeat：7.5.2

redis：最新版

部署方式：rpm+二进制包

使用架构

软件包

logstash-7.5.2.rpm

elasticsearch-7.5.2-x86_64.rpm

filebeat-7.5.2-linux-x86_64.tar.gz

安装

Elasticsearch集群安装

主节点：10.228.83.120 从节点：10.228.83.112、10.228.83.111、10.228.83.66

1. 通过rpm包安装

rpm --install elasticsearch-7.5.2-x86_64.rpm

1. 主节点配置

# ======================== Elasticsearch Configuration =========================

#

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

#       Before you set out to tweak and tune the configuration, make sure you

#       understand what are you trying to accomplish and the consequences.

#

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

#

# Please consult the documentation for further information on configuration options:

# https://www.elastic.co/guide/en/elasticsearch/reference/index.html

#

# ---------------------------------- Cluster -----------------------------------

#

# Use a descriptive name for your cluster:

#

cluster.name: my-application  #集群名

#

# ------------------------------------ Node ------------------------------------

#

# Use a descriptive name for the node:

#

node.name: node-2  #节点名，唯一

#

# Add custom attributes to the node:

#

#node.attr.rack: r1

#

# ----------------------------------- Paths ------------------------------------

#

# Path to directory where to store the data (separate multiple locations by comma):

#

path.data: /var/lib/elasticsearch

#

# Path to log files:

#

path.logs: /var/log/elasticsearch

#

# ----------------------------------- Memory -----------------------------------

#

# Lock the memory on startup:

#

#bootstrap.memory_lock: true

#

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

#

# Elasticsearch performs poorly when the system is swapping the memory.

#

# ---------------------------------- Network -----------------------------------

#

# Set the bind address to a specific IP (IPv4 or IPv6):

#

network.host: 0.0.0.0  #可访问的地址，这里配置所有都可访问

#

# Set a custom port for HTTP:

#

http.port: 9200

transport.tcp.port: 9300

node.master: true

node.data: true

cluster.initial_master_nodes: ["node-2","node-1"]		#以上4行配置都是集群的配置非集群可不配,配置具体含义建议去查一下，大致意思就是配置端口，配置是否可成为master，配置node是否存data。

#

# For more information, consult the network module documentation.

#

# --------------------------------- Discovery ----------------------------------

#

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

#

discovery.seed_hosts: ["10.228.83.120:9300", "10.228.83.66:9300","10.228.83.112:9300"]   #集群话配置非集群可不配

#

# Bootstrap the cluster using an initial set of master-eligible nodes:

#

#cluster.initial_master_nodes: ["node-1", "node-2","node-3"]

#

# For more information, consult the discovery and cluster formation module documentation.

#

# ---------------------------------- Gateway -----------------------------------

#

# Block initial recovery after a full cluster restart until N nodes are started:

#

#gateway.recover_after_nodes: 3

#

# For more information, consult the gateway module documentation.

#

# ---------------------------------- Various -----------------------------------

#

# Require explicit names when deleting indices:

#

#action.destructive_requires_name: true

#http.cors.enabled: true

#http.cors.allow-origin: "*"

#index.number_of_shards: 3   #设置es索引分片数量，分布式存储，提高查询效率等

#index.number_of_replicas: 1 #设置分片副本数量，提供数据高可用

#discovery.zen.ping.unicast.hosts: ["10.228.83.120","10.228.83.111","10.228.83.112","10.228.83.66" ]

#action.auto_create_index: true

xpack.monitoring.enabled: true

xpack.monitoring.elasticsearch.collection.enabled: true

xpack.monitoring.collection.indices: my-application #以上三行是开启kibana监控功能，此行为监控集群名为：my-application

1. 从节点配置

从节点的配置跟主节点配置基本一致，就是监控只需在主节点配置就可显示了，node.name也需不同。

# ======================== Elasticsearch Configuration =========================

#

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

#       Before you set out to tweak and tune the configuration, make sure you

#       understand what are you trying to accomplish and the consequences.

#

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

#

# Please consult the documentation for further information on configuration options:

# https://www.elastic.co/guide/en/elasticsearch/reference/index.html

#

# ---------------------------------- Cluster -----------------------------------

#

# Use a descriptive name for your cluster:

#

cluster.name: my-application

#

# ------------------------------------ Node ------------------------------------

#

# Use a descriptive name for the node:

#

node.name: node-3

#

# Add custom attributes to the node:

#

#node.attr.rack: r1

#

# ----------------------------------- Paths ------------------------------------

#

# Path to directory where to store the data (separate multiple locations by comma):

#

path.data: /var/lib/elasticsearch

#

# Path to log files:

#

path.logs: /var/log/elasticsearch

#

# ----------------------------------- Memory -----------------------------------

#

# Lock the memory on startup:

#

#bootstrap.memory_lock: true

#

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

#

# Elasticsearch performs poorly when the system is swapping the memory.

#

# ---------------------------------- Network -----------------------------------

#

# Set the bind address to a specific IP (IPv4 or IPv6):

#

network.host: 0.0.0.0

#

# Set a custom port for HTTP:

#

http.port: 9200

transport.tcp.port: 9300

node.master: false

cluster.initial_master_nodes: ["node-2","node-1"]

node.data: true

#

# For more information, consult the network module documentation.

#

# --------------------------------- Discovery ----------------------------------

#

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

#

discovery.seed_hosts: ["10.228.83.120:9300", "10.228.83.66:9300","10.228.83.112:9300","10.228.83.111:9300"]

#

# Bootstrap the cluster using an initial set of master-eligible nodes:

#

#cluster.initial_master_nodes: ["node-1", "node-2","node-3"]

#

# For more information, consult the discovery and cluster formation module documentation.

#

# ---------------------------------- Gateway -----------------------------------

#

# Block initial recovery after a full cluster restart until N nodes are started:

#

#gateway.recover_after_nodes: 3

#

# For more information, consult the gateway module documentation.

#

# ---------------------------------- Various -----------------------------------

#

# Require explicit names when deleting indices:

#

#action.destructive_requires_name: true

#http.cors.enabled: true

#http.cors.allow-origin: "*"

1. 启动，排错

#启动

systemctl start elasticsearch

#排错

#在无法启动的状态下查看elasticsearch.log日志是否有报错

tail -f /var/log/elasticsearch/elasticsearch.log

#上面看不出什么的情况下可看一下，你集群名为开头的日志

tail -100f /var/log/elasticsearch/my-application.log

1. 处理报错

如果无法启动，查看日志报错 ERROR: bootstrap checks failed max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]，需修改sysctl.conf。

参考连接

sudo vim /etc/sysctl.conf

#末尾添加这一行

vm.max_map_count=262144

#加载配置

sudo sysctl -p

1. 访问测试

elasticsearch已经启动的状态下，我们通过网页访问一下是否可通，如下图，集群状态

http://10.228.83.120:9200/_cat/nodes?v



注意:如果没有接入集群得先看一下配置是否正确，在配置正确的情况下，服务起来的情况下，访问一下 ip+端口,查看cluster_uuid是否为 _na_，如果是那就是启动不对。

我遇到此问题的解决方案是：先停止此服务，然后删除原有的数据文件，重新启动一边即可解决。

redis 安装

我这也是采用的rpm包安装。

yum -y update

yum list redis

yum -y install redis

systemctl start redis

vim  /etc/redis.conf

#requirepass foobared去掉注释，foobared改为自己的密码，我在这里改为

requirepass 123456

#保存退出，重启

systemctl restart redis

下载一个redis客户端，连接一下是否可通。下载连接如下图：

Kibana安装

个人感觉Kibana没什么东西，为了省事就直接用docker部署了，当然你也可以选择二进制或者yum安装。

# 启动

docker run --name kibana -p 5601:5601   -itd kibana:7.5.2

# 配置

docker exec -it kibana bash

cd config

vim kibana.yml

#如下配置

server.name: kibana

server.host: "0"

elasticsearch.hosts: [ "http://10.228.83.120:9200","http://10.228.83.111:9200","http://10.228.83.111:9200","http://10.228.83.66:9200" ]

xpack.monitoring.ui.container.elasticsearch.enabled: true

xpack.monitoring.enabled: true  #以上两个配置也是可以让kibana监控kibana

#i18n.locale: zh-CN #汉化，不是很好用，我就不用了

kibana.index: ".kibana"

# 保存退出

exit

docker restart kibana

通过网页访问，效果如下图：

Filebeat安装

filebeat使用的是二进制包后台运行的，rpm安装也行，不过我在启动的时候显示已经启动了但无log日志输出，而且日志数据传输也有点问题，最后就使用了二进制包安装了。

1. 安装

#解压

tar -xzvf filebeat-7.5.2-linux-x86_64.tar.gz

#进入解压目录

cd filebeat-7.5.2-linux-x86_64

1. 配置

vim filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

#已下配置log的路径，tags可以在logstash做过滤的时候用到，multiline.pattern对Java日志做一个格式化。

- type: log

  enabled: true

  paths:

    - /var/base/out.log

  tags: ["base","pro"]

  multiline.pattern: '^\s*(\d{4}|\d{2})\-(\d{2}|[a-zA-Z]{3})\-(\d{2}|\d{4})'

  multiline.negate: true

  multiline.match: after

  fields:

        type: plm

  fields_under_root: true

- type: log

  enabled: true

  paths:

    - /var/api/out.log

  tags: ["api","pro"]

  multiline.pattern: '^\s*(\d{4}|\d{2})\-(\d{2}|[a-zA-Z]{3})\-(\d{2}|\d{4})'

  multiline.negate: true

  multiline.match: after

  fields:

        type: plm

  fields_under_root: true

#- type: log

  # Change to true to enable this input configuration.

  # Paths that should be crawled and fetched. Glob based paths.

#  paths:

#     - /var/log/audit/audit.log

#  tags: ["auth","test"]

#  fields:

#       type: system

#  fields_under_root: true

    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  #  level: debug

  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

  reload.enabled: false

  # Period on which files under path should be checked for changes

  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:

  index.number_of_shards: 1

  #index.codec: best_compression

  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

#============================== Dashboards =====================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the artifacts.elastic.co

# website.

#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  #host: "localhost:5601"

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

  # Array of hosts to connect to.

#  hosts: ["10.228.83.120:9200"]

  # Optional protocol and basic auth credentials.

  #protocol: "https"

  #username: "elastic"

  #password: "changeme"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:

  - add_host_metadata: ~

  - add_cloud_metadata: ~

  - add_docker_metadata: ~

  - add_kubernetes_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["*"]

#============================== X-Pack Monitoring ===============================

# filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this

# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch

# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.

#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true

#配置输出地址，此处输入到redis。

output.redis:

        hosts: ["10.228.83.120:6379"]

        password: "123456"

        db: "0"

        key: "logstash"

        timeout: 5

        datatype: list

1. 启动

注意：需要进filebeat解压文件夹目录下启动。

#后台启动

nohup ./filebeat &

logstash安装

1. 安装

rpm --install  logstash-7.5.2.rpm

1. 配置

vim /etc/logstash/logstash.yml

#末尾添加一条配置，是可以让kabina监控logstash的

xpack.monitoring.enabled: true

1. 设置过滤规则

过滤规则是在 /etc/logstash/conf.d目录下，在此目录下创建配置文件。

vim /etc/logstash/conf.d/demo.conf

input {

    redis {

        host => "10.228.83.120"

        port => 6379

        password => "123456"

        db => "0"

        data_type => "list"

        key => "logstash"

    }

}

# 以上是消费logstash里的数据

filter {

        grok {

            match => ["message","%{TIMESTAMP_ISO8601:local_time}\s*%{WORD:condition}"]   #grok匹配Java 时间与状态值

    	}

	date {

 		locale => "en"

		match => ["local_time","yyyy-MM-dd HH:mm:ss.SSS"]  #将日志时间覆盖默认  时间戳

 		target =>"@timestamp"

	}

	ruby {

    		code => "event.timestamp.time.localtime"

	}

}

output {

    if [type] == "plm" {

        if [tags][0] == "base" {

            elasticsearch {

                hosts => ["http://10.228.83.120:9200","http://10.228.83.111:9200","http://10.228.83.66:9200","http://10.228.83.112:9200"]

                index => "demo-base-%{+YYYY.MM.dd}"

            }

        }

        else if [tags][0] == "config" {

            elasticsearch {

                hosts => ["http://10.228.83.120:9200","http://10.228.83.111:9200","http://10.228.83.66:9200","http://10.228.83.112:9200"]

                index => "demo-%{[tags][0]}-%{+YYYY.MM.dd}"

            }

        }

     }

}

可通过 /usr/share/logstash/bin/logstash -t -f demo.conf 测试配置文件是否有问题

kibana配置

1. 查看监控
   
   
2. 配置索引

1. 支持命令行删除索引，支持通配符
   
   

还有很多高阶功能我就不一一述说了，EKL安装基本已经至此结束了，此文档是在我实操后一点点整理记录的，如有哪一步配置，启动失效的欢迎大家指出，我可就此纠正。

注意：

以上使用到的端口一定要记得放开，谨记放开端口，或者操作完后 关闭防火墙尝试一下是否可通。