4.安装 Logstash （本地及 docker)

2021-11-27 23:31:19

1.环境准备

JVM运行环境

Logstash依赖JVM运行环境，本文以Java 8版本进行介绍，支持以下JVM版本：8、11、15。

2.Logstash的下载和安装

linux:

> curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-7.10.0-linux-x86_64.tar.gz
> tar xzvf logstash-7.10.0-linux-x86_64.tar.gz

APT

# 下载安装公钥
> wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
> sudo apt-get install apt-transport-https
# 保存仓库地址到本地
> echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
# 安装Logstash
> sudo apt-get update && sudo apt-get install logstash

YUM

# 下载安装公钥
> sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# 新建文件/etc/yum.repos.d/logstash.repo，并插入以下内容
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

# 安装logstash
> sudo yum install logstash

mac and Homebrew

# 安装Elastic Homebrew仓库
> brew tap elastic/tap
# 安装Logstash
> brew install elastic/tap/logstash-full
# 通过Homebrew设置开机启动logstash服务
> brew services start elastic/tap/logstash-full
# 重启主机后，启动Logstash
> logstash

3.启动Logstash

本文采用tar包安装方式进行阐述。

进入Logstash安装目录
最简配置启动Logstash

# 通过控制台输入输出收集数据
> bin/logstash -e 'input { stdin { } } output { stdout {} }'
# 在控制台中输入 "Hello world!"，然后会看到控制台输出"Hello world!"
hello world
2013-11-21T01:22:14.405+0000 0.0.0.0 hello world

4.收集数据

Logstash包含3个主要部分：输入(inputs)，过滤器(filters)和输出(outputs)。下面以采集log4j日志并输出到ElasticSearch为例进行阐述。

1.创建收集数据的配置文件 bin/log4j2es.conf，插入以下内容

input {
  file {
  # 要采集的log文件路径
    path => "/data/logs/springboot.log"
  }
}
 
filter {
}
 
output {
  stdout {
    codec => rubydebug
  }
 
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

2.指定配置启动Logstash

> bin/logstash -f bin/log4j2es.conf
# 或者后台启动
> nohup bin/logstash -f bin/log4j2es.conf >/dev/null 2>&1 &

3.查看收集到ElasticSearch索引的数据

> curl http://localhost:9200/_cat/indices

默认Logstash生成以logstash开头带有日期的索引

green open logstash-2021.04.09-000001          3UhrpKMlRRCsJ7e5BRzHpA 1 1    0   0    208b    208b

查看索引中的数据

> curl -XPOST 'http://localhost:9200/logstash-2021.04.09-000001/_search'  -H 'Content-Type: application/json' -d '{"query":{"match_all":{}}}'

返回如下结果；

{
    "took": 1,
    "timed_out": false,
    "hits": {
        "hits": [
            {
                "_index": "logstash-2021.04.09-000001",
                "_type": "_doc",
                "_id": "aTL3UHkBSh9MyZ_E_yVB",
                "_score": 1.0,
                "_source": {
                    "host": "elastichost",
                    "path": "/data/logs/springboot.log",
                    "message": "2021-04-09 17:58:47.172  INFO 23556 --- [  restartedMain] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed3.",
                    "@version": "1",
                    "tags": [
                        "_grokparsefailure"
                    ],
                    "@timestamp": "2021-04-09T11:51:40.390Z"
                }
            }
        ]
    }
}

5.docker方式安装

拉取镜像

docker pull docker.elastic.co/logstash/logstash:7.10.0

docker模式运行Logstash，

# 1.参考tar包中logstah/config文件夹下所有配置拷贝一份放在宿主机 /usr/share/logstash/config/
# 2.修改pipeline.yml，增加以下配置
pipeline.id: main
path.config: /usr/share/logstash/config/log4j2es.conf
# -v挂载Logstash的配置/usr/share/logstash/config/到docker的路径~/settings/中
docker run --rm -it -v ~/settings/:/usr/share/logstash/config/ docker.elastic.co/logstash/logstash:7.10.0

码农公寓