使用supervisor 管理logstash

2023-12-31 22:02:52

使用supervisor 管理logstash

supervisor和logstash此处不做介绍，直接进行操作配置

环境准备

需要创建一个普通用户，如jianxp以下均使用该用户部署服务，设用户目录为 /home/jianxp，安装目录为 /opt

准备部署包 jdk/supervisor/logstash 上传至服务器，默认在 /tools 目录下，解压到/opt目录下。

JDK安装

  • 若 已安装 jdk1.8.0，则只需要建立已有JDK的软连接，可跳过后面的步骤：

    mkdir -p /opt/java
ln -sf ${JAVA_HOME} /opt/java/jdk

  • 解压安装包：

    tar zxf jdk-8u191-linux-x64.tar.gz -C /opt
ln -s /opt/jdk1.8.0_191 /opt/jdk

  • 添加环境变量：

    编辑文件 vi ~/.bash_profile，追加如下环境变量：

    export JAVA_HOME=/opt/jdk
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

  • 执行命令 source ~/.bash_profile 使环境变量生效。

  • 执行命令 java -version，若输出 java version "1.8.0_191"，则说明jdk安装成功

Supervisor安装

  • 离线安装supervisor，解压安装包 tar -zxf /tools/supervisor-3.3.4.tar -C /opt/，依次执行下列步骤

    cd /opt/supervisor-3.3.4
tar -zxf setuptools-32.3.1.tar.gz && cd setuptools-32.3.1
python setup.py install --user
cd .. && tar -zxf meld3-1.0.2.tar.gz && cd meld3-1.0.2
python setup.py install --user
cd .. && tar -zxf supervisor-3.3.4.tar.gz && cd supervisor-3.3.4
python setup.py install --user

  • 编辑supervisor配置文件

    mkdir -p /opt/supervisor && cd /opt/supervisor && vi supervisord.conf

    supervisord.conf 文件内容如下

    [unix_http_server]
file=/opt/supervisor/supervisor.sock
; (the path to the socket file)
chmod=0700
; sockef file mode (default 0700)

[supervisord]
logfile=/opt/supervisor/supervisord.log
;  (main log file;default $CWD/supervisord.log)
pidfile=/opt/supervisor/supervisord.pid
; (supervisord pidfile;default supervisord.pid)
childlogdir=/opt/supervisor
; ('AUTO' child log dir, default $TEMP)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections

[rpcinterface:supervisor]
supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///opt/supervisor/supervisor.sock
; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /opt/supervisor/conf.d/*.conf

  • 创建配置文件里涉及到的目录

    • 日志目录

    mkdir -p /opt/supervisor && mkdir -p /opt/supervisor

    • 管理进程的配置文件目录：

    mkdir -p /opt/supervisor/conf.d

  • 建立软连接

    mkdir -p ~/.local/etc
ln -s /opt/supervisor/supervisord.conf ~/.local/etc/

  • 至此，supervisor安装完毕

    • 执行 supervisod 即可启动supervisor
    • 执行 supervisorctl status，无报错则说明启动成功。

    logstash 安装

解压安装包，添加软连接

tar -zxf /tools/logstash-6.8.9-fix.tar.gz -C /opt/
ln -s /opt/logstash-6.8.9 /opt/logstash

配置logstash用supervisor管理（logstash的所有配置均在supervisor管理的配置文件中完成）

编辑配置文件，执行 cd /opt/supervisor/conf.d && vi logstash.conf

; logstash config file

[program:logstash]
command=/opt/logstash-6.8.9/bin/logstash 
directory=/opt/logstash-6.8.9/
user=jianxp
numprocs=1
stdout_logfile=/opt/logstash/logs/stderr.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=1
redirect_stderr=true
autostart=true
autorestart=true
startsecs=1
stopwaitsecs=1
killasgroup=true
priority=1

说明
command=/opt/logstash-6.8.9/bin/logstash   这个没有加-f等参数，默认读config/pipelines.yaml文件，此文件也是后续要配置的。

在/opt/logstash-6.8.9/config/conf.d 创建两个config文件，作为logstash的pipeline配置

vim config/conf.d/test.conf
input{
file{
    path =>"/opt/logstash_test/test.log"
    start_position=>"beginning" 
}
}
output { 
file {
    path =>"/opt/logstash_test/output.log"
} 
}

vim config/conf.d/file.conf
input{
file{
    path =>"/opt/logstash_test/test.log"
    start_position=>"beginning"

}
}

output { 
file {
    path =>"/opt/logstash_test/output33.log"
} 
}

上面两个pipeline时比较简单的，以文件输入和和文件输出为例子，两个pipeline都是以文件test.log为共同输入，然后输出到不同的文件中。

 编辑config/pipelines.yaml 文件，添加如下内容

 - pipeline.id: file
   pipeline.workers: 1
   pipeline.batch.size: 1
   path.config: "/opt/logstash-6.8.9/config/conf.d/file.conf"

 - pipeline.id: test
   pipeline.workers: 1
   pipeline.batch.size: 1
   path.config: "/opt/logstash-6.8.9/config/conf.d/test.conf"

上面的logstash的pipeline配置完成后，并且supervisor目录下的logstash.conf也配置完成，既可以使用supervisor启动logstash

supervisorctl update
supervisorctl status

在/opt/logstash-6.8.9/logs 目录下，可以查看logstash输出的日志，通过日志可以看到，Pipelines running count=2，表示启动两个pipeline。

可以开启另外一个shell窗口，向test.log文件中增加内容，log中也会输出Opening file 等信息。

[2021-03-10T16:18:04,953][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"test", :thread=>"#<Thread:0x24bd4c10 run>"}
[2021-03-10T16:18:04,958][INFO ][logstash.runner          ] Logstash shut down.
[2021-03-10T16:18:42,012][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.9"}
[2021-03-10T16:18:47,360][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"file", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}
[2021-03-10T16:18:47,721][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}
[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}
[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}
[2021-03-10T16:18:47,992][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"file", :thread=>"#<Thread:0x5c0e2cd6 run>"}
[2021-03-10T16:18:47,993][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"test", :thread=>"#<Thread:0x6626055f run>"}
[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-10T16:18:48,108][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:file, :test], :non_running_pipelines=>[]}
[2021-03-10T16:18:48,531][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-03-10T16:19:11,774][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash_test/output.log"}
[2021-03-10T16:19:11,776][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash_test/output33.log"}

查看输出的文件

[jianxp@localhost logstash_test]$ ll
总用量 16
-rw-rw-r--. 1 jianxp jianxp  335 3月  10 15:23 aa
-rw-r--r--. 1 jianxp jianxp 3338 3月  10 16:19 output33.log
-rw-r--r--. 1 jianxp jianxp 3338 3月  10 16:19 output.log
-rw-rw-r--. 1 jianxp jianxp 1340 3月  10 16:19 test.log
上一篇：Virtual Box 6.1 安装 Ubuntu 20.04 LTS


下一篇：Unity 2019 LTS正式推出