使用 supervisor 管理 Celery 服务

使用 supervisor 管理 Celery 服务

Celery 后台运行

  如果我们想让celery worker运行在后台而不是终端上,在后台以守护进程的方式运行,我们可以使用supervisor进行管理。

celery 的安装

pip install celery

celery的安装很简单,直接通过pip就能够安装,并且本身没有任何的配置,如果需要命令行运行,直接以下命令:

celery -A proj(个人的项目,一般为main文件的相对路径) worker -l info

注意:如果你是win10系统,使用上面命令,可能报错:

Celery ValueError: not enough values to unpack (expected 3, got 0)

请查看这篇文章解决问题

使用通用的 init-script

首先获取 Celery 的 GitHub 仓库 extra/generic-init.d/ 文件夹下的文件。

启动 worker

将上面 GitHub 仓库文件夹下的 celeryd 文件复制到本地的 /etc/init.d/celeryd 文件中并赋予可执行的权限: sudo chmod +x /etc/init.d/celeryd,然后在 /etc/default/ 文件夹下创建 celeryd 配置文件:

# Names of nodes to start
# most people will only start one node:
CELERYD_NODES="worker1"
# but you can also start multiple and configure settings
# for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
# alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10 # Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use
# comment out this line if you don't use an app
CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app" # Where to chdir at start.
CELERYD_CHDIR="/opt/Myproject/" # Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1" # Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG" # %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid" # Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery" # If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

这个配置文件中,你可能需要更改 CELERY_BINCELERY_APPCELERYD_CHDIRCELERYD_USERCELERYD_GROUP

根据你自身项目配置好上述配置后就可以通过 /etc/init.d/celeryd {start|stop|restart|graceful|kill|dryrun|create-paths} 这些命令来控制 Celery worker 进程。

启动 beat

如果你的项目中还有周期任务,那么还得把 Celery beat 启动起来。

把 GitHub 仓库中的 celerybeat 文件复制到 /etc/init.d/celerybeat 文件中,然后赋予可执行权限: sudo chmod +x /etc/init.d/celerybeat

然后同样得有一个 /etc/default/celerybeat 配置文件。不过,如果没有什么特殊配置,可以直接使用 /etc/default/celeryd 配置文件,不必创建 /etc/default/celerybeat 配置文件。

如果你依然要创建的,那么这个文件可能是长这样的:

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use
# comment out this line if you don't use an app
CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app" # Where to chdir at start.
CELERYBEAT_CHDIR="/opt/Myproject/" # Extra arguments to celerybeat
CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule"

完成配置后,同样通过 /etc/init.d/celerybeat {start|stop|restart|create-paths|status} 这些命令控制 Celery beat 进程。

参考文档:
Celery daemon
How to run celery as a daemon?

上一篇:Spring-两种配置容器


下一篇:CSS和JavaScript以及Ajax实现预加载图片的方法及优缺点分析