使用 supervisor 管理 Celery 服务
Celery 后台运行
如果我们想让celery worker运行在后台而不是终端上,在后台以守护进程的方式运行,我们可以使用supervisor进行管理。
celery 的安装
pip install celery
celery的安装很简单,直接通过pip就能够安装,并且本身没有任何的配置,如果需要命令行运行,直接以下命令:
celery -A proj(个人的项目,一般为main文件的相对路径) worker -l info
注意:如果你是win10系统,使用上面命令,可能报错:
Celery ValueError: not enough values to unpack (expected 3, got 0)
请查看这篇文章解决问题
使用通用的 init-script
首先获取 Celery 的 GitHub 仓库 extra/generic-init.d/
文件夹下的文件。
启动 worker
将上面 GitHub 仓库文件夹下的 celeryd
文件复制到本地的 /etc/init.d/celeryd
文件中并赋予可执行的权限: sudo chmod +x /etc/init.d/celeryd
,然后在 /etc/default/
文件夹下创建 celeryd
配置文件:
# Names of nodes to start
# most people will only start one node:
CELERYD_NODES="worker1"
# but you can also start multiple and configure settings
# for each in CELERYD_OPTS
#CELERYD_NODES="worker1 worker2 worker3"
# alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10 # Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use
# comment out this line if you don't use an app
CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app" # Where to chdir at start.
CELERYD_CHDIR="/opt/Myproject/" # Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# Configure node-specific settings by appending node name to arguments:
#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1" # Set logging level to DEBUG
#CELERYD_LOG_LEVEL="DEBUG" # %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid" # Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# a user/group combination that already exists (e.g., nobody).
CELERYD_USER="celery"
CELERYD_GROUP="celery" # If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
这个配置文件中,你可能需要更改 CELERY_BIN
, CELERY_APP
, CELERYD_CHDIR
, CELERYD_USER
, CELERYD_GROUP
。
根据你自身项目配置好上述配置后就可以通过 /etc/init.d/celeryd {start|stop|restart|graceful|kill|dryrun|create-paths}
这些命令来控制 Celery worker 进程。
启动 beat
如果你的项目中还有周期任务,那么还得把 Celery beat 启动起来。
把 GitHub 仓库中的 celerybeat
文件复制到 /etc/init.d/celerybeat
文件中,然后赋予可执行权限: sudo chmod +x /etc/init.d/celerybeat
。
然后同样得有一个 /etc/default/celerybeat
配置文件。不过,如果没有什么特殊配置,可以直接使用 /etc/default/celeryd
配置文件,不必创建 /etc/default/celerybeat
配置文件。
如果你依然要创建的,那么这个文件可能是长这样的:
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery" # App instance to use
# comment out this line if you don't use an app
CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app" # Where to chdir at start.
CELERYBEAT_CHDIR="/opt/Myproject/" # Extra arguments to celerybeat
CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule"
完成配置后,同样通过 /etc/init.d/celerybeat {start|stop|restart|create-paths|status}
这些命令控制 Celery beat 进程。