airflow----安装

Airflow配置安装

1.安装前准备工作

安装版本说明

安装工具 版本 用途
Python 3.6.5 安装airflow及其依赖包、开发airflow的dag使用
MySQL 5.7 作为airflow的元数据库
Airflow 1.10.10 任务调度平台

2.安装Python3

#python依赖
yum -y install zlib zlib-devel
yum -y install bzip2 bzip2-devel
yum -y install ncurses ncurses-devel
yum -y install readline readline-devel
yum -y install openssl openssl-devel
yum -y install openssl-static
yum -y install xz lzma xz-devel
yum -y install sqlite sqlite-devel
yum -y install gdbm gdbm-devel
yum -y install tk tk-devel
yum install gcc

#安装wget命令
yum -y install wget
#使用wget下载Python源码压缩包到/root目录下
wget -P /root https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
#在当前目录解压Python源码压缩包
tar -zxvf Python-3.6.5.tgz
#进入解压后的文件目录下
cd /root/Python-3.6.5
#检测及校验平台
./configure --with-ssl --prefix=/service/python3
#编译Python源代码
make
#安装Python
make install
#备份原来的Python软连接
mv /usr/bin/python /usr/bin/python2.backup
#制作新的指向Python3的软连接
ln -s /service/python3/bin/python3 /usr/bin/python
#建立pip的软连接
ln -s /service/python3//bin/pip3 /usr/bin/pip
#查看Python版本
python -V
#检测pip是否可用
pip
#升级pip
pip install --upgrade pip
#获取yum命令所在位置
whereis yum
#yum: /usr/bin/yum /etc/yum /etc/yum.conf /usr/share/man/man8/yum.8
#编辑yum文件
vi /usr/bin/yum /etc/yum /etc/yum.conf /usr/share/man/man8/yum.8
#进入编辑模式
i
#修改第一行内容(看系统版本,centos7对应2.7,centos6对应2.6)
#修改前:
#!/usr/bin/python
#修改后:
#!/usr/bin/python2.7
#退出编辑模式
esc
#保存文件
wq
#按上述方式编辑以下文件,修改第一行内容
/usr/libexec/urlgrabber-ext-down

3.安装MySQL

#卸载mariadb
rpm -qa | grep mariadb
rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64
#sudo rpm -e --nodeps mariadb-libs-5.5.52-1.el7.x86_64
rpm -qa | grep mariadb
#下载mysql的repo源
wget -P /root http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
#通过rpm安装
rpm -ivh mysql-community-release-el7-5.noarch.rpm
#安装mysql
yum install mysql-server
#授权
chown -R mysql:mysql /var/lib/mysql
#开启Mysql服务
service mysqld start
#用root用户连接登录mysql:
mysql -uroot
#重置mysql密码
use mysql;
update user set password=password(‘root‘) where user=‘root‘;
flush privileges;
#为Airflow建库、建用户
#建库:
create database airflow;
#建用户:
create user ‘airflow‘@‘%‘ identified by ‘airflow‘;
create user ‘airflow‘@‘localhost‘ identified by ‘airflow‘;
#为用户授权:
grant all on airflow.* to ‘airflow‘@‘%‘;
grant all on airflow.* to ‘root‘@‘%‘;
flush privileges;
exit;

4.安装Airflow

#设置临时环境变量
export SLUGIFY_USES_TEXT_UNIDECODE=yes
#添加编辑环境变量
vi /etc/profile
#在最后添加以下内容:
----》
export PS1="[\u@\h \w]\$ "
#Python环境变量
export PYTHON_HOME=/service/python3
export PATH=$PATH:$PYTHON_HOME/bin
#Airflow环境变量
export AIRFLOW_HOME=/root/airflow
export SITE_AIRFLOW_HOME=/service/python3/lib/python3.6/site-packages/airflow
export PATH=$PATH:$SITE_AIRFLOW_HOME/bin
----》
#生效环境变量
source /etc/profile
#安装apache-airflow并且指定1.10.15版本
pip install apache-airflow===1.10.15

airflow会被安装到python3下的site-packages目录下,完整目录为:

${PYTHON_HOME}/lib/python3.6/site-packages/airflow
#绝对路径/service/python3/lib/python3.6/site-packages/airflow

 

安装两个包,否则会报错

pip install Flask-SQLAlchemy==2.4.4
pip install SQLAlchemy==1.3.23 



执行airflow命令做初始化操作(肯定会报错)

airflow
####
[2019-07-17 04:40:01,565] {__init__.py:51} INFO - Using executor SequentialExecutor
usage: airflow [-h]
               {backfill,list_tasks,clear,pause,unpause,trigger_dag,delete_dag,pool,variables,kerberos,render,run,initdb,list_dags,dag_state,task_failed_deps,task_state,serve_logs,test,webserver,resetdb,upgradedb,scheduler,worker,flower,version,connections,create_user}
               ...
airflow: error: the following arguments are required: subcommand
####
#到此,airflow会在刚刚的AIRFLOW_HOME目录下生成一些文件。当然,执行该命令时可能会报一些错误,可以不用理会!
#报错如下:
[2019-07-17 04:40:01,565] {__init__.py:51} INFO - Using executor SequentialExecutor
usage: airflow [-h]
               {backfill,list_tasks,clear,pause,unpause,trigger_dag,delete_dag,pool,variables,kerberos,render,run,initdb,list_dags,dag_state,task_failed_deps,task_state,serve_logs,test,webserver,resetdb,upgradedb,scheduler,worker,flower,version,connections,create_user}
               ...
airflow: error: the following arguments are required: subcommand
#生成的文件logs如下所示:
[root@test01 ~]$ cd airflow/
[root@test01 ~/airflow]$ ll
total 28
-rw-r--r--. 1 root root 20572 Jul 17 
上一篇:msys2 git status显示中文文件名问题


下一篇:【leetcode】61: 旋转链表