flink部署操作-flink standalone集群安装部署

2024-02-15 13:33:10

 

flink集群安装部署

 

standalone集群模式

 

  • 必须依赖
  1. 必须的软件
  2. JAVA_HOME配置
  • flink安装
  1. 配置flink
  2. 启动flink
  3. 添加Jobmanager/taskmanager 实例到集群
  • 个人真实环境实践安装步骤

 

必须依赖

必须的软件

flink运行在所有类unix环境中，例如：linux、mac、或者cygwin，并且集群由一个master节点和一个或者多个worker节点。在你开始安装系统之前，确保你有在每个节点上安装以下软件。

 

  • java 1.8.x或者更高
  • ssh

如果你的集群没有这些软件，你需要安装或者升级他们。注意：一般linux服务器上都有ssh，但是java是需要自己安装的。

在集群的所有节点上需要配置SSH免密码登录。

JAVA_HOME配置

flink需要在集群的所有节点(master节点和worker节点)配置JAVA_HOME，指向安装在机器上的java。

你可以在这个文件中进行配置：conf/flink-conf.yaml  通过env.java.home这个key。

 

flink安装

去下载页面随时下载安装包。确保选择flink安装包匹配到你的hadoop版本。如果你不打算使用hadoop的话，可以选择任意版本。

下载最新版本之后，把安装包上传到你的master节点，然后解压：

  1.   tar xzf flink-*.tgz
  2.   cd flink-*

配置flink

解压之后，需要修改conf/flink-conf.yaml

设置jobmanager.rpc.address的值为master节点的ip或者主机名。你也可以定义每个节点上允许jvm申请的最大内存，使用jobmanager.heap.mb和taskmanager.heap.mb

这两个参数的值的单位都是MB，如果有一些节点想要分配更多的内存，可以通过覆盖这个参数的默认值 FLINK_TM_HEAP

最后，你需要提供一个节点列表作为worker节点。因为，类似于HDFS配置，修改文件conf/slaves 然后在里面输入每一个worker节点的ip/hostname 。每一个worker节点将运行一个taskmanager程序。

下面的例子说明了三个节点的配置：(ip地址从10.0.0.1到10.0.0.3 对应的主机名 master worker1 worker2)并显示配置文件的内容(需要访问所有机器的相同路径)

flink部署操作-flink standalone集群安装部署

  1.   vi /path/to/flink/conf/flink-conf.yaml
  2.    
  3.   jobmanager.rpc.address: 10.0.0.1
  4.    
  5.    
  6.   vi /path/to/flink/conf/slaves
  7.    
  8.   10.0.0.2
  9.   10.0.0.3

flink目录必须在每一个worker节点的相同路劲。你可以使用一个共享的NFS目录，或者拷贝整个flink目录到每一个worker节点。

有关配置的详细信息，请参见详细的配置页面进行查看。

下面这几个参数的配置值非常重要。

 

  • Jobmanager可用内存(jobmanager.heap.mb)
  • taskmanager可用内存(taskmanager.heap.mb)
  • 每个机器可用cpu数量(taskmanager.numberOfTaskSlots)
  • 集群中的总cpu数量(parallelism.default)
  • 节点临时目录(taskmanager.tmp.dirs)

启动flink

下面的脚本将会在本机启动一个jobmanager节点，然后通过SSH连接到slaves文件中的所有worker节点，在worker节点上面启动taskmanager。现在flink启动并且运行。在本地运行的jobmanager现在将会通过配置的RPC端口接收任务。

确认你在master节点并且进入flink目录：

bin/start-cluster.sh

停止flink，需要使用stop-cluster.sh脚本

添加jobmanager或者taskmanager实例到集群

你可以通过bin/jobmanager.sh脚本和bin/taskmanager.sh脚本向一个运行中的集群添加jobmanager和taskmanager。

添加jobmanager

bin/jobmanager.sh ((start|start-foreground) cluster)|stop|stop-all

 

添加taskmanager

bin/taskmanager.sh start|start-foreground|stop|stop-all

 

 

 

 

个人真实环境实践安装步骤

以上的内容来源于官网文档翻译

下面的内容来自于本人在真实环境的一个安装步骤：

集群环境规划：三台机器，一主两从

  1.   hadoop100 jobManager
  2.   hadoop101 taskManager
  3.   hadoop102 taskManager
  4.    
  5.   注意：
  6.   1：这几台节点之间需要互相配置好SSH免密码登录。(至少要配置hadoop100可以免密码登录hadoop101和hadoop102)
  7.   2：这几台节点需要安装jdk1.8及以上，并且在/etc/profile中配置环境变量JAVA_HOME
  8.   例如：
  9.   export JAVA_HOME=/usr/local/jdk
  10.   export PATH=.:$JAVA_HOME/bin:$PATH

1：上传flink安装包到hadoop100节点的/usr/local目录下，然后解压

  1.   cd /usr/local
  2.   tar -zxvf flink-1.4.1-bin-hadoop27-scala_2.11.tgz

2：修改hadoop100节点上的flink的配置文件

  1.   cd /usr/local/flink-1.4.1/conf
  2.   vi flink-conf.yaml
  3.   # 修改此参数的值，改为主节点的主机名
  4.   jobmanager.rpc.address: hadoop100
  5.    
  6.    
  7.   vi slaves
  8.   hadoop101
  9.   hadoop102

3：把修改好配置文件的flink目录拷贝到其他两个节点

  1.   scp -rq /usr/local/flink-1.4.1 hadoop101:/usr/local
  2.   scp -rq /usr/local/flink-1.4.1 hadoop102:/usr/local

4：在hadoop100节点启动集群

  1.   cd /usr/local/flink-1.4.1
  2.   bin/start-cluster.sh

执行上面命令以后正常将会看到以下日志输出：

  1.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  2.   Starting cluster.
  3.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  4.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  5.   Starting jobmanager daemon on host hadoop100.
  6.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  7.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  8.   Starting taskmanager daemon on host hadoop101.
  9.   Starting taskmanager daemon on host hadoop102.

5：验证集群启动情况

查看进程：

  1.   在hadoop100节点上执行jps，可以看到：
  2.   3785 JobManager
  3.    
  4.   在hadoop101节点上执行jps，可以看到：
  5.   2534 TaskManager
  6.    
  7.   在hadoop101节点上执行jps，可以看到：
  8.   2402 TaskManager
  9.    
  10.   能看到对应的jobmanager和taskmanager进程即可。

如果启动失败了，请查看对应的日志：

  1.   cd /usr/local/flink-1.4.1/log
  2.    
  3.   针对jobmanager节点：
  4.   more flink-root-jobmanager-0-hadoop100.log
  5.    
  6.   针对taskmanager节点：
  7.   more flink-root-taskmanager-0-hadoop101.log
  8.   more flink-root-taskmanager-0-hadoop102.log
  9.    
  10.   查看此日志文件中是否有异常日志信息

6：访问集群web界面

http://hadoop100:8081

flink部署操作-flink standalone集群安装部署

 

7：停止集群

 

  1.   在hadoop100节点上执行下面命令
  2.   cd /usr/local/flink-1.4.1
  3.   bin/stop-cluster.sh

执行停止命令之后将会看到下面日志输出：

  1.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  2.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  3.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  4.   Stopping taskmanager daemon (pid: 3321) on host hadoop101.
  5.   Stopping taskmanager daemon (pid: 3088) on host hadoop102.
  6.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  7.   Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  8.   Stopping jobmanager daemon (pid: 5341) on host hadoop100.

再去对应的节点上执行jps进程发现对应的jobmanager和taskmanager进程都没有了。

 

 

上一篇：转载一篇makefile，说的很详细


下一篇：Hadoop中standalone模式环境搭建配置