1. 主机规划
|
主机名称 |
IP地址 |
操作系统 |
部署软件 |
运行进程 |
备注 |
|
mini01 |
172.16.1.11【内网】 10.0.0.11 【外网】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7【主】 |
QuorumPeerMain、 |
|
|
mini02 |
172.16.1.12【内网】 10.0.0.12 【外网】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0 |
QuorumPeerMain、 |
|
|
mini03 |
172.16.1.13【内网】 10.0.0.13 【外网】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、kafka_2.11-2.0.0、spark-2.4.0-hadoop2.7 |
QuorumPeerMain、 |
|
|
mini04 |
172.16.1.14【内网】 10.0.0.14 【外网】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、spark-2.4.0-hadoop2.7 |
QuorumPeerMain、 |
|
|
mini05 |
172.16.1.15【内网】 10.0.0.15 【外网】 |
CentOS 7.5 |
Jdk-8、zookeeper-3.4.5、Hadoop2.7.6、hbase-2.0.2、spark-2.4.0-hadoop2.7 |
QuorumPeerMain、 |
说明
该Spark集群安装，但是有一个很大的问题，那就是Master节点存在单点故障，要解决此问题，就要借助zookeeper，并且启动至少两个Master节点来实现高可靠。具体部署下节讲解。
2. 免密码登录
实现mini01到mini02、mini03、mini04、mini05通过秘钥免密码登录。
参见文章：Hadoop2.7.6_01_部署
3. Jdk【java8】
参见文章：Hadoop2.7.6_01_部署
4. Spark部署步骤
4.1. Spark安装
[yun@mini01 software]$ pwd
/app/software
[yun@mini01 software]$ ll
total
-rw-r--r-- yun yun Nov : spark-2.4.-bin-hadoop2..tgz
[yun@mini01 software]$ tar xf spark-2.4.-bin-hadoop2..tgz
[yun@mini01 software]$ mv spark-2.4.-bin-hadoop2. /app/
[yun@mini01 software]$ cd /app/
[yun@mini01 ~]$ ln -s spark-2.4.-bin-hadoop2./ spark
[yun@mini01 ~]$ ll -d spark-*
drwxr-xr-x yun yun Oct : spark-2.4.-bin-hadoop2.
lrwxrwxrwx yun yun Nov : spark -> spark-2.4.-bin-hadoop2./
4.2. 环境变量修改
根据规划，该环境变量的修改包括mini01、mini03、mini04、mini05。
# 需要root权限去添加环境变量
[root@mini01 ~]# tail /etc/profile
………………
# spark环境变量
export SPARK_HOME="/app/spark"
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH [root@mini01 ~]# logout
[yun@mini01 conf]$ source /etc/profile # 重新加载该环境变量
4.3. 配置修改
[yun@mini01 conf]$ pwd
/app/spark/conf
[yun@mini01 conf]$ cp -a spark-env.sh.template spark-env.sh
[yun@mini01 conf]$ tail spark-env.sh # 修改环境变量配置
# Options for native BLAS, like Intel MKL, OpenBLAS, and so on.
# You might get better performance to enable these options if using native BLAS (see SPARK-).
# - MKL_NUM_THREADS= Disable multi-threading of Intel MKL
# - OPENBLAS_NUM_THREADS= Disable multi-threading of OpenBLAS # 添加配置如下
# 配置JAVA_HOME
export JAVA_HOME=/app/jdk
# 设置Master的主机名
export SPARK_MASTER_IP=mini01
# 每一个Worker最多可以使用的内存，我的虚拟机就2g
# 真实服务器如果有128G，你可以设置为100G
# 所以这里设置为1024m或1g
export SPARK_WORKER_MEMORY=1024m
# 每一个Worker最多可以使用的cpu core的个数，我虚拟机就一个...
# 真实服务器如果有32个，你可以设置为32个
export SPARK_WORKER_CORES=
# 提交Application的端口，默认就是这个，万一要改呢，改这里
export SPARK_MASTER_PORT= [yun@mini01 conf]$ pwd
/app/spark/conf
[yun@mini01 conf]$ cp -a slaves.template slaves
[yun@mini01 conf]$ tail slaves # 修改slaves 配置
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# # A Spark Worker will be started on each of the machines listed below.
mini03
mini04
mini05
4.4. 分发到其他机器
分发到mini03、mini04和mini05
[yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini03:/app # 拷贝到mini03
[yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini04:/app # 拷贝到mini04
[yun@mini01 ~]$ scp -pr spark-2.4.-bin-hadoop2./ yun@mini05:/app # 拷贝到mini05
在mini03、mini04和mini05上操作
[yun@mini04 ~]$ pwd
/app
[yun@mini04 ~]$ ll -d spark-2.4.-bin-hadoop2.
drwxr-xr-x yun yun Oct : spark-2.4.-bin-hadoop2.
[yun@mini04 ~]$ ln -s spark-2.4.-bin-hadoop2./ spark
[yun@mini04 ~]$ ll -d spark-*
drwxr-xr-x yun yun Oct : spark-2.4.-bin-hadoop2.
lrwxrwxrwx yun yun Nov : spark -> spark-2.4.-bin-hadoop2./
4.5. 启动spark
在mini01上操作
[yun@mini01 sbin]$ pwd
/app/spark/sbin
[yun@mini01 sbin]$ ./start-all.sh # 关闭使用 stop-all.sh 脚本
starting org.apache.spark.deploy.master.Master, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.master.Master--mini01.out
mini03: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker--mini03.out
mini05: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker--mini05.out
mini04: starting org.apache.spark.deploy.worker.Worker, logging to /app/spark/logs/spark-yun-org.apache.spark.deploy.worker.Worker--mini04.out
[yun@mini01 ~]$
[yun@mini01 ~]$ jps # 查看进程状态
Master
Jps
mini03进程查看
[yun@mini03 ~]$ jps
Worker
Jps
mini04进程查看
[yun@mini04 ~]$ jps
Jps
Worker
mini05进程查看
[yun@mini05 ~]$ jps
Worker
Jps
4.6. 浏览器访问
http://mini01:8080/