CentOS下SparkR安装部署：hadoop2.7.3+spark2.0.0+scale2.11.8+hive2.1.0

2022-12-29 22:05:52

注：之前本人写了一篇SparkR的安装部署文章：SparkR安装部署及数据分析实例，当时SparkR项目还没正式入主Spark，需要自己下载SparkR安装包，但现在spark已经支持R接口，so更新了这篇文章。

1、Hadoop安装

参考：

http://www.linuxidc.com/Linux/2015-11/124800.htm

http://blog.csdn.net/sa14023053/article/details/51952534

yarn-site.xml

   <property>

     <name>yarn.scheduler.maximum-allocation-mb</name>

     <value>57344</value>

   </property>

   <property>

     <name>yarn.scheduler.minimum-allocation-mb</name>

     <value>2048</value>

   </property>

   <property>

     <name>yarn.nodemanager.resource.memory-mb</name>

     <value>57344</value>

   </property>

   <property>

     <name>yarn.app.mapreduce.am.resource.mb</name>

     <value>2048</value>

   </property>

2、Spark安装

参考：

http://blog.csdn.net/sa14023053/article/details/51953836

vim spark-defaults.conf

 spark.master                     spark://hadoopmaster:7077

 spark.eventLog.enabled           true

 spark.eventLog.dir               hdfs://hadoopmaster:9000/directory

 spark.yarn.historyServer.address hadoopmaster:18080

 spark.serializer                 org.apache.spark.serializer.KryoSerializer

 spark.driver.memory              30g

 spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

vim spark-env.sh

 export SCALA_HOME=/xxx/scala-2.11.8

 export JAVA_HOME=/usr/java/jdk1.8.0_101

 export HADOOP_HOME=/xxx/hadoop-2.7.3

 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

 export SPARK_MASTER_IP=XXXX

 export SPARK_LOCAL_DIRS=/xxx/spark-2.0.0-bin-hadoop2.7

 export SPARK_WORKER_MEMORY=110g

 export SPARK_LOCAL_DIRS=/data/sparkdata/local

 SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true"

 export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://hadoopmaster:9000/directory"

3、Mysql安装

参考：

http://blog.csdn.net/wendi_0506/article/details/39478369

grant all privileges on *.* to hive@localhost identified by 'hive' with grant option;

4、Hive安装

参考：

http://blog.csdn.net/lnho2015/article/details/51307438

http://blog.csdn.net/lnho2015/article/details/51355511

http://blog.csdn.net/blue_jjw/article/details/50479263

hadoop fs -mkdir /tmp

hadoop fs -mkdir -p /user/hive/warehouse

hadoop fs -mkdir -p /user/hive/log

hadoop fs -chmod g+w /user/hive/warehouse

hadoop fs -chmod g+w /user/hive/log

hadoop fs -chmod g+w /tmp

hive-site.xml配置如下：

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

    <name>hive.metastore.warehouse.dir</name>

    <value>/user/hive/warehouse</value>

    <description>location of default database for the warehouse</description>

</property>

<property>

    <name>hive.querylog.location</name>

    <value>/user/hive/log</value>

</property>

<property>

   <name>javax.jdo.option.ConnectionURL</name>

   <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>

   <description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>com.mysql.jdbc.Driver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionUserName</name>

  <value>hive</value>

  <description>username to use against metastore database</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionPassword</name>

  <value>hive</value>

  <description>password to use against metastore database</description>

</property>

</configuration>

hive客户端配置（scp hadoopmaster上的hive安装目录到slaver1，修改hive_site.xml配置）：

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

    <property>

       <name>hive.metastore.uris</name>

       <value>thrift://hadoopmaster:9083</value>

    </property>

</configuration>

5、R安装

5.1 安装相应软件

yum -y groupinstall 'Development Tools'
yum -y install gfortran
yum -y install cmake
yum -y groupinstall "X Window System"
yum install gcc
yum install gcc-c++
yum install gcc-gfortran
yum install bzip2-libs
yum install bzip2-devel
yum -y install xz-devel.x86_64
yum install libcurl-devel.x86_64
yum -y install readline-devel tcl tk libX11-devel libXtst-devel xorg-x11-xtrans-devel libpng-devel libXt-devel

5.2 编译和安装

./configure --prefix /usr/R --enable-R-shlib --with-readline=yes --with-x=yes --with-tcltk=yes --with-cairo=yes --with-libpng=yes --with-jpeglib=yes --with-libtiff=yes --with-aqua=yes --with-ICU=yes --with-libcurl=yes --enable-utf8

make

make install

vim /etc/profile:

export R_HOME=/usr/R/lib64/R

然后：

source /etc/profile

R CMD javareconf

source /usr/R/lib64/R/etc/ldpaths

vim ~/.bash_profile

添加PATH=/usr/R/bin:$PATH

然后source ~/.bash_profile

ok，SparkR集群安装完成！

码农公寓