（三）Spark-Hadoop集群搭建-Java&Python版Spark

2023-08-03 12:37:58

Spark-Hadoop集群搭建

视频教程：

1、优酷

2、YouTube

配置java

启动ftp

[root@master ~]# /etc/init.d/vsftpd restart

关闭 vsftpd： [失败]

为 vsftpd 启动 vsftpd： [确定]

默认情况下 root不允许使用ftp

[root@master vsftpd]# pwd

/etc/vsftpd

[root@master vsftpd]# ls

ftpusers user_list

将这两个文件中的root注释掉。

然后重启ftp

在两台机器上创建文件夹

[root@master ~]# mkdir installer

[root@master ~]#

上传jdk

E:\开发工具

jdk-7u79-linux-i586.rpm

到installer目录

Master机器

[root@master installer]# rpm -ivh jdk-7u79-linux-i586.rpm

Preparing... ########################################### [100%]

1:jdk ########################################### [100%]

Unpacking JAR files...

rt.jar...

jsse.jar...

charsets.jar...

tools.jar...

localedata.jar...

jfxrt.jar...

plugin.jar...

javaws.jar...

Deploy.jar...

Slave机器

[root@slave installer]# rpm -ivh jdk-7u79-linux-i586.rpm

Preparing... ########################################### [100%]

1:jdk ########################################### [100%]

Unpacking JAR files...

rt.jar...

jsse.jar...

charsets.jar...

tools.jar...

localedata.jar...

jfxrt.jar...

plugin.jar...

javaws.jar...

deploy.jar...

添加hadoop用户

[root@master ~]# useradd hadoop

[root@master ~]# passwd hadoop

Changing password for user hadoop.

New UNIX password:

BAD PASSWORD: it is based on a dictionary word

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

[root@slave ~]# useradd hadoop

[root@slave ~]# passwd hadoop

Changing password for user hadoop.

New UNIX password:

BAD PASSWORD: it is based on a dictionary word

Retype new UNIX password:

passwd: all authentication tokens updated successfully.

并且在slave1和slave2使用hadoop用户创建installer目录

[root@master ~]# su - hadoop

[hadoop@master ~]$ mkdir installer

[root@slave ~]# su - hadoop

[hadoop@slave ~]$ mkdir installer

配置ssh等效性

一路回车键

[hadoop@master ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

32:da:01:64:6a:48:4c:f0:58:4f:60:2e:ea:18:a0:5d hadoop@master

[hadoop@slave ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):

Created directory '/home/hadoop/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/hadoop/.ssh/id_rsa.

Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.

The key fingerprint is:

80:49:e7:58:18:0b:9c:1f:ee:d8:96:5a:c5:50:a5:bb hadoop@slave

Master

[hadoop@master ~]$ cd .ssh/

[hadoop@master .ssh]$ ls

id_rsa id_rsa.pub

[hadoop@master .ssh]$ cat id_rsa.pub > authorized_keys

[hadoop@master .ssh]$ scp authorized_keys slave:~/.ssh/

The authenticity of host 'slave (192.168.3.101)' can't be established.

RSA key fingerprint is de:5a:2e:67:62:a4:27:2c:d7:de:85:eb:c7:6c:93:c8.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'slave,192.168.3.101' (RSA) to the list of known hosts.

hadoop@slave's password:

authorized_keys

Slave

[hadoop@slave ~]$ cd .ssh/

[hadoop@slave .ssh]$ ls

authorized_keys id_rsa id_rsa.pub

[hadoop@slave .ssh]$ cat id_rsa.pub >> authorized_keys

[hadoop@slave .ssh]$ scp authorized_keys master:~/.ssh/

The authenticity of host 'master (192.168.3.100)' can't be established.

RSA key fingerprint is de:5a:2e:67:62:a4:27:2c:d7:de:85:eb:c7:6c:93:c8.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'master,192.168.3.100' (RSA) to the list of known hosts.

hadoop@master's password:

authorized_keys

最后

[hadoop@master .ssh]$ chmod 600 authorized_keys

[hadoop@slave .ssh]$ chmod 600 authorized_keys

到这里配置完毕，可以直接使用

[hadoop@master ~]$ ssh slave

[hadoop@slave ~]$ ssh master

配置hadoop

解压缩

[hadoop@master installer]$ tar -zxvf hadoop-2.6.0.tar.gz

[hadoop@master installer]$ mv hadoop-2.6.0 hadoop2

[hadoop@master installer]$ ls

hadoop2 hadoop-2.6.0.tar.gz

配置环境变量

[hadoop@master ~]$ vim .bashrc

# .bashrc

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

# User specific aliases and functions

export JAVA_HOME=/usr/java/jdk1.7.0_79

export HADOOP_HOME=/home/hadoop/installer/hadoop2

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export CLASSPATH=$CLASSPATH:$HADOOP_HOME/lib:$JAVA_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

让bashrc生效

[hadoop@master ~]$ . .bashrc

[hadoop@master ~]$ scp .bashrc slave:~

.bashrc 100% 341 0.3KB/s 00:00

配置hadoop文件

[hadoop@master hadoop]$ vim hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_79

#export HADOOP_HEAPSIZE=

export HADOOP_HEAPSIZE=100

[hadoop@master hadoop]$ vim yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.7.0_79

JAVA_HEAP_MAX=-Xmx300m

YARN_HEAPSIZE=100

[hadoop@master hadoop]$ vim mapred-env.sh

export JAVA_HOME=/usr/java/jdk1.6.0_27

export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=100

[hadoop@master hadoop]$ vim slaves

slave

[hadoop@master hadoop]$ vim core-site.xml

<name>fs.defaultFS</name>

<value>hdfs://master:9000</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/hadoop/tmp</value>

</property>

</configuration>

创建tmp目录

[hadoop@master ~]$ mkdir tmp

[hadoop@slave ~]$ mkdir tmp

[hadoop@master hadoop]$ vim hdfs-site.xml

<name>dfs.namenode.secondary.http-address</name>

<value>master:50090</value>

</property>

<name>dfs.namenode.name.dir</name>

<value>/home/hadoop/data/dfs/name</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/home/hadoop/data/dfs/data</value>

</property>

<name>dfs.replication</name>

</property>

<name>dfs.webhdfs.enabled</name>

</property>

</configuration>

创建目录

[hadoop@master ~]$ mkdir -p data/dfs/data

[hadoop@master ~]$ mkdir -p data/dfs/name

[hadoop@slave ~]$ mkdir -p data/dfs/data

[hadoop@slave ~]$ mkdir -p data/dfs/name

[hadoop@master hadoop]$ cp mapred-site.xml.template mapred-site.xml

[hadoop@master hadoop]$ vim mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

<value>master:10020</value>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>master:19888</value>

</property>

<name>mapreduce.map.memory.mb</name>

</property>

<name>mapreduce.reduce.memory.mb</name>

</property>

<name>yarn.app.mapreduce.am.resource.mb</name>

</property>

</configuration>

[hadoop@master hadoop]$ vim yarn-site.xml

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<description>The hostname of the RM.</description>

<name>yarn.resourcemanager.hostname</name>

<value>master</value>

</property>

<description>The address of the applications manager interface in the RM.</description>

<name>yarn.resourcemanager.address</name>

<value>${yarn.resourcemanager.hostname}:8032</value>

</property>

<description>The address of the scheduler interface.</description>

<name>yarn.resourcemanager.scheduler.address</name>

<value>${yarn.resourcemanager.hostname}:8030</value>

</property>

<description>The http address of the RM web application.</description>

<name>yarn.resourcemanager.webapp.address</name>

<value>${yarn.resourcemanager.hostname}:8088</value>

</property>

<description>The https adddress of the RM web application.</description>

<name>yarn.resourcemanager.webapp.https.address</name>

<value>${yarn.resourcemanager.hostname}:8090</value>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>${yarn.resourcemanager.hostname}:8031</value>

</property>

<description>The address of the RM admin interface.</description>

<name>yarn.resourcemanager.admin.address</name>

<value>${yarn.resourcemanager.hostname}:8033</value>

</property>

<description>The minimum allocation for every container request at the RM,

in MBs. Memory requests lower than this won't take effect,

and the specified value will get allocated at minimum.</description>

<name>yarn.scheduler.minimum-allocation-mb</name>

</property>

<description>The maximum allocation for every container request at the RM,

in MBs. Memory requests higher than this won't take effect,

and will get capped to this value.</description>

<name>yarn.scheduler.maximum-allocation-mb</name>

</property>

<description>Ratio between virtual memory to physical memory when

setting memory limits for containers. Container allocations are

expressed in terms of physical memory, and virtual memory usage

is allowed to exceed this allocation by this ratio.

</description>

<name>yarn.nodemanager.vmem-pmem-ratio</name>

</property>

</configuration>

[hadoop@master installer]$ cd installer/

[hadoop@master installer]$ scp -r hadoop2 slave:~/installer/

Slave机器上执行

[hadoop@slave ~]$ . .bashrc

格式化

[hadoop@master ~]$ hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

16/11/03 19:42:11 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = master/192.168.3.100

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 2.6.0

STARTUP_MSG: classpath = /home/hadoop/installer/hadoop2/etc/hadoop:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/activation-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/junit-4.11.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/htrace-core-3.0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-el-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hadoop-annotations-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hadoop-auth-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-common-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-nfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/common/hadoop-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-registry-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-client-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-api-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.6.0.jar:/home/hadoop/installer/hadoop2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.0.jar:/home/hadoop/installer/hadoop2/contrib/capacity-scheduler/*.jar:/home/hadoop/installer/hadoop2/contrib/capacity-scheduler/*.jar

STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on 2014-11-13T21:10Z

STARTUP_MSG: java = 1.7.0_79

************************************************************/

16/11/03 19:42:11 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]

16/11/03 19:42:11 INFO namenode.NameNode: createNameNode [-format]

16/11/03 19:42:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/11/03 19:42:11 WARN common.Util: Path /home/hadoop/data/dfs/name should be specified as a URI in configuration files. Please update hdfs configuration.

Formatting using clusterid: CID-9c79097a-7f44-48cc-9308-ea9c487a685e

16/11/03 19:42:11 INFO namenode.FSNamesystem: No KeyProvider found.

16/11/03 19:42:11 INFO namenode.FSNamesystem: fsLock is fair:true

16/11/03 19:42:11 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000

16/11/03 19:42:11 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true

16/11/03 19:42:11 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000

16/11/03 19:42:11 INFO blockmanagement.BlockManager: The block deletion will start around 2016 十一月 03 19:42:11

16/11/03 19:42:11 INFO util.GSet: Computing capacity for map BlocksMap

16/11/03 19:42:11 INFO util.GSet: VM type = 32-bit

16/11/03 19:42:11 INFO util.GSet: 2.0% max memory 96.7 MB = 1.9 MB

16/11/03 19:42:11 INFO util.GSet: capacity = 2^19 = 524288 entries

16/11/03 19:42:11 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false

16/11/03 19:42:11 INFO blockmanagement.BlockManager: defaultReplication = 1

16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxReplication = 512

16/11/03 19:42:11 INFO blockmanagement.BlockManager: minReplication = 1

16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxReplicationStreams = 2

16/11/03 19:42:11 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false

16/11/03 19:42:11 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000

16/11/03 19:42:11 INFO blockmanagement.BlockManager: encryptDataTransfer = false

16/11/03 19:42:11 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000

16/11/03 19:42:12 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE)

16/11/03 19:42:12 INFO namenode.FSNamesystem: supergroup = supergroup

16/11/03 19:42:12 INFO namenode.FSNamesystem: isPermissionEnabled = true

16/11/03 19:42:12 INFO namenode.FSNamesystem: HA Enabled: false

16/11/03 19:42:12 INFO namenode.FSNamesystem: Append Enabled: true

16/11/03 19:42:12 INFO util.GSet: Computing capacity for map INodeMap

16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit

16/11/03 19:42:12 INFO util.GSet: 1.0% max memory 96.7 MB = 990.1 KB

16/11/03 19:42:12 INFO util.GSet: capacity = 2^18 = 262144 entries

16/11/03 19:42:12 INFO namenode.NameNode: Caching file names occuring more than 10 times

16/11/03 19:42:12 INFO util.GSet: Computing capacity for map cachedBlocks

16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit

16/11/03 19:42:12 INFO util.GSet: 0.25% max memory 96.7 MB = 247.5 KB

16/11/03 19:42:12 INFO util.GSet: capacity = 2^16 = 65536 entries

16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033

16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0

16/11/03 19:42:12 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000

16/11/03 19:42:12 INFO namenode.FSNamesystem: Retry cache on namenode is enabled

16/11/03 19:42:12 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis

16/11/03 19:42:12 INFO util.GSet: Computing capacity for map NameNodeRetryCache

16/11/03 19:42:12 INFO util.GSet: VM type = 32-bit

16/11/03 19:42:12 INFO util.GSet: 0.029999999329447746% max memory 96.7 MB = 29.7 KB

16/11/03 19:42:12 INFO util.GSet: capacity = 2^13 = 8192 entries

16/11/03 19:42:12 INFO namenode.NNConf: ACLs enabled? false

16/11/03 19:42:12 INFO namenode.NNConf: XAttrs enabled? true

16/11/03 19:42:12 INFO namenode.NNConf: Maximum size of an xattr: 16384

16/11/03 19:42:12 INFO namenode.FSImage: Allocated new BlockPoolId: BP-942661971-192.168.3.100-1478173332298

16/11/03 19:42:12 INFO common.Storage: Storage directory /home/hadoop/data/dfs/name has been successfully formatted.

16/11/03 19:42:12 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

16/11/03 19:42:12 INFO util.ExitUtil: Exiting with status 0

16/11/03 19:42:12 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at master/192.168.3.100

************************************************************/

没有任何错误，格式化成功。

[hadoop@master ~]$ start-dfs.sh

[hadoop@master ~]$ start-yarn.sh

[hadoop@master ~]$ jps

17769 ResourceManager

18020 Jps

17443 NameNode

17631 SecondaryNameNode

[hadoop@slave ~]$ jps

13538 Jps

13297 DataNode

13408 NodeManager

[hadoop@master mapreduce]$ ls

hadoop-mapreduce-client-app-2.6.0.jar hadoop-mapreduce-client-hs-plugins-2.6.0.jar hadoop-mapreduce-examples-2.6.0.jar

hadoop-mapreduce-client-common-2.6.0.jar hadoop-mapreduce-client-jobclient-2.6.0.jar lib

hadoop-mapreduce-client-core-2.6.0.jar hadoop-mapreduce-client-jobclient-2.6.0-tests.jar lib-examples

hadoop-mapreduce-client-hs-2.6.0.jar hadoop-mapreduce-client-shuffle-2.6.0.jar sources

[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /data/ /output

16/11/03 20:01:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/11/03 20:01:45 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.3.100:8032

16/11/03 20:01:49 INFO input.FileInputFormat: Total input paths to process : 2

16/11/03 20:01:49 INFO mapreduce.JobSubmitter: number of splits:2

16/11/03 20:01:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478174305039_0001

16/11/03 20:01:54 INFO impl.YarnClientImpl: Submitted application application_1478174305039_0001

16/11/03 20:01:55 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1478174305039_0001/

16/11/03 20:01:55 INFO mapreduce.Job: Running job: job_1478174305039_0001

16/11/03 20:02:06 INFO mapreduce.Job: Job job_1478174305039_0001 running in uber mode : false

16/11/03 20:02:06 INFO mapreduce.Job: map 0% reduce 0%

16/11/03 20:02:06 INFO mapreduce.Job: Job job_1478174305039_0001 failed with state FAILED due to: Application application_1478174305039_0001 failed 2 times due to AM Container for appattempt_1478174305039_0001_000002 exited with exitCode: -103

For more detailed output, check application tracking page:http://master:8088/proxy/application_1478174305039_0001/Then, click on links to logs of each attempt.

Diagnostics: Container [pid=11781,containerID=container_1478174305039_0001_02_000001] is running beyond virtual memory limits. Current usage: 41.4 MB of 100 MB physical memory used; 1.2 GB of 210.0 MB virtual memory used. Killing container.

Dump of the process-tree for container_1478174305039_0001_02_000001 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

|- 11781 11780 11781 11781 (bash) 0 0 65404928 274 /bin/bash -c /usr/java/jdk1.7.0_79/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001/stdout 2>/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001/stderr

|- 11790 11781 11781 11781 (java) 75 38 1248215040 10317 /usr/java/jdk1.7.0_79/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/home/hadoop/installer/hadoop2/logs/userlogs/application_1478174305039_0001/container_1478174305039_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

Failing this attempt. Failing the application.

16/11/03 20:02:06 INFO mapreduce.Job: Counters: 0

[hadoop@master hadoop2]$ vim ./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

<description>Ratio between virtual memory to physical memory when

setting memory limits for containers. Container allocations are

expressed in terms of physical memory, and virtual memory usage

is allowed to exceed this allocation by this ratio.

</description>

<name>yarn.nodemanager.vmem-pmem-ratio</name>

<value>2.1</value> 这里的2.1是一个系数

</property>

标红错误说物理内存乘以2.1 得到210M，因为一开始我们设置了物理内存是100M

调整方法要么把2.1调整大，要么调整大物理内存

修改之前最好先停掉yarn

[hadoop@master hadoop]$ vim yarn-site.xml

<description>The minimum allocation for every container request at the RM,

in MBs. Memory requests lower than this won't take effect,

and the specified value will get allocated at minimum.</description>

<name>yarn.scheduler.minimum-allocation-mb</name>

</property>

<description>Ratio between virtual memory to physical memory when

setting memory limits for containers. Container allocations are

expressed in terms of physical memory, and virtual memory usage

is allowed to exceed this allocation by this ratio.

</description>

<name>yarn.nodemanager.vmem-pmem-ratio</name>

</property>

[hadoop@master hadoop]$ pwd

/home/hadoop/install/hadoop2/etc/hadoop

[hadoop@master hadoop]$ scp yarn-site.xml slave1:/home/hadoop/hadoop2/etc/hadoop

yarn-site.xml 100% 3165 3.1KB/s 00:00

[hadoop@master hadoop]$ scp yarn-site.xml slave2:/home/hadoop/hadoop2/etc/hadoop

yarn-site.xml 100% 3165 3.1KB/s 00:00

可以不用停止hdfs服务

继续测试

[hadoop@master hadoop]$ start-yarn.sh

[hadoop@master mapreduce]$ jps

16511 Jps

14927 NameNode

15112 SecondaryNameNode

[hadoop@master mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /data/ /output

16/11/03 20:10:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/11/03 20:10:01 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.3.100:8032

16/11/03 20:10:02 INFO input.FileInputFormat: Total input paths to process : 2

16/11/03 20:10:02 INFO mapreduce.JobSubmitter: number of splits:2

16/11/03 20:10:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1478174985535_0001

16/11/03 20:10:03 INFO impl.YarnClientImpl: Submitted application application_1478174985535_0001

16/11/03 20:10:03 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1478174985535_0001/

16/11/03 20:10:03 INFO mapreduce.Job: Running job: job_1478174985535_0001

16/11/03 20:10:13 INFO mapreduce.Job: Job job_1478174985535_0001 running in uber mode : false

16/11/03 20:10:13 INFO mapreduce.Job: map 0% reduce 0%

16/11/03 20:10:54 INFO mapreduce.Job: map 100% reduce 0%

16/11/03 20:11:01 INFO mapreduce.Job: map 100% reduce 100%

16/11/03 20:11:02 INFO mapreduce.Job: Job job_1478174985535_0001 completed successfully

16/11/03 20:11:02 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=62

FILE: Number of bytes written=316503

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=226

HDFS: Number of bytes written=30

HDFS: Number of read operations=9

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=2

Launched reduce tasks=1

Data-local map tasks=2

Total time spent by all maps in occupied slots (ms)=75720

Total time spent by all reduces in occupied slots (ms)=3903

Total time spent by all map tasks (ms)=75720

Total time spent by all reduce tasks (ms)=3903

Total vcore-seconds taken by all map tasks=75720

Total vcore-seconds taken by all reduce tasks=3903

Total megabyte-seconds taken by all map tasks=22716000

Total megabyte-seconds taken by all reduce tasks=1170900

Map-Reduce Framework

Map input records=4

Map output records=7

Map output bytes=66

Map output materialized bytes=68

Input split bytes=188

Combine input records=7

Combine output records=5

Reduce input groups=4

Reduce shuffle bytes=68

Reduce input records=5

Reduce output records=4

Spilled Records=10

Shuffled Maps =2

Failed Shuffles=0

Merged Map outputs=2

GC time elapsed (ms)=1281

CPU time spent (ms)=9240

Physical memory (bytes) snapshot=394346496

Virtual memory (bytes) snapshot=1360240640

Total committed heap usage (bytes)=257433600

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=38

File Output Format Counters

Bytes Written=30

[hadoop@master ~]$ hadoop dfs -cat /output/*

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

16/11/03 20:14:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

123 2

hadoop 2

java 2

spark 1

这样表示hadoop安装成功

码农公寓

Spark-Hadoop集群搭建

视频教程：

配置java

添加hadoop用户

配置ssh等效性

配置hadoop

解压缩

配置环境变量

配置hadoop文件

相关文章