CentOS系统下Hadoop 2.4.1集群安装配置（简易版）

2023-01-29 22:51:16

安装配置

1、软件下载

JDK下载:jdk-7u65-linux-i586.tar.gz

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

Hadoop下载：hadoop-2.4.1.tar.gz

http://www.apache.org/dyn/closer.cgi/hadoop/common/

2、/etc/hosts配置

  1. 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
  2. ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
  3. <strong><span style="color:#ff0000;">192.168.1.2 Master.Hadoop
  4. 192.168.1.3 Slave1.Hadoop</span></strong>

3、/etc/profile配置

  1. export JAVA_HOME=/usr/java/jrockit-jdk1.6.0_45-R28.2.7-4.1.0
  2. export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
  3. export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
  4. export HADOOP_HOME=/usr/hadoop
  5. export HADOOP_HOME_WARN_SUPPRESS=1
  6. export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

4、~/etc/hadoop/core-site.xml配置

  1. <configuration>
  2. <property>
  3. <name>fs.defaultFS</name>
  4. <value>hdfs://Master.Hadoop:9000</value>
  5. <description>
  6. Where to find the Hadoop Filesystem through the network.
  7. Note 9000 is not the default port.
  8. (This is slightly changed from previous versions which didnt have "hdfs")
  9. </description>
  10. </property>
  11. <property>
  12. <name>hadoop.tmp.dir</name>
  13. <value>/usr/hadoop/tmp</value>
  14. </property>
  15. </configuration>

5、~/etc/hadoop/mapred-site.xml配置

  1. <configuration>
  2. <property>
  3. <name>mapreduce.framework.name</name>
  4. <value>yarn</value>
  5. </property>
  6. </configuration>

6、etc/hadoop/yarn-site.xml配置

  1. <configuration>
  2. <property>
  3. <name>yarn.resourcemanager.scheduler.address</name>
  4. <value>Master.Hadoop:8030</value>
  5. </property>
  6. <property>
  7. <name>yarn.resourcemanager.resource-tracker.address</name>
  8. <value>Master.Hadoop:8031</value>
  9. </property>
  10. <property>
  11. <name>yarn.resourcemanager.address</name>
  12. <value>Master.Hadoop:8032</value>
  13. </property>
  14. <property>
  15. <name>yarn.resourcemanager.admin.address</name>
  16. <value>Master.Hadoop:8033</value>
  17. </property>
  18. <property>
  19. <name>yarn.resourcemanager.webapp.address</name>
  20. <value>Master.Hadoop:8088</value>
  21. </property>
  22. <property>
  23. <name>yarn.resourcemanager.webapp.https.address</name>
  24. <value>Master.Hadoop:8090</value>
  25. </property>
  26. <property>
  27. <name>yarn.nodemanager.local-dirs</name>
  28. <value>${hadoop.tmp.dir}/nodemanager/local</value>
  29. <description>the local directories used by the nodemanager</description>
  30. </property>
  31. <property>
  32. <name>yarn.nodemanager.remote-app-log-dir</name>
  33. <value>${hadoop.tmp.dir}/nodemanager/remote</value>
  34. <description>directory on hdfs where the application logs are moved to </description>
  35. </property>
  36. <property>
  37. <name>yarn.nodemanager.log-dirs</name>
  38. <value>${hadoop.tmp.dir}/nodemanager/logs</value>
  39. <description>the directories used by Nodemanagers as log directories</description>
  40. </property>
  41. <property>
  42. <name>yarn.nodemanager.aux-services</name>
  43. <value>mapreduce_shuffle</value>
  44. <description>shuffle service that needs to be set for Map Reduce to run </description>
  45. </property>
  46. <property>
  47. <name>mapreduce.jobhistory.address</name>
  48. <value>Master.Hadoop:10020</value>
  49. </property>
  50. <property>
  51. <name>mapreduce.jobhistory.webapp.address</name>
  52. <value>Master.Hadoop:19888</value>
  53. </property>
  54. </configuration>

7、~/etc/hadoop/hdfs-site.xml配置

  1. <configuration>
  2. <property>
  3. <name>dfs.permissions.superusergroup</name>
  4. <value>root</value>
  5. </property>
  6. <property>
  7. <name>dfs.replication</name>
  8. <value>1</value>
  9. </property>
  10. </configuration>

启动与验证

1、格式化HDFS文件系统

hadoop namenode -format

2、启动Hadoop

启动前关闭集群中所有机器的防火墙

service iptables stop

启动命令

start-all.sh

3、验证Hadoop

方式一：jps

方式二：hadoop dfsadmin -report

上一篇：Mysql对自增主键ID进行重新排序


下一篇：System.arraycopy 怎么使用的？