kafka_2.11-0.8.2.2.tgz的3节点集群的下载、安装和配置（图文详解）

2021-09-05 15:34:27

kafka_2.10-0.8.1.1.tgz的1或3节点集群的下载、安装和配置（图文详细教程）绝对干货

一、安装前准备
　　1.1 示例机器

二、 JDK7 安装

1.1 下载地址
　　下载地址： http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

[hadoop@hadoop1 ~]$ cd

[hadoop@hadoop1 ~]$ rz

-bash: rz: command not found

[hadoop@hadoop1 ~]$ su root

Password:

[root@hadoop1 hadoop]# yum -y install lrzsz

　　hadoop2和hadoop3操作一样，不多赘述。

　　在安装jdk之前，先要卸载自带的openjdk。

Centos 6.5下的OPENJDK卸载和SUN的JDK安装、环境变量配置

kafka_2.11-0.8.2.2.tgz的3节点集群的下载、安装和配置（图文详解）

[hadoop@hadoop1 ~]$ pwd

/home/hadoop

[hadoop@hadoop1 ~]$ ll

total 0

[hadoop@hadoop1 ~]$ rz

[hadoop@hadoop1 ~]$ ll

total 149920

-rw-r--r--. 1 hadoop hadoop 153512879 Oct 23  2015 jdk-7u79-linux-x64.tar.gz

[hadoop@hadoop1 ~]$

　　hadoop2和hadoop3操作一样，不多赘述。

1.2 安装
　　解压缩

cd /home/hadoop

tar zxvf jdk-7u79-linux-x64.gz

[hadoop@hadoop1 ~]$ pwd

/home/hadoop

[hadoop@hadoop1 ~]$ ll

total 149920

-rw-r--r--. 1 hadoop hadoop 153512879 Oct 23  2015 jdk-7u79-linux-x64.tar.gz

[hadoop@hadoop1 ~]$ tar -zxvf jdk-7u79-linux-x64.tar.gz

　　hadoop2和hadoop3操作一样，不多赘述。

　　建立软连接

ln -s jdk1.7.0_79 jdk

[hadoop@hadoop1 ~]$ ll

total 4

drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

[hadoop@hadoop1 ~]$ ln -s jdk1.7.0_79/ jdk

[hadoop@hadoop1 ~]$ ll

total 4

lrwxrwxrwx. 1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

[hadoop@hadoop1 ~]$

　　hadoop2和hadoop3操作一样，不多赘述。

　　设置环境变量

vim /etc/profile

　　hadoop2和hadoop3操作一样，不多赘述。

　　添加如下：

#jdk
export JAVA_HOME=/home/hadoop/jdk

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

　　hadoop2和hadoop3操作一样，不多赘述。

　　设置环境变量

source /etc/profile

　　hadoop2和hadoop3操作一样，不多赘述。

执行 java -version，如果有版本显示则说明安装成功

三、安装 zookeeper

1、安装 zookeeper

[hadoop@hadoop1 ~]$ pwd

/home/hadoop

[hadoop@hadoop1 ~]$ ll

total 4

lrwxrwxrwx. 1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

[hadoop@hadoop1 ~]$ rz

[hadoop@hadoop1 ~]$ ll

total 27408

lrwxrwxrwx. 1 hadoop hadoop       12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x. 8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79

-rw-r--r--. 1 hadoop hadoop 28060242 Mar 20 10:24 zookeeper-3.4.5-cdh5.5.4.gz

[hadoop@hadoop1 ~]$

　　hadoop2和hadoop3都去操作，这里不多赘述。

　　1.1 安装
　　解压

cd /home/hadoop

tar zxvf zookeeper-3.4.5-cdh5.5.4.gz

[hadoop@hadoop1 ~]$ ll

total 27408

lrwxrwxrwx. 1 hadoop hadoop       12 Apr 22 22:34 jdk -> jdk1.7.0_79/

drwxr-xr-x. 8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79

-rw-r--r--. 1 hadoop hadoop 28060242 Mar 20 10:24 zookeeper-3.4.5-cdh5.5.4.gz

[hadoop@hadoop1 ~]$ tar -zxvf zookeeper-3.4.5-cdh5.5.4.gz

　　 hadoop2和hadoop3都去操作，这里不多赘述。

　　1.2 建立软连接

[hadoop@hadoop1 ~]$ pwd

/home/hadoop

[hadoop@hadoop1 ~]$ ll

total 8

lrwxrwxrwx.  1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

drwxr-xr-x. 14 hadoop hadoop 4096 Apr 26  2016 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$ ln -s zookeeper-3.4.5-cdh5.5.4/ zookeeper

[hadoop@hadoop1 ~]$ ll

total 8

lrwxrwxrwx.  1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

lrwxrwxrwx.  1 hadoop hadoop   25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 14 hadoop hadoop 4096 Apr 26  2016 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$

　　hadoop2和hadoop3都去操作，这里不多赘述。

#zookeeper

export ZOOKEEPER_HOME=/home/hadoop/zookeeper

export  PATH=$PATH:$ZOOKEEPER_HOME/bin

　　hadoop2和hadoop3都去操作，这里不多赘述。

[root@hadoop1 hadoop]# vim /etc/profile

[root@hadoop1 hadoop]# source /etc/profile

　　hadoop2和hadoop3都去操作，这里不多赘述。

　　1.3 修改配置文件

　　复制配置文件

cd /home/hadoop/zookeeper/conf

cp zoo_sample.cfg zoo.cfg

[hadoop@hadoop1 ~]$ ll

total 8

lrwxrwxrwx.  1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

lrwxrwxrwx.  1 hadoop hadoop   25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 14 hadoop hadoop 4096 Apr 26  2016 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$ cd zookeeper

[hadoop@hadoop1 zookeeper]$ cd conf/

[hadoop@hadoop1 conf]$ pwd

/home/hadoop/zookeeper/conf

[hadoop@hadoop1 conf]$ ll

total 12

-rw-rw-r--. 1 hadoop hadoop  535 Apr 26  2016 configuration.xsl

-rw-rw-r--. 1 hadoop hadoop 2693 Apr 26  2016 log4j.properties

-rw-rw-r--. 1 hadoop hadoop  808 Apr 26  2016 zoo_sample.cfg

[hadoop@hadoop1 conf]$ cp zoo_sample.cfg zoo.cfg

[hadoop@hadoop1 conf]$ ll

total 16

-rw-rw-r--. 1 hadoop hadoop  535 Apr 26  2016 configuration.xsl

-rw-rw-r--. 1 hadoop hadoop 2693 Apr 26  2016 log4j.properties

-rw-rw-r--  1 hadoop hadoop  808 Apr 23 08:45 zoo.cfg

-rw-rw-r--. 1 hadoop hadoop  808 Apr 26  2016 zoo_sample.cfg

[hadoop@hadoop1 conf]$

　　hadoop2和hadoop3都去操作，这里不多赘述。

[hadoop@hadoop1 conf]$ pwd

/home/hadoop/zookeeper/conf

[hadoop@hadoop1 conf]$ ll

total 16

-rw-rw-r--. 1 hadoop hadoop  535 Apr 26  2016 configuration.xsl

-rw-rw-r--. 1 hadoop hadoop 2693 Apr 26  2016 log4j.properties

-rw-rw-r--  1 hadoop hadoop  808 Apr 23 08:45 zoo.cfg

-rw-rw-r--. 1 hadoop hadoop  808 Apr 26  2016 zoo_sample.cfg

[hadoop@hadoop1 conf]$ vim zoo.cfg

　　hadoop2和hadoop3都去操作，这里不多赘述。

　　修改参数

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/home/hadoop/zookeeper/data

dataLogDir=/home/hadoop/zookeeper/logs

clientPort=2181

server.1=hadoop1:2888:3888

server.2=hadoop2:2888:3888

server.3=hadoop3:2888:3888

　tickTime 时长单位为毫秒，为 zk 使用的基本时间度量单位。例如， 1 * tickTime 是客户端与 zk 服务端的心跳时间， 2 * tickTime 是客户端会话的超时时间。 tickTime 的默认值为2000 毫秒，更低的 tickTime 值可以更快地发现超时问题，但也会导致更高的网络流量(心跳消息)和更高的 CPU 使用率(会话的跟踪处理）。
　　clientPort zk 服务进程监听的 TCP 端口，默认情况下，服务端会监听 2181 端口。dataDir 无默认配置，必须配置，用于配置存储快照文件的目录。如果没有配置dataLogDir，那么事务日志也会存储在此目录。

　　hadoop2和hadoop3都去操作，这里不多赘述。

创建目录

mkdir -p  /home/hadoop/zookeeper/data

mkdir -p  /home/hadoop/zookeeper/logs

　　hadoop2和hadoop3都去操作，这里不多赘述。

1.3 创建 ID 文件

　　在 dataDir 目录下（即/home/hadoop/zookeeper/data下）添加 myid 文件，并把 server.x 中的 x 数字写入文件中

[hadoop@hadoop1 data]$ pwd

/home/hadoop/zookeeper/data

[hadoop@hadoop1 data]$ ll

total 0

[hadoop@hadoop1 data]$ vim myid

1

　　　hadoop2和hadoop3都去操作，这里不多赘述。

2 启动 zookeeper
　　2.1 启动

cd /home/hadoop/zookeeper/bin

./zkServer.sh start

　　启动 ZK 服务: ./zkServer.sh start
　　查看 ZK 服务状态: ./zkServer.sh status
　　停止 ZK 服务: ./zkServer.sh stop
　　重启 ZK 服务: ./zkServer.sh restart

　　hadoop2和hadoop3都去操作，这里不多赘述。

　　2.1 测试
　　在 hadoop1 机器上的 zookeeper 中创建节点

cd /home/hadoop/zookeeper/bin

./zkCli.sh

create /hello hehe

　　这里不演示

　　在 hadoop2 机器上查看节点

cd /home/hadoop/zookeeper/bin

./zkCli.sh

　　这里不演示

　　get /hello //如果有值则说明 zookeeper 运行正常

2.1 进程的意义

　　这里不多赘述。

四、kafka的安装
　　4.1 安装
　　在 hadoop1、hadoop2和hadoop3 机器上安装

cd /home/hadoop

tar zxvf kafka_2.11-0.8.2.2.tgz

ln -s kafka_2.11-0.8.2.2 kafka

[hadoop@hadoop1 ~]$ ll

total 28

drwxr-xr-x   8 hadoop hadoop 4096 Apr 26  2016 apache-flume-1.6.0-cdh5.5.4-bin

drwxrwxr-x   5 hadoop hadoop 4096 Apr 23 18:34 data

lrwxrwxrwx   1 hadoop hadoop   32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/

lrwxrwxrwx   1 hadoop hadoop   22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/

drwxr-xr-x  18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop   21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/

drwxr-xr-x  27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop   20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/

drwxr-xr-x  10 hadoop hadoop 4096 Apr 26  2016 hive-1.1.0-cdh5.5.4

lrwxrwxrwx.  1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

lrwxrwxrwx.  1 hadoop hadoop   25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$ rz

[hadoop@hadoop1 ~]$ ll

total 15436

drwxr-xr-x   8 hadoop hadoop     4096 Apr 26  2016 apache-flume-1.6.0-cdh5.5.4-bin

drwxrwxr-x   5 hadoop hadoop     4096 Apr 23 18:34 data

lrwxrwxrwx   1 hadoop hadoop       32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/

lrwxrwxrwx   1 hadoop hadoop       22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/

drwxr-xr-x  18 hadoop hadoop     4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop       21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/

drwxr-xr-x  27 hadoop hadoop     4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop       20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/

drwxr-xr-x  10 hadoop hadoop     4096 Apr 26  2016 hive-1.1.0-cdh5.5.4

lrwxrwxrwx.  1 hadoop hadoop       12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79

-rw-r--r--   1 hadoop hadoop 15773865 Mar 20 10:24 kafka_2.11-0.8.2.2.tgz

lrwxrwxrwx.  1 hadoop hadoop       25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 16 hadoop hadoop     4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$

　　hadoop2和hadoop3操作，不多赘述。

[hadoop@hadoop1 ~]$ ll

total 15436

drwxr-xr-x   8 hadoop hadoop     4096 Apr 26  2016 apache-flume-1.6.0-cdh5.5.4-bin

drwxrwxr-x   5 hadoop hadoop     4096 Apr 23 18:34 data

lrwxrwxrwx   1 hadoop hadoop       32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/

lrwxrwxrwx   1 hadoop hadoop       22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/

drwxr-xr-x  18 hadoop hadoop     4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop       21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/

drwxr-xr-x  27 hadoop hadoop     4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop       20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/

drwxr-xr-x  10 hadoop hadoop     4096 Apr 26  2016 hive-1.1.0-cdh5.5.4

lrwxrwxrwx.  1 hadoop hadoop       12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop     4096 Apr 11  2015 jdk1.7.0_79

-rw-r--r--   1 hadoop hadoop 15773865 Mar 20 10:24 kafka_2.11-0.8.2.2.tgz

lrwxrwxrwx.  1 hadoop hadoop       25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 16 hadoop hadoop     4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$ tar -zxvf kafka_2.11-0.8.2.2.tgz

　　hadoop2和hadoop3操作，不多赘述。

[hadoop@hadoop1 ~]$ ll

total 32

drwxr-xr-x   8 hadoop hadoop 4096 Apr 26  2016 apache-flume-1.6.0-cdh5.5.4-bin

drwxrwxr-x   5 hadoop hadoop 4096 Apr 23 18:34 data

lrwxrwxrwx   1 hadoop hadoop   32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/

lrwxrwxrwx   1 hadoop hadoop   22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/

drwxr-xr-x  18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop   21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/

drwxr-xr-x  27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop   20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/

drwxr-xr-x  10 hadoop hadoop 4096 Apr 26  2016 hive-1.1.0-cdh5.5.4

lrwxrwxrwx.  1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

drwxr-xr-x   5 hadoop hadoop 4096 Sep  3  2015 kafka_2.11-0.8.2.2

lrwxrwxrwx.  1 hadoop hadoop   25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$ ln -s kafka_2.11-0.8.2.2/ kafka

[hadoop@hadoop1 ~]$ ll

total 32

drwxr-xr-x   8 hadoop hadoop 4096 Apr 26  2016 apache-flume-1.6.0-cdh5.5.4-bin

drwxrwxr-x   5 hadoop hadoop 4096 Apr 23 18:34 data

lrwxrwxrwx   1 hadoop hadoop   32 Apr 23 18:14 flume -> apache-flume-1.6.0-cdh5.5.4-bin/

lrwxrwxrwx   1 hadoop hadoop   22 Apr 23 09:08 hadoop -> hadoop-2.6.0-cdh5.5.4/

drwxr-xr-x  18 hadoop hadoop 4096 Apr 23 09:50 hadoop-2.6.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop   21 Apr 23 16:55 hbase -> hbase-1.0.0-cdh5.5.4/

drwxr-xr-x  27 hadoop hadoop 4096 Apr 23 17:27 hbase-1.0.0-cdh5.5.4

lrwxrwxrwx   1 hadoop hadoop   20 Apr 23 15:27 hive -> hive-1.1.0-cdh5.5.4/

drwxr-xr-x  10 hadoop hadoop 4096 Apr 26  2016 hive-1.1.0-cdh5.5.4

lrwxrwxrwx.  1 hadoop hadoop   12 Apr 22 22:27 jdk -> jdk1.7.0_79/

drwxr-xr-x.  8 hadoop hadoop 4096 Apr 11  2015 jdk1.7.0_79

lrwxrwxrwx   1 hadoop hadoop   19 Apr 23 18:41 kafka -> kafka_2.11-0.8.2.2/

drwxr-xr-x   5 hadoop hadoop 4096 Sep  3  2015 kafka_2.11-0.8.2.2

lrwxrwxrwx.  1 hadoop hadoop   25 Apr 22 22:49 zookeeper -> zookeeper-3.4.5-cdh5.5.4/

drwxr-xr-x. 16 hadoop hadoop 4096 Apr 23 08:57 zookeeper-3.4.5-cdh5.5.4

[hadoop@hadoop1 ~]$

　　hadoop2和hadoop3操作，不多赘述。

　　环境变量

[hadoop@hadoop1 ~]$ su root

Password:

[root@hadoop1 hadoop]# vim /etc/profile

　　hadoop2和hadoop3操作，不多赘述。

#kafka

export KAFKA_HOME=/home/hadoop/kafka

export PATH=$PATH:$KAFKA_HOME/bin

　　hadoop2和hadoop3操作，不多赘述。

[hadoop@hadoop1 ~]$ su root

Password:

[root@hadoop1 hadoop]# vim /etc/profile

[root@hadoop1 hadoop]# source /etc/profile

[root@hadoop1 hadoop]#

　　hadoop2和hadoop3操作，不多赘述。

　　4.1 修改配置文件
　　1. hadoop1 上修改 config/server.properties

export HBASE_MANAGES_ZK=false

broker.id=1

port=9092

host.name=hadoop1

log.dirs=/home/kafka-logs

num.partitions=5

log.cleaner.enable=false

offsets.storage=kafka

dual.commit.enabled=true
delete.topic.enable=true
zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181

2. hadoop2 上修改 config/server.properties

export HBASE_MANAGES_ZK=false

broker.id=2

port=9092

host.name=hadoop2

log.dirs=/home/kafka-logs

num.partitions=5

log.cleaner.enable=false

offsets.storage=kafka

dual.commit.enabled=true

delete.topic.enable=true

zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181

3. hadoop3 上修改 config/server.properties

export HBASE_MANAGES_ZK=false

broker.id=3

port=9092

host.name=hadoop3

log.dirs=/home/kafka-logs

num.partitions=5

log.cleaner.enable=false

offsets.storage=kafka

dual.commit.enabled=true

delete.topic.enable=true

zookeeper.connect=hadoop1:2181,hadoop2:2181,hadoop3:2181

创建目录

mkdir -p   /home/kafka-logs

　　hadoop1、hadoop2和hadoop3都执行。

[hadoop@hadoop1 kafka]$ mkdir -p /home/kafka-logs

mkdir: cannot create directory `/home/kafka-logs': Permission denied

[hadoop@hadoop1 kafka]$ su root

Password:

[root@hadoop1 kafka]# mkdir -p /home/kafka-logs

[root@hadoop1 kafka]# chown -R hadoop:hadoop /home/kafka-logs

[root@hadoop1 kafka]# cd /home/

[root@hadoop1 home]# ll

total 8

drwx------. 26 hadoop hadoop 4096 Apr 23 21:18 hadoop

drwxr-xr-x   2 hadoop hadoop 4096 Apr 23 21:18 kafka-logs

[root@hadoop1 home]#

　　注意：在CDH版本的，发行者已经帮我们解决了这个问题。Apache版本，需要解决

首先解决kafka Unrecognized VM option ‘UseCompressedOops’问题

vi /home/hadoop/kafka/bin/kafka-run-class.sh

if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then

  KAFKA_JVM_PERFORMANCE_OPTS="-server  -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"

fi

去掉-XX:+UseCompressedOops即可

[hadoop@hadoop1 bin]$ pwd

/home/hadoop/kafka/bin

[hadoop@hadoop1 bin]$ ll

total 80

-rwxr-xr-x 1 hadoop hadoop  943 Sep  3  2015 kafka-console-consumer.sh

-rwxr-xr-x 1 hadoop hadoop  942 Sep  3  2015 kafka-console-producer.sh

-rwxr-xr-x 1 hadoop hadoop  870 Sep  3  2015 kafka-consumer-offset-checker.sh

-rwxr-xr-x 1 hadoop hadoop  946 Sep  3  2015 kafka-consumer-perf-test.sh

-rwxr-xr-x 1 hadoop hadoop  860 Sep  3  2015 kafka-mirror-maker.sh

-rwxr-xr-x 1 hadoop hadoop  884 Sep  3  2015 kafka-preferred-replica-election.sh

-rwxr-xr-x 1 hadoop hadoop  946 Sep  3  2015 kafka-producer-perf-test.sh

-rwxr-xr-x 1 hadoop hadoop  872 Sep  3  2015 kafka-reassign-partitions.sh

-rwxr-xr-x 1 hadoop hadoop  866 Sep  3  2015 kafka-replay-log-producer.sh

-rwxr-xr-x 1 hadoop hadoop  872 Sep  3  2015 kafka-replica-verification.sh

-rwxr-xr-x 1 hadoop hadoop 4185 Sep  3  2015 kafka-run-class.sh

-rwxr-xr-x 1 hadoop hadoop 1333 Sep  3  2015 kafka-server-start.sh

-rwxr-xr-x 1 hadoop hadoop  891 Sep  3  2015 kafka-server-stop.sh

-rwxr-xr-x 1 hadoop hadoop  868 Sep  3  2015 kafka-simple-consumer-shell.sh

-rwxr-xr-x 1 hadoop hadoop  861 Sep  3  2015 kafka-topics.sh

drwxr-xr-x 2 hadoop hadoop 4096 Sep  3  2015 windows

-rwxr-xr-x 1 hadoop hadoop 1370 Sep  3  2015 zookeeper-server-start.sh

-rwxr-xr-x 1 hadoop hadoop  875 Sep  3  2015 zookeeper-server-stop.sh

-rwxr-xr-x 1 hadoop hadoop  968 Sep  3  2015 zookeeper-shell.sh

[hadoop@hadoop1 bin]$ vim kafka-run-class.sh

　　这里，不需自己去解决。

　　在三台机器上的kafka目录下，分别执行以下命令

nohup bin/kafka-server-start.sh config/server.properties &

[hadoop@hadoop1 kafka]$ pwd

/home/hadoop/kafka

[hadoop@hadoop1 kafka]$ nohup bin/kafka-server-start.sh config/server.properties &

[1] 15091

[hadoop@hadoop1 kafka]$ nohup: ignoring input and appending output to `nohup.out'

[1]+  Exit 1                  nohup bin/kafka-server-start.sh config/server.properties

[hadoop@hadoop1 kafka]$

　　出现以下提示后回车即可。

查看状态

cat nohup.out

[2017-04-23 21:23:35,304] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,304] INFO Client environment:java.home=/home/hadoop/jdk1.7.0_79/jre (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,304] INFO Client environment:java.class.path=.:/home/hadoop/jdk/lib/dt.jar:/home/hadoop/jdk/lib/tools.jar:/home/hadoop/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/home/hadoop/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/hadoop/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/hadoop/kafka/bin/../libs/jopt-simple-3.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-javadoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-scaladoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-sources.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-test.jar:/home/hadoop/kafka/bin/../libs/kafka-clients-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/log4j-1.2.16.jar:/home/hadoop/kafka/bin/../libs/lz4-1.2.0.jar:/home/hadoop/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/hadoop/kafka/bin/../libs/scala-library-2.11.5.jar:/home/hadoop/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/scala-xml_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/hadoop/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/hadoop/kafka/bin/../libs/snappy-java-1.1.1.7.jar:/home/hadoop/kafka/bin/../libs/zkclient-0.3.jar:/home/hadoop/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/hadoop/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,304] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,304] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,304] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,305] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,305] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,305] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,305] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,305] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,305] INFO Client environment:user.dir=/home/hadoop/kafka_2.11-0.8.2.2 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,308] INFO Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@31dbfb7a (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:23:35,391] INFO Opening socket connection to server hadoop1/192.168.80.121:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:23:35,410] INFO Socket connection established to hadoop1/192.168.80.121:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:23:35,530] INFO Session establishment complete on server hadoop1/192.168.80.121:2181, sessionid = 0x15b99fea1e10012, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:23:35,542] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)

[2017-04-23 21:23:36,009] INFO Loading logs. (kafka.log.LogManager)

[2017-04-23 21:23:36,069] INFO Logs loading complete. (kafka.log.LogManager)

[2017-04-23 21:23:36,072] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)

[2017-04-23 21:23:36,110] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)

[2017-04-23 21:23:36,333] INFO Awaiting socket connections on hadoop1:9092. (kafka.network.Acceptor)

[2017-04-23 21:23:36,340] INFO [Socket Server on Broker 1], Started (kafka.network.SocketServer)

[hadoop@hadoop1 kafka]$

r:/home/hadoop/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/hadoop/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/hadoop/kafka/bin/../libs/jopt-simple-3.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-javadoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-scaladoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-sources.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-test.jar:/home/hadoop/kafka/bin/../libs/kafka-clients-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/log4j-1.2.16.jar:/home/hadoop/kafka/bin/../libs/lz4-1.2.0.jar:/home/hadoop/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/hadoop/kafka/bin/../libs/scala-library-2.11.5.jar:/home/hadoop/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/scala-xml_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/hadoop/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/hadoop/kafka/bin/../libs/snappy-java-1.1.1.7.jar:/home/hadoop/kafka/bin/../libs/zkclient-0.3.jar:/home/hadoop/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/hadoop/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,857] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,857] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,857] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,858] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,858] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,858] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,858] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,858] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,858] INFO Client environment:user.dir=/home/hadoop/kafka_2.11-0.8.2.2 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:34,862] INFO Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@3764253e (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:25:35,713] INFO Opening socket connection to server hadoop3/192.168.80.123:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:25:35,811] INFO Socket connection established to hadoop3/192.168.80.123:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:25:35,884] INFO Session establishment complete on server hadoop3/192.168.80.123:2181, sessionid = 0x35b99ff49bf0012, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:25:35,890] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)

[2017-04-23 21:25:37,146] INFO Loading logs. (kafka.log.LogManager)

[2017-04-23 21:25:37,243] INFO Logs loading complete. (kafka.log.LogManager)

[2017-04-23 21:25:37,245] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)

[2017-04-23 21:25:37,300] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)

[2017-04-23 21:25:37,798] INFO Awaiting socket connections on hadoop2:9092. (kafka.network.Acceptor)

[2017-04-23 21:25:37,803] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)

[2017-04-23 21:25:39,744] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2017-04-23 21:25:42,084] INFO Registered broker 2 at path /brokers/ids/2 with address hadoop2:9092. (kafka.utils.ZkUtils$)

[2017-04-23 21:25:42,520] INFO [Kafka Server 2], started (kafka.server.KafkaServer)

[hadoop@hadoop2 kafka]$

r:/home/hadoop/kafka/bin/../examples/build/libs//kafka-examples*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/hadoop/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/hadoop/kafka/bin/../clients/build/libs/kafka-clients*.jar:/home/hadoop/kafka/bin/../libs/jopt-simple-3.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-javadoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-scaladoc.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-sources.jar:/home/hadoop/kafka/bin/../libs/kafka_2.11-0.8.2.2-test.jar:/home/hadoop/kafka/bin/../libs/kafka-clients-0.8.2.2.jar:/home/hadoop/kafka/bin/../libs/log4j-1.2.16.jar:/home/hadoop/kafka/bin/../libs/lz4-1.2.0.jar:/home/hadoop/kafka/bin/../libs/metrics-core-2.2.0.jar:/home/hadoop/kafka/bin/../libs/scala-library-2.11.5.jar:/home/hadoop/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/scala-xml_2.11-1.0.2.jar:/home/hadoop/kafka/bin/../libs/slf4j-api-1.7.6.jar:/home/hadoop/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/home/hadoop/kafka/bin/../libs/snappy-java-1.1.1.7.jar:/home/hadoop/kafka/bin/../libs/zkclient-0.3.jar:/home/hadoop/kafka/bin/../libs/zookeeper-3.4.6.jar:/home/hadoop/kafka/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,239] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,240] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,240] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,241] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,241] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,241] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,241] INFO Client environment:user.name=hadoop (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,241] INFO Client environment:user.home=/home/hadoop (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,241] INFO Client environment:user.dir=/home/hadoop/kafka_2.11-0.8.2.2 (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:10,244] INFO Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@3764253e (org.apache.zookeeper.ZooKeeper)

[2017-04-23 21:26:11,052] INFO Opening socket connection to server hadoop3/192.168.80.123:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:26:11,080] INFO Socket connection established to hadoop3/192.168.80.123:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:26:11,370] INFO Session establishment complete on server hadoop3/192.168.80.123:2181, sessionid = 0x35b99ff49bf0013, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)

[2017-04-23 21:26:11,505] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)

[2017-04-23 21:26:12,561] INFO Loading logs. (kafka.log.LogManager)

[2017-04-23 21:26:12,698] INFO Logs loading complete. (kafka.log.LogManager)

[2017-04-23 21:26:12,702] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)

[2017-04-23 21:26:12,918] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)

[2017-04-23 21:26:14,277] INFO Awaiting socket connections on hadoop3:9092. (kafka.network.Acceptor)

[2017-04-23 21:26:14,623] INFO [Socket Server on Broker 3], Started (kafka.network.SocketServer)

[2017-04-23 21:26:17,519] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2017-04-23 21:26:19,961] INFO Registered broker 3 at path /brokers/ids/3 with address hadoop3:9092. (kafka.utils.ZkUtils$)

[2017-04-23 21:26:20,172] INFO [Kafka Server 3], started (kafka.server.KafkaServer)

[hadoop@hadoop3 kafka]$

　　至此，kafka_2.11-0.8.2.2.tgz的3节点集群的搭建完成！

扩展补充

kafka的server.properties配置文件参考示范（图文详解）（多种方式）

码农公寓

kafka_2.10-0.8.1.1.tgz的1或3节点集群的下载、安装和配置（图文详细教程）绝对干货

Centos 6.5下的OPENJDK卸载和SUN的JDK安装、环境变量配置

kafka的server.properties配置文件参考示范（图文详解）（多种方式）

相关文章