1、将安装文件kafka_2.11-2.4.1.tgz拷贝至/opt/software，并解压
#查看文件 [bigdata@bigdata01 software]$ ll | grep kafka_2.11-2.4.1.tgz -rw-r--r--. 1 bigdata bigdata 70159813 Aug 15 18:46 kafka_2.11-2.4.1.tgz #解压到/opt/module/ [bigdata@bigdata01 software]$ tar -zxvf kafka_2.11-2.4.1.tgz -C /opt/module/
2、配置环境变量
[bigdata@bigdata01 kafka_2.11-2.4.1]$ sudo vim /etc/profile.d/my_env.sh
增加以下内容
#KAFKA_HOME export KAFKA_HOME=/opt/module/kafka_2.11-2.4.1/ export PATH=$PATH:$KAFKA_HOME/bin
生效
[bigdata@bigdata01 kafka_2.11-2.4.1]$ source /etc/profile.d/my_env.sh
3、新建datas目录
[bigdata@bigdata01 kafka_2.11-2.4.1]$ mkdir datas [bigdata@bigdata01 kafka_2.11-2.4.1]$ ll total 56 drwxr-xr-x. 3 bigdata bigdata 4096 Mar 3 2020 bin drwxr-xr-x. 2 bigdata bigdata 4096 Aug 27 19:21 config drwxrwxr-x. 2 bigdata bigdata 6 Aug 27 19:21 datas drwxr-xr-x. 2 bigdata bigdata 8192 Aug 27 19:01 libs -rw-r--r--. 1 bigdata bigdata 32216 Mar 3 2020 LICENSE -rw-r--r--. 1 bigdata bigdata 337 Mar 3 2020 NOTICE drwxr-xr-x. 2 bigdata bigdata 44 Mar 3 2020 site-docs
4、修改配置文件
#进入config目录 [bigdata@bigdata01 kafka_2.11-2.4.1]$ cd /opt/module/kafka_2.11-2.4.1/config/ [bigdata@bigdata01 config]$ ll total 72 -rw-r--r--. 1 bigdata bigdata 906 Mar 3 2020 connect-console-sink.properties -rw-r--r--. 1 bigdata bigdata 909 Mar 3 2020 connect-console-source.properties -rw-r--r--. 1 bigdata bigdata 5321 Mar 3 2020 connect-distributed.properties -rw-r--r--. 1 bigdata bigdata 883 Mar 3 2020 connect-file-sink.properties -rw-r--r--. 1 bigdata bigdata 881 Mar 3 2020 connect-file-source.properties -rw-r--r--. 1 bigdata bigdata 2247 Mar 3 2020 connect-log4j.properties -rw-r--r--. 1 bigdata bigdata 1539 Mar 3 2020 connect-mirror-maker.properties -rw-r--r--. 1 bigdata bigdata 2262 Mar 3 2020 connect-standalone.properties -rw-r--r--. 1 bigdata bigdata 1221 Mar 3 2020 consumer.properties -rw-r--r--. 1 bigdata bigdata 4675 Mar 3 2020 log4j.properties -rw-r--r--. 1 bigdata bigdata 1925 Mar 3 2020 producer.properties -rw-r--r--. 1 bigdata bigdata 6848 Mar 3 2020 server.properties -rw-r--r--. 1 bigdata bigdata 1032 Mar 3 2020 tools-log4j.properties -rw-r--r--. 1 bigdata bigdata 1169 Mar 3 2020 trogdor.conf -rw-r--r--. 1 bigdata bigdata 1205 Mar 3 2020 zookeeper.properties
修改server.properties
[bigdata@bigdata01 config]$ vim server.properties
4.1、修改broker.id=0，全局唯一id，集群中不允许重复（bigdata01为0，bigdata02为1，bigdata03为2，如有其他服务器依次递增）
4.2、记住kafka端口号，后续要用（listeners=PLAINTEXT://:9092，说明是9092）
4.3、修改kafka消息目录（kafka消息即为log）
log.dirs=/opt/module/kafka_2.11-2.4.1/datas/
4.4、修改zookeeper连接
zookeeper.connect=bigdata01:2181,bigdata02:2181,bigdata03:2181
5、将kafka分发到其他机器
[bigdata@bigdata01 module]$ pwd /opt/module [bigdata@bigdata01 module]$ /home/bigdata/xsync kafka_2.11-2.4.1/
6、修改bigdata02、bigdata03机器broker.id
[bigdata@bigdata02 ~]$ cd /opt/module/kafka_2.11-2.4.1/config/ [bigdata@bigdata02 config]$ vim server.properties
bigdata02修改broker.id=1
[bigdata@bigdata03 ~]$ cd /opt/module/kafka_2.11-2.4.1/config/ [bigdata@bigdata03 config]$ vim server.properties
bigdata03修改broker.id=2
7、分发环境变量
[bigdata@bigdata01 module]$ scp /etc/profile.d/my_env.sh root@bigdata02:/etc/profile.d/ root@bigdata02's password: my_env.sh [bigdata@bigdata01 module]$ scp /etc/profile.d/my_env.sh root@bigdata03:/etc/profile.d/ root@bigdata03's password: my_env.sh
环境变量生效
bigdata02
[bigdata@bigdata02 config]$ source /etc/profile.d/my_env.sh
bigdata03
[bigdata@bigdata03 config]$ source /etc/profile.d/my_env.sh
8、启动kafka
需先启动zookeeper
8.1启动zookeeper
cd到/opt/module/zookeeper/bin
[bigdata@bigdata01 bin]$ zkCluster.sh start ============bigdata01================== ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED ============bigdata02================== ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED ============bigdata03================== ZooKeeper JMX enabled by default Using config: /opt/module/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED [bigdata@bigdata01 bin]$ jpscall.sh ===================bigdata01===================== 44656 NameNode 49297 RunJar 45080 NodeManager 58570 Jps 49115 RunJar 44780 DataNode 58524 QuorumPeerMain ===================bigdata02===================== 28196 QuorumPeerMain 28246 Jps 20168 ResourceManager 19993 DataNode 20298 NodeManager ===================bigdata03===================== 44784 Jps 44742 QuorumPeerMain 30599 SecondaryNameNode 30490 DataNode 30683 NodeManager
8.2、启动kafka
[bigdata@bigdata01 bin]$ cd /opt/module/kafka_2.11-2.4.1/bin [bigdata@bigdata01 bin]$ kafka-server-start.sh -daemon /opt/module/kafka_2.11-2.4.1/config/server.properties
依次在bigdata01、bigdata02、bigdata03机器执行上述启动命令。
9、查看kafka集群进程
#显示有kafka进程 [bigdata@bigdata01 bin]$ jpscall.sh ===================bigdata01===================== 44656 NameNode 49297 RunJar 45080 NodeManager 59000 Jps 49115 RunJar 44780 DataNode 58524 QuorumPeerMain 58924 Kafka ===================bigdata02===================== 28672 Jps 28196 QuorumPeerMain 20168 ResourceManager 19993 DataNode 20298 NodeManager 28604 Kafka ===================bigdata03===================== 44742 QuorumPeerMain 30599 SecondaryNameNode 30490 DataNode 45162 Kafka 30683 NodeManager 45228 Jps
10、关闭kafka集群
依次在bigdata01、bigdata02、bigdata03机器执行以下命令
[bigdata@bigdata01 bin]$ kafka-server-stop.sh stop
11、集群启动停止脚本
#在家目录新建bin目录，并将脚本放置到此bin目录下 [bigdata@bigdata01 bin]$ cd ~ [bigdata@bigdata01 bin]$ vim kafka.sh
添加脚本内容
#!/bin/bash
if [ $# -lt 1 ]
then
echo "USAGE:kafka.sh {start|stop}"
exit
fi
case $1 in
start)
for i in bigdata01 bigdata02 bigdata03
do
echo "===================start $i kafka==================="
ssh $i /opt/module/kafka_2.11-2.4.1/bin/kafka-server-start.sh -daemon /opt/module/kafka_2.11-2.4.1/config/server.properties
done
;;
stop)
for i in bigdata01 bigdata02 bigdata03
do
echo "===================stop $i kafka==================="
ssh $i /opt/module/kafka_2.11-2.4.1/bin/kafka-server-stop.sh
done
;;
*)
echo "USAE:kafka.sh {start|stop}"
exit
;;
esac
增加执行权限
[bigdata@bigdata01 bin]$ ll total 4 -rw-rw-r--. 1 bigdata bigdata 573 Aug 27 20:22 kafka.sh [bigdata@bigdata01 bin]$ chmod u+x kafka.sh [bigdata@bigdata01 bin]$ ll total 4 -rwxrw-r--. 1 bigdata bigdata 573 Aug 27 20:22 kafka.sh
示例：停止kafka集群
[bigdata@bigdata01 bin]$ kafka.sh stop ===================stop bigdata01 kafka=================== ===================stop bigdata02 kafka=================== ===================stop bigdata03 kafka===================
示例：启动kafka集群
[bigdata@bigdata01 bin]$ kafka.sh start ===================start bigdata01 kafka=================== ===================start bigdata02 kafka=================== ===================start bigdata03 kafka=================== [bigdata@bigdata01 bin]$ jpscall.sh ===================bigdata01===================== 44656 NameNode 49297 RunJar 59910 Jps 45080 NodeManager 49115 RunJar 44780 DataNode 58524 QuorumPeerMain 59836 Kafka ===================bigdata02===================== 28196 QuorumPeerMain 20168 ResourceManager 19993 DataNode 20298 NodeManager 29146 Kafka 29212 Jps ===================bigdata03===================== 45777 Kafka 45843 Jps 44742 QuorumPeerMain 30599 SecondaryNameNode 30490 DataNode 30683 NodeManager
完成。