kafka-manager配置安装及Kerberos（CDH）认证
1、安装包(已编译)
Kafka-manager安装包版本为1.3.3.22, 低版本可能不支持kerberbos 认证。
2、解压安装包及配置
#unzip -d kafka-manager kafka-manager-1.3.3.22.zip
#cd /user/kafka-manager
bin conf lib logs README.md script share
3、认证配置
(1) 修改 conf/application.conf文件中zk的地址以及启用Kafka-Manager使用账号登录和消费者配置
|
##zkhosts kafka-manager.zkhosts="dcdl-test-datanode1.essence.com:2181,dcdl-test-datanode2.essence.com:2181,dcdl-test-namenode1.essence.com:2181/kafka2" #配置连kafka zk kafka-manager.zkhosts=${?ZK_HOSTS} pinned-dispatcher.type="PinnedDispatcher" pinned-dispatcher.executor="thread-pool-executor" application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature"] akka { loggers = ["akka.event.slf4j.Slf4jLogger"] loglevel = "INFO" } akka.logger-startup-timeout = 60s basicAuthentication.enabled=true #开启登录kafka-manager验证 basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED} basicAuthentication.username="admin" basicAuthentication.username=${?KAFKA_MANAGER_USERNAME} basicAuthentication.password="admin" basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD} basicAuthentication.realm="Kafka-Manager" basicAuthentication.excluded=["/api/health"] # ping the health of your instance without authentification
kafka-manager.consumer.properties.file=/user/kafka-manager/conf/consumer.properties #配置消费者文件 #kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE} |
(2) 修改conf/consumer.properties内容如下：
|
security.protocol=SASL_PLAINTEXT key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer sasl.mechanism=GSSAPI sasl.kerberos.service.name=kafka
|
(3) 配置kerberbos 认证文件，创建conf/manager_jaas.conf
|
# 其中Clinet段用于连接zookeeper认证,KafkaClient用于连接kafka服务器认证 Client{ com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/kafka/kafka.keytab" principal="kafka@ESSENCE.COM"; }; KafkaClient{ com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/kafka/kafka.keytab" principal="kafka@ESSENCE.COM"; }; |
4、创建kafka-manager 启动脚本, /user/kafka-manager/script/start.sh 脚本内容如下：
|
#bin/bash echo ‘-----‘$(date +%F%t%T)> manager.out export ZK_HOSTS="dcdl-test-datanode1.essence.com:2181,dcdl-test-datanode2.essence.com:2181, dcdl-test-namenode1.essence.com:2181/kafka2" # kafka-manager HOME MANAGER_HOME=/user/kafka-manager KAFKA_MANAGER=$MANAGER_HOME/bin/kafka-manager APP_HOME=-Dapplication.home=$MANAGER_HOME HTTP_PORT=-Dhttp.port=8090 # SASL JAAS_CONF=-Djava.security.auth.login.config=$MANAGER_HOME/conf/manager_jaas.conf KRB5_CONF=-Djava.security.krb5.conf=$MANAGER_HOME/conf/krb5.conf
nohup $KAFKA_MANAGER $JAAS_CONF $KRB5_CONF $APP_HOME $HTTP_PORT >manager.out 2>&1 &
echo "$!" echo "$!" >mpid tailf manager.out |
5、添加kafka-manager停止脚本，/user/kafka-manager/script/stop.sh 内容如下：
|
echo ‘----------------------------------‘$(date +%F%t%T)> manager.out ps -ef |grep kafka-manager | grep -v grep |awk ‘{print $2}‘| xargs kill rm -rf /user/kafka-manager/RUNNING_PID tailf manager.out |
6、kafka-manager 监控界面介绍; 登录地址：http://10.2.98.128:8090
(1) 添加kafka集群,kerberbos认证参数设置
|
|
(2) CDH环境修改broker_java_opts和mirror_maker_java_opts两个配置项，开放外部访问jmx_port
|
|
把这两项中的-Dcom.sun.management.jmxremote.local.only参数改为false,并且删除掉-Djava.rmi.server.hostname和-Dcom.sun.management.jmxremote.host这两个配置参数，然后重启kafka，jmx即可对外访问
(3) kafka集群添加保存完成后，进入监控界面\
监控kafka topic的offset,
|
|