前言
kafka集群扩容后,新的broker上面不会数据进入这些节点,也就是说,这些节点是空闲的;它只有在创建新的topic时才会参与工作。除非将已有的partition迁移到新的服务器上面;所以需要将一些topic的分区迁移到新的broker上。
另外一种情况就是集群减少broker节点,对原有的topic信息进行迁移,重新分配broker节点。
简单案例
本次案例将原本存在于两个borker节点(86,87)上的topic:test01(两副本),转移到87,88两个broker上(重新分配)。
生成分配计划
进入kafka的bin目录:
cd /opt/cloudera/parcels/KAFKA/bin
新建文件:
vi topics-to-move.json {"topics":
[{"topic":"test01"}],
"version": 1
}
执行脚本,生成分配计划:(--broker-list "87,88"是目标borker)
kafka-reassign-partitions --zookeeper bigdata102:2181,bigdata103:2181,bigdata104:2181 --topics-to-move-json-file topics-to-move.json --broker-list "87,88" --generate
执行结果如下:
Current partition replica assignment #当前分区的副本分配
{"version":1,"partitions":[{"topic":"test01","partition":0,"replicas":[86,87]},{"topic":"test01","partition":1,"replicas":[86,87]},{"topic":"test01","partition":2,"replicas":[86,87]}]} Proposed partition reassignment configuration #建议的分区配置
{"version":1,"partitions":[{"topic":"test01","partition":0,"replicas":[87,88]},{"topic":"test01","partition":1,"replicas":[88,87]},{"topic":"test01","partition":1,"replicas":[87,88]}]}
新建文件topic-reassignment.json,将建议的分区设置copy进去:
vi topic-reassignment.json {"version":1,"partitions":[{"topic":"test01","partition":0,"replicas":[87,88]},{"topic":"test01","partition":1,"replicas":[88,87]},{"topic":"test01","partition":1,"replicas":[87,88]}]}
执行分配计划
根据建议的分区设置,重新分配topic的分区到broker节点
kafka-reassign-partitions --zookeeper bigdata102:2181,bigdata103:2181,bigdata104:2181 --reassignment-json-file topic-reassignment.json --execute
查看执行状态
kafka-reassign-partitions --zookeeper bigdata102:2181,bigdata103:2181,bigdata104:2181 --reassignment-json-file topic-reassignment.json --verify
最终完成分配