Kafka Offset相关命令总结

2022-04-09 00:34:39

　　　　　　　　　　　　　　　　Kafka Offset相关命令总结

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

一.查询topic的offset的范围

1>.查询某个topic的offset的最小值

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic yinzhengjie --time -

yinzhengjie::　　　　　　　　　　　　#很显然，第一列为topic名称，第二列为partition名称，第三列则是offset。为估计仔细的你应该已经发现了最小的offset为0，分区数为30个

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic yinzhengjie --time - | wc -l

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114:9092 -topic yinzhengjie --time -2

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic data.batch.ad --time -

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic data.batch.ad --time - |wc -l

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114:9092 -topic data.batch.ad --time -2

2>.查询某个topic的offset的最大值

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic yinzhengjie --time -

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

yinzhengjie::

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic yinzhengjie --time - | wc -l

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114:9092 -topic yinzhengjie --time -1

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic data.batch.ad --time -

data.batch.ad::　　　　　　　　#很明显，以冒号为分割，第一列为topic名称，第二列为partition 编号，第三列为当前最大的便宜量位置！

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

data.batch.ad::

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114: -topic data.batch.ad --time - | wc -l

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.1.2.114:9092 -topic data.batch.ad --time -1

二.查看consumer group列表，使用--list参数

　　查看consumer group列表有新、旧两种命令，分别查看新版(信息保存在broker中)consumer列表和老版(信息保存在zookeeper中)consumer列表，因而需要区分指定bootstrap--server和zookeeper参数：（本片博客测试的kafka版本为0.10.2.1）

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --zookeeper 10.1.2.114: --list

Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).　　　　　　　　#很显然，为这里使用的是kafka 0.10..1版本，我们是查询不到对应的信息的！

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --zookeeper 10.1.2.114:2181 --list　　　　　　　　　　　　　　　　#老版本的consumer信息都保存在zookeeper中，很显然，0.10.2.1这个版本已经将consumer信息自己保存啦！

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --bootstrap-server 10.1.2.114: --list　　　　　　　　　　　　#查看新版consumer列表，很显然，我们可以查询到对应的数据

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

id-data.batch.ad

console-consumer-

id-data.batch.env

id-data.batch.er

id-data.batch.pl

console-consumer-

id-data.batch.op

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --bootstrap-server 10.1.2.114:9092 --list　　　　　　　　　　　　#查看新版consumer列表，很显然，我们可以查询到对应的数据

三.查看指定group.id 的消费者消费情况（kafka-consumer-groups.sh）

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --new-consumer --bootstrap-server 10.1.2.114:  --group id-data.batch.ad --describe

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                               consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

data.batch.ad                                              consumer--53498a12-e62b-46c8-9c5c-ad97f807698d   /10.1.2.112                    consumer-

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --new-consumer --bootstrap-server 10.1.2.114:  --group id-data.batch.ad --describe | grep ad | wc -l

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

[root@node101.yinzhengjie.org.cn ~]# 

简单对每列进行说明：

TOPIC：

　　消费者的topic名称　　

PARTITION：

　　分区数的名称　　

CURRENT-OFFSET：

　　consumer group最后一次提交的offset

LOG-END-OFFSET：

　　最后提交的生产消息offset

LAG

　　消费offset与生产offset之间的差值

CONSUMER-ID

　　消费者的ID编号，我们知道消费者组里面可以有最少要有一个消费者，当然也可以有多个消费者。

HOST

　　消费者的主机IP地址。

CLIENT-ID

　　链接的ID编号。

[root@node101.yinzhengjie.org.cn ~]# kafka-consumer-groups.sh --new-consumer --bootstrap-server 10.1.2.114:9092 --group id-data.batch.ad --describe

四. 查询 __consumer_offsets topic 所有内容

1>.运行下面命令前先要在consumer.properties中设置exclude.internal.topics=false

[root@node101.yinzhengjie.org.cn ~]# cat /soft/kafka/config/consumer.properties

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements.  See the NOTICE file distributed with

# this work for additional information regarding copyright ownership.

# The ASF licenses this file to You under the Apache License, Version 2.0

# (the "License"); you may not use this file except in compliance with

# the License.  You may obtain a copy of the License at

#

#    http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# see kafka.consumer.ConsumerConfig for more details

# Zookeeper connection string

# comma separated host:port pairs, each corresponding to a zk

# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"

zookeeper.connect=127.0.0.1:

# timeout in ms for connecting to zookeeper

zookeeper.connection.timeout.ms=

#consumer group id

group.id=test-consumer-group

#consumer timeout

#consumer.timeout.ms=

exclude.internal.topics=false

[root@node101.yinzhengjie.org.cn ~]#

[root@node101.yinzhengjie.org.cn ~]# cat /soft/kafka/config/consumer.properties

2>.0.11.0.0之前版本

[root@node101.yinzhengjie.org.cn ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 10.1.2.114: --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config /soft/kafka/config/consumer.properties --from-beginning

.......

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,96ef03f3-528a--9b56-7b98ea258ba4],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

[-data.batch.ad,data.batch.ad,]::[OffsetMetadata[,b350dc94-868b-42a5-91f3-7fb59ca7ea93],CommitTime ,ExpirationTime ]

.......

[root@node101.yinzhengjie.org.cn ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 10.1.2.114:9092 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config /soft/kafka/config/consumer.properties --from-beginning

3>. 0.11.0.0之后版本(含)

[root@node101.yinzhengjie.org.cn ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 10.1.2.114:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config /soft/kafka/config/consumer.properties --from-beginning

五.

六.

码农公寓

相关文章