Kafka常用命令汇总:

1、列出集群里的所有主题。
kafka-topics.sh --zookeeper localhost:2181 --list

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --list
ClearFdmDirtyDataCheckTopic
ClearFdmDirtyData_CheckTopic
ClearFdmDirtyData_Check_Topic
ClearFdmDirtyData_ExeTopic
TaskTag_Check_topic
TaskTag_Exe_topic



2、 创建一个叫作my-topic的主题,主题包含8分区,每个分区拥有两个副本。
kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --replication-factor 3 --partitions 3

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --replication-factor 3 --partitions 3
Created topic "my-topic".


3、列出集群里所有主题的详细信息。
kafka-topics.sh --zookeeper localhost:2181 --describe


[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --describe
Topic:ClearFdmDirtyDataCheckTopic       PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ClearFdmDirtyDataCheckTopic      Partition: 0    Leader: 163     Replicas: 163   Isr: 163
Topic:ClearFdmDirtyData_CheckTopic      PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: ClearFdmDirtyData_CheckTopic     Partition: 0    Leader: 163     Replicas: 163,164,135   Isr: 163,164,135
Topic:ClearFdmDirtyData_Check_Topic     PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ClearFdmDirtyData_Check_Topic    Partition: 0    Leader: 135     Replicas: 135   Isr: 135
Topic:ClearFdmDirtyData_ExeTopic        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ClearFdmDirtyData_ExeTopic       Partition: 0    Leader: 164     Replicas: 164   Isr: 164
Topic:TaskTag_Check_topic       PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: TaskTag_Check_topic      Partition: 0    Leader: 135     Replicas: 135,163,164   Isr: 135,163,164
        Topic: TaskTag_Check_topic      Partition: 1    Leader: 136     Replicas: 136,164,135   Isr: 136,164,135
        Topic: TaskTag_Check_topic      Partition: 2    Leader: 162     Replicas: 162,135,136   Isr: 162,135,136
Topic:TaskTag_Exe_topic PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: TaskTag_Exe_topic        Partition: 0    Leader: 164     Replicas: 164,162,163   Isr: 164,162,163
        Topic: TaskTag_Exe_topic        Partition: 1    Leader: 135     Replicas: 135,163,164   Isr: 135,163,164
        Topic: TaskTag_Exe_topic        Partition: 2    Leader: 136     Replicas: 136,164,135   Isr: 136,164,135
Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: __consumer_offsets       Partition: 0    Leader: 136     Replicas: 136,162,163   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 1    Leader: 162     Replicas: 162,163,164   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 2    Leader: 163     Replicas: 163,164,135   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 3    Leader: 164     Replicas: 164,135,136   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 4    Leader: 135     Replicas: 135,136,162   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 5    Leader: 136     Replicas: 136,163,164   Isr: 163,164,136
        Topic: __consumer_offsets       Partition: 6    Leader: 162     Replicas: 162,164,135   Isr: 162,164,135
        Topic: __consumer_offsets       Partition: 7    Leader: 163     Replicas: 163,135,136   Isr: 163,135,136
        Topic: __consumer_offsets       Partition: 8    Leader: 164     Replicas: 164,136,162   Isr: 162,164,136
        Topic: __consumer_offsets       Partition: 9    Leader: 135     Replicas: 135,162,163   Isr: 162,163,135
        Topic: __consumer_offsets       Partition: 10   Leader: 136     Replicas: 136,164,135   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 11   Leader: 162     Replicas: 162,135,136   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 12   Leader: 163     Replicas: 163,136,162   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 13   Leader: 164     Replicas: 164,162,163   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 14   Leader: 135     Replicas: 135,163,164   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 15   Leader: 136     Replicas: 136,135,162   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 16   Leader: 162     Replicas: 162,136,163   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 17   Leader: 163     Replicas: 163,162,164   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 18   Leader: 164     Replicas: 164,163,135   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 19   Leader: 135     Replicas: 135,164,136   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 20   Leader: 136     Replicas: 136,162,163   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 21   Leader: 162     Replicas: 162,163,164   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 22   Leader: 163     Replicas: 163,164,135   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 23   Leader: 164     Replicas: 164,135,136   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 24   Leader: 135     Replicas: 135,136,162   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 25   Leader: 136     Replicas: 136,163,164   Isr: 163,164,136
        Topic: __consumer_offsets       Partition: 26   Leader: 162     Replicas: 162,164,135   Isr: 162,164,135
        Topic: __consumer_offsets       Partition: 27   Leader: 163     Replicas: 163,135,136   Isr: 163,135,136
        Topic: __consumer_offsets       Partition: 28   Leader: 164     Replicas: 164,136,162   Isr: 162,164,136
        Topic: __consumer_offsets       Partition: 29   Leader: 135     Replicas: 135,162,163   Isr: 162,163,135
        Topic: __consumer_offsets       Partition: 30   Leader: 136     Replicas: 136,164,135   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 31   Leader: 162     Replicas: 162,135,136   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 32   Leader: 163     Replicas: 163,136,162   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 33   Leader: 164     Replicas: 164,162,163   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 34   Leader: 135     Replicas: 135,163,164   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 35   Leader: 136     Replicas: 136,135,162   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 36   Leader: 162     Replicas: 162,136,163   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 37   Leader: 163     Replicas: 163,162,164   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 38   Leader: 164     Replicas: 164,163,135   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 39   Leader: 135     Replicas: 135,164,136   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 40   Leader: 136     Replicas: 136,162,163   Isr: 162,163,136
        Topic: __consumer_offsets       Partition: 41   Leader: 162     Replicas: 162,163,164   Isr: 162,163,164
        Topic: __consumer_offsets       Partition: 42   Leader: 163     Replicas: 163,164,135   Isr: 163,164,135
        Topic: __consumer_offsets       Partition: 43   Leader: 164     Replicas: 164,135,136   Isr: 164,135,136
        Topic: __consumer_offsets       Partition: 44   Leader: 135     Replicas: 135,136,162   Isr: 162,135,136
        Topic: __consumer_offsets       Partition: 45   Leader: 136     Replicas: 136,163,164   Isr: 163,164,136
        Topic: __consumer_offsets       Partition: 46   Leader: 162     Replicas: 162,164,135   Isr: 162,164,135
        Topic: __consumer_offsets       Partition: 47   Leader: 163     Replicas: 163,135,136   Isr: 163,135,136
        Topic: __consumer_offsets       Partition: 48   Leader: 164     Replicas: 164,136,162   Isr: 162,164,136
        Topic: __consumer_offsets       Partition: 49   Leader: 135     Replicas: 135,162,163   Isr: 162,163,135




4、列出集群里特定主题的详细信息。
kafka-topics.sh --zookeeper localhost:2181 --describe  --topic my-topic

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --describe  --topic my-topic
Topic:my-topic  PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: my-topic Partition: 0    Leader: 135     Replicas: 135,162,163   Isr: 135,162,163
        Topic: my-topic Partition: 1    Leader: 136     Replicas: 136,163,164   Isr: 136,163,164
        Topic: my-topic Partition: 2    Leader: 162     Replicas: 162,164,135   Isr: 162,164,135


5、删除一个叫作my-topic的主题。
kafka-topics.sh --zookeeper localhost:2181 --delete  --topic my-topic

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-topics.sh --zookeeper localhost:2181 --delete  --topic my-topic
Topic my-topic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.


6、列出旧版本的所有消费者群组。
kafka-consumer-groups.sh --zookeeper localhost:2181 --list


7、列出新版本的所有消费者群组。
kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --list

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --list
ClearFDM_Check_ConsumerGroup_1
ClearFDM_Exe_ConsumerGroup_1
consumer_group_TaskTag_Check_topic
consumer_group_TaskTag_Exec_topic


8、获取旧版本消费者群组testgroup的详细信息。
kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group testgroup

9、获取新版本消费者群组testgroup的详细信息。
kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --describe --group testgroup


[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-consumer-groups.sh --new-consumer --bootstrap-server 172.21.50.162:9092 --describe --group consumer_group_TaskTag_Check_topic
GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
consumer_group_TaskTag_Check_topic TaskTag_Check_topic            2          unknown         0               unknown         fda52f5c-3d16-434e-8ba7-d26975dfe2a5_/172.21.50.10
consumer_group_TaskTag_Check_topic TaskTag_Check_topic            1          unknown         0               unknown         cbd08551-8380-40ff-bcda-da59cfb9480c_/172.21.50.10
consumer_group_TaskTag_Check_topic TaskTag_Check_topic            0          unknown         22              unknown         50d7c78c-7528-4aad-ba94-68724628f2cd_/172.21.50.10


10、查看某一个topic对应的消息数量。
kafka-run-class.sh  kafka.tools.GetOffsetShell --broker-list 172.21.50.162:9092 --topic my-topic --time -1

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-run-class.sh  kafka.tools.GetOffsetShell --broker-list 172.21.50.162:9092 --topic consumer_group_TaskTag_Check_topic --time -1
consumer_group_TaskTag_Check_topic:0:0


11、查看log日志片段的内容,显示查看日志的内容。
kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log --print-data-log

[bdp_client@BJLFRZ-Client-50-162 t9-1]$ kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000000000000.log
Dumping 00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 CreateTime: 1537343431271 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 2502058395 keysize: 13
offset: 1 position: 66 CreateTime: 1537343431272 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 2236582723 keysize: 13
offset: 2 position: 132 CreateTime: 1537343431272 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 3270504851 keysize: 13
offset: 3 position: 198 CreateTime: 1537343708182 isvalid: true payloadsize: 19 magic: 1 compresscodec: NoCompressionCodec crc: 450329064 keysize: 13


12、控制台生产者:向主题 my-topic 生成两个消息。
kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic

[bdp_client@BJLFRZ-Client-50-162 ~]$ kafka-console-producer.sh --broker-list localhost:9092 --topic my-topic
1
2
3
4
5
6
7



13、控制台消费者:从主题 my-topic 获取消息。
kafka-console-consumer.sh --zookeeper localhost:2181  --topic my-topic --from-beginning
kafka-console-consumer.sh --new-consumer --bootstrap-server 172.21.50.162:9092  --topic my-topic --from-beginning
Logo
Kafka开源项目指南

Kafka开源项目指南提供详尽教程，助开发者掌握其架构、配置和使用，实现高效数据流管理和实时处理。它高性能、可扩展，适合日志收集和实时数据处理，通过持久化保障数据安全，是企业大数据生态系统的核心。

更多推荐

Kafka入门（一） 概述、部署与API的简单使用

Kafka概述、部署与API的简单使用

avatar Kafka开源项目指南

基于canal和kafka同步，实现binlog同步ElasticSearch

文章目录前言elasticsearch 安装canal安装canal-adapter 安装及配置mysql 安装zk及kafaka安装查看效果注意事项前言中间件版本elasticsearch7.5.2canal1.1.4client-adapter1.1.5-alpha-1zookeeper3.4.13kafka2.6.0mysql5.7.31elasticsearch 安装{"settings"

avatar Kafka开源项目指南

基于 Iceberg 的湖仓一体架构在 B 站的实践

背景在B站，每天都有PB级的数据注入到大数据平台，经过离线或实时的ETL建模后，提供给下游的分析、推荐及预测等场景使用。面对如此大规模的数据，如何高效低成本地满足下游数据的分析需求，一直是我们重点的工作方向。我们之前的数据处理流程基本上是这样的：采集端将客户端埋点、服务端埋点、日志、业务数据库等数据收集到HDFS、Kafka等存储系统中，然后通过Hive、Spark、Fl...

avatar Kafka开源项目指南