背景:最近使用的项目中用到了kafka,并且需要集成kerberos.这边kerberos认证已经通过,但是在消费的时候一直拉不到数据,随后使用shell命令在服务器上消费数据,结果一样。
后面更改日志级别,发现一直循环打印如下日志:

2020-05-21 10:05:06.651 [ForkJoinPool-1-worker-5] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 133 to Cluster(id = YFJdX6kITzuEPD3F4zdZEw, nodes = [master:9092 (id: 39 rack: null)], partitions = [Partition(topic = fuxi_test, partition = 0, leader = 39, replicas = [39], isr = [39], offlineReplicas = [])], controller = master:9092 (id: 39 rack: null))
2020-05-21 10:05:06.651 [ForkJoinPool-1-worker-5] DEBUG o.a.k.c.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=test_consumer] Sending FindCoordinator request to broker master:9092 (id: 39 rack: null)
2020-05-21 10:05:06.655 [ForkJoinPool-1-worker-5] DEBUG o.a.k.c.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=test_consumer] Received FindCoordinator response ClientResponse(receivedTimeMs=1590026706655, latencyMs=4, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=consumer-1, correlationId=265), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=COORDINATOR_NOT_AVAILABLE, node=:-1 (id: -1 rack: null)))
2020-05-21 10:05:06.655 [ForkJoinPool-1-worker-5] DEBUG o.a.k.c.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=test_consumer] Group coordinator lookup failed: The coordinator is not available.
2020-05-21 10:05:06.655 [ForkJoinPool-1-worker-5] DEBUG o.a.k.c.consumer.internals.AbstractCoordinator - [Consumer clientId=consumer-1, groupId=test_consumer] Coordinator discovery failed, refreshing metadata

最终查阅网上资料,发现一位童鞋说:
__consumer_offsets 这个topic是否存在
结果发现该topic真的没有,手动创建

kafka-topics --zookeeper master:2181 --partitions 1 --replication-factor 1 --create --topic __consumer_offsets

重启kafka ok

Logo

Kafka开源项目指南提供详尽教程,助开发者掌握其架构、配置和使用,实现高效数据流管理和实时处理。它高性能、可扩展,适合日志收集和实时数据处理,通过持久化保障数据安全,是企业大数据生态系统的核心。

更多推荐