Kafka报错:

org.apache.kafka.common.errors.RecordTooLargeException: There are some messages at [Partition=Offset]: {testTopic-0=25537140} whose size is larger than the fetch size 1048576 and hence cannot be ever returned. Increase the fetch size on the client (using max.partition.fetch.bytes), or decrease the maximum message size the broker will allow (using message.max.bytes).

根据提示,是需要调整参数max.partition.fetch.bytes或message.max.bytes,解决办法可参考该篇博客。那么有没有可以不需要修改配置文件或参数的方法呢?可参考https://issues.apache.org/jira/browse/KAFKA-4762。这里提出一种解决办法:

Logs show that you are using 0.10.x(or before) where max.partition.fetch.bytes is a hard limit even when you enable the compression. In your case, seems that you have enabled the compression on the producer side. `max.partition.fetch.bytes` also applies to the whole compressed message which is often much larger than a single one. That's why you run into RecordTooLargeException.

0.10.1 which completes KIP-74 already 'fixes' your problem by making `max.partition.fetch.bytes` field in the fetch request much less useful, so you can try with an 0.10.1 build.

简单地说,就是你需要调整一下kafka版本。我将kafka版本0.10.0.1调整为0.10.1.0,问题解决。

Logo

Kafka开源项目指南提供详尽教程,助开发者掌握其架构、配置和使用,实现高效数据流管理和实时处理。它高性能、可扩展,适合日志收集和实时数据处理,通过持久化保障数据安全,是企业大数据生态系统的核心。

更多推荐