引言:

  目前使用golang接入kafka,于是便用到了github.com/Shopify/sarama,一个kafka的golang client library,但官方doc中尚无中文版本,于是本人在这里捉襟见肘做一个简单地翻译,作为自己的一个点滴记录和查阅,如果能够帮助到你一点点,我会非常高兴的。另外,由于本人能力有限,若有不当之处,欢迎指正。

正文:

package sarama

Package sarama is a pure Go client library for dealing with Apache Kafka (versions 0.8 and later). It includes a high-level API for easily producing and consuming messages, and a low-level API for controlling bytes on the wire when the high-level API is insufficient. Usage examples for the high-level APIs are provided inline with their full documentation.

sarama 是一个为了处理apache kafka的纯Go (编写的),客户端库。它包含用于轻松生成和消费消息的高级API,以及当高级API不足时用于控制传输bytes的低级API。本文提供完整使用高级API的示例。

To produce messages, use either the AsyncProducer or the SyncProducer. The AsyncProducer accepts messages on a channel and produces them asynchronously in the background as efficiently as possible; it is preferred in most cases. The SyncProducer provides a method which will block until Kafka acknowledges the message as produced. This can be useful but comes with two caveats: it will generally be less efficient, and the actual durability guarantees depend on the configured value of Producer.RequiredAcks. There are configurations where a message acknowledged by the SyncProducer can still sometimes be lost.

要生成消息,请使用AsyncProducer或SyncProducer。AsyncProducer接收通道上的消息,并在后台以异步方式尽可能高效地生成它们; 在大多数情况下首选此种方式。SyncProducer提供了一种阻塞的等待服务器,直到Kafka确认消息生成为止的方法。这在某些情况下是有用的,但有两个注意事项:它通常效率较低,实际的持久性保证取决于“Producer.RequiredAcks”的配置值。这是一个用于一个消息确认ack多久能够丢失的配置。

To consume messages, use the Consumer. Note that Sarama’s Consumer implementation does not currently support automatic consumer-group rebalancing and offset tracking. For Zookeeper-based tracking (Kafka 0.8.2 and earlier), the https://github.com/wvanbergen/kafka library builds on Sarama to add this support. For Kafka-based tracking (Kafka 0.9 and later), the https://github.com/bsm/sarama-cluster library builds on Sarama to add this support.

要消费消息,请使用消费者。注意Sarama的Consumer实现目前不支持自动消费者组rebalancing和offset偏移量追踪。对于基于Zookeeper的追踪(Kafka 0.8.2及更早版本),https://github.com/wvanbergen/kafka库基于Sarama添加此支持。对于基于kafka的追踪(Kafka 0.9及更高版本),https://github.com/bsm/sarama-cluster库基于Sarama添加此支持。

For lower-level needs, the Broker and Request/Response objects permit precise control over each connection and message sent on the wire; the Client provides higher-level metadata management that is shared between the producers and the consumer. The Request/Response objects and properties are mostly undocumented, as they line up exactly with the protocol fields documented by Kafka at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

对于较低级别的需求,Broker和Request/Response对象允许精确控制每个连接和消息的发送; 此客户端提供在生产者和消费者之间共享的更高级别的元数据管理。请求/响应对象和属性大部分未记录在文档中,因为它们与Kafka 记录的协议字段完全一致,网址为 https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol

Broker related metrics:

Broker 关联指标 :

NameTypeDescription
incoming-byte-ratemeterBytes/second read off all brokers
incoming-byte-rate-for-broker-meterBytes/second read off a given broker
outgoing-byte-ratemeterBytes/second written off all brokers
outgoing-byte-rate-for-broker-meterBytes/second written off a given broker
request-ratemeterRequests/second sent to all brokers
request-rate-for-broker-meterRequests/second sent to a given broker
request-sizehistogramDistribution of the request size in bytes for all brokers
request-size-for-broker-histogramDistribution of the request size in bytes for a given broker
request-latency-in-mshistogramDistribution of the request latency in ms for all brokers
request-latency-in-ms-for-broker-histogramDistribution of the request latency in ms for a given broker
response-ratemeterResponses/second received from all brokers
response-rate-for-broker-meterResponses/second received from a given broker
response-sizehistogramDistribution of the response size in bytes for all brokers
response-size-for-broker-histogramDistribution of the response size in bytes for a given broker

Note that we do not gather specific metrics for seed brokers but they are part of the “all brokers” metrics.
请注意,我们不会收集seed brokers的详细指标,但它们是“所有brokers”指标的一部分。

Producer related metrics:
生产者关联指标:

NameTypeDescription
batch-sizehistogramDistribution of the number of bytes sent per partition per request for all topics
batch-size-for-topic-histogramDistribution of the number of bytes sent per partition per request for a given topic
record-send-ratemeterRecords/second sent to all topics
record-send-rate-for-topic-meterRecords/second sent to a given topic
records-per-requesthistogramDistribution of the number of records sent per request for all topics
records-per-request-for-topic-histogramDistribution of the number of records sent per request for a given topic
compression-ratiohistogramDistribution of the compression ratio times 100 of record batches for all topics
compression-ratio-for-topic-histogramDistribution of the compression ratio times 100 of record batches for a given topic

附录

由于精力有限,这里不包括API中的常量,变量,函数等详细说明,详情请参阅:
sarama github GoDoc

Logo

Kafka开源项目指南提供详尽教程,助开发者掌握其架构、配置和使用,实现高效数据流管理和实时处理。它高性能、可扩展,适合日志收集和实时数据处理,通过持久化保障数据安全,是企业大数据生态系统的核心。

更多推荐