kafka生产者发送消息的方式可以分为:发送并忘记、同步发送、异步发送。

1、发送并忘记

生产者只要调用send方法就可以了,不需要处理send的返回值,大多数情况下消息都可以正常发送出去,但可能会丢失消息。

public class KafkaProducer {

    public static void main(String[] args) {
        KafkaProducer<String, String> producer = new KafkaProducer(producerConfigs());
        ProducerRecord<String, String> record = new ProducerRecord("test_topic", "hello kafka");
        //调用send方法即完成发送
        producer.send(record);
    }

    public static Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.93.132:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        return props;
    }
}

但是当我们获取元数据时出现问题,send的方法依旧会被阻塞。

try {
	//这边会阻塞
    clusterAndWaitTime = waitOnMetadata(record.topic(), record.partition(), maxBlockTimeMs);
} catch (KafkaException e) {
    if (metadata.isClosed())
        throw new KafkaException("Producer closed while send in progress", e);
    throw e;
}

比如指定一个不存在的分区发送消息,就会因为元数据获取问题,导致阻塞,默认是等待60s。

Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic test_topic not present in metadata after 1000 ms.
	at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1269)
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:933)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:743)
	at cn.enjoyedu.sendtype.KafkaFutureProducer.main(KafkaFutureProducer.java:26)
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic test_topic not present in metadata after 1000 ms.

2、同步发送

接收send方法的返回值Future对象,并调用get方法。

public static void main(String[] args) throws ExecutionException, InterruptedException {
        KafkaProducer<String, String> producer = new KafkaProducer(producerConfigs());
        ProducerRecord<String, String> record = new ProducerRecord("test_topic", "hello kafka");
        Future<RecordMetadata> send = producer.send(record);
        //get方法会阻塞
        RecordMetadata recordMetadata = send.get();
        System.out.println("topic: " + recordMetadata.topic() + ", offset: " + recordMetadata.offset() + ", partition: " + recordMetadata.partition());
    }

可以获取发送的一些信息。

topic: test_topic, offset: 232, partition: 3

3、异步发送

使用带Callback的send方法,提供请求完成的异步处理,当发送到服务器的记录已被确认时,将调用此方法。

public Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback) {
    // intercept the record, which can be potentially modified; this method does not throw exceptions
    ProducerRecord<K, V> interceptedRecord = this.interceptors.onSend(record);
    return doSend(interceptedRecord, callback);
}
public static void main(String[] args) {
        KafkaProducer<String, String> producer = new KafkaProducer(producerConfigs());
        ProducerRecord<String, String> record = new ProducerRecord("test_topic", "hello_key", "hello kafka");
        //callback1
        producer.send(record,
                (metadata, e) -> {
                    if (e != null) {
                        e.printStackTrace();
                    } else {
                        System.out.println("The offset of the record we just sent is: " + metadata.offset());
                    }
                });
        //callback2
        producer.send(record,
                (metadata, e) -> {
                    if (e != null) {
                        e.printStackTrace();
                    } else {
                        System.out.println("The offset of the record we just sent is: " + metadata.offset());
                    }
                });
        producer.close();
    }
The offset of the record we just sent is: 98
The offset of the record we just sent is: 99

对于发送到同一分区的记录,保证按顺序执行回调,也就是说,在上面的示例中,保证在callback2之前执行callback1。

Logo

Kafka开源项目指南提供详尽教程,助开发者掌握其架构、配置和使用,实现高效数据流管理和实时处理。它高性能、可扩展,适合日志收集和实时数据处理,通过持久化保障数据安全,是企业大数据生态系统的核心。

更多推荐