flink kafka sink 默认分区器 FlinkFixedPartitioner 原理与注意
FlinkFixedPartitioner源码:package org.apache.flink.streaming.connectors.kafka.partitioner;import org.apache.flink.util.Preconditions;public class FlinkFixedPartitioner<T> extends FlinkKaf...
·
FlinkFixedPartitioner源码:
package org.apache.flink.streaming.connectors.kafka.partitioner;
import org.apache.flink.util.Preconditions;
public class FlinkFixedPartitioner<T> extends FlinkKafkaPartitioner<T> {
private int parallelInstanceId;
public FlinkFixedPartitioner() {
}
public void open(int parallelInstanceId, int parallelInstances) {
Preconditions.checkArgument(parallelInstanceId >= 0, "Id of this subtask cannot be negative.");
Preconditions.checkArgument(parallelInstances > 0, "Number of subtasks must be larger than 0.");
this.parallelInstanceId = parallelInstanceId;
}
public int partition(T record, byte[] key, byte[] value, String targetTopic, int[] partitions) {
Preconditions.checkArgument(partitions != null && partitions.length > 0, "Partitions of the target topic is empty.");
return partitions[this.parallelInstanceId % partitions.length];
}
}
根据源码可以看出:
flink是根据sink的subtask的id和kafka的partition数量进行取余计算的,计算过程如下:
flink并行度为3(F0,F1,F2),partition数量为2(P0,P1),则F0->P0,F1->P1,F2->P0
flink并行度为2(F0,F1),partition数量为3(P0,P1,P2),则F0->P0,F1->P1
因此默认分区器会有2个坑:
-
当 Sink 的并发度低于 Topic 的 partition 个数时,一个 sink task 写一个 partition,会导致部分 partition 完全没有数据。
-
当 topic 的 partition 扩容时,则需要重启作业,以便发现新的 partition。
更多推荐
已为社区贡献3条内容
所有评论(0)