Flink流式处理集成kafka
1:引言对于实时处理当中,我们实际工作当中的数据源一般都是使用kafka,所以我们一起来看看如何通过Flink来集成kafkaFlink提供了一个特有的kafka connector去读写kafka topic的数据。flink消费kafka数据,并不是完全通过跟踪kafka消费组的offset来实现去保证exactly-once的语义,而是flink内部去跟踪offset和做checkpoint
1:引言
对于实时处理当中,我们实际工作当中的数据源一般都是使用kafka,所以我们一起来看看如何通过Flink来集成kafka
Flink提供了一个特有的kafka connector去读写kafka topic的数据。flink消费kafka数据,并不是完全通过跟踪kafka消费组的offset来实现去保证exactly-once的语义,而是flink内部去跟踪offset和做checkpoint去实现exactly-once的语义,而且对于kafka的partition,Flink会启动对应的并行度去处理kafka当中的每个分区的数据。
Flink整合kafka官网介绍
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kafka.html
2:导入pom依赖
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka -->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.11</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-statebackend-rocksdb_2.11</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.25</version>
</dependency>
3:将kafka作为flink的source来使用
实际工作当中一般都是将kafka作为flink的source来使用
3.1:创建kafka的topic
安装好kafka集群,并启动kafka集群,然后在node01执行以下命令创建kafka的topic为test
kafka-topics.sh --create --partitions 3 --topic test --replication-factor 1 --zookeeper node01:2181,node02:2181,node03:2181
3.2:代码实现
import java.util.Properties
import org.apache.flink.contrib.streaming.state.RocksDBStateBackend
import org.apache.flink.streaming.api.CheckpointingMode
import org.apache.flink.streaming.api.environment.CheckpointConfig
import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment}
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
import org.apache.flink.streaming.util.serialization.SimpleStringSchema
/**
* 将kafka作为flink的source来使用
*/
object FlinkKafkaSource {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
//**隐式转换
import org.apache.flink.api.scala._
//checkpoint**配置
env.enableCheckpointing(100)
env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500)
env.getCheckpointConfig.setCheckpointTimeout(60000)
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
env.getCheckpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)
//设置statebackend
env.setStateBackend(new RocksDBStateBackend("hdfs://node01:8020/flink_kafka_sink/checkpoints",true));
val topic = "test"
val prop = new Properties()
prop.setProperty("bootstrap.servers","node01:9092,node02:9092,node03:9092")
prop.setProperty("group.id","con1")
prop.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
prop.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
val kafkaConsumer = new FlinkKafkaConsumer[String]("test",new SimpleStringSchema,prop)
kafkaConsumer.setCommitOffsetsOnCheckpoints(true)
val kafkaSource: DataStream[String] = env.addSource(kafkaConsumer)
kafkaSource.print()
env.execute()
}
}
3.3:kafka生产数据
node01执行以下命令,通过shell命令行来生产数据到kafka当中去
##创建topic
kafka-topics.sh --create --topic test --partitions 3 --replication-factor 2 --zookeeper node01:2181,node02:2181,node03:2181
##发送数据
kafka-console-producer.sh --broker-list node01:9092,node02:9092,node03:9092 --topic test
4:将kafka作为flink的sink来使用
我们也可以将kafka作为flink的sink来使用,就是将flink处理完成之后的数据写入到kafka当中去。
4.1:socket发送数据
node01执行以下命令,从socket当中发送数据
nc -lk 9999
4.2:代码实现
import java.util.Properties
import org.apache.flink.contrib.streaming.state.RocksDBStateBackend
import org.apache.flink.streaming.api.CheckpointingMode
import org.apache.flink.streaming.api.environment.CheckpointConfig
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer
import org.apache.flink.streaming.connectors.kafka.internals.KeyedSerializationSchemaWrapper
import org.apache.flink.streaming.util.serialization.SimpleStringSchema
/**
* 将kafka作为flink的sink来使用
*/
object FlinkKafkaSink {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
//隐式转换
import org.apache.flink.api.scala._
//checkpoint配置
env.enableCheckpointing(5000);
env.getCheckpointConfig.setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500);
env.getCheckpointConfig.setCheckpointTimeout(60000);
env.getCheckpointConfig.setMaxConcurrentCheckpoints(1);
env.getCheckpointConfig.enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
//设置statebackend
env.setStateBackend(new RocksDBStateBackend("hdfs://node01:8020/flink_kafka_sink/checkpoints",true));
val socketStream = env.socketTextStream("node01",9999)
val topic = "test"
val prop = new Properties()
prop.setProperty("bootstrap.servers","node01:9092,node02:9092,node03:9092")
prop.setProperty("group.id","kafka_group1")
//第一种解决方案,设置FlinkKafkaProducer里面的事务超时时间
//设置事务超时时间
prop.setProperty("transaction.timeout.ms",60000*15+"");
//第二种解决方案,设置kafka的最大事务超时时间
//FlinkKafkaProducer011<String> myProducer = new FlinkKafkaProducer<>(brokerList, topic, new SimpleStringSchema());
//使用支持仅一次语义的形式
/**
* defaultTopic: String,
* serializationSchema: KafkaSerializationSchema[IN],
* producerConfig: Properties,
* semantic: FlinkKafkaProducer.Semantic
*/
val kafkaSink = new FlinkKafkaProducer[String](topic,new KeyedSerializationSchemaWrapper[String](new SimpleStringSchema()), prop,FlinkKafkaProducer.Semantic.EXACTLY_ONCE)
socketStream.addSink(kafkaSink)
env.execute("StreamingFromCollectionScala")
}
}
4.3:启动kafka消费者
node01执行以下命令启动kafka消费者,消费数据
kafka-console-consumer.sh --bootstrap-server node01:9092,node02:9092,node03:9092 --topic test
更多推荐
所有评论(0)