Flink SQL Kafka写入Hudi详解

Flink读写Hudi案列详解

一个数据小开发

4548人浏览 · 2022-04-13 14:30:10

一个数据小开发 · 2022-04-13 14:30:10 发布

1、背景

前面Hudi的专栏已经详细讲解过本地如何构建这套Hudi的运行环境，在此就不在重复了，如果感兴趣想了解的可以去学习下，从0到1搭建数据湖Hudi环境_一个数据小开发的博客-CSDN博客

本次重点是为了能够很好的理解Hudi的两种表COW和MOR，下面将对这两种表操作的详情进行demo操作。

2、实操

2.1、启动Flink SQL Client

export HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath`
./sql-client.sh embedded -j ../lib/hudi-flink-bundle_2.11-0.10.1.jar shell

2.2、启动Hadoop

2.3、创建KafkaTopic

kafka-topics.sh --create --bootstrap-server 11.238.116.75:9092 --replication-factor 1 --partitions 3 --topic hudi_flink_demo

2.4、创建Flink Kafka表和Hudi表

-- 构建kafka source表
CREATE TABLE data_gen (
   id STRING,
   name STRING,
   user_age STRING,
   user_other STRING,
   ts TIMESTAMP(3)
) WITH (
 'connector' = 'kafka',
 'topic' = 'hudi_flink_demo',
 'properties.bootstrap.servers' = 'ip:9092',
 'properties.group.id' = 'testGroup3',
 'format' = 'json',
 'scan.startup.mode' = 'earliest-offset'
);

2.4.1、COW表

-- 构建COW模式Hudi表
CREATE TABLE hudi_cow_data_gen(
  id STRING,
  name STRING,
  user_age STRING,
  user_other STRING,
  ts TIMESTAMP(3),
  PRIMARY KEY(id) NOT ENFORCED
)
WITH (
  'connector' = 'hudi',
  'path' = 'hdfs://localhost:9000/user/root/hudi/hudi_cow_data_gen',
  'table.type' = 'COPY_ON_WRITE',
  'write.insert.drop.duplicates' = 'true'
);

-- 启动任务
insert into hudi_cow_data_gen select * from data_gen;

启动成功后，可以在yarn上看到这个任务的启动状态

点击上面的“ApplicationMaster”之后可以跳转到相对应的这个任务在Flink之上运行的一个Dashboard页面

可以看到任务都运行正常，并且在hdfs上看到，数据目录已经创建成功，但是因为还没开始写入数据，所以还没有parquet数据文件产生。

往Kafka发送一条数据之后，可以发现，records received已经有一条数据了

在去看下checkpoint页面

在checkpoint成功之后，发现数据文件已经生成

把parquet文件下载下来查看下里面的内容

parquet文件读取方式


##查看schema:

java -jar  parquet-tools-1.6.0rc3-SNAPSHOT.jar  schema -d bf0ef9e2-79de-4cbf-80a9-7b67325eedfb_3-4-0_20220412203746273.parquet | head -n 10

##查看文件内容：

java -jar  parquet-tools-1.6.0rc3-SNAPSHOT.jar  head -n 100 bf0ef9e2-79de-4cbf-80a9-7b67325eedfb_3-4-0_20220412203746273.parquet

此jar包可以在评论区留言后向我获取

发现插入的数据生效了，下面再插入一条数据，并且主键值还是“qwer1”的这条数据，只是改变下其他列的值，观察数据是否会发生变更。

发现hdfs上已经生成了两个parquet文件了，下面换一种方法读取里面的数据，用spark

读取里面的值

2.4.2、Spark读取COW表

import org.apache.spark.SparkConf;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class SparkReadHudi {
    public static void main(String[] args) {

        SparkConf conf = null;
        SparkSession sparkSession = null;

        conf = new SparkConf()
                .setAppName("SparkHudi")
                .set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=file:log4j.properties")
                .set("spark.logConf", "true")
                .set("spark.debug.maxToStringFields", "1000")
                .set("spark.sql.decimalOperations.allowPrecisionLoss", "false")
                .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
                .setMaster("local[*]");

        sparkSession = SparkSession.builder()
                .config(conf)
                .getOrCreate();

        Dataset<Row> hudiDataset = sparkSession.read()
                .format("hudi")
                .load("hdfs://localhost:9000/user/root/hudi/hudi_cow_data_gen")
                ;

        hudiDataset.createOrReplaceTempView("t_flink_hudi_table");
        sparkSession.sql("select * from t_flink_hudi_table").show();


    }
}

可以对比发现数据发生了变更，证明Hudi 数据更新生效了。

2.4.3、MOR表

delta_commits --指标含义，根据设置的提交次数来合并数据

-- 构建MOR模式的表

CREATE TABLE hudi_mor_data_gen(
    id STRING,
    name STRING,
    user_age STRING,
    user_other STRING,
    ts TIMESTAMP(3),
    PRIMARY KEY(id) NOT ENFORCED
)
WITH (
  'connector' = 'hudi',
  'path' = 'hdfs://localhost:9000/user/root/hudi/hudi_mor_data_gen',
  'table.type' = 'MERGE_ON_READ',
  'read.streaming.enabled' = 'true',
  'write.tasks'= '4',
  'compaction.tasks'= '4',
  'compaction.delta_seconds' = '120',
  'compaction.delta_commits' = '1',
  'read.streaming.check-interval' = '1'
);


CREATE TABLE hudi_mor_data_gen2(
    id STRING,
    name STRING,
    user_age STRING,
    user_other STRING,
    ts TIMESTAMP(3),
    PRIMARY KEY(id) NOT ENFORCED
)
WITH (
  'connector' = 'hudi',
  'path' = 'hdfs://localhost:9000/user/root/hudi/hudi_mor_data_gen2',
  'table.type' = 'MERGE_ON_READ',
  'read.streaming.enabled' = 'true',
  'write.tasks'= '4',
  'compaction.tasks'= '4',
  'compaction.delta_seconds' = '120',
  'compaction.delta_commits' = '3',
  'read.streaming.check-interval' = '1'
);

第一个表构建的是 delta_commits = 1 ，实际测试发现，当发生一次数据提交的时候，后台就会根据checkpoint去触发一次compaction，符合预期；

第二个表构建的是 delta_commits = 3，实际测试发现，当发生三次数据提交的时候，后台就会根据checkpoint去触发一次compaction，符合预期；

2.4.4、Spark增量模式读取MOR表

import org.apache.spark.SparkConf;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.hudi.DataSourceReadOptions;

public class SparkReadHudi {
    public static void main(String[] args) {

        SparkConf conf = null;
        SparkSession sparkSession = null;

        conf = new SparkConf()
                .setAppName("SparkHudi")
                .set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=file:log4j.properties")
                .set("spark.logConf", "true")
                .set("spark.debug.maxToStringFields", "1000")
                .set("spark.sql.decimalOperations.allowPrecisionLoss", "false")
                .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
                .setMaster("local[*]");

        sparkSession = SparkSession.builder()
                .config(conf)
                .getOrCreate();

        Dataset<Row> hudiDataset = sparkSession.read()
                .format("hudi")
                .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL())
                // 设置增量读取数据时开始时间
                .option(DataSourceReadOptions.BEGIN_INSTANTTIME().key(), "20220413101430946")
                .load("hdfs://localhost:9000/user/root/hudi/hudi_mor_data_gen2");


        hudiDataset.createOrReplaceTempView("t_flink_hudi_table");
        sparkSession.sql("select * from t_flink_hudi_table").show();


    }
}

3、本次Spark代码的pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>hudi-flink-spark</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven-jar-plugin.version>3.2.0</maven-jar-plugin.version>
        <maven-surefire-plugin.version>3.0.0-M4</maven-surefire-plugin.version>
        <maven-failsafe-plugin.version>3.0.0-M4</maven-failsafe-plugin.version>
        <maven-shade-plugin.version>3.2.4</maven-shade-plugin.version>
        <maven-javadoc-plugin.version>3.1.1</maven-javadoc-plugin.version>
        <maven-compiler-plugin.version>3.8.0</maven-compiler-plugin.version>
        <maven-deploy-plugin.version>2.4</maven-deploy-plugin.version>
        <genjavadoc-plugin.version>0.15</genjavadoc-plugin.version>
        <build-helper-maven-plugin.version>1.7</build-helper-maven-plugin.version>
        <maven-enforcer-plugin.version>3.0.0-M1</maven-enforcer-plugin.version>
        <maven-docker-plugin.version>0.37.0</maven-docker-plugin.version>

        <java.version>1.8</java.version>
        <fasterxml.version>2.6.7</fasterxml.version>
        <fasterxml.jackson.databind.version>2.6.7.3</fasterxml.jackson.databind.version>
        <fasterxml.jackson.module.scala.version>2.6.7.1</fasterxml.jackson.module.scala.version>
        <fasterxml.jackson.dataformat.yaml.version>2.7.4</fasterxml.jackson.dataformat.yaml.version>
        <fasterxml.spark3.version>2.10.0</fasterxml.spark3.version>
        <kafka.version>2.3.0</kafka.version>
        <confluent.version>5.3.4</confluent.version>
        <glassfish.version>2.17</glassfish.version>
        <parquet.version>1.10.1</parquet.version>
        <junit.jupiter.version>5.7.0-M1</junit.jupiter.version>
        <junit.vintage.version>5.7.0-M1</junit.vintage.version>
        <junit.platform.version>1.7.0-M1</junit.platform.version>
        <mockito.jupiter.version>3.3.3</mockito.jupiter.version>
        <log4j.version>1.2.17</log4j.version>
        <log4j.test.version>2.17.0</log4j.test.version>
        <slf4j.version>1.7.30</slf4j.version>
        <joda.version>2.9.9</joda.version>
        <hadoop.version>3.3.1</hadoop.version>
        <hive.groupid>org.apache.hive</hive.groupid>
        <hive.version>2.3.1</hive.version>
        <hive.exec.classifier>core</hive.exec.classifier>
        <metrics.version>4.1.1</metrics.version>
        <orc.version>1.6.0</orc.version>
        <airlift.version>0.16</airlift.version>
        <prometheus.version>0.8.0</prometheus.version>
        <http.version>4.4.1</http.version>
        <spark.version>${spark2.version}</spark.version>
        <sparkbundle.version></sparkbundle.version>
        <flink.version>1.12.0</flink.version>
        <spark2.version>2.4.4</spark2.version>
        <spark3.version>3.1.2</spark3.version>
        <hudi.spark.module>hudi-spark2</hudi.spark.module>
        <avro.version>1.8.2</avro.version>
        <scala11.version>2.11.12</scala11.version>
        <scala12.version>2.12.10</scala12.version>
        <scala.version>${scala11.version}</scala.version>
        <scala.binary.version>2.11</scala.binary.version>
        <apache-rat-plugin.version>0.12</apache-rat-plugin.version>
        <scala-maven-plugin.version>3.3.1</scala-maven-plugin.version>
        <scalatest.version>3.0.1</scalatest.version>
        <surefire-log4j.file>file://${project.basedir}/src/test/resources/log4j-surefire.properties
        </surefire-log4j.file>
        <thrift.version>0.12.0</thrift.version>
        <jetty.version>9.4.15.v20190215</jetty.version>
        <htrace.version>3.1.0-incubating</htrace.version>
        <hbase.version>1.2.3</hbase.version>
        <codehaus-jackson.version>1.9.13</codehaus-jackson.version>
        <h2.version>1.4.199</h2.version>
        <awaitility.version>3.1.2</awaitility.version>
        <skipTests>false</skipTests>
        <skipUTs>${skipTests}</skipUTs>
        <skipFTs>${skipTests}</skipFTs>
        <skipITs>${skipTests}</skipITs>
        <skip.hudi-spark2.unit.tests>${skipTests}</skip.hudi-spark2.unit.tests>
        <skip.hudi-spark3.unit.tests>${skipTests}</skip.hudi-spark3.unit.tests>
        <skipDocker>${skipTests}</skipDocker>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <main.basedir>${project.basedir}</main.basedir>
        <spark.bundle.hive.scope>provided</spark.bundle.hive.scope>
        <spark.bundle.hive.shade.prefix/>
        <spark.bundle.avro.scope>compile</spark.bundle.avro.scope>
        <spark.bundle.spark.shade.prefix>org.apache.hudi.spark.</spark.bundle.spark.shade.prefix>
        <utilities.bundle.hive.scope>provided</utilities.bundle.hive.scope>
        <utilities.bundle.hive.shade.prefix/>
        <argLine>-Xmx2g</argLine>
        <jacoco.version>0.8.5</jacoco.version>
        <presto.bundle.bootstrap.scope>compile</presto.bundle.bootstrap.scope>
        <presto.bundle.bootstrap.shade.prefix>org.apache.hudi.</presto.bundle.bootstrap.shade.prefix>
        <shadeSources>true</shadeSources>
        <zk-curator.version>2.7.1</zk-curator.version>
        <antlr.version>4.7</antlr.version>
        <aws.sdk.version>1.12.22</aws.sdk.version>
        <proto.version>3.17.3</proto.version>
        <protoc.version>3.1.0</protoc.version>
        <dynamodb.lockclient.version>1.1.0</dynamodb.lockclient.version>
        <dynamodb-local.port>8000</dynamodb-local.port>
        <dynamodb-local.endpoint>http://localhost:${dynamodb-local.port}</dynamodb-local.endpoint>
        <fastjson.verson>1.2.72</fastjson.verson>
    </properties>

    <dependencies>


        <!--        spark-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hudi</groupId>
            <artifactId>hudi-spark-bundle_2.11</artifactId>
            <version>0.10.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-avro_2.11</artifactId>
            <version>2.4.4</version>
        </dependency>


        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hudi</groupId>
            <artifactId>hudi-common</artifactId>
            <version>0.10.1</version>
        </dependency>

    </dependencies>

</project>

Kafka开源项目指南

Kafka开源项目指南提供详尽教程，助开发者掌握其架构、配置和使用，实现高效数据流管理和实时处理。它高性能、可扩展，适合日志收集和实时数据处理，通过持久化保障数据安全，是企业大数据生态系统的核心。

更多推荐

Kafka入门（一）概述、部署与API的简单使用

Kafka概述、部署与API的简单使用

Kafka开源项目指南

基于canal和kafka同步，实现binlog同步ElasticSearch

文章目录前言elasticsearch 安装canal安装canal-adapter 安装及配置mysql 安装zk及kafaka安装查看效果注意事项前言中间件版本elasticsearch7.5.2canal1.1.4client-adapter1.1.5-alpha-1zookeeper3.4.13kafka2.6.0mysql5.7.31elasticsearch 安装{"settings"

Kafka开源项目指南

基于 Iceberg 的湖仓一体架构在 B 站的实践

背景在B站，每天都有PB级的数据注入到大数据平台，经过离线或实时的ETL建模后，提供给下游的分析、推荐及预测等场景使用。面对如此大规模的数据，如何高效低成本地满足下游数据的分析需求，一直是我们重点的工作方向。我们之前的数据处理流程基本上是这样的：采集端将客户端埋点、服务端埋点、日志、业务数据库等数据收集到HDFS、Kafka等存储系统中，然后通过Hive、Spark、Fl...