转载自：http://blog.csdn.net/xueba207/article/details/51135423

问题描述

笔者使用Spark streaming读取Kakfa中的数据，做进一步处理，用到了KafkaUtil的createDirectStream()方法；该方法不会自动保存topic partition的offset到zk，需要在代码中编写提交逻辑，此处介绍了保存offset的方法。 
当删除已经使用过的kafka topic，然后新建同名topic，使用该方式时出现了"numRecords must not be negative"异常 
详细信息如下图: 
异常截图
是不合法的参数异常，RDD的记录数目必须不能是负数。 
下文详细分析该问题的出现的场景，以及解决方法。

异常分析

numRecords确定

首先，定位出异常出现的问题，和大致原因。异常中打印出了出现的位置org.apache.spark.streaming.scheduler.StreamInputInfo.InputInfoTracker的第38行，此处代码：

InputInfoTracker

代码38行，判断了numRecords是否大于等于0，当不满足条件时抛出异常，可判断此时numRecords<0。 
numRecords的解释： 
numRecords: the number of records in a batch 
应该是当前rdd中records 数目计算出了问题。 
numRecords 构造StreamInputInfo时的参数，结合异常中的信息，找到了DirectKafkaInputDStream中的构造InputInfo的位置： 
DirectKafkaInputDStream

可知 numRecords是rdd.count()的值。

rdd.count的计算

根据以上分析可知rdd.count()值为负值，因此需要分析rdd的是如何生成的。 
同样在DirectKafkaInputDStream中找到rdd的生成代码：

create Kafka rdd

从此处一路跟踪代码，可在KafkaRDD.scala中找到rdd.count的赋值逻辑：

KafkaRDD.count

offsetRanges的计算逻辑

offsetRanges的定义

offsetRanges: offset ranges that define the Kafka data belonging to this RDD

在KafkaRDDPartition 40行找到kafka partition offsetRange的计算逻辑：

def count(): Long = untilOffset - fromOffset 
fromOffset： per-topic/partition Kafka offset defining the (inclusive) starting point of the batch 
untilOffset： per-topic/partition Kafka offset defining the (inclusive) ending point of the batch

fromOffset来自zk中保存； 
untilOffset通过DirectKafkaInputDStream第145行：

val untilOffsets = clamp(latestLeaderOffsets(maxRetries))

计算得到，计算过程得到最新的offset，然后使用spark.streaming.kafka.maxRatePerPartition做clamp，得到允许的最大untilOffsets，##而此时新建的topic，如果topic中没有数据，untilOffsets应该为0##

原因总结

当删除一个topic时，zk中的offset信息并没有被清除，因此KafkaDirectStreaming再次启动时仍会得到旧的topic offset为old_offset，作为fromOffset。 
当新建了topic后，使用untiloffset计算逻辑，得到untilOffset为0（如果topic已有数据则>0）； 
再次被启动的KafkaDirectStreaming Job通过异常的计算逻辑得到的rdd numRecords值为可计算为： 
numRecords = untilOffset - fromOffset(old_offset) 
当untilOffset < old_offset时，此异常会出现，对于新建的topic这种情况的可能性很大

解决方法

思路

根据以上分析，可在确定KafkaDirectStreaming 的fromOffsets时判断fromOffset与untiloffset的大小关系，当untilOffset < fromOffset时，矫正fromOffset为offset初始值0。

流程

  • 从zk获取topic/partition 的fromOffset（获取方法链接）
  • 利用SimpleConsumer获取每个partiton的lastOffset（untilOffset ）
  • 判断每个partition lastOffset与fromOffset的关系
  • 当lastOffset < fromOffset时，将fromOffset赋值为0 
    通过以上步骤完成fromOffset的值矫正。


获取kafka topic partition lastoffset代码：

<code class="language-java hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> org.frey.example.utils.kafka;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> com.google.common.collect.Lists;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> com.google.common.collect.Maps;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> kafka.api.PartitionOffsetRequestInfo;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> kafka.cluster.Broker;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> kafka.common.TopicAndPartition;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> kafka.javaapi.*;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> kafka.javaapi.consumer.SimpleConsumer;

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Date;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.HashMap;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.List;
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> java.util.Map;

<span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
 * KafkaOffsetTool
 *
 *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @author</span> v1-daddy
 *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @date</span> 2016/4/11
 */</span>
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">KafkaOffsetTool</span> {</span>

  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> KafkaOffsetTool instance;
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> TIMEOUT = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100000</span>;
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">final</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> BUFFERSIZE = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">64</span> * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1024</span>;

  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-title" style="box-sizing: border-box;">KafkaOffsetTool</span>() {
  }

  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">synchronized</span> KafkaOffsetTool <span class="hljs-title" style="box-sizing: border-box;">getInstance</span>() {
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (instance == <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>) {
      instance = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> KafkaOffsetTool();
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> instance;
  }

  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> Map<TopicAndPartition, Long> <span class="hljs-title" style="box-sizing: border-box;">getLastOffset</span>(String brokerList, List<String> topics,
      String groupId) {

    Map<TopicAndPartition, Long> topicAndPartitionLongMap = Maps.newHashMap();

    Map<TopicAndPartition, Broker> topicAndPartitionBrokerMap =
        KafkaOffsetTool.getInstance().findLeader(brokerList, topics);

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (Map.Entry<TopicAndPartition, Broker> topicAndPartitionBrokerEntry : topicAndPartitionBrokerMap
        .entrySet()) {
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// get leader broker</span>
      Broker leaderBroker = topicAndPartitionBrokerEntry.getValue();

      SimpleConsumer simpleConsumer = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SimpleConsumer(leaderBroker.host(), leaderBroker.port(),
          TIMEOUT, BUFFERSIZE, groupId);

      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> readOffset = getTopicAndPartitionLastOffset(simpleConsumer,
          topicAndPartitionBrokerEntry.getKey(), groupId);

      topicAndPartitionLongMap.put(topicAndPartitionBrokerEntry.getKey(), readOffset);

    }

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> topicAndPartitionLongMap;

  }

  <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
   * 得到所有的 TopicAndPartition
   *
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> brokerList
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> topics
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @return</span> topicAndPartitions
   */</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> Map<TopicAndPartition, Broker> <span class="hljs-title" style="box-sizing: border-box;">findLeader</span>(String brokerList, List<String> topics) {
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// get broker's url array</span>
    String[] brokerUrlArray = getBorkerUrlFromBrokerList(brokerList);
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// get broker's port map</span>
    Map<String, Integer> brokerPortMap = getPortFromBrokerList(brokerList);

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// create array list of TopicAndPartition</span>
    Map<TopicAndPartition, Broker> topicAndPartitionBrokerMap = Maps.newHashMap();

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (String broker : brokerUrlArray) {

      SimpleConsumer consumer = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>;
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> {
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// new instance of simple Consumer</span>
        consumer = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SimpleConsumer(broker, brokerPortMap.get(broker), TIMEOUT, BUFFERSIZE,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"leaderLookup"</span> + <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Date().getTime());

        TopicMetadataRequest req = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> TopicMetadataRequest(topics);

        TopicMetadataResponse resp = consumer.send(req);

        List<TopicMetadata> metaData = resp.topicsMetadata();

        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (TopicMetadata item : metaData) {
          <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (PartitionMetadata part : item.partitionsMetadata()) {
            TopicAndPartition topicAndPartition =
                <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> TopicAndPartition(item.topic(), part.partitionId());
            topicAndPartitionBrokerMap.put(topicAndPartition, part.leader());
          }
        }
      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">catch</span> (Exception e) {
        e.printStackTrace();
      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">finally</span> {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (consumer != <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">null</span>)
          consumer.close();
      }
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> topicAndPartitionBrokerMap;
  }

  <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
   * get last offset
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> consumer
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> topicAndPartition
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> clientName
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @return</span>
   */</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span> <span class="hljs-title" style="box-sizing: border-box;">getTopicAndPartitionLastOffset</span>(SimpleConsumer consumer,
      TopicAndPartition topicAndPartition, String clientName) {
    Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo =
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> HashMap<TopicAndPartition, PartitionOffsetRequestInfo>();

    requestInfo.put(topicAndPartition, <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> PartitionOffsetRequestInfo(
        kafka.api.OffsetRequest.LatestTime(), <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>));

    OffsetRequest request = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> OffsetRequest(
        requestInfo, kafka.api.OffsetRequest.CurrentVersion(),
        clientName);

    OffsetResponse response = consumer.getOffsetsBefore(request);

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (response.hasError()) {
      System.out
          .println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Error fetching data Offset Data the Broker. Reason: "</span>
              + response.errorCode(topicAndPartition.topic(), topicAndPartition.partition()));
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>;
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">long</span>[] offsets = response.offsets(topicAndPartition.topic(), topicAndPartition.partition());
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> offsets[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>];
  }
  <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
   * 得到所有的broker url
   *
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> brokerlist
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @return</span>
   */</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> String[] <span class="hljs-title" style="box-sizing: border-box;">getBorkerUrlFromBrokerList</span>(String brokerlist) {
    String[] brokers = brokerlist.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>);
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">int</span> i = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>; i < brokers.length; i++) {
      brokers[i] = brokers[i].split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">":"</span>)[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>];
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> brokers;
  }

  <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
   * 得到broker url 与 其port 的映射关系
   *
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @param</span> brokerlist
   *<span class="hljs-javadoctag" style="color: rgb(102, 0, 102); box-sizing: border-box;"> @return</span>
   */</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">private</span> Map<String, Integer> <span class="hljs-title" style="box-sizing: border-box;">getPortFromBrokerList</span>(String brokerlist) {
    Map<String, Integer> map = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> HashMap<String, Integer>();
    String[] brokers = brokerlist.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>);
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (String item : brokers) {
      String[] itemArr = item.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">":"</span>);
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (itemArr.length > <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) {
        map.put(itemArr[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>], Integer.parseInt(itemArr[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]));
      }
    }
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">return</span> map;
  }

  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">public</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">static</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">void</span> <span class="hljs-title" style="box-sizing: border-box;">main</span>(String[] args) {
    List<String> topics = Lists.newArrayList();
    topics.add(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"ys"</span>);
    topics.add(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bugfix"</span>);
    Map<TopicAndPartition, Long> topicAndPartitionLongMap =
        KafkaOffsetTool.getInstance().getLastOffset(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"broker001:9092,broker002:9092"</span>, topics, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my.group.id"</span>);

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (Map.Entry<TopicAndPartition, Long> entry : topicAndPartitionLongMap.entrySet()) {
     System.out.println(entry.getKey().topic() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"-"</span>+ entry.getKey().partition() + <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">":"</span> + entry.getValue());
    }
  }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li><li style="box-sizing: border-box; padding: 0px 5px;">142</li><li style="box-sizing: border-box; padding: 0px 5px;">143</li><li style="box-sizing: border-box; padding: 0px 5px;">144</li><li style="box-sizing: border-box; padding: 0px 5px;">145</li><li style="box-sizing: border-box; padding: 0px 5px;">146</li><li style="box-sizing: border-box; padding: 0px 5px;">147</li><li style="box-sizing: border-box; padding: 0px 5px;">148</li><li style="box-sizing: border-box; padding: 0px 5px;">149</li><li style="box-sizing: border-box; padding: 0px 5px;">150</li><li style="box-sizing: border-box; padding: 0px 5px;">151</li><li style="box-sizing: border-box; padding: 0px 5px;">152</li><li style="box-sizing: border-box; padding: 0px 5px;">153</li><li style="box-sizing: border-box; padding: 0px 5px;">154</li><li style="box-sizing: border-box; padding: 0px 5px;">155</li><li style="box-sizing: border-box; padding: 0px 5px;">156</li><li style="box-sizing: border-box; padding: 0px 5px;">157</li><li style="box-sizing: border-box; padding: 0px 5px;">158</li><li style="box-sizing: border-box; padding: 0px 5px;">159</li><li style="box-sizing: border-box; padding: 0px 5px;">160</li><li style="box-sizing: border-box; padding: 0px 5px;">161</li><li style="box-sizing: border-box; padding: 0px 5px;">162</li><li style="box-sizing: border-box; padding: 0px 5px;">163</li><li style="box-sizing: border-box; padding: 0px 5px;">164</li><li style="box-sizing: border-box; padding: 0px 5px;">165</li><li style="box-sizing: border-box; padding: 0px 5px;">166</li><li style="box-sizing: border-box; padding: 0px 5px;">167</li><li style="box-sizing: border-box; padding: 0px 5px;">168</li><li style="box-sizing: border-box; padding: 0px 5px;">169</li><li style="box-sizing: border-box; padding: 0px 5px;">170</li><li style="box-sizing: border-box; padding: 0px 5px;">171</li><li style="box-sizing: border-box; padding: 0px 5px;">172</li><li style="box-sizing: border-box; padding: 0px 5px;">173</li><li style="box-sizing: border-box; padding: 0px 5px;">174</li><li style="box-sizing: border-box; padding: 0px 5px;">175</li><li style="box-sizing: border-box; padding: 0px 5px;">176</li><li style="box-sizing: border-box; padding: 0px 5px;">177</li><li style="box-sizing: border-box; padding: 0px 5px;">178</li><li style="box-sizing: border-box; padding: 0px 5px;">179</li><li style="box-sizing: border-box; padding: 0px 5px;">180</li><li style="box-sizing: border-box; padding: 0px 5px;">181</li><li style="box-sizing: border-box; padding: 0px 5px;">182</li><li style="box-sizing: border-box; padding: 0px 5px;">183</li><li style="box-sizing: border-box; padding: 0px 5px;">184</li><li style="box-sizing: border-box; padding: 0px 5px;">185</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li><li style="box-sizing: border-box; padding: 0px 5px;">90</li><li style="box-sizing: border-box; padding: 0px 5px;">91</li><li style="box-sizing: border-box; padding: 0px 5px;">92</li><li style="box-sizing: border-box; padding: 0px 5px;">93</li><li style="box-sizing: border-box; padding: 0px 5px;">94</li><li style="box-sizing: border-box; padding: 0px 5px;">95</li><li style="box-sizing: border-box; padding: 0px 5px;">96</li><li style="box-sizing: border-box; padding: 0px 5px;">97</li><li style="box-sizing: border-box; padding: 0px 5px;">98</li><li style="box-sizing: border-box; padding: 0px 5px;">99</li><li style="box-sizing: border-box; padding: 0px 5px;">100</li><li style="box-sizing: border-box; padding: 0px 5px;">101</li><li style="box-sizing: border-box; padding: 0px 5px;">102</li><li style="box-sizing: border-box; padding: 0px 5px;">103</li><li style="box-sizing: border-box; padding: 0px 5px;">104</li><li style="box-sizing: border-box; padding: 0px 5px;">105</li><li style="box-sizing: border-box; padding: 0px 5px;">106</li><li style="box-sizing: border-box; padding: 0px 5px;">107</li><li style="box-sizing: border-box; padding: 0px 5px;">108</li><li style="box-sizing: border-box; padding: 0px 5px;">109</li><li style="box-sizing: border-box; padding: 0px 5px;">110</li><li style="box-sizing: border-box; padding: 0px 5px;">111</li><li style="box-sizing: border-box; padding: 0px 5px;">112</li><li style="box-sizing: border-box; padding: 0px 5px;">113</li><li style="box-sizing: border-box; padding: 0px 5px;">114</li><li style="box-sizing: border-box; padding: 0px 5px;">115</li><li style="box-sizing: border-box; padding: 0px 5px;">116</li><li style="box-sizing: border-box; padding: 0px 5px;">117</li><li style="box-sizing: border-box; padding: 0px 5px;">118</li><li style="box-sizing: border-box; padding: 0px 5px;">119</li><li style="box-sizing: border-box; padding: 0px 5px;">120</li><li style="box-sizing: border-box; padding: 0px 5px;">121</li><li style="box-sizing: border-box; padding: 0px 5px;">122</li><li style="box-sizing: border-box; padding: 0px 5px;">123</li><li style="box-sizing: border-box; padding: 0px 5px;">124</li><li style="box-sizing: border-box; padding: 0px 5px;">125</li><li style="box-sizing: border-box; padding: 0px 5px;">126</li><li style="box-sizing: border-box; padding: 0px 5px;">127</li><li style="box-sizing: border-box; padding: 0px 5px;">128</li><li style="box-sizing: border-box; padding: 0px 5px;">129</li><li style="box-sizing: border-box; padding: 0px 5px;">130</li><li style="box-sizing: border-box; padding: 0px 5px;">131</li><li style="box-sizing: border-box; padding: 0px 5px;">132</li><li style="box-sizing: border-box; padding: 0px 5px;">133</li><li style="box-sizing: border-box; padding: 0px 5px;">134</li><li style="box-sizing: border-box; padding: 0px 5px;">135</li><li style="box-sizing: border-box; padding: 0px 5px;">136</li><li style="box-sizing: border-box; padding: 0px 5px;">137</li><li style="box-sizing: border-box; padding: 0px 5px;">138</li><li style="box-sizing: border-box; padding: 0px 5px;">139</li><li style="box-sizing: border-box; padding: 0px 5px;">140</li><li style="box-sizing: border-box; padding: 0px 5px;">141</li><li style="box-sizing: border-box; padding: 0px 5px;">142</li><li style="box-sizing: border-box; padding: 0px 5px;">143</li><li style="box-sizing: border-box; padding: 0px 5px;">144</li><li style="box-sizing: border-box; padding: 0px 5px;">145</li><li style="box-sizing: border-box; padding: 0px 5px;">146</li><li style="box-sizing: border-box; padding: 0px 5px;">147</li><li style="box-sizing: border-box; padding: 0px 5px;">148</li><li style="box-sizing: border-box; padding: 0px 5px;">149</li><li style="box-sizing: border-box; padding: 0px 5px;">150</li><li style="box-sizing: border-box; padding: 0px 5px;">151</li><li style="box-sizing: border-box; padding: 0px 5px;">152</li><li style="box-sizing: border-box; padding: 0px 5px;">153</li><li style="box-sizing: border-box; padding: 0px 5px;">154</li><li style="box-sizing: border-box; padding: 0px 5px;">155</li><li style="box-sizing: border-box; padding: 0px 5px;">156</li><li style="box-sizing: border-box; padding: 0px 5px;">157</li><li style="box-sizing: border-box; padding: 0px 5px;">158</li><li style="box-sizing: border-box; padding: 0px 5px;">159</li><li style="box-sizing: border-box; padding: 0px 5px;">160</li><li style="box-sizing: border-box; padding: 0px 5px;">161</li><li style="box-sizing: border-box; padding: 0px 5px;">162</li><li style="box-sizing: border-box; padding: 0px 5px;">163</li><li style="box-sizing: border-box; padding: 0px 5px;">164</li><li style="box-sizing: border-box; padding: 0px 5px;">165</li><li style="box-sizing: border-box; padding: 0px 5px;">166</li><li style="box-sizing: border-box; padding: 0px 5px;">167</li><li style="box-sizing: border-box; padding: 0px 5px;">168</li><li style="box-sizing: border-box; padding: 0px 5px;">169</li><li style="box-sizing: border-box; padding: 0px 5px;">170</li><li style="box-sizing: border-box; padding: 0px 5px;">171</li><li style="box-sizing: border-box; padding: 0px 5px;">172</li><li style="box-sizing: border-box; padding: 0px 5px;">173</li><li style="box-sizing: border-box; padding: 0px 5px;">174</li><li style="box-sizing: border-box; padding: 0px 5px;">175</li><li style="box-sizing: border-box; padding: 0px 5px;">176</li><li style="box-sizing: border-box; padding: 0px 5px;">177</li><li style="box-sizing: border-box; padding: 0px 5px;">178</li><li style="box-sizing: border-box; padding: 0px 5px;">179</li><li style="box-sizing: border-box; padding: 0px 5px;">180</li><li style="box-sizing: border-box; padding: 0px 5px;">181</li><li style="box-sizing: border-box; padding: 0px 5px;">182</li><li style="box-sizing: border-box; padding: 0px 5px;">183</li><li style="box-sizing: border-box; padding: 0px 5px;">184</li><li style="box-sizing: border-box; padding: 0px 5px;">185</li></ul>

矫正offset核心代码：

<code class="language-java hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">    <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** 以下 矫正 offset */</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 得到Topic/partition 的lastOffsets</span>
    Map<TopicAndPartition, Long> topicAndPartitionLongMap =
        KafkaOffsetTool.getInstance().getLastOffset(kafkaParams.get(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"metadata.broker.list"</span>),
            topicList, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"my.group.id"</span>);

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 遍历每个Topic.partition</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> (Map.Entry<TopicAndPartition, Long> topicAndPartitionLongEntry : fromOffsets.entrySet()) {
      <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// fromOffset > lastOffset时</span>
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (topicAndPartitionLongEntry.getValue() >
          topicAndPartitionLongMap.get(topicAndPartitionLongEntry.getKey())) {
         <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//矫正fromoffset为offset初始值0</span>
        topicAndPartitionLongEntry.setValue(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>L);
      }
    }
    <span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/** 以上 矫正 offset */</span></code>

Logo
Kafka开源项目指南

Kafka开源项目指南提供详尽教程，助开发者掌握其架构、配置和使用，实现高效数据流管理和实时处理。它高性能、可扩展，适合日志收集和实时数据处理，通过持久化保障数据安全，是企业大数据生态系统的核心。

更多推荐

Kafka入门（一） 概述、部署与API的简单使用

Kafka概述、部署与API的简单使用

avatar Kafka开源项目指南

基于canal和kafka同步，实现binlog同步ElasticSearch

文章目录前言elasticsearch 安装canal安装canal-adapter 安装及配置mysql 安装zk及kafaka安装查看效果注意事项前言中间件版本elasticsearch7.5.2canal1.1.4client-adapter1.1.5-alpha-1zookeeper3.4.13kafka2.6.0mysql5.7.31elasticsearch 安装{"settings"

avatar Kafka开源项目指南

基于 Iceberg 的湖仓一体架构在 B 站的实践

背景在B站，每天都有PB级的数据注入到大数据平台，经过离线或实时的ETL建模后，提供给下游的分析、推荐及预测等场景使用。面对如此大规模的数据，如何高效低成本地满足下游数据的分析需求，一直是我们重点的工作方向。我们之前的数据处理流程基本上是这样的：采集端将客户端埋点、服务端埋点、日志、业务数据库等数据收集到HDFS、Kafka等存储系统中，然后通过Hive、Spark、Fl...

avatar Kafka开源项目指南