Partitions and Memory Usage

replica.fetch.max.bytes 

每个partition都分配一个buffer给replica,如果分配1M,如果有1000个partition就需要1G的内存需求考虑有充足的内存

fetch.message.max.bytes

同样的考虑也适用于consumer，对于大文件需要确保有充足的内存接收，大文件可能需要较少的partition或者更多的内存

Partition Reassignment

当系统的负载过大需要增加broker时，kafka不会自动将数据加载到该broker

需要手动执行

创建想要移动的topic list

topics-to-move.json
{"topics": [{"topic": "foo1"},
            {"topic": "foo2"}],
 "version":1
}

保存为json文件，我们这里是topics-to-move.json

执行 --generate将产生推荐的列表

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 
    --topics-to-move-json-file topics-to-move.json 
    --broker-list "4" 
    --generate

执行完成屏幕将显示推荐的信息

Current partition replica assignment

{"version":1,
 "partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
               {"topic":"foo1","partition":0,"replicas":[3,1]},
               {"topic":"foo2","partition":2,"replicas":[1,2]},
               {"topic":"foo2","partition":0,"replicas":[3,2]},
               {"topic":"foo1","partition":1,"replicas":[2,3]},
               {"topic":"foo2","partition":1,"replicas":[2,3]}]
}

{"version":1,
 "partitions":[{"topic":"foo1","partition":3,"replicas":4},
               {"topic":"foo1","partition":1,"replicas":4},
               {"topic":"foo2","partition":2,"replicas":4}]
}

校验信息并且保存json文件，这里是expand-cluster-reassignment.json

执行动作

bin/kafka-reassign-partitions.sh \
    --zookeeper localhost:2181 \
    --reassignment-json-file expand-cluster-reassignment.json 
    --execute

最后验证结果

bin/kafka-reassign-partitions.sh \
    --zookeeper localhost:2181 \
    --reassignment-json-file expand-cluster-reassignment.json 
    --verify

Garbage Collection