kafka安装(二)
1、Create a topiclocalhost:bin jack$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testCreated topic "test".localhost:bin jack$ kafka-topics.sh -
1、Create a topic
localhost:bin jack$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
localhost:bin jack$ kafka-topics.sh --list --zookeeper localhost:2181
test
2、Send some messages
localhost:bin jack$ kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2016-06-22 14:35:14,700] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
This is a message
This is another message
3、Start a consumer
localhost:bin jack$ kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message
如果要最新的数据,可以不带 --from-beginning 参数即可。
4、Setting up a multi-broker cluster(单节点多个Broker)
将kafka_2.10-0.8.2.2文件夹再复制两份分别为kafka_2,kafka_3
scp -r kafka_2.10-0.8.2.2 kafka_2
scp -r kafka_2.10-0.8.2.2 kafka_3
分别修改kafka_2/config/server.properties以及kafka_3/config/server.properties 文件中的broker.id,以及port属性,确保唯一性
kafka_2.10-0.8.2.2/config/server.properties
broker.id=1
port=9092
kafka_2/config/server.properties
broker.id=2
port=9093
kafka_3/config/server.properties
broker.id=3
port=9094
因为多个Broker使用同一个目录,会报错,所以同时修改下log.dirs
log.dirs=/tmp/kafka2-logs
log.dirs=/tmp/kafka3-logs
kafka.common.KafkaException: Failed to acquire lock on file .lock in /tmp/kafka-logs. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98)
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:335)
at kafka.server.KafkaServer.startup(KafkaServer.scala:85)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
启动3个Broker
kafka-server-start.sh ../config/server.properties &
创建一个replication factor为3的topic,并查看Topic的状态
localhost:bin jack$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
Created topic "my-replicated-topic".
localhost:bin jack$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
从上面的内容可以看出,该topic包含1个part,replicationfactor为3,且Node1 是leador
解释如下:
- "leader" is the node responsible for all reads and writes for the given partition. Each node will be the leader for a randomly selected portion of the partitions.
- "replicas" is the list of nodes that replicate the log for this partition regardless of whether they are the leader or even if they are currently alive.
- "isr" is the set of "in-sync" replicas. This is the subset of the replicas list that is currently alive and caught-up to the leader.
Let's publish a few messages to our new topic:
kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
my test message 1
my test message 2
^C
Now let's consume these messages:
kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
...
my test message 1
my test message 2
^C
参考:
http://kafka.apache.org/documentation.html#quickstart_send
http://www.centoscn.com/CentosServer/cluster/2015/0312/4863.html
更多推荐
所有评论(0)