环境准备

jdk1.8、kafka2.11(kafka-manage可选)、zookeeper3.4.10、logstash5.4.6、kibana5.4.6、elasticsearch(head插件,ik中文分词插件)

1、jdk安装

yum install java-1.8.0-openjdk


2、安装kafka

修改配置文件

vi /usr/local/kafka/config/server.properties 

port=9092
log.dirs=/data/kafka
zookeeper.connect=localhost:2181

启用JMX

vi /usr/local/kafka/bin/kafka-server-start.sh 

增加:

export JMX_PORT=${JMX_PORT:-9999}

启动:cd /usr/local/kafka && ./bin/kafka-server-start.sh -daemon config/server.properties 

3、安装zookeeper

配置:

cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg

修改配置:

vi /usr/local/zookeeper/conf/zoo.cfg


dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
server.1=localhost:2888:3888

启动:

/usr/local/zookeeper/bin/zkServer.sh start

4、安装logstash

直接安装rpm

rpm -ivh logstash-5.6.4.rpm

软件目录:/usr/share/logstash

配置目录:/etc/logstash/conf.d

配置文件:/etc/logstash/logstash.yml

日志目录:/var/log/logstash

LS_HOME:/var/lib/logstash


指定文件启动:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstack-template.conf --path.data=/root/


5、安装elasticSearch

直接安装rpm

rpm -ivh elasticsearch-5.6.4.rpm

软件目录:/usr/share/elasticsearch

配置目录:/etc/elasticsearch

日志目录:/var/log/elasticsearch

数据目录:/var/lib/elasticsearch

PID目录:/var/run/elasticsearch


修改配置:

vi /etc/elasticsearch/elasticsearch.yml

#禁用swap
bootstrap.mlockall: true
#绑定所有网卡
network.host: 0.0.0.0
#
# Enable CORS in elasticsearch (elasticsearch-head)
#
http.cors.enabled: true
http.cors.allow-origin: "*"

启动 systemctl start elasticsearch

(head插件和ik可自选)

6、安装kibana

直接rpm安装

rpm -ivh kibana-5.6-4.rpm 

修改配置

vi /etc/kibana/kibana.yml

server.host: "kibana-node"

启动:systemctl start kibana

------------------------------------------------------------------

单机环境搭建完成,可以进入测试阶段,过程中会遇到一些坑,欢迎评论。

Logo

Kafka开源项目指南提供详尽教程,助开发者掌握其架构、配置和使用,实现高效数据流管理和实时处理。它高性能、可扩展,适合日志收集和实时数据处理,通过持久化保障数据安全,是企业大数据生态系统的核心。

更多推荐