问题背景

前面几篇文章讲了kafka和udp收集日志的方式，本文介绍使用filebeat+sidecars收集本地log文件

注意事项：

• sidecars运行在不同的服务器进行日志采集，发送到graylog-server的服务器进行存储
• 默认已安装JDK
• filebeat+sidecars安装包下载链接，也可以去官网下载最新版
• filebeat下载官网
• sidecars下载官网
• graylog-sidecar官方文档

graylog+kafka+zookeeper（单机测试及源码），graylog组件部署，查找问题分析（一）

graylog+kafka+zookeeper（单机测试及源码），kafka+zookeeper组件部署（二）

graylog+kafka+zookeeper（单机测试及源码），graylog测试用例及源码（三）

graylog+kafka+zookeeper（单机测试及源码），graylog收集kafka（脚本创建发布订阅方式）存储的消息（四）

graylog+kafka+zookeeper（单机测试及源码），graylog设置URL报警方式（五）

graylog+kafka+zookeeper（单机测试及源码），graylog+filebeat+sidecars收集log日志（六）

graylog+kafka+zookeeper（单机测试及源码），微服务日志查询使用（七）

graylog+kafka+zookeeper（单机测试及源码），仪表板Dashboards创建及使用（八）

graylog+kafka+zookeeper（单机测试及源码），indices索引和streams流创建及使用，日志定期删除功能（九）

filebeat和sidecar安装部署以及基本使用方法

1 下载sidecar，选择x86_64版本，https://github.com/Graylog2/collector-sidecar/releases
2 安装sidecar

rpm -ivh graylog-sidecar-1.1.0-1.x86_64.rpm

3 创建token，等一下会使用，点击红框create or reuse a token

4 输入名字，点击create token

5 可以看到成功了，点击复制按钮：12vbtksuen10f8dcvf2dum4ul9qhnb5kj6ouhru7q2gjsm53kjbu

6 显示Never used

7 更改sidecar配置

vim /etc/graylog/sidecar/sidecar.yml

# The URL to the Graylog server API.
server_url: "http://10.10.196.242:9000/api/"

# The API token to use to authenticate against the Graylog server API.
# This field is mandatory
server_api_token: "12vbtksuen10f8dcvf2dum4ul9qhnb5kj6ouhru7q2gjsm53kjbu"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:/etc/graylog/sidecar/node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
#node_id: "file:/etc/graylog/sidecar/node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
node_name: "authentication"

# The update interval in seconds. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
#tls_skip_verify: false

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []

# Directory where the sidecar stores internal data.
#cache_path: "/var/cache/graylog-sidecar"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "/var/log/graylog-sidecar"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "/var/lib/graylog-sidecar/generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_accesslist:
#       - "/usr/bin/filebeat"
#       - "/opt/collectors/*"
#
# Example disable access listing:
#     collector_binaries_accesslist: []
#
# Default:
# collector_binaries_accesslist:
#  - "/usr/bin/filebeat"
#  - "/usr/bin/packetbeat"
#  - "/usr/bin/metricbeat"
#  - "/usr/bin/heartbeat"
#  - "/usr/bin/auditbeat"
#  - "/usr/bin/journalbeat"
#  - "/usr/share/filebeat/bin/filebeat"
#  - "/usr/share/packetbeat/bin/packetbeat"
#  - "/usr/share/metricbeat/bin/metricbeat"
#  - "/usr/share/heartbeat/bin/heartbeat"
#  - "/usr/share/auditbeat/bin/auditbeat"
#  - "/usr/share/journalbeat/bin/journalbeat"
#  - "/usr/bin/nxlog"
#  - "/opt/nxlog/bin/nxlog"

主要更改以下几项

server_url: "http://10.10.196.242:9000/api/"    # api的graylog外网地址
# api token 必要的，不然启动不了，token需要在web界面上进行手动创建
server_api_token: "12vbtksuen10f8dcvf2dum4ul9qhnb5kj6ouhru7q2gjsm53kjbu"
node_name: "authentication"    # 自定义节点名称
update_interval: 10
send_status: true

8 安装sidescar

graylog-sidecar -service install

9 启动sidecar

systemctl start graylog-sidecar

10 查看sidecar

ps -ef | grep sidecar

11 下载filebeat，选择x86_64版本，https://www.elastic.co/cn/downloads/beats/filebeat

12 安装filebeat

rpm -ivh filebeat-7.16.2-x86_64.rpm

13 创建beat的input输入，使用filebeat采集

14 选择输入模式，点击launch new input

15 设置beats名字和端口，选择global，一开始Active connections为0，等后面的所有配置完成后，会变成1，可以点击show received messages直接查看接收的消息

16 添加收集配置项，选择configuration

17 点击create configuration

18 输入名字，选择filebeat on linux

19 填写收集的log文件路径，点击create

20 显示成功创建

21 点击overview，选择manage sidecar

22 选择配置

23 点击confirm

24 显示成功

25 使用sidecar的collector收集器查看日志

26 查看日志

心得

• 网上没有太多的资料，要学会看官方文档

作为程序员第 22 篇文章，每次写一句歌词记录一下，看看人生有几首歌的时间，wahahaha …

Lyric： 摘下麦芽糖熟透