fluentd

Fluentd 安装配置

1. Fluentd配置文件详解1
2. Fluentd配置文件详解2和k8s部署
3. Fluentd配置grep用法详解
4. Fluentd配置用法详解es

前提需要创建下面两个目录(路径可自定义)

mkdir -p /home/fluentd/container-logs  /home/fluentd/conf/

制作含有elasticsearch插件的fluentd镜像

创建Dockerfile 方法1

FROM fluentd:latest
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch"]

创建Dockerfile 方法2

FROM fluentd:latest
RUN fluent-gem install fluent-plugin-elasticsearch --user-install

创建Dockerfile 方法3 (这个对接es 7.*)

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "7.10.1"]
RUN gem install excon -v 0.109.0
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.3"]
USER root

创建Dockerfile 方法4

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN  gem install faraday-net_http -v 3.0.2
RUN  gem install faraday -v 2.8.1
RUN  gem install excon -v 0.109.0
RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "7.10.1"]
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.3"]
USER fluent

创建Dockerfile 方法5 (这个对接es 8.*)

FROM fluentd
USER root
RUN gem install fluent-plugin-elasticsearch \
fluent-plugin-tail-ex \
fluent-plugin-tail-multiline

USER root
#多留个端口出来供以后转发用
EXPOSE 24224 24224/udp 25225

制作镜像

docker build -t fluent .

值得注意的是

fluent-plugin-elasticsearch 插件的版本应该和 elasticsearch 对应，我这边使用的elasticsearch 为7.10.1 则 fluent-plugin-elasticsearch 应该在 5.0.3，版本无法对应的话，fluent 启动时 将报 无法连接至elasticsearch，如下:

报错

"The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product."

启动服务

Docker启动

docker启动命令

docker run -d \
    -p 24224:24224 \
    -v /home/fluentd/container-logs:/fluentd/log \
    -v /home/fluentd/conf/fluent.conf:/fluentd/etc/fluent.conf \   
    fluentd:latest

Docker-Compose启动

首先创建docker-compose.yml文件

version: '3'
services:

  server-fluentd:
    image: fluentd:latest
    container_name: server-fluentd
    ports:
      - "24224:24224"
    restart: always
    privileged: true
    environment:
      - FLUENTD_CONF=fluentd.conf
    volumes:
      -  ./log:/var/log  # 如果是tail方式,flaunted配置文件中的日志路径应该是这个容器中的路径
      -  ./fluentd.conf:/fluentd/etc/fluentd.conf

docker-compose 启动命令

 docker-compose up -d

 #docker 启动命令
sudo docker run \
-d -p 24224:24224  \
-v /home/fluentd/container-logs:/fluentd/log \
-v /home/fluentd/conf/fluent.conf:/fluentd/etc/fluent.conf  \  
fluentd:latest

编辑配置文件后重启服务

vi /home/fluentd/conf/fluent.conf

 <source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *>
  @type elasticsearch
  host 43.143.134.152
  port 9200
  index_name fluentd.${tag}
  <buffer>
    flush_interval 5s
  </buffer>
</match>

其他容器启动命令参考

docker run -d \
--log-driver fluentd \
--log-opt fluentd-address=localhost:24224 \
--log-opt tag="nginx-test" \
--log-opt fluentd-async-connect \
--name nginx-test \
-p 8080:80 \
nginx

#--log-driver: 配置log驱动
#--log-opt: 配置log相关的参数
#tag：配置match tag
#fluentd-address: fluentd服务地址
#fluentd-async-connect：fluentd-docker异步设置,避免fluentd挂掉之后导致Docker容器也挂了