Docker ELK日志系统搭建（单机）

一、Docker镜像列表

组件名称	版本	说明	拉取镜像命令
elasticsearch	7.17.9	日志分析、检索	docker pull elasticsearch:7.17.9
logstash	7.17.9	日志收集	docker pull logstash:7.17.9
kibana	7.17.9	ES可视化	docker pull kibana:7.17.9
Nginx	1.21.1	http代理	docker pull nginx

二、创建Docker网络

docker network create --driver bridge biznet

三、Docker部署，初始化docker-compose.yml

部署的步骤最终采用docker-compose.yml方式，集中部署容器。
步骤都是类似：

• 先对各个容器初始化执行一次，获取一些必要的目录和文件，将其复制到宿主机当中
• 删除刚刚创建的容器
• 在宿主机当中创建映射所需的目录和文件，并配置到docker-compose.yml文件当中

1、nginx部署

初次执行，获取相关的目录和文件

docker run -d --name nginx -p 8080:80 nginx

nginx所需的文件和目录如下

• 配置文件目录/etc/nginx/conf.d
• 自定义的证书文件目录/etc/nginx/cert
• 日志文件目录/var/log/nginx
• html静态资源目录/usr/share/nginx/html
• web网站资源目录/var/www

docker cp elasticsearch:/etc/nginx/conf.d /data/elk/nginx/conf.d
docker cp elasticsearch:/etc/nginx/cert /data/elk/nginx/cert
docker cp elasticsearch:/var/log/nginx /data/elk/nginx/log
docker cp elasticsearch:/usr/share/nginx/html /data/elk/nginx/html
docker cp elasticsearch:/var/www /data/elk/nginx/www

删除nginx容器

docker rm -f nginx

2、ES部署

初次执行，获取相关的目录和文件

docker run -it --privileged=true -d -p 9200:9200 -p 9300:9300 --name elasticsearch --net elk_net --ip 172.18.0.2  -e ES_JAVA_OPTS="-Xms512m -Xmx512m" -e "discovery.type=single-node" elasticsearch:7.17.9

ES所需的文件和目录如下

• 配置文件存放目录 /usr/share/elasticsearch/config
• ES日志存放目录 /usr/share/elasticsearch/logs
• ES数据存放目录 /usr/share/elasticsearch/data

#docker cp 容器名:容器内部需要拷贝的文件地址 本地存放地址
docker cp elasticsearch:/usr/share/elasticsearch/config /data/elk/es/
docker cp elasticsearch:/usr/share/elasticsearch/logs /data/elk/es/
docker cp elasticsearch:/usr/share/elasticsearch/data /data/elk/es/

删除ES容器

#删除elasticsearch容器
docker rm -f elasticsearch

3、kibana部署

初次执行，获取相关的目录和文件

docker run -d  --privileged=true --name kibana -p 5601:5601 --net elk_net --ip 172.18.0.4 kibana:7.17.9

kibana所需文件和目录如下

• 配置目录/usr/share/kibana/config
• 配置文件/usr/share/kibana/config/kibana.yml

docker cp kibana:/usr/share/kibana/config/kibana.yml /data/elk/kibana/config/

删除Kibana容器

docker rm -f kibana

4、logstash部署

初次执行，获取相关的目录和文件

docker run -d -it --privileged=true --name=logstash  --net elk_net  --ip 172.18.0.3 -p 5047:5047 -p 9600:9600 logstash:7.17.9

kibana所需文件和目录如下：

• 配置文件存放目录/usr/share/logstash/config
• 数据文件存放目录/usr/share/logstash/data
• 管道相关存放目录/usr/share/logstash/pipeline

docker cp logstash:/usr/share/logstash/config /data/elk/logstash/
docker cp logstash:/usr/share/logstash/data /data/elk/logstash/
docker cp logstash:/usr/share/logstash/pipeline /data/elk/logstash/

删除logstash容器

docker rm -f logstash

5、初始化docker-compose.yml文件

version: '1.0'
services:
  nginx:
    # 容器的名称
    container_name: nginx
    # 使用的镜像名称
    image: nginx
    # 当 docker 服务启动后，自动启动 nginx 容器
    restart: always
    # 使用自定义的网络
    networks:
      - biznet
    # 对外映射的端口 外部端口:内部端口
    ports:
     - "80:80"
     - "443:443"
    # network_mode: host
    environment:
      - NGINX_PORT=80
      - TZ=Asia/Shanghai
    privileged: true
    # 宿主机硬盘目录映射容器内的存放数据的目录
    volumes:
      - /opt/docker/nginx/conf.d:/etc/nginx/conf.d
      - /opt/docker/nginx/cert:/etc/nginx/cert
      - /opt/docker/nginx/logs:/var/log/nginx
      #- /opt/docker/nginx/html:/usr/share/nginx/html
      - /opt/docker/nginx/www:/var/www
  elasticsearch:
    # 容器的名称
    container_name: elasticsearch
    # 使用的镜像名称
    image: elasticsearch:7.17.9
    # 当 docker 服务启动后，自动启动容器
    restart: always
    # 使用自定义的网络
    networks:
      - biznet
    # 对外映射的端口 外部端口:内部端口
    ports:
     - "9200:9200"
     - "9300:9300"
    environment:
      environment:
      - "ELASTICSEARCH_ENABLE_REST_TLS=false"
      - "ELASTICSEARCH_SKIP_TRANSPORT_TLS=true"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.type=single-node"
      - "ELASTICSEARCH_PASSWORD=elk@123"
      - "TAKE_FILE_OWNERSHIP=true"
    privileged: true
    # 宿主机硬盘目录映射容器内的存放数据的目录
    volumes:
      - /opt/docker/elk/es/config:/usr/share/elasticsearch/config
      - /opt/docker/elk/es/logs:/usr/share/elasticsearch/logs
      - /opt/docker/elk/es/data:/usr/share/elasticsearch/data
  kibana:
    # 容器的名称
    container_name: kibana
    # 使用的镜像名称
    image: kibana:7.17.9
    # 当 docker 服务启动后，自动启动容器
    restart: always
    # 使用自定义的网络
    networks:
      - biznet
    # 对外映射的端口 外部端口:内部端口
    ports:
     - "5601:5601"
    environment: 
      #- "KIBANA_ELASTICSEARCH_URL=elasticsearch"
      #- "KIBANA_ELASTICSEARCH_PORT_NUMBER=9200"
      #- "KIBANA_HOST=0.0.0.0"
      - "KIBANA_PORT_NUMBER=5601"
      #- "KIBANA_CREATE_USER=true"
      #- "KIBANA_PASSWORD=elk@123"
      #- "KIBANA_ELASTICSEARCH_PASSWORD=elk@123"
      #- "KIBANA_VOLUME_DIR=/usr/share/kibana/kibana"
    privileged: true
    # 宿主机硬盘目录映射容器内的存放数据的目录
    volumes:
      - /opt/docker/elk/kibana/config:/usr/share/kibana/config
  logstash:
    # 容器的名称
    container_name: logstash
    # 使用的镜像名称
    image: logstash:7.17.9
    # 当 docker 服务启动后，自动启动容器
    restart: always
    # 使用自定义的网络
    networks:
      - biznet
    # 对外映射的端口 外部端口:内部端口
    ports:
     - "5047:5047"
     - "9600:9600"
    privileged: true
    # 宿主机硬盘目录映射容器内的存放数据的目录
    volumes:
      - /opt/docker/elk/logstash/config:/usr/share/logstash/config
      - /opt/docker/elk/logstash/data:/usr/share/logstash/data
      - /opt/docker/elk/logstash/pipeline:/usr/share/logstash/pipeline
      # 将宿主机nginx日志所在目录映射到logstash容器中，用作nginx日志收集
      - /opt/docker/nginx/logs:/usr/share/logs/nginx

networks:
  biznet:
    driver: bridge

四、各容器配置文件初始化

注意：这里的配置文件都在宿主机目录中

1、Nginx配置文件初始化

注意：nginx这里主要是容器创建完成后的工作

主要是做一下access_log日志json格式化

进入容器内，修改文件/etc/nginx/nginx.conf内容如下。修改完成后重启nginx容器

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    log_format json '{"@timestamp":"$time_iso8601",'
                                  '"host":"$server_addr",'
                                  '"clientip":"$remote_addr",'
                                  '"SendBytes":$body_bytes_sent,'
                                  '"responsetime":$request_time,'
                                  '"upstreamtime":"$upstream_response_time",'
                                  '"upstreamhost":"$upstream_addr",'
                                  '"http_host":"$host",'
                                  '"uri":"$uri",'
                                  '"domain":"$host",'
                                  '"xff":"$http_x_forwarded_for",'
                                  '"referer":"$http_referer",'
                                  '"tcp_xff":"$proxy_protocol_addr",'
                                  '"http_user_agent":"$http_user_agent",'
                                  '"status":"$status"}';

    access_log  /var/log/nginx/access.log  json;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

2、ES配置文件

修改elasticsearch.yml配置文件

cluster.name: "docker-elastic-cluster" # 集群名称
network.host: 0.0.0.0
http.cors.enabled: true # 跨域
http.cors.allow-origin: "*"     #允许所有的访问
# X-Pack是Elastic Stack的一个扩展插件，包含了安全控制，报警，监控，报表和画图功能。X-Pack能够方便地启用或禁用
xpack.security.enabled: true    #开启x-pack扩展插件 

注意: 当容器创建完成后，需要进入elasticsearch容器配置X-Pack插件密码

默认账号（用户名）是：elastic

#进入容器
docker exec -it elasticsearch /bin/bash
#配置密码
./bin/elasticsearch-setup-passwords interactive

3、kibana配置文件初始化

修改kibana.yml配置文件

#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
server.name: kibana #kibana名称
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
# server.publicBaseUrl: "http://172.16.0.242:5601"           #kiban的地址，ip是主服务器外部可以访问的IP，非容器IP
elasticsearch.hosts: [ "http://elasticsearch:9200" ]       #es地址
elasticsearch.username: "elastic"                          #es账号
elasticsearch.password: "elk@123"                         #es密码
monitoring.ui.container.elasticsearch.enabled: true

i18n.locale: "zh-CN" #设置中文网页

xpack.monitoring.ui.container.elasticsearch.enabled: true  #开启x-pack

注意：elasticsearch.hosts的主机地址，需要保证kibana和elasticsearch两者是同一个docker网络

4、logstash配置文件初始化

修改config/jvm.options配置文件

## JVM configuration

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

# -Xms1g
# -Xmx1g

-Xms512m
-Xmx512m

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## Locale
# Set the locale language
#-Duser.language=en

# Set the locale country
#-Duser.country=US

# Set the locale variant, if any
#-Duser.variant=

## basic

# set the I/O temp directory
#-Djava.io.tmpdir=$HOME

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# Set enableADS to true to enable Logstash to run on certain versions of the JDK
-Djdk.io.File.enableADS=true

# use our provided JNA always versus the system one
#-Djna.nosys=true

# Turn on JRuby invokedynamic
-Djruby.compile.invokedynamic=true
# Force Compilation
-Djruby.jit.threshold=0
# Make sure joni regexp interruptability is enabled
-Djruby.regexp.interruptible=true

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${LOGSTASH_HOME}/heapdump.hprof

## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime

# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${LS_GC_LOG_FILE}

# Entropy source for randomness
-Djava.security.egd=file:/dev/urandom

# Copy the logging context from parent threads to children
-Dlog4j2.isThreadContextMapInheritable=true

17-:--add-opens java.base/sun.nio.ch=ALL-UNNAMED
17-:--add-opens java.base/java.io=ALL-UNNAMED

修改config/logstash.yml配置文件

node.name: "logstash001"            #节点名称
http.host: "0.0.0.0"
xpack.monitoring.enabled: false     #禁用x-pack
# xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]

新建pipeline/logstash_nginx_access_log.conf配置文件，收集nginx_access_log日志

input {
  file {
    path => ['/usr/share/logs/nginx/access.log']  #文件所在地址 容器内地址
    start_position => "end"
    type => "nginx-access-log"            #类型名称
  }
}
 
filter {
  json {
     source => "message"
     remove_field => ["beat","offset","tags","prospector"] #移除字段，不需要采集
  }
  #date {
  #  match => ["@timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] #匹配timestamp字段
  #  target => "@timestamp"  #将匹配到的数据写到@timestamp字段中
  #}
  mutate {
     remove_field => ["message"]
  }
}
 
output {
  if[type] == "nginx-access-log"{
    elasticsearch {
      hosts => [ "elasticsearch:9200" ]      #es地址
      index => "nginx-log-%{+YYYY-MM-dd}"    #索引名称
      user => "elastic"                      #es用户名
      password => "elk@123"                 #es密码
    }
  }
}

新建pipeline/logstash_nginx_error_log.conf配置文件，收集nginx_error_log日志

input {
  file {
    path => ['/usr/share/logs/nginx/error.log']  #文件所在地址 容器内地址
    start_position => "end"
    type => "nginx-error-log"            #类型名称
  }
}
 
filter {
  
}
 
output {
  if[type] == "nginx-error-log"{
    elasticsearch {
      hosts => [ "elasticsearch:9200" ]      #es地址
      index => "nginx-log-%{+YYYY-MM-dd}"    #索引名称
      user => "elastic"                      #es用户名
      password => "elk@123"                  #es密码
    }
  }
}

五、创建容器并验证

创建所有容器

docker-compose up -d

呈现效果如下

验证

访问kibanahttp://localhost:5601

参考文章

• docker搭建ELK以及监控nginx日志
• 详解 logstash.conf
• Kibana7.9.2设置elasticsearch索引过期时间，到期自动删除