k8s部署elfk来实现pod运行elfk
本文详细介绍了在Kubernetes集群中部署ELFK（Elasticsearch、Logstash、Filebeat、Kibana）日志系统的实验过程。实验内容包括：1）部署3节点Elasticsearch集群并配置X-Pack安全认证；2）部署Kibana可视化平台；3）配置Logstash日志处理管道；4）部署Nginx和Tomcat应用，并使用Filebeat作为Sidecar容器收集日志
k8s 中部署elfk
实验要求
1. 部署一套elk
2. 部署两个Web pod，nginx和tomcat。
3. 在两个Web pod 中部署一个sidecar(边车服务)，来收集nginx和tomcat的日志到es中并在kibana中展示
实验前置
1. 数据持久化部分，使用动态存储类，NFS或者ceph均可，本次使用172.27.9.50 公共的NFS
2. 镜像仓库使用172.27.9.50公共仓库
实验过程
1. 为了实现ELK有状态服务的部署，需要准备数据持久化相关工作（部署供应商、动态存储类）
2. 部署ES集群
3. 部署kibana
4. 部署logstash组件
5. 部署2个的Web 中间件。同时带有filebeat边车服务
6. 创建定期删除ES中索引的cronjob pod 同时编写 索引删除脚本
创建在部署NFS供应商时使用的harbor服务器的密钥（默认命名空间）
kubectl create secret docker-registry \
harbor-secret \
--docker-server=172.27.9.50 \
--docker-username=admin \
--docker-password=Harbor12345
部署NFS供应商
# 创建SA账户并对 sa 授权 kubectl create serviceaccount nfs-provisioner kubectl create clusterrolebinding nfs-provisioner --clusterrole=cluster-admin --serviceaccount=default:nfs-provisioner # 安装 nfs-provisioner 程序 cat > ./nfs-deployment.yaml << EOF kind: Deployment apiVersion: apps/v1 metadata: name: nfs-provisioner spec: selector: matchLabels: c app: nfs-provisioner replicas: 1 template: metadata: labels: app: nfs-provisioner spec: serviceAccount: nfs-provisioner imagePullSecrets: - name: harbor-secret containers: - name: nfs-provisioner image: 172.27.9.50/base/mydlq/nfs-subdir-external-provisioner:v4.0.0 imagePullPolicy: IfNotPresent volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: bw-wg.com/nfs - name: NFS_SERVER value: 172.27.9.50 - name: NFS_PATH value: /mnt/nfs_pro volumes: - name: nfs-client-root nfs: server: 172.27.9.50 path: /mnt/nfs_pro EOF # 创建动态类 cat > ./sc.yaml << EOF kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: nfs-storageclass provisioner: bw-wg.com/nfs reclaimPolicy: Retain EOF
部署ES集群
# 创建一个命名空间，并为要开启x-pace所以要准备出一个P12证书文件
kubectl create ns log
kubectl create configmap -n log elastic-certificates --from-file=elastic-certificates.p12=elastic-certificates.p12
# 创建log命名空间下的 secret
kubectl create secret docker-registry \
harbor-secret \
--docker-server=172.27.9.50 \
--docker-username=admin \
--docker-password=Harbor12345 -n log
# 部署ES 有状态服务
cat > ./elasticsearch-sts.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
namespace: log
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
ports:
- name: api
port: 9200
- name: discovery
port: 9300
clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
namespace: log
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
initContainers:
- name: fix-permissions
image: 172.27.9.50/base/busybox:1.28.3
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: es-data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: 172.27.9.50/base/busybox:1.28.3
imagePullPolicy: IfNotPresent
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: 172.27.9.50/base/busybox:1.28.3
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
containers:
- name: elasticsearch
image: 172.27.9.50/elfk/elasticsearch:7.17.7
imagePullPolicy: IfNotPresent
env:
- name: "cluster.name"
value: "elk"
- name: "node.master"
value: "true"
- name: "node.data"
value: "true"
- name: "http.host"
value: "0.0.0.0"
- name: "network.host"
value: "_eth0_"
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: "bootstrap.memory_lock"
value: "false"
- name: "http.port"
value: "9200"
- name: "transport.tcp.port"
value: "9300"
- name: "discovery.seed_hosts"
value: "elasticsearch"
- name: "cluster.initial_master_nodes"
value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
- name: "discovery.seed_resolver.timeout"
value: "10s"
- name: "discovery.zen.minimum_master_nodes"
value: "2"
- name: "gateway.recover_after_nodes"
value: "3"
- name: "http.cors.enabled"
value: "true"
- name: "http.cors.allow-origin"
value: "*"
- name: "http.cors.allow-methods"
value: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
- name: "http.cors.allow-headers"
value: "kbn-version,kbn-xsrf,Origin,X-Requested-With,Content-Type,Accept,Engaged-Auth-Token,Content-Length,Authorization"
- name: "http.cors.allow-credentials"
value: "true"
- name: "xpack.security.enabled"
value: "true"
- name: "xpack.security.transport.ssl.enabled"
value: "true"
- name: "xpack.security.transport.ssl.verification_mode"
value: "certificate"
- name: "xpack.security.transport.ssl.keystore.path"
value: "elastic-certificates.p12"
- name: "xpack.security.transport.ssl.truststore.path"
value: "elastic-certificates.p12"
- name: "ES_JAVA_OPTS"
value: "-Xms512m -Xmx512m"
ports:
- containerPort: 9200
name: api
protocol: TCP
- containerPort: 9300
name: discovery
protocol: TCP
resources:
limits:
cpu: 1000m
requests:
cpu: 500m
volumeMounts:
- name: cert
mountPath: /usr/share/elasticsearch/config/elastic-certificates.p12
subPath: elastic-certificates.p12
readOnly: true
- name: es-data
mountPath: /usr/share/elasticsearch/data
imagePullSecrets:
- name: harbor-secret
volumes:
- name: cert
configMap:
name: elastic-certificates
volumeClaimTemplates:
- metadata:
name: es-data
namespace: log
spec:
accessModes: [ "ReadWriteMany" ]
storageClassName: "nfs-storageclass"
resources:
requests:
storage: 10Gi
EOF
# 初始化密码
kubectl exec -it -n log elasticsearch-0 -- bash
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
部署kibana
cat > kibana-config.yaml << EOF apiVersion: v1 kind: ConfigMap metadata: name: kibana-config namespace: log data: kibana.yml: | server.port: 5601 server.host: "0" kibana.index: ".kibana" elasticsearch.hosts: ["http://elasticsearch.log:9200"] elasticsearch.username: elastic elasticsearch.password: abc123 i18n.locale: "zh-CN" EOF cat > kibana.yaml << EOF apiVersion: v1 kind: Service metadata: name: kibana namespace: log labels: app: kibana spec: type: NodePort selector: app: kibana ports: - port: 5601 protocol: TCP targetPort: 5601 --- apiVersion: apps/v1 kind: Deployment metadata: name: kibana namespace: log labels: app: kibana spec: replicas: 1 selector: matchLabels: app: kibana template: metadata: labels: app: kibana spec: imagePullSecrets: - name: harbor-secret containers: - name: kibana image: 172.27.9.50/elfk/kibana:7.17.7 imagePullPolicy: IfNotPresent ports: - containerPort: 5601 resources: limits: cpu: 500m memory: 500Mi requests: cpu: 500m memory: 500Mi volumeMounts: - name: config mountPath: /usr/share/kibana/config volumes: - name: config configMap: name: kibana-config EOF
部署logstash组件
# 生成logstash的配置文件
cat > ./logstash-conf.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-config
namespace: log
data:
logstash.yml: |
http.host: "0"
http.port: 9600
path.config: /usr/share/logstash/pipeline
config.reload.automatic: true
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: abc123
xpack.monitoring.elasticsearch.hosts: ["http://elasticsearch:9200"]
xpack.monitoring.collection.interval: 10s
logstash.conf: |
input {
beats {
port => 5040
}
}
filter {
if [type] == "nginx_access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
if [type] == "tomcat_access" {
grok {
match => { "message" => "(?<timestamp>%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}) %{LOGLEVEL:level} \[%{DATA:exception_info}\] %{GREEDYDATA:message}" }
}
}
}
output {
if [type] == "nginx_access" {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => "elastic"
password => "abc123"
index => "nginx-log-%{+YYYY.MM.dd}"
}
}
if [type] == "tomcat_access" {
elasticsearch {
hosts => ["elasticsearch:9200"]
user => "elastic"
password => "abc123"
index => "tomcat-log-%{+YYYY.MM.dd}"
}
}
}
EOF
# 生成logstash的部署文件
cat > logstash.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: logstash
namespace: log
spec:
selector:
app: logstash
ports:
- protocol: TCP
port: 5040
nodePort: 30040
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
namespace: log
spec:
replicas: 1
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
imagePullSecrets:
- name: harbor-secret
containers:
- name: logstash
image: 172.27.9.50/elfk/logstash:7.17.7
ports:
- containerPort: 9600
- containerPort: 5040
resources:
limits:
cpu: 300m
memory: 1000Mi
requests:
cpu: 300m
memory: 500Mi
volumeMounts:
- name: config
mountPath: /usr/share/logstash/config
- name: pipeline
mountPath: /usr/share/logstash/pipeline
volumes:
- name: config
configMap:
name: logstash-config
items:
- key: logstash.yml
path: logstash.yml
- name: pipeline
configMap:
name: logstash-config
items:
- key: logstash.conf
path: logstash.conf
EOF
生成filebeat配置文件
# 生成filebeat配置文件
# vi filebeat_conf.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
path: ${path.config}/inputs.d/*.yml
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: log
enabled: true
tail_files: true
backoff: "1s"
paths:
- /nginxlog/*.log
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
type: nginx_access
fields_under_root: true
- type: log
enabled: true
tail_files: true
backoff: "1s"
paths:
- /tomcatlog/*.log
- /tomcatlog/*.txt
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
type: tomcat_access
fields_under_root: true
output.logstash:
hosts: ["logstash.log:5040"]
enabled: true
worker: 1
compression_level: 3
部署2个的Web 中间件。同时带有filebeat边车服务
# 部署一个Nginx服务，同时部署sidecar filebeat服务
cat > ./nginx.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
selector:
app: nginx
ports:
- port: 80
nodePort: 30080
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: default
spec:
replicas: 3
minReadySeconds: 15
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
imagePullSecrets:
- name: harbor-secret
containers:
- name: nginx
image: 172.27.9.50/base/nginx:1.17.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
- name: filebeat
image: 172.27.9.50/elfk/filebeat:7.17.7
imagePullPolicy: IfNotPresent
args: [
"-c", "/etc/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
volumeMounts:
- name: config
mountPath: /etc/filebeat
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: nginx-log
mountPath: /nginxlog
volumes:
- name: config
configMap:
name: filebeat-config
items:
- key: filebeat.yml
path: filebeat.yml
- name: data
emptyDir: {}
- name: nginx-log
emptyDir: {}
EOF
# 部署一个Tomcat服务，同时部署sidecar filebeat服务
cat > ./tomcat.yaml << EOF
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
labels:
app: tomcat
spec:
selector:
app: tomcat
ports:
- port: 8080
nodePort: 30880
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat
namespace: default
spec:
replicas: 3
minReadySeconds: 15
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
imagePullSecrets:
- name: harbor-secret
containers:
- name: tomcat
image: 172.27.9.50/base/tomcat:8.0.51-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
volumeMounts:
- name: tomcat-log
mountPath: /usr/local/tomcat/logs
- name: filebeat
image: 172.27.9.50/elfk/filebeat:7.17.7
imagePullPolicy: IfNotPresent
args: [
"-c", "/etc/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
runAsUser: 0
volumeMounts:
- name: config
mountPath: /etc/filebeat
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: tomcat-log
mountPath: /tomcatlog
volumes:
- name: config
configMap:
name: filebeat-config
items:
- key: filebeat.yml
path: filebeat.yml
- name: data
emptyDir: {}
- name: tomcat-log
emptyDir: {}
EOF
ES定期删除策略
nginx与tomcat的访问日志是按天生成到ES中。 现在部署一个Cronjob
# 制作一个用于运行计划任务的ubuntu镜像
cat > dockerfile << EOF
From ubuntu:latest
RUN apt-get update && apt-get install -y curl telnet net-tools
EOF
# 打包
docker build -t 172.27.9.50/base/bwtools:v1 .
# 上传到9.50
docker push 172.27.9.50/base/bwtools:v1
# 编写cronjob的yaml文件
cat > es-index-del.yaml << EOF
apiVersion: batch/v1
kind: CronJob
metadata:
name: es-index-del
namespace: log
spec:
# 并发策略
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
schedule: '*/2 * * * *'
# 表示如果Job因为某种原因无法按调度准时启动，在startingDeadlineSeconds时间段之内，CronJob仍然试图重新启动Job
# 如果在startingDeadlineSeconds时间之内没有启动成功，则不再试图重新启动。
# 如果startingDeadlineSeconds的值没有设置，则没有按时启动的任务不会被尝试重新启动。
startingDeadlineSeconds: 60
suspend: false
jobTemplate:
spec:
# 该字段限定了 Job 对象在集群中的存活时长，一旦达到activeDeadlineSeconds 指定的时长，该 Job 创建的所有的 Pod 都将被终止。
activeDeadlineSeconds: 600
# 设定 Job 最大的重试次数。该字段的默认值为 6；
# 一旦重试次数达到了 backoffLimit 中的值，Job 将被标记为失败;
backoffLimit: 6
# Job结束需要成功运行的Pods。 默认为1
completions: 1
# 并行运行的Pod个数，默认为1
parallelism: 1
template:
spec:
restartPolicy: OnFailure
imagePullSecrets:
- name: harbor-secret
containers:
- command:
- 'sh'
- '/root/script/es-index-del.sh'
image: 172.27.9.50/base/bwtools:v1
imagePullPolicy: IfNotPresent
name: es-backup
resources:
requests:
cpu: 250m
memory: 512Mi
volumeMounts:
- name: es-index-del-script
mountPath: /root/script/es-index-del.sh
subPath: es-index-del.sh
readOnly: true
- name: localtime-volume
mountPath: /etc/localtime
readOnly: true
volumes:
- name: es-index-del-script
configMap:
name: es-del-script-cm
items:
- key: es-index-del.sh
path: es-index-del.sh
- name: localtime-volume
hostPath:
path: /etc/localtime
EOF
# 定期删除索引的脚本
# vi es-index-del.sh
#!/bin/bash
# create by songjia
# date: 20230302
# description: ES索引的清理
ES_ip='elasticsearch.log'
es_user='elastic'
es_pw='abc123'
# 返回2天前的日期
check_day=`date -d '-2 days' '+%F'`
echo "预计将会删除 ${check_day} 日前的所有索引！ "
# 将日期转换为时间戳
check_day_timestamp=`date -d "$check_day" +%s`
del_log(){
index_day=$1
index_day_timestamp=`date -d "$index_day" +%s`
# 当索引的时间戳值小于当前日期7天前的时间戳时，删除此索引
if [ ${index_day_timestamp} -lt ${check_day_timestamp} ];then
# 将横线的日期格式转换成ES中点形式的日期格式
es_format_date=`echo ${index_day} | sed 's/-/\./g'`
echo "当前开始删除以 ${es_format_date} 结尾的所有索引!"
curl -s -XDELETE -u ${es_user}:${es_pw} http://${ES_ip}:9200/*${es_format_date}
else
echo "${index_day} 日期的索引未到删除阀值，跳过。"
fi
}
curl -s -XGET -u ${es_user}:${es_pw} http://${ES_ip}:9200/_cat/indices |awk -F" " '{print $3}'|grep -vE '^\.' |awk -F"-" '{print $NF}'|sort|uniq|sed 's/\./-/g'|while read ES_DATE
do
echo "ES中存在 ${ES_DATE} 日期的索引"
del_log ${ES_DATE}
done
# 将脚本以文件的方式写入ConfigMap资源
kubectl create configmap es-del-script-cm --from-file=es-index-del.sh -n log
火山引擎开发者社区是火山引擎打造的AI技术生态平台，聚焦Agent与大模型开发，提供豆包系列模型（图像/视频/视觉）、智能分析与会话工具，并配套评测集、动手实验室及行业案例库。社区通过技术沙龙、挑战赛等活动促进开发者成长，新用户可领50万Tokens权益，助力构建智能应用。
更多推荐
25
0- 0
已为社区贡献1条内容
所有评论(0)