k8s filebeat sidecar模式收集ingress nginx日志并可视化展示

2021-08-05 14:04:05 阅读：231 来源： 互联网

标签：filebeat name ingress nginx true log

本文默认k8s环境以及已经部署ingress controller

公司所用ingress监控是由prometheus+grafana进行，但是监控不够全面，故使用filebeat去采集ingress日志，并自主进行可视化展示

1、ingress nginx日志数据落盘

在ingress controller中将configmap改为

kind: ConfigMap
apiVersion: v1
metadata:
  name: ingress-nginx-controller
data:
  access-log-path: /var/log/nginx/access.log
  compute-full-forwarded-for: 'true'
  enable-vts-status: 'true'
  error-log-path: /var/log/nginx/error.log
  forwarded-for-header: X-Forwarded-For
  log-format-upstream: >-
    {"@timestamp":
    "$time_iso8601","remote_addr":"$remote_addr","x-forward-for":"$http_x_forwarded_for","request_id":"$req_id","remote_user":"$remote_user","bytes_sent":$bytes_sent,"request_time":$request_time,"status":$status,"vhost":"$host","request_proto":"$server_protocol","path":"$uri","request_query":"$args","request_length":$request_length,"duration":$request_time,"method":"$request_method","http_referrer":"$http_referer","http_user_agent":"$http_user_agent","upstream-sever":"$proxy_upstream_name","proxy_alternative_upstream_name":"$proxy_alternative_upstream_name","upstream_addr":"$upstream_addr","upstream_response_length":$upstream_response_length,"upstream_response_time":$upstream_response_time,"upstream_status":$upstream_status}
  use-forwarded-headers: 'true'

2、生成filebeat镜像

dockerfile

FROM  million12/centos-supervisor:4.0.2
WORKDIR /usr/local
ADD filebeat-7.5.0-linux-x86_64.tar.gz .
RUN ln -s filebeat-7.5.0-linux-x86_64  filebeat \
 && cd filebeat       \
 && mkdir  config     \
 && chmod +x filebeat \
 && cp filebeat.yml config/ \ 
 && yum -y install logrotate crontabs


COPY supervisord.conf /etc/supervisord.conf

RUN mkdir -p /var/log/supervisor
EXPOSE 22 80
CMD ["/usr/bin/supervisord"]

因为需要使用logrotate进行日志轮转，需要安装

logrotate crontabs

supervisord.conf配置如下

[supervisord]
nodaemon=true

[program:cron]
command=/usr/sbin/crond -i

[program:filebeat]
command=/usr/local/filebeat/filebeat -c /usr/local/filebeat/config/filebeat.yml

3、修改原有ingress controller depl，将filebeat与ingress controller放到同一pod中，使用emptydir卷共享ingress日志，使filebeat能够读取，另外一个是面对日志的持续正常如何处理，这里使用logrotate，将logrotate在filebeat中配置，尽量对ingress影响小点，首先增加filebeat configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: filebeat-config
data:
  filebeat.yml: |
    filebeat.inputs:
      - type: log
        enabled: true 
        paths:
          - /var/log/nginx/access.log
        json.keys_under_root: true
        json.overwrite_keys: true
        json.add_error_key: true      
        json.ignore_decoding_error: true      
        tags: ["access"]
      - type: log
        enabled: true 
        paths:
          - /var/log/nginx/error.log
        json.keys_under_root: true
        json.overwrite_keys: true
        json.add_error_key: true      
        json.ignore_decoding_error: true      
        tags: ["error"]
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
    setup.template.settings:
      index.number_of_shards: 3
    output.elasticsearch:
      hosts: ["es-local.nxgp.svc.cluster.local:9200"]
      index: "nginx_log-%{+yyyy.MM.dd}"
      indices:
        - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM.dd}"
          when.contains:
            tags: "access"
        - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM.dd}"
          when.contains:
            tags: "error"
    setup.template.name: "nginx_log"
    setup.template.pattern: "nginx_*"
    setup.template.enabled: true
    setup.ilm.enabled: false
    setup.template.overwrite: false

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-logrotate
data:
  nginx: |
    /var/log/nginx/*.log {
      su root root
      size 50M
      notifempty
      copytruncate
      rotate 3
      missingok
      compress
      dateext
      dateformat .%Y%m%d-%H
  }

然后进行depl更新，只展示新增部分

      volumes:
        - name: ingress-log
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config
            defaultMode: 420
        - name: logrotateconf
          configMap:
            name: nginx-ingress-logrotate
            items:
              - key: nginx
                path: nginx
            defaultMode: 420
      containers:
        - name: controller
          volumeMounts:
            - name: ingress-log
              mountPath: /var/log/nginx/
        - name: filebeat
          image: 'xxx/filebeat:7.5.0'
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
            requests:
              cpu: '1'
              memory: 1Gi
          volumeMounts:
            - name: filebeat-config
              mountPath: /usr/local/filebeat/config/
            - name: ingress-log
              mountPath: /var/log/nginx/
            - name: logrotateconf
              mountPath: /etc/logrotate.d/nginx
              subPath: nginx
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 300
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      serviceAccount: ingress-nginx
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

logrotate是按天更新，更新时间不定

轮转效果内存占用高的是还没有进行打包的







4、es可视化展示

(1)PV

(2)UV

(3)Top10(接口访问量)

(4)Top10(客户端IP访问占比)

(5)Top10(最慢接口)

(6)后端upstream占比

(7)实时流量

(8)客户端访问占比

(9)平均并发数

(10)异常状态码统计

(11)总流量

(12)接口异常响应码

(13)接口访问耗时占比

(14)每10秒接口访问平均耗时

(15)每10秒接口访问最大耗时

(16)状态码统计

(17)访问量趋势图

(18)超过30秒以上的接口

(19)超过30秒以上的接口出现次数

标签：filebeat,name,ingress,nginx,true,log
来源： https://www.cnblogs.com/wxw7blog/p/15102962.html

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

ICode9

k8s filebeat sidecar模式收集ingress nginx日志并可视化展示

1、ingress nginx日志数据落盘

2、生成filebeat镜像