标签：elk http log request server nginx 172.17 日志 图形化

配置nginx日志格式，输出为json格式

 log_format json '{ "@timestamp":"$time_iso8601",'
                  '"@source":"$server_addr",'
                  '"time_local":"$time_local",'
                  '"remote_addr":"$remote_addr",'
                  '"remote_port":"$remote_port",'
                  '"remote_user":"$remote_user",'
                  '"server_name":"$server_name",'
                  '"server_port":"$server_port",'
                  '"server_protocol":"$server_protocol",'
                  '"request":"$request",'
                  '"request_uri":"$request_uri",'
                  '"uri":"$uri",'
                  '"request_time":"$request_time",'
                  '"request_method":"$request_method",'
                  '"request_length":$request_length,'
                  '"status":"$status",'
                  '"scheme":"$scheme",'
                  '"body_bytes_sent":"$body_bytes_sent",'
                  '"bytes_sent":"$bytes_sent",'
                  '"request_body":"$request_body",'
                  '"upstream_addr":"$upstream_addr",'
                  '"upstream_response_time":"$upstream_response_time",'
                  '"upstream_status":"$upstream_status",'
                  '"http_host":"$http_host",'
                  '"http_referrer":"$http_referer",'
                  '"http_user_agent":"$http_user_agent",'
                  '"http_x_forwarded_for":"$http_x_forwarded_for",'
                  '"connection":"$connection",'
                  '"connection_requests":"$connection_requests",'
                  '"content_length":"$content_length",'
                  '"content_type":"$content_type",'
                  '"cookie_name":"$cookie_name",'
                  '"limit_rate":"$limit_rate",'
                  '"hostname":"$hostname",'
                  '"args":"$args",'
                  '"https":"$https",'
                  '"http_cookie":"$http_cookie",'
                  '"msec":"$msec",'
                  '"pid":"$pid"}';
 
access_log  /usr/local/nginx/logs/access.log json;

配置filebeat收集nginx日志缓存到redis中

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - "/usr/local/nginx/logs/access.log"
  fields:
    app_id: "access.log"

- type: log
  enabled: true
  paths:
    - "/usr/local/nginx/logs/error.log"
  fields:
    app_id: "error.log"

output.redis:
  hosts: ["192.168.10.46:6380"]
  password: ""
  db: 2
  key: "nginx"
  keys:
    - key: "%{[fields.list]}"
      mappings:
        app_id: "access.log"
        app_id: "error.log"
  worker: 4 
  timeout: 20
  max_retries: 3
  codec.json:
    pretty: false

monitoring.enabled: true
monitoring.elasticsearch:
  hosts: ["http://172.17.9.31:9200","http://172.17.9.31:9200"]

 使用logstash从redis中取出日志，格式化输出到elasticsearch集群中

input {
    redis {
        host => "172.17.9.33"
        port => 6379
        db => 2
        key => "nginx"
        
        data_type => "list"
        threads => 4
        tags => "nginx"
    }
}

filter {
    if "nginx" in [tags] {
        json { 
            source => "message" 
        }
        grok { 
            match => [ "message", "%{HTTPDATE:[@metadata][timestamp]}" ] 
        }
        date { 
            match => [ "[@metadata][timestamp]", "dd/MMM/yyyy:HH:mm:ss Z" ] 
        }

    }
}

output {

    if "nginx" in [tags] {
        if [fields][app_id] == "access.log" {
            elasticsearch {
                hosts => ["172.17.9.31:9200","172.17.9.32:9200"]
                index => "logstash-nginx-access.log-%{+YYYY.MM.dd}"
            }
        }

        if [fields][app_id] == "error.log" {
            elasticsearch {
                hosts => ["172.17.9.31:9200","172.17.9.32:9200"]
                index => "logstash-nginx-error.log%{+YYYY.MM.dd}"
            }
        }

    }


}

 配置kibana，新增索引

 创建索引模式

 访问nginx，查看kibana数据，正常返回即可

 然后点击overview，创建仪表盘，根据自己需求创建即可。

下边为我创建的仪表盘，统计了每个IP不同时间访问次数，访问网站url次数，访问网站来源IP次数，返回状态码比例。

标签：elk,http,log,request,server,nginx,172.17,日志,图形化
来源： https://www.cnblogs.com/oliver-yt/p/15561046.html

本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享；
2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关；
3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关；
4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除；
5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。