ICode9

精准搜索请尝试: 精确搜索
首页 > 编程语言> 文章详细

Logstash深入收集Java日志

2022-09-07 19:02:04  阅读:244  来源: 互联网

标签:file Java log tomcat logstash json conf 日志 Logstash


Logstash深入收集Java日志

没有修改Json格式

在企业中,我们看到tomcat日志遇到异常(exception)一条日志可能是几行或者十几行甚至几十行,组成的,那么,我们需要将多行日志变成一行日志,来收集。

这里我们有几种方式可以实现:
1.将日志改成Json格式
在企业中,想要将java日志改成json格式,并没有那么容易。
格式不是你想改,想改就能改,让我挣开,让我明白,放手你的爱~~~~
因为将日志改成Json格式,查看起来会很难受,有些开发人员不希望将日志格式改成Json的,所以,在改日志格式之前需要跟开发人员进行沟通,那么将tomcat日志格式改成Json格式也有两种方式。
1)开发自己更改,通过程序代码,或者log4j
2)运维修改tomcat的server配置文件

准备tomcat环境

# 1.安装tomcat
[root@elkstack03 ~]# yum install -y tomcat

# 2.部署tomcat代码
[root@elkstack03 ~]# vim /usr/share/tomcat/webapps/ROOT/index.jsp
test tomcat

# 3.启动tomcat
[root@elkstack03 ~]# systemctl start tomcat

1662511941975

使用Logstash收集java日志

[root@elkstack03 tomcat]# vim /etc/logstash/conf.d/tomcat_file_es.conf
input{
        file{
                type => "tomcat_access_log"
                path => "/var/log/tomcat/localhost_access_log.2022-09-07.txt"
                start_position => "beginning"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
        }
}


[root@elkstack03 tomcat]# /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_file_es -f /etc/logstash/conf.d/tomcat_file_es.conf &

修改tomcat日志格式为Json

[root@elkstack03 tomcat]# vim /etc/tomcat/server.xml
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_log" suffix=".log"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/> 
               
               
               
               
               
137         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
138                prefix="localhost_access_log." suffix=".txt"
139                pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;
    %u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&
    quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&
    quot;%{User-Agent}i&quot;}" />

使用Logstash来解析json格式

vim tomcat_file_es_json.conf 
input{
        file{
                type => "tomcat_access_log_json"
                path => "/var/log/tomcat/localhost_access_log.*.txt"
                start_position => "end"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
                codec => "json"
        }
}

[root@elkstack03 tomcat]# /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_access_json -f /etc/logstash/conf.d/tomcat_file_es_json.conf &

还是一坨

解析Json格式

[root@elkstack03 conf.d]# vim tomcat_file_es_json.conf 
input{
        file{
                type => "tomcat_access_log_json"
                path => "/var/log/tomcat/localhost_access_log.*.txt"
                start_position => "end"
        }
}

filter{
        json{
                source => "message"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
                codec => "json"
        }
}

删除多余的message字段

[root@elkstack03 conf.d]# vim tomcat_file_es_json.conf 
input{
        file{
                type => "tomcat_access_log_json"
                path => "/var/log/tomcat/localhost_access_log.*.txt"
                start_position => "end"
        }
}

filter{
        json{   
                source => "message"
                remove_field => ["message"]
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
                codec => "json"
        }
}

[root@elkstack03 conf.d]#  /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_access_json -f /etc/logstash/conf.d/tomcat_file_es_json.conf &

Logstash收集catlina日志(异常错误日志)

[root@elkstack03 conf.d]# vim /etc/logstash/conf.d/catlina_file_es.conf
input{
        file{
                type => "tomcat_catlina_log"
                path => "/var/log/tomcat/catalina.*.log"
                start_position => "beginning"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
        }
}

[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_catlina/ -f /etc/logstash/conf.d/catlina_file_es.conf &

logstash多行合并

[root@elkstack03 conf.d]# vim /etc/logstash/conf.d/catlina_file_es.conf 
input{
        file{
                type => "tomcat_catlina_log"
                path => "/var/log/tomcat/catalina.*.log"
                start_position => "beginning"
                codec => multiline {
                    pattern => "^[A-Z]"
                    negate => true
                    what => "previous"
                }
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
        }
}

标签:file,Java,log,tomcat,logstash,json,conf,日志,Logstash
来源: https://www.cnblogs.com/wangchengww/p/16666873.html

本站声明: 1. iCode9 技术分享网(下文简称本站)提供的所有内容,仅供技术学习、探讨和分享;
2. 关于本站的所有留言、评论、转载及引用,纯属内容发起人的个人观点,与本站观点和立场无关;
3. 关于本站的所有言论和文字,纯属内容发起人的个人观点,与本站观点和立场无关;
4. 本站文章均是网友提供,不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属;如您发现该文章侵犯了您的权益,可联系我们第一时间进行删除;
5. 本站为非盈利性的个人网站,所有内容不会用来进行牟利,也不会利用任何形式的广告来间接获益,纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。

专注分享技术,共同学习,共同进步。侵权联系[81616952@qq.com]

Copyright (C)ICode9.com, All Rights Reserved.

ICode9版权所有