标签：ELK 31 kibana elasticsearch nfs 日志 root logstash 搭建
ELK 日志管理平台
1>Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它是基于Lucene（一个全文检索引擎的架构）开发的分布式存储检索引擎，可用来存储各类日志，采用 Java 开发，可通过 RESTful Web 接口，通过浏览器来与 Elasticsearch 通信。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等;负责存储最终数据、建立索引、提供搜索功能。
2>Logstash 主要是用来日志的搜集、分析、过滤日志的工具，负责采集日志，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。
另外，它使用JRuby 语言编写，运行在Java虚拟机（JVM）上，是一款强大的数据处理工具，可以实现=数据传输、格式处理、格式化输出。 Logstash 还具有强大的插件功能，常用于日志处理。
3>Kibana 也是一个开源和免费的工具，它基于Node.js开发，负责提供可视化界面，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。
4>Filebeat隶属于Beats，其作为原logstash-forwarder的替代来完成。Filebeat轻量级的日志传输工具， ，也可称为：是一个轻量级的日志采集器。它可以读取系统、nignx、apache等logs文件，监控日志文件，传输数据到Elasticsearch或者Logstash，最后在Kibana中实现可视化。目前Beats包含四种工具：
Packetbeat（搜集网络流量数据）
Topbeat（搜集系统、进程和文件系统级别的 CPU 和内存使用情况等数据）
Filebeat（搜集文件数据）：FileBeat采集数据时是Json化的，这个日志采集工具相当轻量级，对系统资源的消耗很少。而LogStash的优点则是有丰富的Filter插件，用于对数据作粗处理，占用资源高。一般，我们使用FileBeat采集数据之后会传输给Kafka消息队列，然后LogStash采集消息队列中的数据，作过滤处理，最后将数据传输给ES。
Winlogbeat（搜集 Windows 事件日志数据）
Elastic Stack的优点
Elastic Stack 目前已成为机器数据分析，或者说实时日志处理领域，开源界的第一选择，和传统的日志处理方案相比，Elastic Stack 具有如下几个优点：
1）处理方式灵活。Elasticsearch 是实时全文索引，不需要像 storm 那样预先编程才能使用；
2）配置简易上手。Elasticsearch 全部采用 JSON 接口，Logstash 是 Ruby DSL 设计，都是目前业界最通用的配置语法设计；
3）检索性能高效。虽然每次查询都是实时计算，但是优秀的设计和实现基本可以达到全天数据查询的秒级响应；
4）集群线性扩展。不管是 Elasticsearch 集群还是 Logstash 集群都是可以线性扩展的；
5）前端操作炫丽。Kibana 界面上，只需要点击鼠标，就可以完成搜索、聚合功能，生成炫丽的仪表板。
一、介绍
1、组成
ELK由Elasticsearch、Logstash和Kibana三部分组件组成；
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用
kibana 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。
2、四大组件
Logstash: logstash server端用来搜集日志；
Elasticsearch: 存储各类日志；
Kibana: web化接口用作查寻和可视化日志；
Logstash Forwarder: logstash client端用来通过lumberjack 网络协议发送日志到logstash server；
3、工作流程
在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。
下面是在两台节点上都安装一下环境。
nfs-31
rsync-41
二、安装JDk（两个节点都要安装）
配置阿里源：wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
Logstash的运行依赖于Java运行环境，Elasticsearch 要求至少 Java 7。
[root@controller ~]# yum install java-1.8.0-openjdk -y
[root@controller ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
1、关闭防火墙
systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动
2、关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
三、安装Elasticsearch
基础环境安装（两个节点都要安装，配置文件不一样）
1）下载并安装GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
2）添加yum仓库
[root@elk-node1 ~]# vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
3）安装elasticsearch
[root@elk-node1 ~]# yum install -y elasticsearch
4）添加自启动
chkconfig --add elasticsearch
5）启动命令
systemctl daemon-reload
systemctl enable elasticsearch.service
6）修改配置
[root@nfs-31 ~]#cd /etc/elasticsearch/
[root@nfs-31 /etc/elasticsearch]#ls
elasticsearch.yml logging.yml scripts
[root@nfs-31 /etc/elasticsearch]#cp elasticsearch.yml{,.bak}
[root@nfs-31 /etc/elasticsearch]#mkdir -p /data/es-data
[root@nfs-31 /etc/elasticsearch]#vim elasticsearch.yml
[root@nfs-31 /etc/elasticsearch]# grep '^[a-z]' elasticsearch.yml
cluster.name: hejianlai //集群名称
node.name: elk-node1 //节点名称
path.data: /data/es-data //数据存放目录
path.logs: /var/log/elasticsearch/ //日志存放目录
bootstrap.memory_lock: true //打开内存
network.host: 0.0.0.0 //监听网络
http.port: 9200 //端口
discovery.zen.ping.multicast.enabled: false //改为单播
discovery.zen.ping.unicast.hosts: ["10.0.0.31", "10.0.0.41"]
#注意
#data/es-data没权限，赋权限即可
chown -R elasticsearch:elasticsearch /data/es-data/
#启动
systemctl start elasticsearch
[root@nfs-31 /etc/elasticsearch]#systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2022-08-20 20:46:06 CST; 5s ago
Docs: http://www.elastic.co
Process: 2912 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 2913 (java)
CGroup: /system.slice/elasticsearch.service
└─2913 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+...
访问地址10.0.0.31:9200
安装ES插件
#统计索引数
[root@nfs-31 /etc/elasticsearch]#curl -i -XGET 'http://10.0.0.31:9200/_count?pretty' -d '
> "query":{
> "match_all":{}
> }'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 95
{
"count" : 0,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
}
}
#es插件，收费的不建议使用(这个不安装)
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install marvel-agent
#安装开源的elasticsearch-head插件
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/share/elasticsearch/plugins/head
访问：10.0.0.31:9200/_plugin/head/
使用POST方法创建查询
使用GET方法查询数据
基本查询
41机器配置
[root@rsync-41 /etc/elasticsearch]#cat elasticsearch.yml
cluster.name: hejianlai
node.name: elk-node2
path.data: /data/es-data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.31", "10.0.0.41"]
#data/es-data没权限，赋权限即可
chown -R elasticsearch:elasticsearch /data/es-data/
在构建Elasticsearch(ES)多节点集群的时候，通常情况下只需要将elasticsearch.yml中的cluster.name设置成相同即可，ES会自动匹配并构成集群。但是很多时候可能由于不同的节点在不同的网段下，导致无法自动获取集群。此时可以将启用单播，显式指定节点的发现。具体做法是在elasticsearch.yml文件中设置如下两个参数：
重启31机器 开启41机器
[root@nfs-31 ~]#systemctl restart elasticsearch
[root@rsync-41 /etc/elasticsearch]#systemctl start elasticsearch
访问 10.0.0.31:9200/_plugin/head/
安装监控kopf
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ......................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed kopf into /usr/share/elasticsearch/plugins/kopf
访问 10.0.0.31:9200/_plugin/kopf/#!/cluster
四、安装Logstash(客户端即需要收集日志的机子，ES节点上都要安装)
官方文档地址：https://www.elastic.co/guide/en/logstash/current/index.html
31机器
#1、下载安装GPG Key
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
#2、添加yum仓库
vim /etc/yum.repos.d/logstash.repo
[logstash-2.1]
name=Logstash repository for 2.1.x packages
baseurl=http://packages.elastic.co/logstash/2.1/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
#3、安装logstash
yum install -y logstash
#4、测试数据
#简单的输入输出
[root@nfs-31 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
hello world
2022-08-20T13:35:44.716Z nfs-31 hello world
hi afei
2022-08-20T13:35:57.028Z nfs-31 hi afei
你好啊 渣渣辉
Received an event that has a different character encoding than you configured. {:text=>"你好啊 渣渣\\xE7辉", :expected_charset=>"UTF-8", :level=>:warn}
2022-08-20T13:36:14.242Z nfs-31 你好啊 渣渣\xE7辉
2022-08-20T13:36:19.675Z nfs-31
logstash shutdown completed
#可以使用rubydebug详细输出
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
{
"message" => "",
"@version" => "1",
"@timestamp" => "2022-08-20T13:37:34.060Z",
"host" => "nfs-31"
}
mimi
{
"message" => "mimi",
"@version" => "1",
"@timestamp" => "2022-08-20T13:37:40.516Z",
"host" => "nfs-31"
}
#内容写进elasticsearch中
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{hosts=>["10.0.0.31"]} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
hello afei
你好帅
啦啦啦啦
dasda
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["10.0.0.31:9200"]} stdout{ codec => rubydebug}}'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
北京
{
"message" => "北京",
"@version" => "1",
"@timestamp" => "2022-08-20T13:41:44.333Z",
"host" => "nfs-31"
}
afei
{
"message" => "afei",
"@version" => "1",
"@timestamp" => "2022-08-20T13:42:05.132Z",
"host" => "nfs-31"
}
Logstash shutdown completed
logstash日志收集配置文件编写
#1、交换式输入信息
[root@nfs-31 ~]#vim /etc/logstash/conf.d/logstash-01.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/logstash-01.conf
input { stdin { } }
output {
elasticsearch { hosts => ["10.0.0.31:9200"]}
stdout { codec => rubydebug }
}
#执行命令
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-01.conf
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
wuhan
{
"message" => "wuhan",
"@version" => "1",
"@timestamp" => "2022-08-20T13:50:19.731Z",
"host" => "nfs-31"
}
小阿飞
Received an event that has a different character encoding than you configured. {:text=>"\\xE9\\x98小阿飞", :expected_charset=>"UTF-8", :level=>:warn}
{
"message" => "\\xE9\\x98小阿飞",
"@version" => "1",
"@timestamp" => "2022-08-20T13:50:34.473Z",
"host" => "nfs-31"
}
#收集系统日志
[root@nfs-31 ~]#vim /etc/logstash/conf.d/systemlog.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/systemlog.conf
input{
file {
path => "/var/log/messages"
type => "sysstem"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.31:9200"]
index => "systemlog-%{+YYYY.MM.dd}"
}
}
#放在后台执行
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/systemlog.conf &
[1] 4471
[root@nfs-31 ~]#OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
收集elk错误日志配置文件编写
[root@nfs-31 ~]#vim /etc/logstash/conf.d/elk_log.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/elk_log.conf
input {
file {
path => "/var/log/messages"
type => "system"
start_position => "beginning"
}
}
input {
file {
path => "/var/log/elasticsearch/hejianlai.log"
type => "es-error"
start_position => "beginning"
codec => multiline {
pattern => "^\[" //正则匹配[开头的为一个事件
negate => true
what => "previous"
}
}
}
output {
if [type] == "system"{
elasticsearch {
hosts => ["10.0.0.31:9200"]
index => "systemlog-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error"{
elasticsearch {
hosts => ["10.0.0.31:9200"]
index => "es-error-%{+YYYY.MM.dd}"
}
}
}
#放入后台运行
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf &
[2] 4523
五、安装Kibana
官方下载地址：https://www.elastic.co/downloads/kibana
官方最新的版本出来了6.3.1太新了，下载后出现很多坑后来就下了4.3.1的·先用着吧
#1、kibana的安装：
[root@nfs-31 ~]#cd /usr/local/
[root@nfs-31 /usr/local]#wget https://download.elastic.co/kibana/kibana/kibana-4.6.3-linux-x86_64.tar.gz
[root@nfs-31 /usr/local]# tar -xf kibana-4.6.3-linux-x86_64.tar.gz
[root@nfs-31 /usr/local]#ln -s /usr/local/kibana-4.6.3-linux-x86_64 /usr/local/kibana
[root@nfs-31 /usr/local]#cd kibana
[root@nfs-31 /usr/local/kibana]#ls
bin data LICENSE.txt node_modules package.json src
config installedPlugins node optimize README.txt webpackShims
#2、修改配置文件
[root@nfs-31 /usr/local/kibana]#cd config/
[root@nfs-31 /usr/local/kibana/config]#pwd
/usr/local/kibana/config
[root@nfs-31 /usr/local/kibana/config]#cp kibana.yml kibana.yml.bb
[root@nfs-31 /usr/local/kibana/config]#vim kibana.yml
[root@nfs-31 /usr/local/kibana/config]#grep -Ev "^#|^$" kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.0.0.31:9200"
kibana.index: ".kibana"
#3、screen是一个全屏窗口管理器，它在几个进程（通常是交互式shell）之间复用物理终端。每个虚拟终端提供DEC VT100的功能。
yum install -y screen
#4、启动screen命令后运行kibana最后按ctrl+a+d组合键让其在单独的窗口里运行。
[root@nfs-31 /usr/local/kibana/config]#/usr/local/kibana/bin/kibana
log [22:21:42.921] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
log [22:21:42.940] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
[root@nfs-31 /usr/local/kibana/config]#screen -ls
There is a screen on:
4834.pts-0.nfs-31 (Attached)
1 Socket in /var/run/screen/S-root.
[root@nfs-31 /usr/local/kibana/config]#/usr/local/kibana/bin/kibana
log [22:22:27.423] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.440] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [22:22:27.448] [info][status][plugin:kbn_vislib_vis_types@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.456] [info][status][plugin:markdown_vis@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.459] [info][status][plugin:metric_vis@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.463] [info][status][plugin:spyModes@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.465] [info][status][plugin:statusPage@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.466] [info][status][plugin:table_vis@1.0.0] Status changed from uninitialized to green - Ready
log [22:22:27.469] [info][listening] Server running at http://0.0.0.0:5601
log [22:22:27.480] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to green - Kibana index ready
六、kibana简单使用
访问kibana地址：10.0.0.31:5601
第一次登录我们创建一个elk的es-error索引
添加之前写的systemlog索引
*为正则匹配
添加message和path字段
运用搜索栏功能，我们搜soft关键字
ELK日志平台搭建基本搞掂
标签：ELK,31,kibana,elasticsearch,nfs,日志,root,logstash,搭建 来源： https://www.cnblogs.com/Xafei/p/16634895.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。