ELK 日志管理平台

1>Elasticsearch是个开源分布式搜索引擎，提供搜集、分析、存储数据三大功能。它是基于Lucene（一个全文检索引擎的架构）开发的分布式存储检索引擎，可用来存储各类日志，采用 Java 开发，可通过 RESTful Web 接口，通过浏览器来与 Elasticsearch 通信。它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等;负责存储最终数据、建立索引、提供搜索功能。

2>Logstash 主要是用来日志的搜集、分析、过滤日志的工具，负责采集日志，支持大量的数据获取方式。一般工作方式为c/s架构，client端安装在需要收集日志的主机上，server端负责将收到的各节点日志进行过滤、修改等操作在一并发往elasticsearch上去。

另外，它使用JRuby 语言编写，运行在Java虚拟机（JVM）上，是一款强大的数据处理工具，可以实现=数据传输、格式处理、格式化输出。 Logstash 还具有强大的插件功能，常用于日志处理。

3>Kibana 也是一个开源和免费的工具，它基于Node.js开发，负责提供可视化界面，Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助汇总、分析和搜索重要数据日志。

4>Filebeat隶属于Beats，其作为原logstash-forwarder的替代来完成。Filebeat轻量级的日志传输工具，，也可称为：是一个轻量级的日志采集器。它可以读取系统、nignx、apache等logs文件，监控日志文件，传输数据到Elasticsearch或者Logstash，最后在Kibana中实现可视化。目前Beats包含四种工具：

Packetbeat（搜集网络流量数据）
Topbeat（搜集系统、进程和文件系统级别的 CPU 和内存使用情况等数据）
Filebeat（搜集文件数据）：FileBeat采集数据时是Json化的，这个日志采集工具相当轻量级，对系统资源的消耗很少。而LogStash的优点则是有丰富的Filter插件，用于对数据作粗处理，占用资源高。一般，我们使用FileBeat采集数据之后会传输给Kafka消息队列，然后LogStash采集消息队列中的数据，作过滤处理，最后将数据传输给ES。
Winlogbeat（搜集 Windows 事件日志数据）

Elastic Stack的优点

Elastic Stack 目前已成为机器数据分析，或者说实时日志处理领域，开源界的第一选择，和传统的日志处理方案相比，Elastic Stack 具有如下几个优点：
1）处理方式灵活。Elasticsearch 是实时全文索引，不需要像 storm 那样预先编程才能使用；
2）配置简易上手。Elasticsearch 全部采用 JSON 接口，Logstash 是 Ruby DSL 设计，都是目前业界最通用的配置语法设计；
3）检索性能高效。虽然每次查询都是实时计算，但是优秀的设计和实现基本可以达到全天数据查询的秒级响应；
4）集群线性扩展。不管是 Elasticsearch 集群还是 Logstash 集群都是可以线性扩展的；
5）前端操作炫丽。Kibana 界面上，只需要点击鼠标，就可以完成搜索、聚合功能，生成炫丽的仪表板。

一、介绍

1、组成

ELK由Elasticsearch、Logstash和Kibana三部分组件组成；
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用
kibana 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

2、四大组件
Logstash: logstash server端用来搜集日志；
Elasticsearch: 存储各类日志；
Kibana: web化接口用作查寻和可视化日志；
Logstash Forwarder: logstash client端用来通过lumberjack 网络协议发送日志到logstash server；

3、工作流程

在需要收集日志的所有服务上部署logstash，作为logstash agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana 来结合自定义搜索进行页面展示。

下面是在两台节点上都安装一下环境。

nfs-31 
rsync-41

二、安装JDk（两个节点都要安装）

配置阿里源：wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
Logstash的运行依赖于Java运行环境，Elasticsearch 要求至少 Java 7。
[root@controller ~]# yum install java-1.8.0-openjdk -y
[root@controller ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
1、关闭防火墙
systemctl stop firewalld.service #停止firewall
systemctl disable firewalld.service #禁止firewall开机启动
2、关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

三、安装Elasticsearch

基础环境安装（两个节点都要安装，配置文件不一样）

1）下载并安装GPG Key
[root@elk-node1 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
 
2）添加yum仓库
[root@elk-node1 ~]# vim /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
 
3）安装elasticsearch
[root@elk-node1 ~]# yum install -y elasticsearch
4）添加自启动
chkconfig --add elasticsearch
5）启动命令
systemctl daemon-reload
systemctl enable elasticsearch.service

6）修改配置
[root@nfs-31 ~]#cd /etc/elasticsearch/
[root@nfs-31 /etc/elasticsearch]#ls
elasticsearch.yml  logging.yml  scripts
[root@nfs-31 /etc/elasticsearch]#cp elasticsearch.yml{,.bak}
[root@nfs-31 /etc/elasticsearch]#mkdir -p /data/es-data
[root@nfs-31 /etc/elasticsearch]#vim elasticsearch.yml

[root@nfs-31 /etc/elasticsearch]# grep '^[a-z]' elasticsearch.yml
cluster.name: hejianlai               //集群名称
node.name: elk-node1                  //节点名称
path.data: /data/es-data              //数据存放目录
path.logs: /var/log/elasticsearch/    //日志存放目录
bootstrap.memory_lock: true           //打开内存
network.host: 0.0.0.0                 //监听网络
http.port: 9200                       //端口
discovery.zen.ping.multicast.enabled: false                    //改为单播
discovery.zen.ping.unicast.hosts: ["10.0.0.31", "10.0.0.41"]

#注意
#data/es-data没权限，赋权限即可
 chown -R elasticsearch:elasticsearch /data/es-data/
 
#启动
systemctl start elasticsearch

[root@nfs-31 /etc/elasticsearch]#systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-08-20 20:46:06 CST; 5s ago
     Docs: http://www.elastic.co
  Process: 2912 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 2913 (java)
   CGroup: /system.slice/elasticsearch.service
           └─2913 /bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+...

访问地址10.0.0.31:9200

安装ES插件

#统计索引数
[root@nfs-31 /etc/elasticsearch]#curl -i -XGET 'http://10.0.0.31:9200/_count?pretty' -d '
> "query":{
>     "match_all":{}
> }'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 95

{
  "count" : 0,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  }
}

#es插件，收费的不建议使用(这个不安装)
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install marvel-agent

#安装开源的elasticsearch-head插件
[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/share/elasticsearch/plugins/head

访问：10.0.0.31:9200/_plugin/head/

使用POST方法创建查询

使用GET方法查询数据

基本查询

41机器配置

[root@rsync-41 /etc/elasticsearch]#cat elasticsearch.yml
cluster.name: hejianlai
node.name: elk-node2
path.data: /data/es-data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.31", "10.0.0.41"]

#data/es-data没权限，赋权限即可
 chown -R elasticsearch:elasticsearch /data/es-data/

在构建Elasticsearch(ES)多节点集群的时候，通常情况下只需要将elasticsearch.yml中的cluster.name设置成相同即可，ES会自动匹配并构成集群。但是很多时候可能由于不同的节点在不同的网段下，导致无法自动获取集群。此时可以将启用单播，显式指定节点的发现。具体做法是在elasticsearch.yml文件中设置如下两个参数：

重启31机器开启41机器

[root@nfs-31 ~]#systemctl restart elasticsearch
[root@rsync-41 /etc/elasticsearch]#systemctl start elasticsearch

访问 10.0.0.31:9200/_plugin/head/

安装监控kopf

[root@nfs-31 ~]#/usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf
-> Installing lmenezes/elasticsearch-kopf...
Trying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip ...
Downloading ......................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed kopf into /usr/share/elasticsearch/plugins/kopf

访问 10.0.0.31:9200/_plugin/kopf/#!/cluster

四、安装Logstash(客户端即需要收集日志的机子，ES节点上都要安装)

官方文档地址：https://www.elastic.co/guide/en/logstash/current/index.html

31机器

#1、下载安装GPG Key
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

#2、添加yum仓库
vim /etc/yum.repos.d/logstash.repo
[logstash-2.1]
name=Logstash repository for 2.1.x packages
baseurl=http://packages.elastic.co/logstash/2.1/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

#3、安装logstash
yum install -y logstash
#4、测试数据
#简单的输入输出
[root@nfs-31 ~]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
hello world
2022-08-20T13:35:44.716Z nfs-31 hello world
hi afei 
2022-08-20T13:35:57.028Z nfs-31 hi afei
你好啊 渣渣辉  
Received an event that has a different character encoding than you configured. {:text=>"你好啊 渣渣\xE7辉", :expected_charset=>"UTF-8", :level=>:warn}
2022-08-20T13:36:14.242Z nfs-31 你好啊 渣渣xE7辉

2022-08-20T13:36:19.675Z nfs-31 
logstash shutdown completed
#可以使用rubydebug详细输出
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed

{
       "message" => "",
      "@version" => "1",
    "@timestamp" => "2022-08-20T13:37:34.060Z",
          "host" => "nfs-31"
}
mimi
{
       "message" => "mimi",
      "@version" => "1",
    "@timestamp" => "2022-08-20T13:37:40.516Z",
          "host" => "nfs-31"
}


#内容写进elasticsearch中
[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{hosts=>["10.0.0.31"]} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
hello afei
你好帅
啦啦啦啦
dasda 

[root@nfs-31 ~]#/opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["10.0.0.31:9200"]} stdout{ codec => rubydebug}}'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
北京
{
       "message" => "北京",
      "@version" => "1",
    "@timestamp" => "2022-08-20T13:41:44.333Z",
          "host" => "nfs-31"
}
afei
{
       "message" => "afei",
      "@version" => "1",
    "@timestamp" => "2022-08-20T13:42:05.132Z",
          "host" => "nfs-31"
}

Logstash shutdown completed

logstash日志收集配置文件编写

#1、交换式输入信息
[root@nfs-31 ~]#vim /etc/logstash/conf.d/logstash-01.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/logstash-01.conf
input { stdin { } }
output {
        elasticsearch { hosts => ["10.0.0.31:9200"]}
        stdout { codec => rubydebug }
}

#执行命令
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-01.conf
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Settings: Default filter workers: 1
Logstash startup completed
wuhan
{
       "message" => "wuhan",
      "@version" => "1",
    "@timestamp" => "2022-08-20T13:50:19.731Z",
          "host" => "nfs-31"
}
小阿飞  
Received an event that has a different character encoding than you configured. {:text=>"\xE9\x98小阿飞", :expected_charset=>"UTF-8", :level=>:warn}
{
       "message" => "\xE9\x98小阿飞",
      "@version" => "1",
    "@timestamp" => "2022-08-20T13:50:34.473Z",
          "host" => "nfs-31"
}

#收集系统日志
[root@nfs-31 ~]#vim /etc/logstash/conf.d/systemlog.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/systemlog.conf
input{
    file {
    path => "/var/log/messages"
    type => "sysstem"
    start_position => "beginning"
    }
}
output{
    elasticsearch{
    hosts => ["10.0.0.31:9200"]
    index => "systemlog-%{+YYYY.MM.dd}"
    }
}

#放在后台执行
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/systemlog.conf &
[1] 4471
[root@nfs-31 ~]#OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

收集elk错误日志配置文件编写

[root@nfs-31 ~]#vim /etc/logstash/conf.d/elk_log.conf
[root@nfs-31 ~]#cat /etc/logstash/conf.d/elk_log.conf
input {
    file {
      path => "/var/log/messages"
      type => "system"
      start_position => "beginning"
    }
}
input {
    file {
       path => "/var/log/elasticsearch/hejianlai.log"
       type => "es-error"
       start_position => "beginning"
       codec => multiline {
          pattern => "^["                                 //正则匹配[开头的为一个事件
          negate => true
          what => "previous"
        }
    }
}
output {
  
    if [type] == "system"{
        elasticsearch {
           hosts => ["10.0.0.31:9200"]
           index => "systemlog-%{+YYYY.MM.dd}"
        }
    }
  
    if [type] == "es-error"{
        elasticsearch {
           hosts => ["10.0.0.31:9200"]
           index => "es-error-%{+YYYY.MM.dd}"
        }
    }
}

#放入后台运行
[root@nfs-31 ~]#/opt/logstash/bin/logstash -f /etc/logstash/conf.d/elk_log.conf &
[2] 4523

五、安装Kibana

官方下载地址：https://www.elastic.co/downloads/kibana

官方最新的版本出来了6.3.1太新了，下载后出现很多坑后来就下了4.3.1的·先用着吧

#1、kibana的安装：
[root@nfs-31 ~]#cd /usr/local/
[root@nfs-31 /usr/local]#wget https://download.elastic.co/kibana/kibana/kibana-4.6.3-linux-x86_64.tar.gz
[root@nfs-31 /usr/local]# tar -xf kibana-4.6.3-linux-x86_64.tar.gz 
[root@nfs-31 /usr/local]#ln -s /usr/local/kibana-4.6.3-linux-x86_64 /usr/local/kibana
[root@nfs-31 /usr/local]#cd kibana
[root@nfs-31 /usr/local/kibana]#ls
bin     data              LICENSE.txt  node_modules  package.json  src
config  installedPlugins  node         optimize      README.txt    webpackShims

#2、修改配置文件
[root@nfs-31 /usr/local/kibana]#cd config/
[root@nfs-31 /usr/local/kibana/config]#pwd
/usr/local/kibana/config
[root@nfs-31 /usr/local/kibana/config]#cp kibana.yml kibana.yml.bb 
[root@nfs-31 /usr/local/kibana/config]#vim kibana.yml
[root@nfs-31 /usr/local/kibana/config]#grep -Ev "^#|^$" kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.0.0.31:9200"
kibana.index: ".kibana"

#3、screen是一个全屏窗口管理器，它在几个进程（通常是交互式shell）之间复用物理终端。每个虚拟终端提供DEC VT100的功能。
yum install -y screen

#4、启动screen命令后运行kibana最后按ctrl+a+d组合键让其在单独的窗口里运行。
[root@nfs-31 /usr/local/kibana/config]#/usr/local/kibana/bin/kibana
  log   [22:21:42.921] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:21:42.940] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
[root@nfs-31 /usr/local/kibana/config]#screen -ls
There is a screen on:
        4834.pts-0.nfs-31       (Attached)
1 Socket in /var/run/screen/S-root.

[root@nfs-31 /usr/local/kibana/config]#/usr/local/kibana/bin/kibana
  log   [22:22:27.423] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.440] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [22:22:27.448] [info][status][plugin:kbn_vislib_vis_types@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.456] [info][status][plugin:markdown_vis@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.459] [info][status][plugin:metric_vis@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.463] [info][status][plugin:spyModes@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.465] [info][status][plugin:statusPage@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.466] [info][status][plugin:table_vis@1.0.0] Status changed from uninitialized to green - Ready
  log   [22:22:27.469] [info][listening] Server running at http://0.0.0.0:5601
  log   [22:22:27.480] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to green - Kibana index ready