ELK的安装配置

来源:本站原创 Linux 超过44 views围观 0条评论

官方文档
https://www.elastic.co/guide/en/kibana/current/rpm.html

Logstash
1、安装jdk

Logstash的运行依赖于Java运行环境。
# yum -y install java-1.8.0
[root@elk01 ~]# java -v
Unrecognized option: -v
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[root@elk01 ~]# java -version
openjdk version “1.8.0_121”
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
[root@elk01 ~]#

运行报错
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one,
then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

步骤:

1 准备工作
2 安装配置 Elasticsearch
3 安装配置 Kibana
4 安装配置 Filebeat
5 安装配置 Logstash (可选)
6 安装配置 Nginx (可选)
7 单独配置一个客户端

直接YUM.安装
配置 ES 的 yum 源  所有都可以yum.使用这个源

导入签名:
导入elasticsearch PGP key
rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch
设置 yum 源
vim /etc/yum.repo.d/elk.repo
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

1 yum intall -y e/l/k/f
2 systemctl enable e/1/k/f
3 开防火墙
4 修改配置

Elasticsearch 配置文件在 /etc/elasticsearch/elasticsearch.yml ,
如果不使用 Logstash 或者 Logstash 与 Elasticsearch 不在同一服务器,
那么需要使 Elasticsearch 监听到指定的 IP 地址和端口,例如修改 elasticsearch.yml 中的下边两行:
network.host: 0.0.0.0   –允许所有IP访问
http.port: 9200   –端口
检查安装,执行
[root@elk01 ~]# curl 192.168.142.135 9200   –elasticsearch
<script>var hashRoute = ‘/app/kibana’;
var defaultRoute = ‘/app/kibana’;

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>curl: (7) Failed to connect to 0.0.35.240: Invalid argument

编辑配置文件 /etc/kibana/kibana.yml ,修改下列两行
server.port 5601   –这个配置有错自己排查
server.host 0.0.0.0  –这个配置有错自己排查

[root@elk01 ~]# curl 192.168.142.135 5601 —kibana
<script>var hashRoute = ‘/app/kibana’;
var defaultRoute = ‘/app/kibana’;

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>curl: (7) Failed to connect to 0.0.21.225: Invalid argument
[root@elk01 ~]#

配置 Logstash 的输入和输出,新建输入输出配置文件:
你会发现logstash 5.x里面 logstash调试命令没有了.坑爹

find / -name logstash -type f 查一下程序放在这里
/usr/share/logstash/bin/logstash

测试一下
/usr/share/logstash/bin/logstash -e ‘input { stdin{} } output { stdout{ codec => rubydebug} }’
结果
{
    “@timestamp” => 2017-04-21T04:51:03.336Z,
      “@version” => “1”,
          “host” => “elk02”,
       “message” => “”
}

——-报错处理——-
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
c[ERROR] 2017-09-26 19:06:06.370 [LogStash::Runner] elasticsearch – Unknown setting ‘host’ for elasticsearch
[ERROR] 2017-09-26 19:06:06.381 [LogStash::Runner] agent – Cannot create pipeline {:reason=>”Something is wrong with your configuration.”}

agent配置错误修改一下.本例是host字段写错了.

[2017-09-26T20:09:56,369][INFO ][org.logstash.beats.BeatsHandler] Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: -12, from: /172.30.100.139:48966
[2017-09-26T20:09:56,374][INFO ][org.logstash.beats.BeatsHandler] Exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: -1, from: /172.30.100.139:48966

Invalid Frame Type—-无效类型,本例是升级filebeat后解决.

——报错处理——-

 

我们可以使用curl命令发送请求来查看ES是否接收到了数据:
curl ‘http://localhost:9200/_search?pretty’

使用配置文件

使用-e参数 在命令行中指定配置是很常用的方式,不过如果需要配置更多设置则需要很长的内容。这种情况,
我们首先创建一个简单的配置文件,并且指定logstash使用这个配置文件。如我们创建一个文件名是”logstash-simple.conf”
的配置文件并且保存在和Logstash相同的目录中。内容如下:

input { stdin { } }
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}
接下来,执行命令:
bin/logstash -f logstash-simple.conf

测试一下写入es
[root@elk02 ~]# /usr/share/logstash/bin/logstash -e ‘input { stdin{} } output { elasticsearch { hosts => [“192.168.142.137:9200”] index => “logstash-%{+YYYY.MM.dd}” } }’
在ES上可以看到
@timestamp:April 21st 2017, 13:15:59.151 @version:1 host:elk02 message: _id:AVuO8DvRQjoNvLqUPbr9 _type:logs _index:logstash-2017.04.21 _score: –

vi /etc/logstash/conf.d/first-logstash.conf
文件内容如下:
输入:设置监听 5044 端口,接收 beats 的输入数据
输出:将数据输出到 Elasticsearch
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => “localhost:9200”
    manage_template => false
    index => “%{[@metadata][beat]}-%{+YYYY.MM.dd}”
    document_type => “%{[@metadata][type]}”
  }
}
测试
[root@elk01 ~]# curl 192.168.142.135 5044 –filebeat
<script>var hashRoute = ‘/app/kibana’;
var defaultRoute = ‘/app/kibana’;

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>curl: (7) Failed to connect to 0.0.19.180: Invalid argument
[root@elk01 ~]#

修改 Filebeat 将日志发送给 Logstash

Filebeat 可以将日志输入到 Elasticsearh,如刚才的配置。它也可以将日志输入给 Logstash,由 Logstash 处理日志,
再将处理后的日志数据输入到 Elasticsearch。下边配置 Filebeat 将日志 输入到 Logstash。

编辑 Filebeat 配置文件:

vi /etc/filebeat/filebeat.yml
注释掉 Elasticsearch output 的相关设置:

#—————————– Logstash output ——————————–
output.logstash:
  # The Logstash hosts
  hosts: [“localhost:5044”]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: [“/etc/pki/root/ca.pem”]

  # Certificate for SSL client authentication
  #ssl.certificate: “/etc/pki/client/cert.pem”

  # Client Certificate Key
  #ssl.key: “/etc/pki/client/cert.key”

3 ysstemctl restart e/1/k/f

配置完你会发现kibana上没办法看到日志

index默认显示这个因为是默认用logstash传送日志,但配置是用filebeat传送日志,所以这里改一下

logstash-*

——报错信息—
Unable to fetch mapping. Do you have indices matching the pattern?
Patterns allow you to define dynamic index names using * as a wildcard. Example: logstash-*
——报错处理——-
index 改成
filebeat-*即可
Create a new default index ‘filebeat-*’ and click on the ‘Create’ button.
——-完成——-

排错
问题1
5601端口是起来的 但只能本地telnet 防火墙是开的

netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      11175/nginx: master
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      972/sshd           
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      91600/node         

tcp6       0      0 :::9200                 :::*                    LISTEN      85301/java         
tcp6       0      0 :::5044                 :::*                    LISTEN      60427/java         
tcp6       0      0 :::9300                 :::*                    LISTEN      85301/java         

[root@elk01 ~]# telnet 127.0.0.1 5601
Trying 127.0.0.1…
Connected to 127.0.0.1.

问题1  kibana提示
Index Patterns: Please specify a default index pattern
且不能create index

问题2
客户端报错
blish.write_bytes=273
2017-04-19T06:48:43-04:00 ERR Failed to publish events caused by: read tcp 192.168.142.134:33392->192.168.142.137:5044: i/o timeout
2017-04-19T06:48:43-04:00 INFO Error publishing events (retrying): read tcp 192.168.142.134:33392->192.168.142.137:5044: i/o timeout

官方文档
http://udn.yyuap.com/doc/logstash-best-practice-cn/codec/json.html
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7
https://www.howtoforge.com/tutorial/how-to-install-elastic-stack-on-centos-7/#step-install-and-configure-elasticsearch

文章出自:CCIE那点事 http://www.jdccie.com/ 版权所有。本站文章除注明出处外,皆为作者原创文章,可自由引用,但请注明来源。 禁止全文转载。
本文标题:ELK的安装配置
本文链接:http://www.jdccie.com/?p=3571转载请注明转自CCIE那点事
如果喜欢:点此订阅本站
  • 相关文章
  • 为您推荐
  • 各种观点