elk安装(这个是初级的可以把这个套件安上)

2022-02-21 22:14:57

http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html

ELK其实并不是一款软件，而是一整套解决方案，是三个开源软件Elasticsearch,Logstash和Kibana。

Elasticsearch是一个实时的分布式搜索和分析引擎，它可以用于全文搜索以及分析。它是一个建立在全文搜索引擎ApacheLucene基础上的搜索引擎，便用java语言编写。

主要特点：实时分析；分布式实时文件　存储，并将每个字段都编入索引；文档导向，所有的对象全部是文档；高可用性，易扩展，支持集群（cluster）分片和复制（shards和replicas）;支持JSON

集群：

分片和复制

Logstash:是一个具有实时渠道能力的数据收集引擎。

主要特点：几乎可以访问任何数据；可以和多种外部应用结合；支持弹性扩展

它由三个主要部分组成：shipper(发送日志数据);broker(收集数据，缺省内置redis)indexer(数据写入)

shipper是托运者的意思；很多运用场景都没有用logstash作为shipper,或者说没有用elasticsearch作为数据存储也就是说没有indexer（indexer有索引器的意思）；logstash怎么配置logstash进程，然后运用到斧的日志管理架构中最合适它位置。

kibana:是一款基于apache开源协议，使用javascript语言编写，为elasticsearch提供分析和可视代的web平台。它可以在elasticsearch的索引中查找，交互数据，并生成各种维度的表图。

ELK协义体系结构。基本流程是shipper负责从各种数据源里采集数据，然后发送到broker,indexer将存放在broker中的数据再写入Elasticsearch,elasticsearch对这些数据创建索引，然后由kibana对其进行各种分析并以图表展示：

安装elasticsearch需要java8:先安装java8

解压即可,然后修改参数：

sh-4.1$ egrep -v '^#' config/elasticsearch.yml

cluster.name: iseastar

node.name: node_1

path.data: /usr/local/elasticsearch/data

path.logs: /usr/local/elasticsearch/logs

network.host: 123.xx.xx.xxx

http.port:

sh-4.1$ pwd

/usr/local/elasticsearch

对系统参数调整：

sh-4.1# tail - /etc/sysctl.conf

vm.max_map_count =

#在/etc/sysctl.conf追加上面一条

#并执行命令:

sysctl -p

天配置文件中追加如下两行

sh-4.1# tail - /etc/security/limits.conf

* hard nofile

* soft nofile

# cat /etc/security/limits.d/-nproc.conf

# Default limit for number of user's processes to prevent

# accidental fork bombs.

# See rhbz # for reasoning.

*          soft    nproc

root       soft    nproc     unlimited

#在这里修改

elasticsearch是一个近似实时的搜索平台，从索引文档到可搜索有些延迟，通常为１秒。

启动elasticsearch不能以root开启动，只能以普通用户启动

useradd elastic

 passwd elastic

改变目录的属组履主

chown -R elastic:elastic ELK/

以DEMO的方式启动：

./bin/elasticsearch

logstash安装很简单，解压做个软链接，然后修改一下配置文件：

lrwxrwxrwx    root    root       Nov  : logstash -> /usr/local/ELK/logstash-5.0.

测试logstash

sh-4.1# pwd

/usr/local/logstash

sh-4.1# ./bin/logstash -e 'input { stdin { } } output { stdout { } }'

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

The stdin plugin is now waiting for input:

[--24T09::,][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}

[--24T09::,][INFO ][logstash.pipeline        ] Pipeline main started

[--24T09::,][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>}

--24T01::.694Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.716Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.718Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.719Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.279Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.499Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.692Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.136Z iZ2zeiqn00z9x5uu126tjpZ

--24T01::.992Z iZ2zeiqn00z9x5uu126tjpZ

配置　logstash

sh-4.1# mkdir /usr/local/logstash/etc

sh-4.1# vim /usr/local/logstash/etc/hello_search.conf

sh-4.1# cat /usr/local/logstash/etc/hello_search.conf

input {

    stdin {

        type =>"human"

 }

}

output {

    stdout {

        codec =>rubydebug

    }

    elasticsearch {

        host =>"123.57.15.154"

        port =>

    }

}

sh-4.1#

启动：有报错！！！！（先不管最后再解决）

sh-4.1# ./logstash -e 'input {stdin {} } output {stdout {} }'

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

The stdin plugin is now waiting for input:

[--25T16::,][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}

[--25T16::,][INFO ][logstash.pipeline        ] Pipeline main started

[--25T16::,][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>}

输入hello world

sh-4.1# ./logstash -e 'input {stdin {} } output {stdout {} }'

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

The stdin plugin is now waiting for input:

[--25T16::,][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}

[--25T16::,][INFO ][logstash.pipeline        ] Pipeline main started

[--25T16::,][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>}

hello world!!!!

--25T08::.336Z iZ2zeiqn00z9x5uu126tjpZ hello world!!!!

这里通过stdin输入信息，然后通过stdout输出信息。在输入hello world后logstash将处理后的信息输出到屏幕

# ./bin/logstash -e "input {stdin{}}output{stdout{codec=>rubydebug}}"

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

The stdin plugin is now waiting for input:

[--27T00::,][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}

[--27T00::,][INFO ][logstash.pipeline        ] Pipeline main started

[--27T00::,][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>}

hello world

{

    "@timestamp" => --26T16::.365Z,

      "@version" => "",

          "host" => "iZ2zeiqn00z9x5uu126tjpZ",

       "message" => "hello world",

          "tags" => []

}

logstash提供了一个shell脚本叫logstash方便快速运行。它支持一下参数：

-e:意即执行。我们在"Hello World"的时候已经用过这个参数了。事实上你可以不写任何具体配置，直接运行:

./bin/logstash -e ''

达到相同效果。这个参数的默认值是下面这样：

input {

    stdin {}

}

output {

    stdout {}

}

--config或-f:意即文件。真实运用中，我们会写很长的配置，甚至可能超过shell所能支持的1024个字符长度。所以我们必把配置固化到文件，然后通过:

./bin/logstash -f file_name.conf

#这样的形式来运行

些外，logstash还担供一个方便我们规划和书写配置的小功能。你可以直接用:

./bin/logstash -f /etc/logstash.d/

来运行。loogstash会自动读取

/etc/logstsh.d/目录下所有的文本文件，然后在自己内存里拼接成一个完整的大配置文件，再去执行

--configtest 或 -t:意即测试。用来测试logstash读取到的配置文件语法是否能正常解析。logstash配置语法是用grammar.treetop定义的尤其是使用了上一条提到的读取目录方式折读者，尤其要提前测试。

./bin/logstash -t -f conf/st.conf

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

Configuration OK

[--27T01::,][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

sh-4.1# /usr/local/logstash/bin/logstash -f /usr/local/logstash/etc/hello_search.conf

Sending Logstash's logs to /usr/local/logstash/logs which is now configured via log4j2.properties

[--24T10::,][ERROR][logstash.outputs.elasticsearch] Unknown setting 'host' for elasticsearch

[--24T10::,][ERROR][logstash.outputs.elasticsearch] Unknown setting 'port' for elasticsearch

[--24T10::,][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n\tstdin {\n\t\ttype => \"human\"\n\t}\n}\n\noutput {\n\tstdout {\n\t\tcodec => rubydebug\n\t}\n\telasticsearch {\n\t\thost => \"123.xxx.xxx.xxx\"\n\t\tport => 9200\n\t}\n\n}\n\n", :reason=>"Something is wrong with your configuration."}

再次配置logstash:

# cat conf/logstash.conf

input {

    file {

        type => "stagefalcon"

        tags => "falcon"

        path => "/var/log/facon/access.log"

        start_position => beginning

        sincedb_path => "/usr/local/logstash/conf/access.sincedb"

    }

}

output {

    stdout {}

    redis {

        host => "127.0.0.1"

        port => ""

        data_type => "list"

        key => "stagefalcon"

    }

}

logstash启动成功：

/usr/local/logstash/bin/logstash -f /usr/local/logstash/conf/logstash.conf

stdin: 标准输入

安装kibana

这个很简单，也是解压就可以了，配置暂时用默认的，但要做两处修改：下面的ip是本机的外网ip

# egrep -v "(^#|^$)" /usr/local/kibana/config/kibana.yml

server.host: "123.xx.xx.xx"

elasticsearch.url: "http://123.xx.xx.xx:9200"

输入ip加端口:5601

logstash的相关插件下载:

https://github.com/logstash-plugins

注意配置hosts文件:(布属elasticsearch的服务器)

# cat /etc/hosts

127.0.0.1 localhost

::         localhost localhost.localdomain localhost6 localhost6.localdomain6

.17x1.xx.1xx iZ2zx

xx171.6x105 elasticsearch

xx.57.15.xx elasticsearch

码农公寓

相关文章