Logstash传输给ES的数据会自动映射为5索引，5备份，字段都为text的的索引。这样基本上无法进行数据分析。
所以必须将Logstash的数据按照既定的格式存储在ES中，这时候就要使用到ES模板技术了。在ES中可以定义自定义模板和动态模板，之后es会自动将相关索引映射为模板规定的格式
编译动态映射模板文件bigdata.template：
在Json日志文件中的KEY的位置不固定、或字段数不明确时使用动态映射模板
{
"template": "bigdata-template",
"settings": {
"index.number_of_shards": ,
"number_of_replicas":
},
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"omit_norms": true
},
"dynamic_templates": [{
"message_field": {
"match": "message",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "analyzed",
"omit_norms": true,
"fielddata": {
"format": "disabled"
}
}
}
}, {
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed",
"doc_values": true
}
}
}],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
dynamic_templates 就是配置具体的动态模板匹配项
"match_mapping_type": "string" 是匹配固定的类型
"match": "time" 匹配字段名为time的数据
"unmatch": "data" 不匹配字段名为data的数据
mapping 就是将匹配的数据项映射为定义的数据类型
Logstash配置文件 nginx.conf：
input {
file {
path => "/usr/local/openresty/nginx/logs/user2.log"
type => "nginx-bigdata"
codec => "json"
}
}
filter {
json {
source => "u_data"
}
}
output {
if [type] == "nginx-bigdata" {
elasticsearch {
hosts => ["172.17.213.60:9200", "172.17.213.61:9200"]
index => "nginx-bigdata"
manage_template => false
template_overwrite => true
template_name => "bigdata-template"
template => "/usr/local/logstash-6.2.4/bigdata.template"
document_type => "nginx-bigdata"
}
}
}
Nginx的配置文件中关于JSON日志格式转换的配置：(此处我只保留了需要的一个字段范围)
escape=json :nginx 1.11.8版本后才提供此参数
log_format userlog escape=json '{"u_data":"$u_data","@timestamp":"$time_iso8601"}';
...
access_log logs/user.log userlog;
产生的日志格式：
{"u_data":"{\"appid\":\"nchaopai\",\"args\":{\"contentId\":0,\"duration\":111811,\"parentId\":0,\"totaltime\":0,\"type\":0},\"bk\":\"-\",\"cp_ver\":\"3.0.5\",\"duid\":\"2cba98f8ddc18464\",\"e\":\"nchaopai.main.stay-duration\",\"os\":\"A\",\"ts\":1572584611,\"ver\":\"8.11.11\"}"}
之后在Kibana里看到就是这样的：
常用格式如下：
log_format log_json escape=json '{"timestamp": "$time_local",'
'"remote_addr": "$remote_addr",'
'"referer": "$http_referer",'
'"request": "$request",'
'"statu": "$status",'
'"byte": "$body_bytes_sent",'
'"agen": "$http_user_agent",'
'"x_forwarded": "$http_x_forwarded_for",'
'"up_resp_time": "$upstream_response_time",'
'"request_time": "$request_time"}';
参考资料：https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/filter/json.html