处理原始日志

日志从moogoo导出来的

{ "mobile" : "13612345678", "isp" : "中国移动_广东", "time" : ISODate("2017-10-16T18:39:51.245Z"), "success" : true }

要解决时间问题:

logstash修改时间

发现日志已是json格式,想用date插件实现time字段赋值给@timestap字段,未果.(因为之前做的都是非json格式日志的时间匹配,先grok 后date)

改造日志为:

{"mobile" : "15812345606", "province": "上海", "isp": "中国移动","@timestamp" : "2017-12-06T09:30:51.244Z", "success" : "false"}

{"mobile" : "15812345607", "province": "河北", "isp": "中国移动","@timestamp" : "2017-12-06T09:20:51.244Z", "success" : "true"}

{"mobile" : "15812345607", "province": "河北", "isp": "中国联通","@timestamp" : "2017-12-06T09:22:51.244Z", "success" : "false"}

{"mobile" : "15812345608", "province": "广东", "isp": "中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "true"}

{"mobile" : "15812345608", "province": "广东", "isp": "中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "false"}

{"mobile" : "15812345608", "province": "广东", "isp": "中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "false"}

这样日志@timestamp的时间就是日志文件的时间了.

统一日志统计需求

上面的几幅图基本说明了问题:三级分, 第一级: 按照省份分 第二级: 按照isp分 第三级: 每个isp的标出成功失败比例

进一步处理日志:

mutate 拆封字段

input { stdin { codec => "json" } }

filter {

    if [success] == "true" {   // 这里true必须是字符串,否则lg启动会报错

        mutate { rename => ["sucess", "status_true"] }

    }

    else {

        mutate { rename => ["sucess", "status_false"] }

    }

}

output {

    stdout { codec => rubydebug }

    elasticsearch {

        hosts => [ "localhost:9200" ]

    }

}

即把日志的 "success" : "false" 拆分成2个字段:

status_true:true

status_false:false

中途遇到的问题:

1. codec => json失效.
   
   原因是: json数据中间本来逗号 不小心少了个逗号

2. mutate 没成功
   
   日志是

{"mobile" : "15812345608", "province": "广东", "isp": "中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : false}

改日志为

# 最后一个字段改成字符串即可

{"mobile" : "15812345608", "province": "广东", "isp": "中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : "false"}

最终日志入库展示

接下来就是kibana出图了

但目标是



目前还没实现百分比.

既然他能分两级我就这样排序: 能看到个数了

todo: 研究百分比

{"mobile" : "15812345606", "isp": "上海_中国移动","@timestamp" : "2017-12-06T09:30:51.244Z", "success" : false}

{"mobile" : "15812345607", "isp": "河北_中国移动","@timestamp" : "2017-12-06T09:20:51.244Z", "success" : true}

{"mobile" : "15812345607", "isp": "河北_中国联通","@timestamp" : "2017-12-06T09:22:51.244Z", "success" : false}

{"mobile" : "15812345608", "isp": "广东_中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : true}

{"mobile" : "15812345608", "isp": "广东_中国移动","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : false}

{"mobile" : "15812345608", "isp": "广东_中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : false}

{"mobile" : "15812345608", "isp": "广东_中国电信","@timestamp" : "2017-12-06T09:23:51.244Z", "success" : true}

最后折衷了下,采用目前方案

绘制方法: