前言：本文源于天天是雾霾新闻，我想利用kibana画一下一线城市雾霾图，希望对想利用经纬度在kibana绘图和获取日志本身时间绘图的同学有所帮助。有什么疑问或者纠错，可以给我发邮件
一、数据准备
为了方便起见，我模拟臆造了json格式的数据
{"timestamp":"2017-01-13T13:13:32.2516955+08:00","deviceId":"myFirstDevice","windSpeed":17,"haze":284,"city":"Beijing","lat":33.9402,"lon":116.40739}
模拟数据我用的是c#，大概如下：
static void SendingRandomMessages()
{
//var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);
int len = 4;
string[] citys = { "Beijing", "Shangjhai", "Guangzhou", "Shenzhen" };
int[] avgWindSpeed = { 10, 16, 5, 7 };
int[] avgWindSpeed1 = { 10, 16, 5, 7 };
int[] avgHaze1 = { 200, 100, 50, 49 };
int[] avgHaze = { 200, 100, 50, 49 };
double[] latitude = { 39.3402, 31.23042, 23.13369, 22.54310 };
double[] longitude = { 116.40739, 121.47370, 113.28880, 114.057860 };
Random rand = new Random();
while (true)
{
try
{
for (int i = 0; i < len; i++)
{
avgWindSpeed[i] = avgWindSpeed1[i] + rand.Next(1, 11);
avgHaze[i] = avgHaze1[i] + rand.Next(10, 100);
var telemetryDataPoint = new
{
timestamp = DateTime.Now,
deviceId = "myFirstDevice",
windSpeed = avgWindSpeed[i],
haze = avgHaze[i],
city = citys[i],
lat = latitude[i],
lon = longitude[i]
};
var message = JsonConvert.SerializeObject(telemetryDataPoint);
//eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes(message)));
Console.WriteLine("{0} > Get message: {1}", "eventHubName", message);
}
}
catch (Exception exception)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message);
Console.ResetColor();
}
Thread.Sleep(200);
}
}
此处我是作为消息发到一个eventhub中，你正确的做法可以将json写到文本文件中，再通过logstash读取即可
我的目的有两个：
- 获取数据中的lat、lon经纬度数据在kibana Map中进行绘图
- 获取数据中的timestamp作为我在kibana中的搜索时间，默认情况下是@timestamp
二、解决问题的整体思路
- lat、lon本质上是float类型，此处需要设计一个mapping
- 日志内的时间，本质上应该是个字符串。我们得先卡出这个字段，然后用date match进行转换
三、解决实例
1. mapping的设计，我给出一个template
{
"template": "geo-*",
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"_default_": {
"_all": {"enabled": true, "omit_norms": true},
"dynamic_templates": [ {
"message_field": {
"match": "message",
"match_mapping_type": "string",
"mapping": {
"type": "string", "index": "analyzed", "omit_norms": true
}
}
}, {
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string", "index": "analyzed", "omit_norms": true,
"fields": {
"raw": {"type": "string", "index": "not_analyzed", "ignore_above": 256}
}
}
}
} ],
"properties": {
"@version": { "type": "string", "index": "not_analyzed" },
"lonlat": { "type": "geo_point" }
}
}
}
}
大概解释如下：
"template": "geo-*"，所有geo开头的索引，都将会套用这个template配置
"lonlat": { "type": "geo_point" } 这个定义了lonlat为geo_point类型，为以后Map绘制奠定基础，这个是关键。
NOTE: 这个lonlat名字不能取成特定的关键名字？，我取成location一直报错。
更加详细的介绍你可以查看官网文档
2. 给出logstash的配置文件
input {
file {
path => "/opt/logstash/1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
json {
source => "message"
}
mutate {
add_field => [ "[lonlat]", "%{lon}" ]
add_field => [ "[lonlat]", "%{lat}" ]
}
date{
match=>["timestamp","ISO8601"]
timezone => "Asia/Shanghai"
"target" => "logdate" }
}
output {
stdout { codec => rubydebug }
elasticsearch
{
hosts =>"wb-elk"
index => "geo-%{+YYYY.MM.dd}"
# template => "/opt/logstash/monster.json"
# template_overwrite => true
}
}
大概解释一下：
json {source => "message"}这个能将json数据格式分解出一个个字段
mutate 这个是向geo_point中加入经纬度数据
date {match=>}这个是将匹配json数据分解出来的timestamp，并以时间格式赋值给logdate
output中注释掉的是template文件。我采用的是直接put template的方式，因此注释掉了。两个方法都可行
更加细致的理解，需要你去查看文档，努力学习
3.运行过程
上图中的location 应该为lonlat。大致stdout应该如上