iLogtail使用入门-iLogtail 采集Prometheus 数据

2022-03-31 05:23:22

采集配置

iLogtail 的采集配置全面兼容Prometheus 配置文件（以下介绍为1.0.30版本+）。

参数

描述

默认值

Yaml

yaml格式的采集配置

ConfigFilePath

采集配置文件路径，当Yaml参数生效时，此字段被忽略。

AuthorizationPath

鉴权路径，如 consul_sd_config 配置中authorization 参数生效时的鉴权文件查找路径。

当Yaml 参数生效时，默认路径为程序运行目录；

当ConfigFilePath参数生效时，默认路径为其配置文件所在路径。

以下是一个简单的prometheus 采集配置。

{

    "inputs":[

        {

            "detail":{

                "Yaml":"global:

  scrape_interval: 15s 

  evaluation_interval: 15s 

scrape_configs:

  - job_name: "prometheus"

    static_configs:

      - targets: ["exporter:18080"]"

            },

            "type":"service_prometheus"

        }

    ]

}

采集数据格式

iLogtail Prometheus 采集的Metrics 数据与日志同样遵循iLogtail 的传输层协议，目前传输数据字段为以下格式。

传输字段

含义

__name__

指标名，与Prometheus exporter 指标名相同。

__labels__

指标Label 熟悉，数据格式为{label_a_nane}#$#{label_a_value}|{label_b_nane}#$#{label_b_value}，例如instance#$#exporter:18080|job#$#prometheus

__time_nano__

采集时间

__value__

指标值

E2E 快速上手

目前iLogtail 已经集成了prometheus 的E2E测试，可以在iLogtail 的根路径快速进行上手验证。

测试命令：TEST_SCOPE=input_prometheus TEST_DEBUG=true make e2e（开启DEBUG 选项可以查看传输数据明细）

TEST_DEBUG=true TEST_PROFILE=false  ./scripts/e2e.sh behavior input_prometheus

=========================================

input_prometheus testing case

=========================================

load log config /home/liujiapeng.ljp/data/ilogtail/behavior-test/plugin_logger.xml

2022-01-20 10:46:46 [INF] [load.go:75] [load] load config from: /home/liujiapeng.ljp/data/ilogtail/test/case/behavior/input_prometheus/ilogtail-e2e.yaml

2022-01-20 10:46:46 [INF] [controller.go:129] [WithCancelChain] httpcase controller is initializing....:

2022-01-20 10:46:46 [INF] [controller.go:129] [WithCancelChain] ilogtail controller is initializing....:

2022-01-20 10:46:46 [INF] [validator_control.go:53] [Init] validator controller is initializing....:

2022-01-20 10:46:46 [DBG] [validator_control.go:57] [Init] stage:add rule:fields-check

2022-01-20 10:46:46 [DBG] [validator_control.go:65] [Init] stage:add rule:counter-check

2022-01-20 10:46:46 [INF] [controller.go:129] [WithCancelChain] subscriber controller is initializing....:

2022-01-20 10:46:46 [INF] [controller.go:129] [WithCancelChain] boot controller is initializing....:

2022-01-20 10:46:46 [INF] [boot_control.go:37] [Start] boot controller is starting....:

Creating network "ilogtail-e2e_default" with the default driver

Building exporter

Step 1/8 : FROM golang:1.16

 ---> 71f1b47263fc

Step 2/8 : WORKDIR /src

 ---> Using cache

 ---> d76e92450cbb

Step 3/8 : COPY exporter/* ./

 ---> Using cache

 ---> 55c76b7af4a1

Step 4/8 : RUN go env -w GO111MODULE=on

 ---> Using cache

 ---> 0bbd054e5ca3

Step 5/8 : RUN go env -w GOPROXY=https://goproxy.cn,direct

 ---> Using cache

 ---> 907a360df1d5

Step 6/8 : RUN go build

 ---> Using cache

 ---> 6ef458eccfc2

Step 7/8 : EXPOSE 18080

 ---> Using cache

 ---> ab9ad470b110

Step 8/8 : CMD ["/src/exporter"]

 ---> Using cache

 ---> 6eafc56b0059


Successfully built 6eafc56b0059

Successfully tagged ilogtail-e2e_exporter:latest

Creating ilogtail-e2e_goc_1      ... done

Creating ilogtail-e2e_exporter_1 ... done

Creating ilogtail-e2e_ilogtail_1 ... done

2022-01-20 10:46:50 [INF] [subscriber_control.go:51] [Start] subscriber controller is starting....:

2022-01-20 10:46:50 [INF] [validator_control.go:81] [Start] validator controller is starting....:

2022-01-20 10:46:50 [INF] [logtailplugin_control.go:63] [Start] ilogtail controller is starting....:

2022-01-20 10:46:50 [INF] [logtailplugin_control.go:70] [Start] the 1 times load config operation is starting ...

2022-01-20 10:46:50 [INF] [grpc.go:69] [func1] the grpc server would start in 0s

2022-01-20 10:46:50 [INF] [httpcase_control.go:67] [Start] httpcase controller is starting....:

2022-01-20 10:46:50 [INF] [controller.go:129] [WithCancelChain] testing has started and will last 15s

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"promhttp_metric_handler_requests_in_flight" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"1" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"promhttp_metric_handler_requests_total" > Contents:<Key:"__labels__" Value:"code#$#200|instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"0" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"promhttp_metric_handler_requests_total" > Contents:<Key:"__labels__" Value:"code#$#500|instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"0" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"promhttp_metric_handler_requests_total" > Contents:<Key:"__labels__" Value:"code#$#503|instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"0" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"test_counter" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"0" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"up" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"1" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"scrape_duration_seconds" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"0.002" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"scrape_samples_scraped" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"5" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"scrape_samples_post_metric_relabeling" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"5" >

2022-01-20 10:46:54 [DBG] [validator_control.go:107] [func2] Time:1642646811 Contents:<Key:"__name__" Value:"scrape_series_added" > Contents:<Key:"__labels__" Value:"instance#$#exporter:18080|job#$#prometheus" > Contents:<Key:"__time_nano__" Value:"1642646811274" > Contents:<Key:"__value__" Value:"5" >

2022-01-20 10:47:05 [INF] [httpcase_control.go:107] [func2] httpcase controller is closing....:

2022-01-20 10:47:05 [INF] [httpcase_control.go:108] [func2] httpcase controller is cleaning....:

2022-01-20 10:47:05 [INF] [logtailplugin_control.go:101] [func1] ilogtail controller is closing....:

2022-01-20 10:47:05 [INF] [logtailplugin_control.go:102] [func1] ilogtail controller is cleaning....:

2022-01-20 10:47:05 [INF] [logtailplugin_control.go:112] [func1] ilogtail controller would wait 5s to deal with the logs on the way

2022-01-20 10:47:10 [INF] [validator_control.go:89] [func1] validator controller is closing....:

2022-01-20 10:47:10 [INF] [validator_control.go:90] [func1] validator controller is cleaning....:

2022-01-20 10:47:10 [INF] [subscriber_control.go:54] [func1] subscriber controller is closing....:

2022-01-20 10:47:10 [INF] [subscriber_control.go:74] [Clean] subscriber controller is cleaning....:

2022-01-20 10:47:10 [INF] [boot_control.go:40] [func1] boot controller is stoping....:

2022-01-20 10:47:10 [INF] [boot_control.go:48] [Clean] boot controller is cleaning....:

Stopping ilogtail-e2e_ilogtail_1 ... done

Stopping ilogtail-e2e_exporter_1 ... done

Stopping ilogtail-e2e_goc_1      ... done

Removing ilogtail-e2e_ilogtail_1 ... done

Removing ilogtail-e2e_exporter_1 ... done

Removing ilogtail-e2e_goc_1      ... done

Removing network ilogtail-e2e_default

2022-01-20 10:47:11 [INF] [controller.go:112] [Start] Testing is completed:

2022-01-20 10:47:11 [INF] [controller.go:122] [Start] the E2E testing is passed:

v1.3.8

=========================================

All testing cases are passed

========================================

本地Node Exporter 采集实战

  1. 准备Linux 环境。
  2. 下载NodeExporter，下载地址：https://prometheus.io/download/#node_exporter，并进行启动，启动后可以通过curl 127.0.0.1:9100/metrics 查看NodeExporter 的Metrics指标。
  3. 下载最新的ilogtail版本进行安装。

# 解压tar包

$ tar zxvf logtail-linux64.tar.gz


# 查看目录结构

$ ll logtail-linux64

drwxr-xr-x   3 500 500  4096 bin

drwxr-xr-x 184 500 500 12288 conf

-rw-r--r--   1 500 500   597 README

drwxr-xr-x   2 500 500  4096 resources


# 进入bin目录

$ cd logtail-linux64/bin

$ ll

-rwxr-xr-x 1 500 500 10052072 ilogtail_1.0.28 # ilogtail可执行文件

-rwxr-xr-x 1 500 500     4191 ilogtaild  

-rwxr-xr-x 1 500 500     5976 libPluginAdapter.so

-rw-r--r-- 1 500 500 89560656 libPluginBase.so

-rwxr-xr-x 1 500 500  2333024 LogtailInsight

  1. 创建采集配置目录。

# 1. 创建sys_conf_dir

$ mkdir sys_conf_dir


# 2. 创建ilogtail_config.json并完成配置。

##### logtail_sys_conf_dir取值为：$pwd/sys_conf_dir/

##### config_server_address固定取值，保持不变。

$ pwd

/root/bin/logtail-linux64/bin

$ cat ilogtail_config.json

{

     "logtail_sys_conf_dir": "/root/bin/logtail-linux64/bin/sys_conf_dir/",  


     "config_server_address" : "http://logtail.cn-zhangjiakou.log.aliyuncs.com"

}


# 3. 此时的目录结构

$ ll

-rwxr-xr-x 1  500  500 ilogtail_1.0.28

-rw-r--r-- 1 root root ilogtail_config.json

-rwxr-xr-x 1  500  500 ilogtaild

-rwxr-xr-x 1  500  500 libPluginAdapter.so

-rw-r--r-- 1  500  500 libPluginBase.so

-rwxr-xr-x 1  500  500 LogtailInsight

drwxr-xr-x 2 root root sys_conf_dir

  1. 设置采集配置文件，将下列内如写入sys_conf_dir/user_local_config.json文件，上述核心配置为plugin部分，配置说明我们启动了Prometheus 采集插件，采集端口为9100，并且我们将采集到的数据保存于node_exporter.log 文件。

{

    "metrics":{

        "##1.0##k8s-log-custom-test-project-helm-0":{

            "aliuid":"1654218965343050",

            "category":"container_stdout_logstore",

            "create_time":1640692891,

            "defaultEndpoint":"cn-beijing-b-intranet.log.aliyuncs.com",

            "delay_alarm_bytes":0,

            "enable":true,

            "enable_tag":false,

            "filter_keys":[


            ],

            "filter_regs":[


            ],

            "group_topic":"",

            "local_storage":true,

            "log_type":"plugin",

            "log_tz":"",

            "max_send_rate":-1,

            "merge_type":"topic",

            "plugin":{

                "inputs":[

                    {

                        "detail":{

                            "Yaml":"global:\n  scrape_interval: 15s\n  evaluation_interval: 15s\nscrape_configs:\n  - job_name: \"prometheus\"\n    static_configs:\n      - targets: [\"localhost:9100\"]"

                        },

                        "type":"service_prometheus"

                    }

                ],

                "flushers":[

                    {

                        "detail":{

                            "FileName":"./node_exporter.log"

                        },

                        "type":"flusher_stdout"

                    }

                ]

            },

            "priority":0,

            "project_name":"k8s-log-custom-test-project-helm",

            "raw_log":false,

            "region":"cn-beijing-b",

            "send_rate_expire":0,

            "sensitive_keys":[


            ],

            "shard_hash_key":[


            ],

            "tz_adjust":false,

            "version":1

        }

    }

}

  1. 启动iLogtail，查看采集数据。

# 启动采集

$ ./ilogtail_1.0.28

$ ps -ef|grep logtail

root       48453       1   ./ilogtail_1.0.28

root       48454   48453   ./ilogtail_1.0.28


# 查看采集数据

tailf node_exporter.json

2022-01-20 11:38:52 {"__name__":"promhttp_metric_handler_errors_total","__labels__":"cause#$#gathering|instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"0","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"promhttp_metric_handler_requests_in_flight","__labels__":"instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"1","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"promhttp_metric_handler_requests_total","__labels__":"code#$#200|instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"1393","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"promhttp_metric_handler_requests_total","__labels__":"code#$#500|instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"0","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"promhttp_metric_handler_requests_total","__labels__":"code#$#503|instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"0","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"up","__labels__":"instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"1","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"scrape_duration_seconds","__labels__":"instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"0.011","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"scrape_samples_scraped","__labels__":"instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"1189","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"scrape_samples_post_metric_relabeling","__labels__":"instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"1189","__time__":"1642649932"}

2022-01-20 11:38:52 {"__name__":"scrape_series_added","__labels__":"instance#$#localhost:9100|job#$#prometheus","__time_nano__":"1642649932033","__value__":"0","__time__":"1642649932"}

日志服务NodeExporter 采集实战

iLogtail 采集Prometheus数据

  1. 参考主机环境日志采集到SLS 建立主机iLogtail与阿里云日志服务的链接。
  2. 主机下载NodeExporter，下载地址：https://prometheus.io/download/#node_exporter，并进行启动，启动后可以通过curl 127.0.0.1:9100/metrics 查看NodeExporter 的Metrics指标。
  3. 创建日志服务MetricStore。iLogtail使用入门-iLogtail 采集Prometheus 数据
  4. 创建Prometheus采集配置。iLogtail使用入门-iLogtail 采集Prometheus 数据iLogtail使用入门-iLogtail 采集Prometheus 数据
  5. 查看采集数据

如下图所知，iLogtail 采集的NodeExporter 指标采用图表化展现，日志服务Metrics查询语言全面兼容PromQL，更多可视化用户请参考https://help.aliyun.com/document_detail/252810.html。

iLogtail使用入门-iLogtail 采集Prometheus 数据

如计算5分钟内SysLoad：

iLogtail使用入门-iLogtail 采集Prometheus 数据

使用Grafana 对接日志服务MetricStore

  1. 安装Grafana，参考Grafana安装指南。
  2. 使用Grafana Prometheus 数据源对接日志服务，参考时序数据对接Grafana。
  3. 下载NodeExporter 看板Json文件，如Node Exporter Full 看板。
  4. 导入看板。

iLogtail使用入门-iLogtail 采集Prometheus 数据

  1. 查看指标数据，如下图所示，Grafana 展示了iLogtail 采集的指标数据。

iLogtail使用入门-iLogtail 采集Prometheus 数据

总结

iLogtail 提供了完整Prometheus 指标采集能力，无需改造Exporter 指标，即可完成Prometheus 指标的采集。而通过日志服务MetricStore的能力，用户也可以使用其作为Prometheus 替代选项，通过的Grafana 商店丰富的看板模板快速构建自己的监控大盘。

参考文档

Grafana安装指南

时序数据对接Grafana

iLogtail 的传输层协议

主机环境日志采集到SLS


上一篇：武林高手？敏捷开发，唯velocity取胜


下一篇：关于代码抽象那点事