prometheus 服务器重启后之"Error on ingesting samples that are too old or are too far into the future"问题解决

背景描述:aws中的prometheus实例状态检查出现异常,重启实例后实例正常,此时发现grafana中主机为空,经查看日志发现如下:

level=warn ts=2021-08-16T03:20:04.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78
level=warn ts=2021-08-16T03:20:04.961Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1507 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=491
level=warn ts=2021-08-16T03:20:06.868Z caller=scrape.go:1203 component="scrape manager" scrape_pool=node target=http://18.166.55.107:9100/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=959
level=warn ts=2021-08-16T03:20:08.140Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://18.166.83.165:9104/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=941
level=warn ts=2021-08-16T03:20:09.037Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mysql-mater-slave-luxe target=http://16.162.107.109:9104/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1507 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=78
level=warn ts=2021-08-16T03:20:09.960Z caller=scrape.go:1203 component="scrape manager" scrape_pool=mtail target=http://18.167.146.20:3903/metrics msg="Appending scrape report failed" err="out of bounds"
level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1507 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=558
level=warn ts=2021-08-16T03:20:10.168Z caller=scrape.go:1203 component="scrape manager" scrape_pool=prometheus target=http://18.166.55.107:9090/metrics msg="Appending scrape report failed" err="out of bounds"

解决方法:

备份prometheus数据库,新建data目录,重启prometheus和grafana服务
[root@prometheus /opt/prometheus]# mv data/ data_bak
[root@prometheus /opt/prometheus]# mkdir -pv data && chown -R prometheus.prometheus data
[root@prometheus /opt/prometheus]# systemctl restart prometheus
[root@prometheus /opt/prometheus]# systemctl restart grafana-server
web界面访问prometheus正常,grafana访问正常

 

prometheus 服务器重启后之"Error on ingesting samples that are too old or are too far into the future"问题解决

上一篇:HDOJ2013蟠桃记


下一篇:I. Sequence 题解(思维+矩阵内全0正方形的数量)