1. 操作

暂略

2. 问题记录

2.1. filebeat往logstash传输数据报错

报错信息：
filebeat：

2021-11-19T10:50:43.056+0800	ERROR	pipeline/output.go:121	Failed to publish events: write tcp 192.168.11.178:53849->192.168.31.180:5046: write: connection reset by peer
2021-11-19T10:50:43.095+0800	ERROR	pipeline/output.go:121	Failed to publish events: write tcp 192.168.11.178:43347->192.168.31.180:5047: write: connection reset by peer

logstash：

[2021-11-19T11:22:18,817][WARN ][logstash.filters.grok    ][log02] Timeout executing grok '%{IPORHOST:clientip} (%{IPORHOST:ip}|-) (%{DATA:remoteUser}|-) \[%{HTTPDATE:httpDate}\] \"%{WORD:method} %{DATA:request} %{NOTSPACE:httpVersion}\" %{NUMBER:statusCode} (?:%{NUMBER:bodyBytesSent}|-) \"(?:%{DATA:httpReferrer}|-)\" %{QS:agent} \"(%{XFORWARDEDFOR:xforwardedfor}|-)\" (%{BASE16FLOAT:requestTime}|-) (%{UPSTREAMADDR:upstreamAddr}|-) (%{HOSTORPORT:serverHost}|-) (%{UPSTREAMTIMES:upstreamResponseTime}|-)' against field 'message' with value 'Value too large to output (566 bytes)! First 255 chars are: 168.158.194.146 10.181.2.116 - [19/Nov/2021:00:06:24 +0800] "GET /index.html HTTP/1.1" 200 76163 "-" "colly - https://github.com/gocolly/colly/v2" "192.168.31.199'!

问题原因

logstash的grok正则和实际的内容不匹配，导致lo gstash hang住，不再接受filebeat过来的请求；另外grok正则的效率hui影响filebeat传输的速率