Go语言之高级篇beego框架之日志收集系统

2022-12-02 11:58:24

一、日志收集系统架构设计

图1

图2

二、开发环境

1、安装jdk

jdk-8u51-windows-x64.exe

安装目录：C:\Program Files\jdk8

2、安装zookeeper

zookeeper-3.3.6.tar.gz

安装目录： C:\Program Files\zookeeper-3.3.6

配置：

C:\Program Files\zookeeper-3.3.6\conf

把配置文件：zoo_sample.cfg，改名成：zoo.cfg

修改部署路径为你安装的路径。

配置环境变量：

C:\Program Files\zookeeper-3.3.6\bin

#启动服务：

进行目录，直接启动服务：C:\Program Files\zookeeper-3.3.6\bin

直接运行服务名称: zkServer.cmd

h had sessionid 0x16908b8a4920000

2019-02-20 10:52:39,153 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53568

2019-02-20 10:52:39,154 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53568

2019-02-20 10:52:39,155 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53568

2019-02-20 10:52:39,158 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:39,161 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:39,163 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53568 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:41,214 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53570

2019-02-20 10:52:41,216 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53570

2019-02-20 10:52:41,217 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53570

2019-02-20 10:52:41,220 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:41,224 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:41,226 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53570 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:42,438 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53571

2019-02-20 10:52:42,439 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53571

2019-02-20 10:52:42,441 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53571

2019-02-20 10:52:42,444 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:42,448 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:42,449 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53571 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:44,219 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53572

2019-02-20 10:52:44,220 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53572

2019-02-20 10:52:44,221 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53572

2019-02-20 10:52:44,223 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:44,227 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:44,229 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53572 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:45,462 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53573

2019-02-20 10:52:45,464 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0

:0:0:0:1:53573

2019-02-20 10:52:45,465 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /0:0:0:0:0:0:0:1:53573

2019-02-20 10:52:45,468 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:45,473 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:45,475 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53573 whic

h had sessionid 0x16908b8a4920000

2019-02-20 10:52:46,769 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53574

2019-02-20 10:52:46,770 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53574

2019-02-20 10:52:46,772 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53574

2019-02-20 10:52:46,775 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:46,779 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:46,781 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53574 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:48,284 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53575

2019-02-20 10:52:48,285 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53575

2019-02-20 10:52:48,286 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53575

2019-02-20 10:52:48,289 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:48,293 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:48,295 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53575 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:50,069 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53576

2019-02-20 10:52:50,071 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53576

2019-02-20 10:52:50,072 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53576

2019-02-20 10:52:50,075 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:50,080 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:50,081 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53576 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:51,547 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53577

2019-02-20 10:52:51,548 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53577

2019-02-20 10:52:51,550 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53577

2019-02-20 10:52:51,552 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:51,557 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:51,559 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53577 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:53,315 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53579

2019-02-20 10:52:53,316 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0

:0:0:0:1:53579

2019-02-20 10:52:53,318 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /0:0:0:0:0:0:0:1:53579

2019-02-20 10:52:53,321 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:53,325 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@639] - Exception causing close of session 0x16908b8a4920000 due to java

.io.IOException: 远程主机强迫关闭了一个现有的连接。

2019-02-20 10:52:53,326 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53579 whic

h had sessionid 0x16908b8a4920000

2019-02-20 10:52:54,537 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53580

2019-02-20 10:52:54,539 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0

:0:0:0:1:53580

2019-02-20 10:52:54,540 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /0:0:0:0:0:0:0:1:53580

2019-02-20 10:52:54,545 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:54,551 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:52:54,552 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53580 whic

h had sessionid 0x16908b8a4920000

2019-02-20 10:52:56,153 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53581

2019-02-20 10:52:56,154 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53581

2019-02-20 10:52:56,156 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53581

2019-02-20 10:52:56,159 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:56,163 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:52:56,164 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53581 which had

sessionid 0x16908b8a4920000

2019-02-20 10:52:58,023 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53582

2019-02-20 10:52:58,024 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0

:0:0:0:1:53582

2019-02-20 10:52:58,026 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /0:0:0:0:0:0:0:1:53582

2019-02-20 10:52:58,028 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:58,032 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:52:58,034 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53582 whic

h had sessionid 0x16908b8a4920000

2019-02-20 10:52:59,424 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53584

2019-02-20 10:52:59,425 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53584

2019-02-20 10:52:59,425 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53584

2019-02-20 10:52:59,428 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:52:59,431 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:52:59,431 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53584 which had

sessionid 0x16908b8a4920000

2019-02-20 10:53:01,246 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53585

2019-02-20 10:53:01,247 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53585

2019-02-20 10:53:01,249 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53585

2019-02-20 10:53:01,252 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:53:01,259 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:53:01,260 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53585 which had

sessionid 0x16908b8a4920000

2019-02-20 10:53:02,677 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53587

2019-02-20 10:53:02,678 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0

:0:0:0:1:53587

2019-02-20 10:53:02,679 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /0:0:0:0:0:0:0:1:53587

2019-02-20 10:53:02,682 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:53:02,685 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:53:02,686 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53587 whic

h had sessionid 0x16908b8a4920000

2019-02-20 10:53:04,384 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53588

2019-02-20 10:53:04,385 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53588

2019-02-20 10:53:04,387 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53588

2019-02-20 10:53:04,390 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:53:04,399 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:53:04,400 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /127.0.0.1:53588 which had

sessionid 0x16908b8a4920000

2019-02-20 10:53:06,078 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /0:0:0:0:0:0:0:1:53589

2019-02-20 10:53:06,079 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /0:0:0:0

:0:0:0:1:53589

2019-02-20 10:53:06,081 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /0:0:0:0:0:0:0:1:53589

2019-02-20 10:53:06,084 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

2019-02-20 10:53:06,089 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@634] - EndOfStreamException: Unable to read additional data from client

 sessionid 0x16908b8a4920000, likely client has closed socket

2019-02-20 10:53:06,090 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1435] - Closed socket connection for client /0:0:0:0:0:0:0:1:53589 whic

h had sessionid 0x16908b8a4920000

2019-02-20 10:53:07,834 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn$Factory@251] - Accepted socket connection from /127.0.0.1:53590

2019-02-20 10:53:07,835 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@770] - Client attempting to renew session 0x16908b8a4920000 at /127.0.0

.1:53590

2019-02-20 10:53:07,836 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOSe

rverCnxn@1580] - Established session 0x16908b8a4920000 with negotiated timeout 6

000 for client /127.0.0.1:53590

2019-02-20 10:53:07,838 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKe

eperServer@593] - Dropping packet at server of type 14

3、kafka

kafka_2.12-2.1.1.tgz

安装目录：D:\run_file\kafka_2.12-2.1.1 （不能这在C:\Program Files\ 下面启动服务会报错。）

#修改配置文件

D:\run_file\kafka_2.12-2.1.1\config

修改为本机IP地址：

advertised.host.name=172.28.5.25

#配置日志地址：

########### Log Basics #############

# A comma separated list of directories under which to store log files

#log.dirs=/tmp/kafka-logs

log.dirs=D:/run_file/kafka_2.12-2.1.1/kafka-logs   #增加这行代码

启动服务：

进入安装目录中：

在命令行下：

cd D:\run_file\kafka_2.12-2.1.1

 ./bin/windows/kafka-server-start.bat ./config/server.preperties

        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSock

NIO.java:366)

        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145

[2019-02-20 10:53:06,087] INFO [ZooKeeperClient] Waiting until connected. (kaf

.zookeeper.ZooKeeperClient)

[2019-02-20 10:53:06,089] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z

KeeperClient)

[2019-02-20 10:53:06,189] INFO [ZooKeeperClient] Waiting until connected. (kaf

.zookeeper.ZooKeeperClient)

[2019-02-20 10:53:06,190] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z

KeeperClient)

[2019-02-20 10:53:06,190] INFO [ZooKeeperClient] Waiting until connected. (kaf

.zookeeper.ZooKeeperClient)

[2019-02-20 10:53:06,191] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.Z

KeeperClient)

[2019-02-20 10:53:07,832] INFO Opening socket connection to server localhost/1

.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.

ache.zookeeper.ClientCnxn)

[2019-02-20 10:53:07,834] INFO Socket connection established to localhost/127.

0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)

[2019-02-20 10:53:07,836] WARN Connected to an old server; r-o mode will be un

ailable (org.apache.zookeeper.ClientCnxnSocket)

[2019-02-20 10:53:07,837] INFO Session establishment complete on server localh

t/127.0.0.1:2181, sessionid = 0x16908b8a4920000, negotiated timeout = 6000 (or

apache.zookeeper.ClientCnxn)

[2019-02-20 10:53:07,844] INFO [SocketServer brokerId=0] Started processors fo

1 acceptors (kafka.network.SocketServer)

[2019-02-20 10:53:07,853] INFO Kafka version : 2.1.1 (org.apache.kafka.common.

ils.AppInfoParser)

[2019-02-20 10:53:07,853] INFO Kafka commitId : 21234bee31165527 (org.apache.k

ka.common.utils.AppInfoParser)

[2019-02-20 10:53:07,856] INFO [KafkaServer id=0] started (kafka.server.KafkaS

ver)

[2019-02-20 10:53:12,993] INFO [GroupMetadataManager brokerId=0] Removed 0 exp

ed offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

二、tail

在cmd中安装：

go get github.com/hpcloud/tail

#代码：

创建一个my.log的文件

路径：

D:\GoFiles\my.log

示例：

vi my.log

############################# Producer Basics #############################

# list of brokers used for bootstrapping knowledge about the rest of the cluster

# format: host1:port1,host2:port2 ...

bootstrap.servers=localhost:9092

# specify the compression codec for all data generated: none, gzip, snappy, lz4, zstd

compression.type=none

# name of the partitioner class for partitioning events; default partition spreads data randomly

#partitioner.class=

# the maximum amount of time the client will wait for the response of a request

#request.timeout.ms=

# how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for

#max.block.ms=

# the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together

#linger.ms=

# the maximum size of a request in bytes

#max.request.size=

# the default batch size in bytes when batching multiple records sent to a partition

#batch.size=

# the total bytes of memory the producer can use to buffer records waiting to be sent to the server

#buffer.memory=

示例：把文件中内容读出来

vi main.go

package main

import (

	"fmt"

	"time"

	"github.com/hpcloud/tail"

)

func main() {

	filename := "D:\\GoFiles\\my.log"

	tails, err := tail.TailFile(filename, tail.Config{

		ReOpen: true,

		Follow: true,

		//Location:  &tail.SeekInfo{Offset: 0, Whence: 2},

		MustExist: false,

		Poll:      true,

	})

	if err != nil {

		fmt.Println("tail file err:", err)

		return

	}

	var msg *tail.Line

	var ok bool

	for true {

		msg, ok = <-tails.Lines

		if !ok {

			fmt.Printf("tail file close reopen,filename:%s\n", tails, filename)

			time.Sleep(100 * time.Millisecond)

			continue

		}

		fmt.Println("msg:", msg.Text)

	}

}

执行结果：

#编译
#在终端中输入：

D:\GoFiles\src\log_collect\tail>go build D:\GoFiles\src\log_collect\tail

#运行

D:\GoFiles\src\log_collect\tail>tail.exe

#返回结果：

############################# Producer Basics #############################

msg:

msg: # list of brokers used for bootstrapping knowledge about the rest of the cl

uster

msg: # format: host1:port1,host2:port2 ...

msg: bootstrap.servers=localhost:9092

msg:

msg: # specify the compression codec for all data generated: none, gzip, snappy,

 lz4, zstd

msg: compression.type=none

msg:

msg: # name of the partitioner class for partitioning events; default partition

spreads data randomly

msg: #partitioner.class=

msg:

msg: # the maximum amount of time the client will wait for the response of a req

uest

msg: #request.timeout.ms=

msg:

msg: # how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will bloc

k for

msg: #max.block.ms=

msg:

msg: # the producer will wait for up to the given delay to allow other records t

o be sent so that the sends can be batched together

msg: #linger.ms=

msg:

msg: # the maximum size of a request in bytes

msg: #max.request.size=

msg:

msg: # the default batch size in bytes when batching multiple records sent to a

partition

msg: #batch.size=

msg:

msg: # the total bytes of memory the producer can use to buffer records waiting

to be sent to the server

msg: #buffer.memory=

二、els安装

1、安装Elasticsearch

elasticsearch-5.5.0.msi

安装目录

C:\Program Files\Elastic\Elasticsearch

#启动目录

C:\Program Files\Elastic\Elasticsearch\bin

运行：

elasticsearch.exe

#访问： http://127.0.0.1:9200/

2、kafka（windows版本安装）

软件部署：

kibana-5.5.1-windows-x86.zip

解压到下面目录中：

C:\Program Files (x86)\kibana

#修改配置文件

elasticsearch.url: "http://127.0.0.1:9200" #指向es的启动页面

#启动kibana

C:\Program Files (x86)\kibana\bin

运行：

kibana.bat

示例：

运行程序的前提条件：

先安装elastic.v2

安装包：

go get gopkg.in/olivere/elastic.v2

使用方法：

import "gopkg.in/olivere/elastic.v2"

目录：

代码：
vi main.go

package main

import (

	"fmt"

	elastic "gopkg.in/olivere/elastic.v2"

)

type Tweet struct {

	User    string

	Message string

}

func main() {

	client, err := elastic.NewClient(elastic.SetSniff(false), elastic.SetURL("http://127.0.0.1:9200/"))

	if err != nil {

		fmt.Println("connect es error", err)

		return

	}

	fmt.Println("conn es succ")

	for i := 0; i < 10000; i++ {

		tweet := Tweet{User: "olivere", Message: "Take Five"}

		_, err = client.Index().

			Index("twitter").

			Type("tweet").

			Id(fmt.Sprintf("%d", i)).

			BodyJson(tweet).

			Do()

		if err != nil {

			// Handle error

			panic(err)

			return

		}

	}

	fmt.Println("insert success")

}

#打开页面：

http://localhost:5601

输入：　twitter --->再点Discover就可以看到下面页面。

3、

码农公寓

一、日志收集系统架构设计

二、开发环境

相关文章