用STRACE解决公司真实故障一例

2021-08-24 10:08:37

这是相关分析文档。为了职业操守，已修改相关公司敏感信息~~~

~~~~~~~~~~~~~~~~~~~~~~~~

关于论坛每五分钟左右，会有warning.html跳转出现的原因调查

（warning.html）页面显示的即服务器正繁忙，请稍后重试

最近两三周，收到客服及公司同事反映，打开BBS时，会时不时出现warning.html页面的提示跳转。

本周开始正规此问题，但调查一直没有明显头绪。数据库，连接数，触发器，SQL语句抓取，PHP-FPM配置，等等都彻查了一次。

最后，用STRACE进程调用跟踪，发现主要原因，现在反馈给大家。当然，这是不是终极原因，有待进一步验证。

STRACE命令对比产生WARNING.HTML和没有产生的情况，会发现，WARING.HTML页出跳转前，出现了服务器500的未知错误，原因是调用相关的/SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/服务出错。

线索如下：

一，以下为正常调用（关键顺序蓝色标出）：

0.001946 getsockopt(6, SOL_SOCKET, SO_ERROR, [-8916466799302672384], [4]) = 0 <0.000010>

0.000044 fcntl(6, F_SETFL, O_RDWR) = 0 <0.000016>

0.000077 sendto(6, "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.1\r\nHost: 1.2.3.4:8080\r\nConnection: Keep-Alive\r\nUser-Agent: PHP-SOAP/5.3.10\r\nContent-Type: text/xml; charset=utf-8\r\nSOAPAction: \"urn:search\"\r\nContent-Length: 525\r\n\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns1=\"http://service.webservice.abc.com\"><SOAP-ENV:Body><ns1:search><ns1:args0>eyJrZXkiOiJcdTMwMTA5OVx1OGQyMlx1NWJjY1x1MzAxMVAyUFx1ODg0Y1x1NGUxYVx1NWMwNlx1NGYxYVx1NjYyZlx1MjAxY1x1NGUwMFx1NWUyNlx1NGUwMFx1OGRlZlx1MjAxZFx1NjIxOFx1NzU2NVx1NGUyZFx1NzY4NFx1OTFjZFx1ODk4MVx1NGUwMFx1OTBlOFx1NTIwNiIsInN0YXR1cyI6IndkemoiLCJzdHlsZSI6IjIifQ==</ns1:args0></ns1:search></SOAP-ENV:Body></SOAP-ENV:Envelope>\n", 786, MSG_DONTWAIT, NULL, 0) = 786 <0.000034>

0.000185 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.076195>

0.076290 recvfrom(6, "HTTP/1.1 200 OK\r\nServer: nginx\r\nDate: Sat, 13 Jun 2015 06:18:59 GMT\r\nContent-Type: text/xml;charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nVary: Accept-Encoding\r\n\r\n26a7\r\n<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"><soapenv:Body><ns:searchResponse xmlns:ns=\"http://service.webservice.abc.com\"><ns:return>{\"key\":\"\343\200\22099\350\264\242\345\257\214\343\200\221P2P\350\241\214\344\270\232\345\260\206\344\274\232\346\230\257\342\200\234\344\270\200\345\270\246\344\270\200\350\267\257\342\200\235\346\210\230\347\225\245\344\270\255\347\232\204\351\207\215\350\246\201\344\270\200\351\203\250\345\210\206\",\"list\":[{\"id\":8918,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1392090431000,\"shortLink\":\"guonei\",\"title\":\"\344\272\222\350\201\224\347\275\221\351\207\221\350\236\215\347\220\206\350\264\242\344\272\247\345\223\201\346\224\266\347\233\212\347\216\207\344\270\200\350\267\257\350\265\260\344\275\216\",\"type\":1},{\"id\":4158,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1366334071000,\"shortLink\":\"guonei\",\"title\":\"\347\275\221\347\273\234\350\236\215\350\265\204 \342\200\234\345\260\217\345\276\256\342\200\235\347\246\217\351\237\263\",\"type\":1},{\"id\":16403,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1420901203000,\"shortLink\":\"hangye\",\"title\":\"\347\275\221\350\264\267\342\200\234\346\207\222\344\272\272\342\200\235\347\246\217\351\237\263\357\274\232\346\217\255\347\247\230P2P\346\234\272\346\236\204\346\212\225\350\265\204\350\200\205"..., 8192, MSG_DONTWAIT, NULL, NULL) = 2920 <0.000018>

0.000461 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.000020>

0.000068 recvfrom(6, "0,\"shortLink\":\"guowai\",\"title\":\"\346\257\224\347\211\271\345\270\201\346\234\200\345\244\247\344\272\244\346\230\223\346\211\200Mt. Gox \347\224\263\350\257\267\347\240\264\344\272\247\346\270\205\347\256\227\",\"type\":1},{\"id\":2875,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1352164359000,\"shortLink\":\"guonei\",\"title\":\"\345\273\272\350\241\214\347\224\263\350\257\267\350\257\201\345\210\270\347\211\214\347\205\247\345\217\257\350\203\275\346\200\247\346\236\201\345\244\247\",\"type\":1},{\"id\":9545,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1394250574000,\"shortLink\":\"guonei\",\"title\":\"\347\231\276\345\272\246\345\267\262\347\224\263\350\257\267\344\270\255\345\233\275\346\260\221\350\220\245\351\223\266\350\241\214\347\211\214\347\205\247\",\"type\":1},{\"id\":19378,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1433040598000,\"shortLink\":\"hydongtai\",\"title\":\"\344\270\212\346\265\267\345\270\202\351\225\277\357\274\232\344\272\222\351\207\221\346\230\257\344\270\212\346\265\267\351\207\221\350\236\215\344\270\255\345\277\203\347\232\204\346\234\200\345\244\247\346\234\272\351\201\207\",\"type\":1},{\"id\":12248,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1403327611000,\"shortLink\":\"guonei\",\"title\":\"\344\272\222\350\201\224\347\275\221\351\207\221\350\236\215\345\217\221\345\261\225\346\230\257\344\270\200\346\254\241\345\205\250\346\226\260\347\232\204\346\234\272\351\201\207\",\"type\":1},{\"id\":11467,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1400424772000,\"shortLink\":\"guonei\",\"title\":\"\344\270\201\344\277\212\345\263\260\357\274\232\351\207\221\350\236\215\346\212\221\345\210\266\346\230\257\346\234\272\351\201\207 \344\272\222\350\201\224\347\275\221\351\207\221\350\236\215\346\214\221\346\210\230\345\244\232\",\"type\":1},{\"id\""..., 8192, MSG_DONTWAIT, NULL, NULL) = 5840 <0.000037>

0.000276 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.001508>

0.001584 recvfrom(6, "0,\"publishDate\":1406076160000,\"shortLink\":\"hangye\",\"title\":\"\344\270\212\345\270\202\345\205\254\345\217\270\346\266\214\345\205\245P2P\350\241\214\344\270\232 \346\230\257\347\202\222\344\275\234\350\277\230\346\230\257\344\275\234\347\247\200\357\274\237\",\"type\":1},{\"id\":19619,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1433465298000,\"shortLink\":\"hangye\",\"title\":\"P2P\346\210\220\350\202\241\344\273\267\347\202\222\344\275\234\347\245\236\345\231\250 \344\270\212\345\270\202\345\205\254\345\217\270\347\253\236\347\233\270\346\224\266\350\264\255\",\"type\":1},{\"id\":7938,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"null\",\"platGrade\":0,\"publishDate\":1387868704000,\"shortLink\":\"licai\",\"title\":\"\344\270\232\345\206\205\344\272\272\345\243\253\347\233\264\350\250\200\345\237\272\351\207\221\345\205\254\345\217\270\344\270\212\345\270\202\346\227\266\346\234\272\345\260\232\346\227\251\",\"type\":1},{\"id\":17738,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1426483246000,\"shortLink\":\"pingtai\",\"title\":\"\344\270\255\345\233\275\345\271\263\345\256\211\357\274\232\351\231\206\351\207\221\346\211\200\344\270\232\345\212\241\346\250\241\345\274\217\344\270\216P2P\345\205\254\345\217\270\344\270\215\345\220\214\",\"type\":1},{\"id\":17816,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1426810621000,\"shortLink\":\"pingtai\",\"title\":\"\345\271\263\345\256\211\345\271\264\346\212\245\357\274\232\351\231\206\351\207\221\346\211\200\347\275\221\350\264\267\344\272\244\346\230\223\350\247\204\346\250\241\344\270\255\345\233\275\347\254\254\344\270\200\",\"type\":1},{\"id\":18246,\"newsReviewNum\":0,\"outHtmlFileUrl\":\"\",\"platGrade\":0,\"publishDate\":1428823514000,\"shortLink\":\"zhongchou\",\"title\":\"\345\271\263\345\256\211\350\220\275\345\255\220\342\200\234\345\211\215\346\265\267\344\274\227\347\255\271\342\200\235"..., 8192, MSG_DONTWAIT, NULL, NULL) = 1332 <0.000030>

0.001170 close(6)

二，以下为出现warning.html的调用（关键顺序蓝色标出）：

0.002344 getsockopt(6, SOL_SOCKET, SO_ERROR, [-8916466799302672384], [4]) = 0 <0.000012>

0.000044 fcntl(6, F_SETFL, O_RDWR) = 0 <0.000011>

0.000071 sendto(6, "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.1\r\nHost: 1.2.3.4:8080\r\nConnection: Keep-Alive\r\nUser-Agent: PHP-SOAP/5.3.10\r\nContent-Type: text/xml; charset=utf-8\r\nSOAPAction: \"urn:search\"\r\nContent-Length: 525\r\n\r\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns1=\"http://service.webservice.abc.com\"><SOAP-ENV:Body><ns1:search><ns1:args0>eyJrZXkiOiJcdTk4ODRcdTZkNGJcdTc5ZjBcdTUxNjhcdTc0MDNcdTY3MDBcdTViY2MxJVx1NGViYVx1NTNlM1x1OGQyMlx1NWJjY1x1NjIxNlx1NWMwNlx1OGQ4NVx1OGZjN1x1NTE3Nlx1NGY1OTk5JVx1NGU0Ylx1NTQ4Y1x1MzAxMFx1OTQ2Ylx1NTM0ZVx1NThlYlx1MzAxMSIsInN0YXR1cyI6IndkemoiLCJzdHlsZSI6IjIifQ==</ns1:args0></ns1:search></SOAP-ENV:Body></SOAP-ENV:Envelope>\n", 786, MSG_DONTWAIT, NULL, 0) = 786 <0.000031>

0.000131 poll([{fd=6, events=POLLIN|POLLERR|POLLHUP}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) <0.010941>

0.011081 recvfrom(6, "HTTP/1.1 500 Internal Server Error\r\nServer: nginx\r\nDate: Sat, 13 Jun 2015 03:15:06 GMT\r\nContent-Type: text/xml;charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\n\r\n14a\r\n<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"><soapenv:Body><soapenv:Fault><faultcode>soapenv:Server</faultcode><faultstring>Exception occurred while trying to invoke service method search</faultstring><detail /></soapenv:Fault></soapenv:Body></soapenv:Envelope>\r\n0\r\n\r\n", 8192, MSG_DONTWAIT, NULL, NULL) = 522 <0.000020>

0.000346 close(6) = 0 <0.000036>

0.000184 stat("/home/abc/abc_bbs//source//class/discuz/discuz_error.php", {st_mode=S_IFREG|0644, st_size=12895, ...}) = 0 <0.000027>

0.000341 lstat("/var/log/discuz/exception_error.log", {st_mode=S_IFREG|0644, st_size=2050807, ...}) = 0 <0.000023>

0.000143 lstat("/var/log/discuz", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000018>

0.000076 lstat("/var/log", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 <0.000015>

0.000076 open("/var/log/discuz/exception_error.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 6 <0.000023>

0.000068 fstat(6, {st_mode=S_IFREG|0644, st_size=2050807, ...}) = 0 <0.000014>

0.000068 lseek(6, 0, SEEK_CUR) = 0 <0.000014>

0.000054 lseek(6, 0, SEEK_CUR) = 0 <0.000014>

0.000049 write(6, "2015-06-13 11:14:32 http://bbs.abc.com/forum.php?mod=viewthread&tid=379354&extra=page%3D120&page=1 \nException occurred while trying to invoke service method search\nArray\n:\350\241\214,\350\260\203\347\224\250\346\226\271\346\263\225:handleException\n\narray (\n 'mod' => 'viewthread',\n 'tid' => '379354',\n 'extra' => 'page=120',\n 'page' => '1',\n)\narray (\n)\n------------------\n", 344) = 344 <0.000022>

0.000107 close(6) = 0 <0.000015>

0.000074 chdir("/home/abc/abc_bbs") = 0 <0.000020>

0.000066 times({tms_utime=47093, tms_stime=6066, tms_cutime=0, tms_cstime=0}) = 3696665972 <0.000015>

0.000134 fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0 <0.000018>

0.002718 rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, 8) = 0 <0.000018>

0.000114 write(5, "\5\0\0\0\0\0\0\1\0\0\0\1", 12) = 12 <0.000046>

0.000095 shutdown(5, 2 /* send and receive */) = 0 <0.000028>

0.000071 close(5) = 0 <0.000022>

0.000084 rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f3652e676a0}, 8) = 0 <0.000018>

0.000246 write(4, "\1\6\0\1\3\331\7\0Status: 302 Moved Temporarily\r\nX-Powered-By: PHP/5.3.10\r\nSet-Cookie: Z6wq_f5c1_saltkey=SOo43Zag; expires=Mon, 13-Jul-2015 03:14:32 GMT; path=/; domain=.abc.com; httponly\r\nContent-Type: text/html; charset=gbk\r\nSet-Cookie: Z6wq_f5c1_lastvisit=1434161672; expires=Mon, 13-Jul-2015 03:14:32 GMT; path=/; domain=.abc.com\r\nSet-Cookie: Z6wq_f5c1_sid=fkJJhh; expires=Sun, 14-Jun-2015 03:14:32 GMT; path=/; domain=.abc.com\r\nExpires: -1\r\nCache-Control: no-store, private, post-check=0, pre-check=0, max-age=0\r\nPragma: no-cache\r\nSet-Cookie: Z6wq_f5c1_lastact=1434165272%09forum.php%09viewthread; expires=Sun, 14-Jun-2015 03:14:32 GMT; path=/; domain=.abc.com\r\nSet-Cookie: Z6wq_f5c1_stats_qc_reg=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.abc.com\r\nSet-Cookie: Z6wq_f5c1_cloudstatpost=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.abc.com\r\nlocation:http://bbs.abc.com/abc/warning.html\r\n\r\n\0\0\0\0\0\0\0\1\3\0\1\0\10\0\0\0\0\0\0\0ZaZ", 1016) = 1016 <0.000083>

0.000234 shutdown(4, 1 /* send */) = 0 <0.000023>

三，对比可以看出，最重要的出错信息如下：

POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.1

Host: 1.2.3.4:8080

500 Internal Server Error

<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">

<soapenv:Body>

<soapenv:Fault>

<faultcode>soapenv:Server

</faultcode>

<faultstring>Exception occurred while trying to invoke service method search

</faultstring>

</soapenv:Fault>

</soapenv:Body>

</soapenv:Envelope>

四，寻着此线索继续追踪

Host: 1.2.3.4:8080在NGINX里指向2.3.4.5:8080。

其中的TOMCAT日志和NGINX对比日志如下（已重启），从中可以发现很强的相关性。

NGINX：

1.190.229.13 - - [13/Jun/2015:14:54:30 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://www.abc.com/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 1.0" "1.190.229.13" "0.002" "-" "-"

101.23.157.6 - - [13/Jun/2015:14:54:30 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"

101.23.157.6 - - [13/Jun/2015:14:54:31 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"

221.204.196.144 - - [13/Jun/2015:14:54:31 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://bbs.abc.com/thread-81726-1-210.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "221.204.196.144" "0.002" "-" "-"

101.23.157.6 - - [13/Jun/2015:14:54:32 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.001" "-" "-"

116.21.109.46 - - [13/Jun/2015:14:54:32 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://www.abc.com/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" "116.21.109.46" "0.002" "-" "-"

124.130.149.182 - - [13/Jun/2015:14:54:33 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "http://bbs.abc.com/thread-489495-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36 2345chrome v3.0.0.6869" "124.130.149.182" "0.002" "-" "-"

101.23.157.6 - - [13/Jun/2015:14:54:33 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.003" "-" "-"

79.171.81.130 - - [13/Jun/2015:14:54:34 +0800] "GET /abc/warning.html HTTP/1.0" 200 713 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" "79.171.81.130" "0.002" "-" "-"

101.23.157.6 - - [13/Jun/2015:14:54:50 +0800] "GET /abc/warning.html HTTP/1.0" 200 1255 "http://bbs.abc.com/thread-489398-1-1.html" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" "101.23.157.6" "0.002" "-" "-"

TOMCAT：

115.29.174.159 - - [13/Jun/2015:14:55:00 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:00 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 9752

115.29.174.159 - - [13/Jun/2015:14:55:01 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:01 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 9590

115.29.174.159 - - [13/Jun/2015:14:55:01 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:01 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 10446

115.29.174.159 - - [13/Jun/2015:14:55:01 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:01 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 200 9754

115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330

10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl HTTP/1.0" 500 330

10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330

115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330

10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl HTTP/1.0" 500 330

115.29.174.159 - - [13/Jun/2015:14:55:02 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342

10.132.44.218 - - [13/Jun/2015:14:55:02 +0800] "POST /SolrWebService/services/search?wsdl/services/search.searchHttpSoap11Endpoint/ HTTP/1.0" 500 330

115.29.174.159 - - [13/Jun/2015:14:55:03 +0800] "GET /SolrWebService/services/search?wsdl HTTP/1.1" 200 4342