最近遇到一些诡异的事情，1Apache + 4Tomcat的系统中，每天偶尔会发现其中的一台或几台Tomcat宕机了。

今天特别频繁，宕了有6次之多。

   环境:

Windows Server 2003 standard  Service Pack 2

JDK 7.0_11-b21

Tomcat 7.0.42

Apache 2.2.24

mod_jk 1.2.37

查看具体的错误日志如下：

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000001800057b2, pid=6400, tid=10300
#
# JRE version: 7.0_11-b21
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [tcnative-1.dll+0x57b2] Java_org_apache_tomcat_jni_Socket_sendbb+0x72
#
# Core dump written. Default location: D:\Tomcat7\bin\hs_err_pid6400.mdmp
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x0000000014ac1800): JavaThread "ajp-apr-7020-exec-145" daemon [_thread_in_native, id=10300, stack(0x00000000227f0000,0x00000000228f0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000040

Registers:
RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0x0000000000000000
RSP=0x00000000228ee310, RBP=0x0000000000000000, RSI=0x0000000016ecc360, RDI=0x00000000000000c1
R8 =0x00000000228ee348, R9 =0x0000000000000000, R10=0x00000000013f237c, R11=0x000000006f002070
R12=0x0000000000000000, R13=0x00000006f0b780a8, R14=0x00000000228ee400, R15=0x0000000014ac1800
RIP=0x00000001800057b2, EFLAGS=0x0000000000010246

Top of Stack: (sp=0x00000000228ee310)
0x00000000228ee310: 00000000228ee328 00000000228ee378
0x00000000228ee320: 00000000013e61f8 0000000000000000
0x00000000228ee330: 0000000000004000 00000000013f23a8
0x00000000228ee340: 00000000228ee3c8 00000000000000c1
0x00000000228ee350: 00000006f0b780a8 0000000014c4de50
0x00000000228ee360: 00000006000000c1 0000000000000000
0x00000000228ee370: 00000000228ee388 00000000228ee3e8
0x00000000228ee380: 00000000013e61f8 00000000228ee388
0x00000000228ee390: 00000006f0b780a8 00000000228ee400
0x00000000228ee3a0: 00000006f0b796e8 0000000000000000
0x00000000228ee3b0: 00000006f0b780a8 0000000000000000
0x00000000228ee3c0: 00000000228ee410 00000000e028e72a
0x00000000228ee3d0: 00000000018e8308 0000000700418ba8
0x00000000228ee3e0: 00000000013ef1d6 00000000000000c1
0x00000000228ee3f0: 0000000000000000 0000000016ecc360
0x00000000228ee400: 0000000000000000 00000000018e8308

Instructions: (pc=0x00000001800057b2)
0x0000000180005792: 56 20 48 8b 4e 18 48 8b c7 48 2b c3 48 03 d5 4c
0x00000001800057a2: 8d 44 24 38 48 89 44 24 38 48 8b 46 30 48 03 d3
0x00000001800057b2: ff 50 40 85 c0 75 29 48 8b 44 24 38 48 85 c0 74
0x00000001800057c2: 08 48 03 d8 48 3b df 72 c5 8b c3 48 8b 6c 24 30

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x0000000000000000 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x00000000228ee310 is pointing into the stack for thread: 0x0000000014ac1800
RBP=0x0000000000000000 is an unknown value
RSI=0x0000000016ecc360 is an unknown value
RDI=0x00000000000000c1 is an unknown value
R8 =0x00000000228ee348 is pointing into the stack for thread: 0x0000000014ac1800
R9 =0x0000000000000000 is an unknown value
R10=0x00000000013f237c is an Interpreter codelet
method entry point (kind = native) [0x00000000013f2100, 0x00000000013f2980] 2176 bytes
R11=0x000000006f002070 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x00000006f0b780a8 is an oop
{method}
- klass: {other class}
R14=0x00000000228ee400 is pointing into the stack for thread: 0x0000000014ac1800
R15=0x0000000014ac1800 is a thread

Stack: [0x00000000227f0000,0x00000000228f0000], sp=0x00000000228ee310, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [tcnative-1.dll+0x57b2] Java_org_apache_tomcat_jni_Socket_sendbb+0x72

[error occurred during error reporting (printing native stack), id 0xc0000005]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.apache.tomcat.jni.Socket.sendbb(JII)I+0
J org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Lorg/apache/tomcat/util/buf/ByteChunk;Lorg/apache/coyote/Response;)I
j org.apache.coyote.Response.doWrite(Lorg/apache/tomcat/util/buf/ByteChunk;)V+6
j org.apache.catalina.connector.OutputBuffer.realWriteBytes([BII)V+38
J org.apache.tomcat.util.buf.ByteChunk.append([BII)V
j org.apache.catalina.connector.OutputBuffer.writeBytes([BII)V+15
j org.apache.catalina.connector.OutputBuffer.write([BII)V+12
j org.apache.catalina.connector.CoyoteOutputStream.write([BII)V+7
j com.XXXXXXX.executeExcelDownload(Ljava/lang/String;)Ljava/lang/String;+361
j com.XXXXXX.getCommonContentData(Ljava/lang/String;Ljava/lang/String;ILjava/lang/String;I[Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;+472
j com.XXXXXX.getTableContent(Lcom/tpm/actions/ActionContext;)Ljava/lang/String;+1356
j com.XXXXXXX.executeGetTableContent(Lcom/tpm/actions/ActionContext;)Ljava/lang/String;+5
j sun.reflect.GeneratedMethodAccessor267.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+40
J java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
j com.XXXXX.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+1171
j javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30
j org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+446
j org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101
j com.XXXXX.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+13
j org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+151
j org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+101
j org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+783
j org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+166
j org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+449
j org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+148
j org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+6
j org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+71
j org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V+196
J org.apache.coyote.ajp.AjpAprProcessor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
j org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;+158
j org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run()V+32
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+46
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x0000000029414000 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=10596, stack(0x000000002f9b0000,0x000000002fab0000)]
0x0000000014ac5000 JavaThread "ajp-apr-7020-exec-150" daemon [_thread_in_native, id=10320, stack(0x0000000022cf0000,0x0000000022df0000)]
0x0000000014ac4800 JavaThread "ajp-apr-7020-exec-149" daemon [_thread_blocked, id=10316, stack(0x0000000022bf0000,0x0000000022cf0000)]
0x0000000014ac3800 JavaThread "ajp-apr-7020-exec-148" daemon [_thread_blocked, id=10312, stack(0x0000000022af0000,0x0000000022bf0000)]
0x0000000014ac3000 JavaThread "ajp-apr-7020-exec-147" daemon [_thread_blocked, id=10308, stack(0x00000000229f0000,0x0000000022af0000)]
0x0000000014ac2000 JavaThread "ajp-apr-7020-exec-146" daemon [_thread_blocked, id=10304, stack(0x00000000228f0000,0x00000000229f0000)]
=>0x0000000014ac1800 JavaThread "ajp-apr-7020-exec-145" daemon [_thread_in_native, id=10300, stack(0x00000000227f0000,0x00000000228f0000)]
0x0000000014ac0800 JavaThread "ajp-apr-7020-exec-144" daemon [_thread_blocked, id=10296, stack(0x00000000226f0000,0x00000000227f0000)]
0x0000000014ac0000 JavaThread "ajp-apr-7020-exec-143" daemon [_thread_blocked, id=10292, stack(0x00000000225f0000,0x00000000226f0000)]
0x0000000014abf000 JavaThread "ajp-apr-7020-exec-142" daemon [_thread_blocked, id=10288, stack(0x00000000224f0000,0x00000000225f0000)]
0x0000000014abe800 JavaThread "ajp-apr-7020-exec-141" daemon [_thread_blocked, id=10284, stack(0x00000000223f0000,0x00000000224f0000)]
0x0000000014abd800 JavaThread "ajp-apr-7020-exec-140" daemon [_thread_blocked, id=10280, stack(0x00000000222f0000,0x00000000223f0000)]
0x0000000014abd000 JavaThread "ajp-apr-7020-exec-139" daemon [_thread_blocked, id=10276, stack(0x00000000221f0000,0x00000000222f0000)]
0x00000000154e4800 JavaThread "ajp-apr-7020-exec-138" daemon [_thread_blocked, id=10272, stack(0x00000000220f0000,0x00000000221f0000)]
0x00000000154e4000 JavaThread "ajp-apr-7020-exec-137" daemon [_thread_blocked, id=10268, stack(0x0000000021ff0000,0x00000000220f0000)]
0x00000000154e3000 JavaThread "ajp-apr-7020-exec-136" daemon [_thread_blocked, id=10264, stack(0x0000000021ef0000,0x0000000021ff0000)]
0x00000000154e2800 JavaThread "ajp-apr-7020-exec-135" daemon [_thread_blocked, id=10260, stack(0x0000000021df0000,0x0000000021ef0000)]
0x00000000154e1800 JavaThread "ajp-apr-7020-exec-134" daemon [_thread_blocked, id=10256, stack(0x0000000021cf0000,0x0000000021df0000)]
0x00000000154e1000 JavaThread "ajp-apr-7020-exec-133" daemon [_thread_blocked, id=10252, stack(0x0000000021bf0000,0x0000000021cf0000)]
0x00000000154e0000 JavaThread "ajp-apr-7020-exec-132" daemon [_thread_blocked, id=10248, stack(0x0000000021af0000,0x0000000021bf0000)]
0x00000000154df800 JavaThread "ajp-apr-7020-exec-131" daemon [_thread_blocked, id=10244, stack(0x00000000219f0000,0x0000000021af0000)]
0x00000000154de800 JavaThread "ajp-apr-7020-exec-130" daemon [_thread_blocked, id=708, stack(0x00000000218f0000,0x00000000219f0000)]
0x00000000154de000 JavaThread "ajp-apr-7020-exec-129" daemon [_thread_blocked, id=9780, stack(0x00000000217f0000,0x00000000218f0000)]
0x0000000014aa5800 JavaThread "ajp-apr-7020-exec-128" daemon [_thread_blocked, id=8508, stack(0x00000000216f0000,0x00000000217f0000)]
0x0000000014aa5000 JavaThread "ajp-apr-7020-exec-127" daemon [_thread_blocked, id=936, stack(0x00000000215f0000,0x00000000216f0000)]
0x0000000014aa4000 JavaThread "ajp-apr-7020-exec-126" daemon [_thread_blocked, id=8472, stack(0x00000000214f0000,0x00000000215f0000)]
0x0000000014aa3800 JavaThread "ajp-apr-7020-exec-125" daemon [_thread_blocked, id=8476, stack(0x00000000213f0000,0x00000000214f0000)]
0x0000000014aa2800 JavaThread "ajp-apr-7020-exec-124" daemon [_thread_blocked, id=10212, stack(0x00000000212f0000,0x00000000213f0000)]
0x0000000014aa2000 JavaThread "ajp-apr-7020-exec-123" daemon [_thread_in_native, id=8436, stack(0x00000000211f0000,0x00000000212f0000)]
0x0000000014aa1000 JavaThread "ajp-apr-7020-exec-122" daemon [_thread_blocked, id=8432, stack(0x00000000210f0000,0x00000000211f0000)]
0x0000000014aa0800 JavaThread "ajp-apr-7020-exec-121" daemon [_thread_blocked, id=1656, stack(0x0000000020ff0000,0x00000000210f0000)]
0x0000000014a9f800 JavaThread "ajp-apr-7020-exec-120" daemon [_thread_blocked, id=10192, stack(0x0000000020ef0000,0x0000000020ff0000)]
0x0000000014a9f000 JavaThread "ajp-apr-7020-exec-119" daemon [_thread_blocked, id=8668, stack(0x0000000020df0000,0x0000000020ef0000)]
0x0000000014b05800 JavaThread "ajp-apr-7020-exec-118" daemon [_thread_blocked, id=10016, stack(0x0000000020cf0000,0x0000000020df0000)]
0x0000000014b05000 JavaThread "ajp-apr-7020-exec-117" daemon [_thread_blocked, id=10028, stack(0x0000000020bf0000,0x0000000020cf0000)]
0x0000000014b04000 JavaThread "ajp-apr-7020-exec-116" daemon [_thread_blocked, id=10032, stack(0x0000000020af0000,0x0000000020bf0000)]
0x0000000014b03800 JavaThread "ajp-apr-7020-exec-115" daemon [_thread_blocked, id=9500, stack(0x00000000209f0000,0x0000000020af0000)]
0x0000000014b02800 JavaThread "ajp-apr-7020-exec-114" daemon [_thread_blocked, id=9504, stack(0x00000000208f0000,0x00000000209f0000)]
0x0000000014b02000 JavaThread "ajp-apr-7020-exec-113" daemon [_thread_blocked, id=9524, stack(0x00000000207f0000,0x00000000208f0000)]
0x0000000014b01000 JavaThread "ajp-apr-7020-exec-112" daemon [_thread_blocked, id=10112, stack(0x00000000206f0000,0x00000000207f0000)]
0x0000000014b00800 JavaThread "ajp-apr-7020-exec-111" daemon [_thread_blocked, id=364, stack(0x00000000205f0000,0x00000000206f0000)]
0x0000000014aff800 JavaThread "ajp-apr-7020-exec-110" daemon [_thread_blocked, id=9520, stack(0x00000000204f0000,0x00000000205f0000)]
0x0000000014aff000 JavaThread "ajp-apr-7020-exec-109" daemon [_thread_blocked, id=9512, stack(0x00000000203f0000,0x00000000204f0000)]
0x0000000015459000 JavaThread "ajp-apr-7020-exec-108" daemon [_thread_blocked, id=9496, stack(0x00000000202f0000,0x00000000203f0000)]
0x0000000015458800 JavaThread "ajp-apr-7020-exec-107" daemon [_thread_blocked, id=9492, stack(0x00000000201f0000,0x00000000202f0000)]
0x0000000015457800 JavaThread "ajp-apr-7020-exec-106" daemon [_thread_in_native, id=9484, stack(0x00000000200f0000,0x00000000201f0000)]
0x0000000015457000 JavaThread "ajp-apr-7020-exec-105" daemon [_thread_blocked, id=9672, stack(0x000000001fff0000,0x00000000200f0000)]
0x0000000015456000 JavaThread "ajp-apr-7020-exec-104" daemon [_thread_blocked, id=9676, stack(0x000000001fef0000,0x000000001fff0000)]
0x0000000015455800 JavaThread "ajp-apr-7020-exec-103" daemon [_thread_blocked, id=9712, stack(0x000000001fdf0000,0x000000001fef0000)]
0x0000000015454800 JavaThread "ajp-apr-7020-exec-102" daemon [_thread_blocked, id=10184, stack(0x000000001fcf0000,0x000000001fdf0000)]
0x0000000015454000 JavaThread "ajp-apr-7020-exec-101" daemon [_thread_blocked, id=9936, stack(0x000000001fbf0000,0x000000001fcf0000)]
0x0000000015453000 JavaThread "ajp-apr-7020-exec-100" daemon [_thread_blocked, id=9940, stack(0x000000001faf0000,0x000000001fbf0000)]
0x0000000015452800 JavaThread "ajp-apr-7020-exec-99" daemon [_thread_blocked, id=9768, stack(0x000000001f9f0000,0x000000001faf0000)]
0x0000000015451800 JavaThread "ajp-apr-7020-exec-98" daemon [_thread_blocked, id=9700, stack(0x000000001f8f0000,0x000000001f9f0000)]
0x0000000015450800 JavaThread "ajp-apr-7020-exec-97" daemon [_thread_blocked, id=9684, stack(0x000000001f7f0000,0x000000001f8f0000)]
0x0000000015450000 JavaThread "ajp-apr-7020-exec-96" daemon [_thread_blocked, id=9680, stack(0x000000001f6f0000,0x000000001f7f0000)]
0x000000001544f000 JavaThread "ajp-apr-7020-exec-95" daemon [_thread_blocked, id=9764, stack(0x000000001f5f0000,0x000000001f6f0000)]
0x000000001544e800 JavaThread "ajp-apr-7020-exec-94" daemon [_thread_blocked, id=9508, stack(0x000000001f4f0000,0x000000001f5f0000)]
0x000000001544d800 JavaThread "ajp-apr-7020-exec-93" daemon [_thread_blocked, id=10072, stack(0x000000001f3f0000,0x000000001f4f0000)]
0x000000001544d000 JavaThread "ajp-apr-7020-exec-92" daemon [_thread_blocked, id=10160, stack(0x000000001f2f0000,0x000000001f3f0000)]
0x000000001544c000 JavaThread "ajp-apr-7020-exec-91" daemon [_thread_blocked, id=9744, stack(0x000000001f1f0000,0x000000001f2f0000)]
0x000000001544b800 JavaThread "ajp-apr-7020-exec-90" daemon [_thread_blocked, id=9740, stack(0x000000001f0f0000,0x000000001f1f0000)]
0x000000001544a800 JavaThread "ajp-apr-7020-exec-89" daemon [_thread_blocked, id=9720, stack(0x000000001eff0000,0x000000001f0f0000)]
0x000000001544a000 JavaThread "ajp-apr-7020-exec-88" daemon [_thread_blocked, id=9944, stack(0x000000001eef0000,0x000000001eff0000)]
0x0000000014df8000 JavaThread "ajp-apr-7020-exec-87" daemon [_thread_blocked, id=9716, stack(0x000000001edf0000,0x000000001eef0000)]
0x0000000014df7800 JavaThread "ajp-apr-7020-exec-86" daemon [_thread_blocked, id=8612, stack(0x000000001ecf0000,0x000000001edf0000)]
0x0000000014df6800 JavaThread "ajp-apr-7020-exec-85" daemon [_thread_blocked, id=8620, stack(0x000000001ebf0000,0x000000001ecf0000)]
0x0000000014df6000 JavaThread "ajp-apr-7020-exec-84" daemon [_thread_blocked, id=1296, stack(0x000000001eaf0000,0x000000001ebf0000)]
0x0000000014df5000 JavaThread "ajp-apr-7020-exec-83" daemon [_thread_blocked, id=8640, stack(0x000000001e9f0000,0x000000001eaf0000)]
0x0000000014df4800 JavaThread "ajp-apr-7020-exec-82" daemon [_thread_blocked, id=292, stack(0x000000001e2e0000,0x000000001e3e0000)]
0x0000000014df3800 JavaThread "ajp-apr-7020-exec-81" daemon [_thread_blocked, id=10176, stack(0x000000001e8f0000,0x000000001e9f0000)]
0x0000000014df3000 JavaThread "ajp-apr-7020-exec-80" daemon [_thread_blocked, id=10148, stack(0x000000001e7f0000,0x000000001e8f0000)]
0x0000000014df2000 JavaThread "ajp-apr-7020-exec-79" daemon [_thread_blocked, id=10140, stack(0x000000001e6f0000,0x000000001e7f0000)]
0x0000000014df1800 JavaThread "ajp-apr-7020-exec-78" daemon [_thread_in_native, id=10136, stack(0x000000001e5f0000,0x000000001e6f0000)]
0x0000000014df0800 JavaThread "ajp-apr-7020-exec-77" daemon [_thread_blocked, id=10128, stack(0x000000001e4f0000,0x000000001e5f0000)]
0x0000000014df0000 JavaThread "ajp-apr-7020-exec-76" daemon [_thread_blocked, id=10116, stack(0x000000001e3f0000,0x000000001e4f0000)]
0x0000000014def000 JavaThread "ajp-apr-7020-exec-75" daemon [_thread_blocked, id=10144, stack(0x000000001e1e0000,0x000000001e2e0000)]
0x0000000014dee800 JavaThread "ajp-apr-7020-exec-74" daemon [_thread_blocked, id=10168, stack(0x000000001e0e0000,0x000000001e1e0000)]
0x0000000014ded800 JavaThread "ajp-apr-7020-exec-73" daemon [_thread_blocked, id=10172, stack(0x000000001dfe0000,0x000000001e0e0000)]
0x0000000014ded000 JavaThread "ajp-apr-7020-exec-72" daemon [_thread_blocked, id=10132, stack(0x000000001dee0000,0x000000001dfe0000)]
0x0000000014dec000 JavaThread "ajp-apr-7020-exec-71" daemon [_thread_blocked, id=10076, stack(0x000000001dde0000,0x000000001dee0000)]
0x0000000014deb800 JavaThread "ajp-apr-7020-exec-70" daemon [_thread_blocked, id=10104, stack(0x000000001dce0000,0x000000001dde0000)]
0x0000000014dea800 JavaThread "ajp-apr-7020-exec-69" daemon [_thread_blocked, id=10108, stack(0x000000001dbe0000,0x000000001dce0000)]
0x0000000014dea000 JavaThread "ajp-apr-7020-exec-68" daemon [_thread_blocked, id=10124, stack(0x000000001dae0000,0x000000001dbe0000)]
0x0000000014de9000 JavaThread "ajp-apr-7020-exec-67" daemon [_thread_blocked, id=10060, stack(0x000000001d9e0000,0x000000001dae0000)]
0x0000000015168000 JavaThread "ajp-apr-7020-exec-66" daemon [_thread_blocked, id=10100, stack(0x000000001d8e0000,0x000000001d9e0000)]
0x0000000015167000 JavaThread "ajp-apr-7020-exec-65" daemon [_thread_blocked, id=10064, stack(0x000000001d7e0000,0x000000001d8e0000)]
0x0000000015166800 JavaThread "ajp-apr-7020-exec-64" daemon [_thread_blocked, id=8528, stack(0x000000001d6e0000,0x000000001d7e0000)]
0x0000000015165800 JavaThread "ajp-apr-7020-exec-63" daemon [_thread_blocked, id=8956, stack(0x000000001d5e0000,0x000000001d6e0000)]
0x0000000015165000 JavaThread "ajp-apr-7020-exec-62" daemon [_thread_blocked, id=8960, stack(0x000000001d4e0000,0x000000001d5e0000)]
0x0000000015164000 JavaThread "ajp-apr-7020-exec-61" daemon [_thread_blocked, id=8880, stack(0x000000001d3e0000,0x000000001d4e0000)]
0x0000000015163800 JavaThread "ajp-apr-7020-exec-60" daemon [_thread_blocked, id=556, stack(0x000000001d2e0000,0x000000001d3e0000)]
0x0000000015162800 JavaThread "ajp-apr-7020-exec-59" daemon [_thread_blocked, id=480, stack(0x000000001d1e0000,0x000000001d2e0000)]
0x0000000015162000 JavaThread "ajp-apr-7020-exec-58" daemon [_thread_blocked, id=8868, stack(0x000000001d0e0000,0x000000001d1e0000)]
0x0000000015161000 JavaThread "ajp-apr-7020-exec-57" daemon [_thread_blocked, id=8532, stack(0x000000001cfe0000,0x000000001d0e0000)]
0x0000000015160800 JavaThread "ajp-apr-7020-exec-56" daemon [_thread_blocked, id=980, stack(0x000000001cee0000,0x000000001cfe0000)]
0x000000001515f800 JavaThread "ajp-apr-7020-exec-55" daemon [_thread_blocked, id=7652, stack(0x000000001cde0000,0x000000001cee0000)]
0x000000001515f000 JavaThread "ajp-apr-7020-exec-54" daemon [_thread_blocked, id=10196, stack(0x000000001cce0000,0x000000001cde0000)]
0x000000001515e000 JavaThread "ajp-apr-7020-exec-53" daemon [_thread_blocked, id=10204, stack(0x000000001cbe0000,0x000000001cce0000)]
0x000000001515d800 JavaThread "ajp-apr-7020-exec-52" daemon [_thread_blocked, id=10200, stack(0x000000001cae0000,0x000000001cbe0000)]
0x000000001515c800 JavaThread "ajp-apr-7020-exec-51" daemon [_thread_blocked, id=10040, stack(0x000000001c9e0000,0x000000001cae0000)]
0x000000001515c000 JavaThread "ajp-apr-7020-exec-50" daemon [_thread_blocked, id=10044, stack(0x000000001c8e0000,0x000000001c9e0000)]
0x000000001515b000 JavaThread "ajp-apr-7020-exec-49" daemon [_thread_blocked, id=10080, stack(0x000000001c7e0000,0x000000001c8e0000)]
0x000000001515a800 JavaThread "ajp-apr-7020-exec-48" daemon [_thread_blocked, id=10008, stack(0x000000001c6e0000,0x000000001c7e0000)]
0x0000000015159800 JavaThread "ajp-apr-7020-exec-47" daemon [_thread_blocked, id=10004, stack(0x000000001c5e0000,0x000000001c6e0000)]
0x0000000015159000 JavaThread "ajp-apr-7020-exec-46" daemon [_thread_blocked, id=10000, stack(0x000000001c4e0000,0x000000001c5e0000)]
0x0000000014fc9000 JavaThread "ajp-apr-7020-exec-45" daemon [_thread_blocked, id=9980, stack(0x000000001c3e0000,0x000000001c4e0000)]
0x0000000014fc8000 JavaThread "ajp-apr-7020-exec-44" daemon [_thread_in_native, id=9580, stack(0x000000001c2e0000,0x000000001c3e0000)]
0x0000000014fc7800 JavaThread "ajp-apr-7020-exec-43" daemon [_thread_blocked, id=9604, stack(0x000000001c1e0000,0x000000001c2e0000)]
0x0000000014fc6800 JavaThread "ajp-apr-7020-exec-42" daemon [_thread_blocked, id=9536, stack(0x000000001c0e0000,0x000000001c1e0000)]
0x0000000014fc6000 JavaThread "ajp-apr-7020-exec-41" daemon [_thread_blocked, id=9564, stack(0x000000001bfe0000,0x000000001c0e0000)]
0x0000000014fc5000 JavaThread "ajp-apr-7020-exec-40" daemon [_thread_blocked, id=9544, stack(0x000000001bee0000,0x000000001bfe0000)]
0x0000000014fc4800 JavaThread "ajp-apr-7020-exec-39" daemon [_thread_in_native, id=9164, stack(0x000000001bde0000,0x000000001bee0000)]
0x0000000014fc3800 JavaThread "ajp-apr-7020-exec-38" daemon [_thread_blocked, id=9344, stack(0x000000001bce0000,0x000000001bde0000)]
0x0000000014fc3000 JavaThread "ajp-apr-7020-exec-37" daemon [_thread_blocked, id=9340, stack(0x000000001bbe0000,0x000000001bce0000)]
0x0000000014fc2000 JavaThread "ajp-apr-7020-exec-36" daemon [_thread_blocked, id=896, stack(0x000000001bae0000,0x000000001bbe0000)]
0x0000000014fc1800 JavaThread "ajp-apr-7020-exec-35" daemon [_thread_blocked, id=9988, stack(0x000000001b9e0000,0x000000001bae0000)]
0x0000000014fc0800 JavaThread "ajp-apr-7020-exec-34" daemon [_thread_blocked, id=10012, stack(0x000000001b8e0000,0x000000001b9e0000)]
0x0000000014fc0000 JavaThread "ajp-apr-7020-exec-33" daemon [_thread_blocked, id=9348, stack(0x000000001b7e0000,0x000000001b8e0000)]
0x0000000014fbf000 JavaThread "ajp-apr-7020-exec-32" daemon [_thread_blocked, id=9352, stack(0x000000001b6e0000,0x000000001b7e0000)]
0x0000000014fbe800 JavaThread "ajp-apr-7020-exec-31" daemon [_thread_blocked, id=7236, stack(0x000000001b5e0000,0x000000001b6e0000)]
0x0000000014fbd800 JavaThread "ajp-apr-7020-exec-30" daemon [_thread_blocked, id=720, stack(0x000000001b4e0000,0x000000001b5e0000)]
0x0000000014fbd000 JavaThread "ajp-apr-7020-exec-29" daemon [_thread_blocked, id=8464, stack(0x000000001b3e0000,0x000000001b4e0000)]
0x0000000014fbc000 JavaThread "ajp-apr-7020-exec-28" daemon [_thread_blocked, id=8468, stack(0x000000001b2e0000,0x000000001b3e0000)]
0x0000000014fbb800 JavaThread "ajp-apr-7020-exec-27" daemon [_thread_blocked, id=8480, stack(0x000000001b1e0000,0x000000001b2e0000)]
0x0000000014fba800 JavaThread "ajp-apr-7020-exec-26" daemon [_thread_blocked, id=8484, stack(0x000000001b0e0000,0x000000001b1e0000)]
0x0000000014fba000 JavaThread "ajp-apr-7020-exec-25" daemon [_thread_blocked, id=8488, stack(0x000000001afe0000,0x000000001b0e0000)]
0x00000000151a5000 JavaThread "ajp-apr-7020-exec-24" daemon [_thread_blocked, id=956, stack(0x000000001ade0000,0x000000001aee0000)]
0x00000000151a4800 JavaThread "ajp-apr-7020-exec-23" daemon [_thread_blocked, id=8512, stack(0x000000001a9e0000,0x000000001aae0000)]
0x00000000151a8000 JavaThread "ajp-apr-7020-exec-22" daemon [_thread_blocked, id=8516, stack(0x000000001a8e0000,0x000000001a9e0000)]
0x00000000151a9000 JavaThread "AWT-Windows" daemon [_thread_in_native, id=10208, stack(0x000000001aee0000,0x000000001afe0000)]
0x00000000151a7800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=10216, stack(0x000000001ace0000,0x000000001ade0000)]
0x00000000151a6800 JavaThread "ajp-apr-7020-exec-21" daemon [_thread_blocked, id=8448, stack(0x000000001abe0000,0x000000001ace0000)]
0x00000000151a6000 JavaThread "ajp-apr-7020-exec-20" daemon [_thread_blocked, id=8428, stack(0x000000001aae0000,0x000000001abe0000)]
0x00000000151a3800 JavaThread "ajp-apr-7020-exec-19" daemon [_thread_blocked, id=8440, stack(0x000000001a7e0000,0x000000001a8e0000)]
0x00000000151a3000 JavaThread "ajp-apr-7020-exec-18" daemon [_thread_blocked, id=8452, stack(0x000000001a6e0000,0x000000001a7e0000)]
0x00000000151a2000 JavaThread "TimerQueue" daemon [_thread_blocked, id=10220, stack(0x000000001a5e0000,0x000000001a6e0000)]
0x00000000151a1800 JavaThread "ajp-apr-7020-exec-17" daemon [_thread_blocked, id=10224, stack(0x000000001a4e0000,0x000000001a5e0000)]
0x00000000151a0800 JavaThread "ajp-apr-7020-exec-16" daemon [_thread_blocked, id=10228, stack(0x000000001a3e0000,0x000000001a4e0000)]
0x00000000151a0000 JavaThread "ajp-apr-7020-exec-15" daemon [_thread_blocked, id=8928, stack(0x000000001a2e0000,0x000000001a3e0000)]
0x000000001519f000 JavaThread "ajp-apr-7020-exec-14" daemon [_thread_blocked, id=8916, stack(0x000000001a1e0000,0x000000001a2e0000)]
0x000000001519e800 JavaThread "ajp-apr-7020-exec-13" daemon [_thread_blocked, id=10164, stack(0x000000001a0e0000,0x000000001a1e0000)]
0x000000001519d800 JavaThread "ajp-apr-7020-exec-12" daemon [_thread_blocked, id=10156, stack(0x0000000019fe0000,0x000000001a0e0000)]
0x000000001519d000 JavaThread "ajp-apr-7020-exec-11" daemon [_thread_blocked, id=10152, stack(0x0000000019ee0000,0x0000000019fe0000)]
0x000000001519c000 JavaThread "ajp-apr-7020-exec-10" daemon [_thread_blocked, id=10120, stack(0x0000000019de0000,0x0000000019ee0000)]
0x000000001519b800 JavaThread "ajp-apr-7020-exec-9" daemon [_thread_blocked, id=10092, stack(0x0000000019ce0000,0x0000000019de0000)]
0x000000001519a800 JavaThread "ajp-apr-7020-exec-8" daemon [_thread_blocked, id=10096, stack(0x0000000019be0000,0x0000000019ce0000)]
0x000000001519a000 JavaThread "ajp-apr-7020-exec-7" daemon [_thread_blocked, id=10052, stack(0x0000000019ae0000,0x0000000019be0000)]
0x000000001579a000 JavaThread "ajp-apr-7020-exec-6" daemon [_thread_blocked, id=9652, stack(0x00000000199e0000,0x0000000019ae0000)]
0x0000000015799800 JavaThread "ajp-apr-7020-exec-5" daemon [_thread_blocked, id=9664, stack(0x00000000198e0000,0x00000000199e0000)]
0x0000000015798800 JavaThread "ajp-apr-7020-exec-4" daemon [_thread_blocked, id=9480, stack(0x00000000197e0000,0x00000000198e0000)]
0x0000000015798000 JavaThread "ajp-apr-7020-exec-3" daemon [_thread_blocked, id=9460, stack(0x0000000016ae0000,0x0000000016be0000)]
0x0000000015797000 JavaThread "ajp-apr-7020-exec-2" daemon [_thread_blocked, id=9464, stack(0x00000000146b0000,0x00000000147b0000)]
0x0000000015796800 JavaThread "ajp-apr-7020-exec-1" daemon [_thread_blocked, id=1356, stack(0x00000000145b0000,0x00000000146b0000)]
0x0000000015795800 JavaThread "ajp-apr-7020-AsyncTimeout" daemon [_thread_blocked, id=9932, stack(0x00000000196e0000,0x00000000197e0000)]
0x0000000015795000 JavaThread "ajp-apr-7020-Acceptor-0" daemon [_thread_in_native, id=9928, stack(0x00000000195e0000,0x00000000196e0000)]
0x0000000015794000 JavaThread "ajp-apr-7020-CometPoller-7" daemon [_thread_blocked, id=9924, stack(0x00000000194e0000,0x00000000195e0000)]
0x0000000015793800 JavaThread "ajp-apr-7020-CometPoller-6" daemon [_thread_blocked, id=9920, stack(0x00000000193e0000,0x00000000194e0000)]
0x0000000015792800 JavaThread "ajp-apr-7020-CometPoller-5" daemon [_thread_blocked, id=9916, stack(0x00000000192e0000,0x00000000193e0000)]
0x0000000015792000 JavaThread "ajp-apr-7020-CometPoller-4" daemon [_thread_blocked, id=9912, stack(0x00000000191e0000,0x00000000192e0000)]
0x0000000015791000 JavaThread "ajp-apr-7020-CometPoller-3" daemon [_thread_blocked, id=9908, stack(0x00000000190e0000,0x00000000191e0000)]
0x0000000015790800 JavaThread "ajp-apr-7020-CometPoller-2" daemon [_thread_blocked, id=9904, stack(0x0000000018fe0000,0x00000000190e0000)]
0x000000001578f800 JavaThread "ajp-apr-7020-CometPoller-1" daemon [_thread_blocked, id=9900, stack(0x0000000018ee0000,0x0000000018fe0000)]
0x000000001578f000 JavaThread "ajp-apr-7020-CometPoller-0" daemon [_thread_blocked, id=9896, stack(0x0000000018de0000,0x0000000018ee0000)]
0x000000001578e000 JavaThread "ajp-apr-7020-Poller-7" daemon [_thread_blocked, id=9892, stack(0x0000000018ce0000,0x0000000018de0000)]
0x000000001578d800 JavaThread "ajp-apr-7020-Poller-6" daemon [_thread_blocked, id=9888, stack(0x0000000018be0000,0x0000000018ce0000)]
0x000000001578c800 JavaThread "ajp-apr-7020-Poller-5" daemon [_thread_blocked, id=9884, stack(0x0000000018ae0000,0x0000000018be0000)]
0x000000001578c000 JavaThread "ajp-apr-7020-Poller-4" daemon [_thread_blocked, id=9880, stack(0x00000000189e0000,0x0000000018ae0000)]
0x000000001578b000 JavaThread "ajp-apr-7020-Poller-3" daemon [_thread_blocked, id=9876, stack(0x00000000188e0000,0x00000000189e0000)]
0x0000000014949800 JavaThread "ajp-apr-7020-Poller-2" daemon [_thread_blocked, id=9872, stack(0x00000000187e0000,0x00000000188e0000)]
0x0000000014948800 JavaThread "ajp-apr-7020-Poller-1" daemon [_thread_blocked, id=9868, stack(0x00000000186e0000,0x00000000187e0000)]
0x0000000014948000 JavaThread "ajp-apr-7020-Poller-0" daemon [_thread_blocked, id=9864, stack(0x00000000185e0000,0x00000000186e0000)]
0x0000000014947000 JavaThread "http-apr-8020-AsyncTimeout" daemon [_thread_blocked, id=9860, stack(0x00000000184e0000,0x00000000185e0000)]
0x0000000014946800 JavaThread "http-apr-8020-Acceptor-0" daemon [_thread_in_native, id=9856, stack(0x00000000183e0000,0x00000000184e0000)]
0x0000000014945800 JavaThread "http-apr-8020-Sendfile-0" daemon [_thread_blocked, id=9852, stack(0x00000000182e0000,0x00000000183e0000)]
0x0000000014945000 JavaThread "http-apr-8020-CometPoller-7" daemon [_thread_blocked, id=9848, stack(0x00000000181e0000,0x00000000182e0000)]
0x0000000014944000 JavaThread "http-apr-8020-CometPoller-6" daemon [_thread_blocked, id=9844, stack(0x00000000180e0000,0x00000000181e0000)]
0x0000000014943800 JavaThread "http-apr-8020-CometPoller-5" daemon [_thread_blocked, id=9840, stack(0x0000000017fe0000,0x00000000180e0000)]
0x0000000014942800 JavaThread "http-apr-8020-CometPoller-4" daemon [_thread_blocked, id=9836, stack(0x0000000017ae0000,0x0000000017be0000)]
0x0000000014942000 JavaThread "http-apr-8020-CometPoller-3" daemon [_thread_blocked, id=9832, stack(0x00000000179e0000,0x0000000017ae0000)]
0x0000000014941000 JavaThread "http-apr-8020-CometPoller-2" daemon [_thread_blocked, id=9828, stack(0x00000000178e0000,0x00000000179e0000)]
0x0000000014940800 JavaThread "http-apr-8020-CometPoller-1" daemon [_thread_blocked, id=9824, stack(0x00000000177e0000,0x00000000178e0000)]
0x000000001493f800 JavaThread "http-apr-8020-CometPoller-0" daemon [_thread_blocked, id=9820, stack(0x00000000176e0000,0x00000000177e0000)]
0x000000001493f000 JavaThread "http-apr-8020-Poller-7" daemon [_thread_blocked, id=9816, stack(0x00000000175e0000,0x00000000176e0000)]
0x000000001493e000 JavaThread "http-apr-8020-Poller-6" daemon [_thread_blocked, id=9812, stack(0x00000000174e0000,0x00000000175e0000)]
0x000000001493d800 JavaThread "http-apr-8020-Poller-5" daemon [_thread_blocked, id=9808, stack(0x00000000173e0000,0x00000000174e0000)]
0x000000001493c800 JavaThread "http-apr-8020-Poller-4" daemon [_thread_blocked, id=9804, stack(0x00000000172e0000,0x00000000173e0000)]
0x000000001493c000 JavaThread "http-apr-8020-Poller-3" daemon [_thread_blocked, id=9800, stack(0x00000000171e0000,0x00000000172e0000)]
0x000000001493b000 JavaThread "http-apr-8020-Poller-2" daemon [_thread_blocked, id=9796, stack(0x00000000170e0000,0x00000000171e0000)]
0x000000001493a000 JavaThread "http-apr-8020-Poller-1" daemon [_thread_blocked, id=9792, stack(0x0000000016fe0000,0x00000000170e0000)]
0x00000000155b9800 JavaThread "http-apr-8020-Poller-0" daemon [_thread_blocked, id=9788, stack(0x0000000016ce0000,0x0000000016de0000)]
0x00000000156ec800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=9784, stack(0x0000000016be0000,0x0000000016ce0000)]
0x0000000012572000 JavaThread "GC Daemon" daemon [_thread_blocked, id=9336, stack(0x0000000014480000,0x0000000014580000)]
0x000000001225e800 JavaThread "Service Thread" daemon [_thread_blocked, id=7120, stack(0x0000000012a80000,0x0000000012b80000)]
0x0000000012254000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=7116, stack(0x0000000012980000,0x0000000012a80000)]
0x0000000012251800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=7112, stack(0x0000000012880000,0x0000000012980000)]
0x0000000012251000 JavaThread "Attach Listener" daemon [_thread_blocked, id=7108, stack(0x0000000012780000,0x0000000012880000)]
0x0000000012206000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=7104, stack(0x0000000012680000,0x0000000012780000)]
0x0000000012186800 JavaThread "Finalizer" daemon [_thread_blocked, id=7048, stack(0x0000000012580000,0x0000000012680000)]
0x00000000102c8800 JavaThread "Reference Handler" daemon [_thread_blocked, id=7044, stack(0x0000000012080000,0x0000000012180000)]
0x0000000000d1c800 JavaThread "main" [_thread_in_native, id=6720, stack(0x0000000000da0000,0x0000000000ea0000)]

Other Threads:
0x00000000102c1800 VMThread [stack: 0x0000000011f80000,0x0000000012080000] [id=7040]
0x000000001226b000 WatcherThread [stack: 0x0000000012b80000,0x0000000012c80000] [id=7124]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
PSYoungGen total 1394240K, used 15900K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390528K, 1% used [0x00000007aaab0000,0x00000007aba37358,0x00000007ff8a0000)
from space 3712K, 0% used [0x00000007ff8a0000,0x00000007ff8a0000,0x00000007ffc40000)
to space 3840K, 0% used [0x00000007ffc40000,0x00000007ffc40000,0x0000000800000000)
ParOldGen total 2796224K, used 54944K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 1% used [0x0000000700000000,0x00000007035a8320,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)

Card table byte_map: [0x00000000043e0000,0x0000000004c70000] byte_map_base: 0x0000000000c60000

Polling page: 0x0000000000ea0000

Code Cache [0x00000000013e0000, 0x0000000001ee0000, 0x00000000043e0000)
total_blobs=2775 nmethods=2054 adapters=671 free_code_cache=38083Kb largest_free_block=38845632

Compilation events (10 events):
Event: 53946.893 Thread 0x0000000012251800 2230 java.util.LinkedList::clear (60 bytes)
Event: 53946.895 Thread 0x0000000012251800 nmethod 2230 0x0000000001dc3c90 code [0x0000000001dc3dc0, 0x0000000001dc3e58]
Event: 54018.818 Thread 0x0000000012254000 2231 com.tpm.monitors.monitors.genericmonitor.service.GenericMonitorCommonService::toShowByXml (269 bytes)
Event: 54018.853 Thread 0x0000000012254000 nmethod 2231 0x0000000001ec7b90 code [0x0000000001ec7ee0, 0x0000000001ec9600]
Event: 54125.263 Thread 0x0000000012251800 2232 ! com.tpm.monitors.monitors.common.utils.MonitorsUtils::RS2TableString4DropdownStr (194 bytes)
Event: 54125.335 Thread 0x0000000012251800 nmethod 2232 0x0000000001ece810 code [0x0000000001eced60, 0x0000000001ed1010]
Event: 54135.477 Thread 0x0000000012254000 2233 sun.java2d.SunGraphics2D::addRenderingHints (144 bytes)
Event: 54135.489 Thread 0x0000000012254000 nmethod 2233 0x0000000001e894d0 code [0x0000000001e896c0, 0x0000000001e89d70]
Event: 54149.940 Thread 0x0000000012251800 2234 java.awt.geom.Path2D$Float::append (216 bytes)
Event: 54149.950 Thread 0x0000000012251800 nmethod 2234 0x0000000001e83510 code [0x0000000001e836c0, 0x0000000001e84010]

GC Heap History (10 events):
Event: 54288.899 GC heap before
{Heap before GC invocations=240 (full 102):
PSYoungGen total 1393920K, used 2258K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff820000)
from space 3904K, 57% used [0x00000007ffc30000,0x00000007ffe64ab0,0x0000000800000000)
to space 4032K, 0% used [0x00000007ff820000,0x00000007ff820000,0x00000007ffc10000)
ParOldGen total 2796224K, used 56898K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 2% used [0x0000000700000000,0x0000000703790bc0,0x00000007aaab0000)
PSPermGen total 131072K, used 51547K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f3256c98,0x00000006f8000000)
Event: 54289.071 GC heap after
Heap after GC invocations=240 (full 102):
PSYoungGen total 1393920K, used 0K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff820000)
from space 3904K, 0% used [0x00000007ffc30000,0x00000007ffc30000,0x0000000800000000)
to space 4032K, 0% used [0x00000007ff820000,0x00000007ff820000,0x00000007ffc10000)
ParOldGen total 2796224K, used 58167K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 2% used [0x0000000700000000,0x00000007038cde80,0x00000007aaab0000)
PSPermGen total 131072K, used 51547K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f3256c98,0x00000006f8000000)
}
Event: 54299.230 GC heap before
{Heap before GC invocations=241 (full 102):
PSYoungGen total 1393920K, used 28044K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 2% used [0x00000007aaab0000,0x00000007ac613128,0x00000007ff820000)
from space 3904K, 0% used [0x00000007ffc30000,0x00000007ffc30000,0x0000000800000000)
to space 4032K, 0% used [0x00000007ff820000,0x00000007ff820000,0x00000007ffc10000)
ParOldGen total 2796224K, used 58167K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 2% used [0x0000000700000000,0x00000007038cde80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
Event: 54299.235 GC heap after
Heap after GC invocations=241 (full 102):
PSYoungGen total 1394048K, used 769K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff820000)
from space 4032K, 19% used [0x00000007ff820000,0x00000007ff8e0780,0x00000007ffc10000)
to space 3840K, 0% used [0x00000007ffc40000,0x00000007ffc40000,0x0000000800000000)
ParOldGen total 2796224K, used 58167K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 2% used [0x0000000700000000,0x00000007038cde80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
}
Event: 54299.235 GC heap before
{Heap before GC invocations=242 (full 103):
PSYoungGen total 1394048K, used 769K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff820000)
from space 4032K, 19% used [0x00000007ff820000,0x00000007ff8e0780,0x00000007ffc10000)
to space 3840K, 0% used [0x00000007ffc40000,0x00000007ffc40000,0x0000000800000000)
ParOldGen total 2796224K, used 58167K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 2% used [0x0000000700000000,0x00000007038cde80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
Event: 54299.400 GC heap after
Heap after GC invocations=242 (full 103):
PSYoungGen total 1394048K, used 0K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff820000)
from space 4032K, 0% used [0x00000007ff820000,0x00000007ff820000,0x00000007ffc10000)
to space 3840K, 0% used [0x00000007ffc40000,0x00000007ffc40000,0x0000000800000000)
ParOldGen total 2796224K, used 52499K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 1% used [0x0000000700000000,0x0000000703344e80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
}
Event: 54299.531 GC heap before
{Heap before GC invocations=243 (full 103):
PSYoungGen total 1394048K, used 1177K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390016K, 0% used [0x00000007aaab0000,0x00000007aabd6618,0x00000007ff820000)
from space 4032K, 0% used [0x00000007ff820000,0x00000007ff820000,0x00000007ffc10000)
to space 3840K, 0% used [0x00000007ffc40000,0x00000007ffc40000,0x0000000800000000)
ParOldGen total 2796224K, used 52499K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 1% used [0x0000000700000000,0x0000000703344e80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
Event: 54299.536 GC heap after
Heap after GC invocations=243 (full 103):
PSYoungGen total 1394368K, used 288K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390528K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff8a0000)
from space 3840K, 7% used [0x00000007ffc40000,0x00000007ffc88000,0x0000000800000000)
to space 3712K, 0% used [0x00000007ff8a0000,0x00000007ff8a0000,0x00000007ffc40000)
ParOldGen total 2796224K, used 52499K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 1% used [0x0000000700000000,0x0000000703344e80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
}
Event: 54299.536 GC heap before
{Heap before GC invocations=244 (full 104):
PSYoungGen total 1394368K, used 288K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390528K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff8a0000)
from space 3840K, 7% used [0x00000007ffc40000,0x00000007ffc88000,0x0000000800000000)
to space 3712K, 0% used [0x00000007ff8a0000,0x00000007ff8a0000,0x00000007ffc40000)
ParOldGen total 2796224K, used 52499K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 1% used [0x0000000700000000,0x0000000703344e80,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
Event: 54299.670 GC heap after
Heap after GC invocations=244 (full 104):
PSYoungGen total 1394368K, used 0K [0x00000007aaab0000, 0x0000000800000000, 0x0000000800000000)
eden space 1390528K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007ff8a0000)
from space 3840K, 0% used [0x00000007ffc40000,0x00000007ffc40000,0x0000000800000000)
to space 3712K, 0% used [0x00000007ff8a0000,0x00000007ff8a0000,0x00000007ffc40000)
ParOldGen total 2796224K, used 52263K [0x0000000700000000, 0x00000007aaab0000, 0x00000007aaab0000)
object space 2796224K, 1% used [0x0000000700000000,0x0000000703309d10,0x00000007aaab0000)
PSPermGen total 131072K, used 51878K [0x00000006f0000000, 0x00000006f8000000, 0x0000000700000000)
object space 131072K, 39% used [0x00000006f0000000,0x00000006f32a9b28,0x00000006f8000000)
}

Deoptimization events (10 events):
Event: 53031.567 Thread 0x0000000015454800 Uncommon trap -58 fr.pc 0x0000000001d87a9c
Event: 53031.567 Thread 0x0000000015454800 Uncommon trap -58 fr.pc 0x0000000001d87a9c
Event: 53256.938 Thread 0x0000000029411000 Uncommon trap -83 fr.pc 0x000000000176911c
Event: 53739.239 Thread 0x0000000014ac0800 Uncommon trap -42 fr.pc 0x0000000001e7c564
Event: 53739.242 Thread 0x0000000014ac0800 Uncommon trap -42 fr.pc 0x0000000001e7c5e4
Event: 53739.244 Thread 0x0000000014ac0800 Uncommon trap -42 fr.pc 0x0000000001e7c564
Event: 53739.269 Thread 0x0000000014ac0800 Uncommon trap -34 fr.pc 0x0000000001d554a0
Event: 53739.294 Thread 0x0000000014ac0800 Uncommon trap -34 fr.pc 0x0000000001d5cc74
Event: 53739.294 Thread 0x0000000014ac0800 Uncommon trap -34 fr.pc 0x0000000001d5cc74
Event: 53739.315 Thread 0x0000000014ac0800 Uncommon trap -42 fr.pc 0x0000000001e4d574

Internal exceptions (10 events):
Event: 54294.906 Thread 0x0000000014df0800 Threw 0x00000007abd373c8 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54294.906 Thread 0x0000000014df0800 Threw 0x00000007abd37fe8 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54295.421 Thread 0x0000000014df0800 Threw 0x00000007abe27d88 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54295.421 Thread 0x0000000014df0800 Threw 0x00000007abe28948 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54295.427 Thread 0x0000000014df0800 Threw 0x00000007abe6b638 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54295.427 Thread 0x0000000014df0800 Threw 0x00000007abe6c1f8 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54298.431 Thread 0x0000000014fba000 Threw 0x00000007ac01e4f8 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54298.431 Thread 0x0000000014fba000 Threw 0x00000007ac01ee28 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54298.432 Thread 0x0000000014fba000 Threw 0x00000007ac025380 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166
Event: 54298.432 Thread 0x0000000014fba000 Threw 0x00000007ac025d70 at C:\jdk7u1_64p\jdk7u11\hotspot\src\share\vm\prims\jvm.cpp:1166

Events (10 events):
Event: 54298.555 Thread 0x0000000029414000 Thread added: 0x0000000029414000
Event: 54299.002 Executing VM operation: RevokeBias
Event: 54299.002 Executing VM operation: RevokeBias done
Event: 54299.002 Thread 0x0000000029414000 Thread exited: 0x0000000029414000
Event: 54299.212 Thread 0x0000000029413800 Thread exited: 0x0000000029413800
Event: 54299.230 Executing VM operation: ParallelGCSystemGC
Event: 54299.401 Executing VM operation: ParallelGCSystemGC done
Event: 54299.519 Thread 0x000000002940f800 Thread exited: 0x000000002940f800
Event: 54299.531 Executing VM operation: ParallelGCSystemGC
Event: 54299.670 Executing VM operation: ParallelGCSystemGC done

Dynamic libraries:
0x0000000140000000 - 0x000000014001e000 D:\Tomcat7\bin\tomcat7.exe
0x0000000077ec0000 - 0x0000000077ffc000 C:\WINDOWS\system32\ntdll.dll
0x0000000077d40000 - 0x0000000077eb3000 C:\WINDOWS\system32\kernel32.dll
0x000007ff7fee0000 - 0x000007ff7ffe6000 C:\WINDOWS\system32\ADVAPI32.dll
0x000007ff7fd30000 - 0x000007ff7fed5000 C:\WINDOWS\system32\RPCRT4.dll
0x000007ff7e9c0000 - 0x000007ff7e9e2000 C:\WINDOWS\system32\Secur32.dll
0x000007ff7f190000 - 0x000007ff7fb9c000 C:\WINDOWS\system32\SHELL32.dll
0x000007ff7fc00000 - 0x000007ff7fc86000 C:\WINDOWS\system32\msvcrt.dll
0x000007ff7fc90000 - 0x000007ff7fd2c000 C:\WINDOWS\system32\GDI32.dll
0x0000000077c20000 - 0x0000000077d2c000 C:\WINDOWS\system32\USER32.dll
0x000007ff7ef60000 - 0x000007ff7effb000 C:\WINDOWS\system32\SHLWAPI.dll
0x000007ff7d500000 - 0x000007ff7d539000 C:\WINDOWS\system32\IMM32.DLL
0x000007ff69300000 - 0x000007ff6930d000 C:\WINDOWS\system32\LPK.DLL
0x000007ff78e80000 - 0x000007ff78f0e000 C:\WINDOWS\system32\USP10.dll
0x000007ff7f000000 - 0x000007ff7f187000 C:\WINDOWS\WinSxS\amd64_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.5190_x-ww_639C4A9E\comctl32.dll
0x000007ff77310000 - 0x000007ff77340000 C:\WINDOWS\system32\WS2_32.dll
0x000007ff77140000 - 0x000007ff7714c000 C:\WINDOWS\system32\WS2HELP.dll
0x000000006ee40000 - 0x000000006f55f000 D:\jdk1.7\jre\bin\server\jvm.dll
0x000007ff770f0000 - 0x000007ff770fa000 C:\WINDOWS\system32\WSOCK32.dll
0x000007ff7e240000 - 0x000007ff7e290000 C:\WINDOWS\system32\WINMM.dll
0x000007ff7e380000 - 0x000007ff7e390000 C:\WINDOWS\system32\PSAPI.DLL
0x0000000079730000 - 0x0000000079801000 D:\jdk1.7\jre\bin\MSVCR100.dll
0x000000006eb70000 - 0x000000006eb7f000 D:\jdk1.7\jre\bin\verify.dll
0x000000006d660000 - 0x000000006d688000 D:\jdk1.7\jre\bin\java.dll
0x000000006ebe0000 - 0x000000006ebf5000 D:\jdk1.7\jre\bin\zip.dll
0x000000006e970000 - 0x000000006e989000 D:\jdk1.7\jre\bin\net.dll
0x000000006e990000 - 0x000000006e9a1000 D:\jdk1.7\jre\bin\nio.dll
0x000000006e8b0000 - 0x000000006e8bb000 D:\jdk1.7\jre\bin\management.dll
0x0000000180000000 - 0x000000018018c000 D:\Tomcat7\bin\tcnative-1.dll
0x000007ff771b0000 - 0x000007ff7722d000 C:\WINDOWS\system32\MSWSOCK.dll
0x0000000068000000 - 0x000000006804c000 C:\WINDOWS\system32\rsaenh.dll
0x000000006eac0000 - 0x000000006eae4000 D:\jdk1.7\jre\bin\sunec.dll
0x000000006eaf0000 - 0x000000006eafb000 D:\jdk1.7\jre\bin\sunmscapi.dll
0x000007ff48230000 - 0x000007ff48390000 C:\WINDOWS\system32\CRYPT32.dll
0x000007ff7d340000 - 0x000007ff7d36c000 C:\WINDOWS\system32\MSASN1.dll
0x000007ff6d200000 - 0x000007ff6d293000 C:\WINDOWS\system32\hnetcfg.dll
0x000007ff77170000 - 0x000007ff7717b000 C:\WINDOWS\System32\wshtcpip.dll
0x000007ff7c680000 - 0x000007ff7c78a000 C:\WINDOWS\system32\USERENV.dll
0x000007ff77370000 - 0x000007ff77409000 C:\WINDOWS\system32\netapi32.dll
0x000007ff7e8f0000 - 0x000007ff7e93f000 C:\WINDOWS\system32\DNSAPI.dll
0x000007ff7e9f0000 - 0x000007ff7e9fb000 C:\WINDOWS\System32\winrnr.dll
0x000007ff7e950000 - 0x000007ff7e9b6000 C:\WINDOWS\system32\WLDAP32.dll
0x000007ff7ea00000 - 0x000007ff7ea07000 C:\WINDOWS\system32\rasadhlp.dll
0x000000006d130000 - 0x000000006d2c3000 D:\jdk1.7\jre\bin\awt.dll
0x000007ff7ebc0000 - 0x000007ff7ecd6000 C:\WINDOWS\system32\OLEAUT32.dll
0x000007ff57140000 - 0x000007ff573c7000 C:\WINDOWS\system32\ole32.dll
0x000000006d400000 - 0x000000006d434000 D:\jdk1.7\jre\bin\fontmanager.dll
0x000000006eb00000 - 0x000000006eb41000 D:\jdk1.7\jre\bin\t2k.dll
0x000007ff7ea10000 - 0x000007ff7eaee000 C:\WINDOWS\system32\CLBCatQ.DLL
0x000007ff7eaf0000 - 0x000007ff7ebb6000 C:\WINDOWS\system32\COMRes.dll
0x000007ff7fbf0000 - 0x000007ff7fbfb000 C:\WINDOWS\system32\VERSION.dll
0x000000006d2d0000 - 0x000000006d2fa000 D:\jdk1.7\jre\bin\dcpr.dll
0x000007ff724e0000 - 0x000007ff7261e000 C:\WINDOWS\system32\DBGHELP.DLL

VM Arguments:
jvm_args: -Dcatalina.base=D:\Tomcat7 -Dcatalina.home=D:\Tomcat7 -Djava.endorsed.dirs=D:\Tomcat7\endorsed -Djava.io.tmpdir=D:\Tomcat7\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=D:\Tomcat7\conf\logging.properties -XX:PermSize=128m -XX:MaxPermSize=256m exit -Xms4096m -Xmx4096m
java_command: <unknown>
Launcher Type: generic

Environment Variables:
JAVA_HOME=D:\jdk
PATH=C:\Program Files\HP\NCU;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;
OS=Windows_NT
PROCESSOR_IDENTIFIER=EM64T Family 6 Model 23 Stepping 10, GenuineIntel

--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, tsc

Memory: 4k page, physical 12580512k(1509624k free), swap 16234840k(7403256k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (23.6-b04) for windows-amd64 JRE (1.7.0_11-b21), built on Jan 12 2013 02:38:42 by "java_re" with unknown MS VC++:1600

time: Mon Oct 28 09:54:45 2013
elapsed time: 54397 seconds

Google 搜索关键字

tcnative-1.dll  Java_org_apache_tomcat_jni_Socket_sendbb

https://issues.apache.org/bugzilla/show_bug.cgi?id=51813

据说是tomcat native liberary 的一个bug。我们目前用的版本是1.1.27，据说1.1.29解决了这个问题.

今天升级了Tomcat7.0.47，该版本的 tomcat native liberary 是1.1.29，待运行几天后查看该问题是否已经解决。

已经正常运行一个多月，从无发生一次类似现象，标明该问题已经彻底解决。