问题描述:系统不定期报出Oracle“未处理的内部错误(-2)”,严重时IIS会Crash
典型异常日志如下:
Exception type: System.AccessViolationException
Message: 尝试读取或写入受保护的内存。这通常指示其他内存已损坏。
InnerException: <none>
StackTrace (generated):
SP IP Function
000000000C388FE0 0000000000000001 System_Data_OracleClient_ni!System.Data.Common.UnsafeNativeMethods.OCILobCreateTemporary(System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, UInt16, CHARSETFORM, LOB_TYPE, Int32, DURATION)+0x2
000000000C3890F0 000007FEF4627DEF System_Data_OracleClient_ni!System.Data.OracleClient.TracedNativeMethods.OCILobCreateTemporary(System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, UInt16, CHARSETFORM, LOB_TYPE, Int32, DURATION)+0xef
000000000C3891A0 000007FEF464FC82 System_Data_OracleClient_ni!System.Data.OracleClient.OracleLob..ctor(System.Data.OracleClient.OracleConnection, System.Data.OracleClient.OracleType)+0xc2
000000000C389210 000007FEF463ED33 System_Data_OracleClient_ni!System.Data.OracleClient.OracleParameterBinding.CreateTemporaryLobForValue(System.Data.OracleClient.OracleConnection, System.Data.OracleClient.OracleType, System.Object)+0x63
000000000C389260 000007FEF463FE68 System_Data_OracleClient_ni!System.Data.OracleClient.OracleParameterBinding.PrepareForBind(System.Data.OracleClient.OracleConnection, Int32 ByRef)+0x178
000000000C3892D0 000007FEF46322A1 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.Execute(System.Data.OracleClient.OciStatementHandle, System.Data.CommandBehavior, Boolean, System.Data.OracleClient.O*widDescriptor ByRef, System.Collections.ArrayList ByRef)+0x381
000000000C3893C0 000007FEF46328C6 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteNonQueryInternal(Boolean, System.Data.OracleClient.O*widDescriptor ByRef)+0xa6
000000000C389450 000007FEF46327DE System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteNonQuery()+0x10e
。。。。。。。。。。。
。。。。。。。。。。。
000000000C38E590 000007FEF8FCF8A6 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x286
000000000C38E6F0 000007FEF8FCF60A mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xa
000000000C38E720 000007FEF8F7B220 mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+0x70
000000000C38E770 000007FEF8F7B82B mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+0x1eb
Nested exception -------------------------------------------------------------
Exception object: 0000000182836198
Exception type: System.InvalidOperationException
Message: 未处理的内部错误(-2)。
InnerException: <none>
StackTrace (generated):
SP IP Function
000000000C38DDD0 000007FEF469DE51 System_Data_OracleClient_ni!System.Data.OracleClient.OracleConnection.CheckError(System.Data.OracleClient.OciErrorHandle, Int32)+0x71951
000000000C38DE10 000007FEF463244B System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.Execute(System.Data.OracleClient.OciStatementHandle, System.Data.CommandBehavior, Boolean, System.Data.OracleClient.O*widDescriptor ByRef, System.Collections.ArrayList ByRef)+0x52b
000000000C38DF00 000007FEF4631F14 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.Execute(System.Data.OracleClient.OciStatementHandle, System.Data.CommandBehavior, System.Collections.ArrayList ByRef)+0x24
000000000C38DF50 000007FEF4633E00 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteReader(System.Data.CommandBehavior)+0x130
000000000C38E010 000007FEF46326B9 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteDbDataReader(System.Data.CommandBehavior)+0x9
000000000C38E040 000007FEF60C67A5 System_Data_ni!System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(System.Data.CommandBehavior)+0x15
000000000C38E070 000007FEF60AA555 System_Data_ni!System.Data.Common.DbDataAdapter.FillInternal(System.Data.DataSet, System.Data.DataTable[], Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)+0x145
000000000C38E120 000007FEF60AA394 System_Data_ni!System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)+0x1a4
000000000C38E1D0 000007FEF60AA196 System_Data_ni!System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet)+0x116
原因分析:
查阅资料发现使用DBLink查询时,oracle会开启隐式分布式事务,进而会引发ORA-24784和内部错误(-2)的异常。
与业务组同事交流,项目上确实使用了DBLink;
分析服务器的事件查看器和业务系统日志,也发现了ORA-24784的Exception,如下:
{"code":24784,"ClassName":"System.Data.OracleClient.OracleException","Message":"ORA-24784: 事务处理已存在\u0000","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"
System.Data.OracleClient.OciEnlistContext.Join(OracleInternalConnection internalConnection, Transaction indigoTransaction)\r\n
System.Data.OracleClient.OracleInternalConnection.Enlist(String userName, String password, String serverName, Transaction transaction, Boolean manualEnlistment)\r\n
System.Data.OracleClient.OracleInternalConnection.Activate(Transaction transaction)\r\n
System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)\r\n
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)\r\n
System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)\r\n
System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)\r\n
System.Data.OracleClient.OracleConnection.Open()\r\n
Demo验证:
经过分析测试验证,使用DBLink的查询确实会开启一个隐式的事务,该数据库连接被重用后如果再显示的开启一个分布式事务时,系统会产生ORA-24784: 事务处理已存在的错误。
问题结论
在并发的混合场景下,通过抓取first chance Exception dump,发现ORA-24784与(-2)、IIS Crash相伴而生,具有很强的关联性。
代码排查所有DBLink的使用,DBLink查询后增加显式的commit调用,以提交因oracle DBLink产生的隐式事务。
相关资料:
oracle官方文档:http://files.cnblogs.com/files/zhaoguan_wang/ORA-24784_Oracle_XA%E6%96%87%E6%A1%A3_ID_1271726.1.pdf