110T oracle故障恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:110T oracle故障恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

有客户一套110T的数据库由于存储控制器故障导致数据库无法正常启动,启动报错如下:

Wed Feb  2 23:16:13 2022

Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0

  Mem# 0: /dev/vgredo6/rredo7b

  Mem# 1: /dev/vgredo4/rredo7a

Wed Feb  2 23:16:14 2022

Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:

ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:16:15 2022

Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:

ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []

ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:16:16 2022

Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:

ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []

ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []

ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:16:16 2022

Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:

ORA-00600: internal error code, arguments: [kghstack_free2], [], [], [], [], [], [], []

ORA-00607: Internal error occurred while making a change to a data block

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []

ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []

ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:16:26 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc:

ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:16:27 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] []

ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:16:27 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc:

ORA-00602: internal programming exception

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] []

ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] []

Wed Feb  2 23:17:11 2022

PSP0: terminating instance due to error 472

Instance terminated by PSP0, pid = 26724

该错误原因是由于redo信息和数据文件block信息不匹配导致无法正常应用日志,从而出现异常,在后续的recover 中还出现以下错误

Fri Feb 18 16:09:59 2022

ALTER DATABASE RECOVER  datafile 609,610,611,612,613,614,615,602,603,604,605,606,607,608 

Fri Feb 18 16:09:59 2022

Media Recovery Start

 parallel recovery started with 16 processes

Fri Feb 18 16:10:00 2022

Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0

  Mem# 0: /dev/vgredo6/rredo7b

  Mem# 1: /dev/vgredo4/rredo7a

Fri Feb 18 16:12:17 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc:

ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], []

Fri Feb 18 16:18:51 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc:

ORA-10562: Error occurred while applying redo to data block (file# 602, block# 1693691)

ORA-10564: tablespace DBS_DCDL_PT

ORA-01110: data file 602: '/dev/vgora12/rdbs_dcdl_pt0155'

ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 33682645

ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], []

Fri Feb 18 16:18:55 2022

Media Recovery failed with error 12801

Fri Feb 18 18:23:59 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw1_22483.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] []

ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []

ORA-00472: PMON  process terminated with error

Fri Feb 18 18:24:04 2022

DBW3: terminating instance due to error 472

Fri Feb 18 18:24:04 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc:

ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []

ORA-00472: PMON  process terminated with error

Fri Feb 18 18:24:04 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc:

ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV] [] []

ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []

ORA-00472: PMON  process terminated with error

Fri Feb 18 18:24:09 2022

LGWR: terminating instance due to error 472

Fri Feb 18 18:24:09 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_lgwr_22489.trc:

ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []

ORA-00472: PMON  process terminated with error

从sqlplus中看到类似一些报错

SQL>  recover datafile 601;

ORA-03113: end-of-file on communication channel

SQL> recover datafile 1066;

ORA-00283: recovery session canceled due to errors

ORA-12801: error signaled in parallel query server P015

ORA-00600: internal error code, arguments: [2037], [207064103], [207064103],

[162], [6], [1], [1833009883], [1130705717]

SQL> recover datafile 1065;

ORA-00283: recovery session canceled due to errors

ORA-12801: error signaled in parallel query server P004

ORA-00600: internal error code, arguments: [kcbzpb_1], [142189139], [3], [0],

[], [], [], []

SQL> recover datafile 2042;

ORA-00283: recovery session canceled due to errors

ORA-12801: error signaled in parallel query server P014

ORA-00600: internal error code, arguments: [3020], [627], [3234156],

[2633062764], [], [], [], []

ORA-10567: Redo is inconsistent with data block

通过屏蔽一致性,强制open库成功

Sun Feb 20 21:20:06 2022

SMON: enabling tx recovery

Sun Feb 20 21:20:06 2022

Database Characterset is ZHS16GBK

Sun Feb 20 21:20:07 2022

ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (74, 17) on object 34131051.

Sun Feb 20 21:20:07 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-00376: file 1416 cannot be read at this time

ORA-01110: data file 1416: '/dev/vgora14/rdbs_icdl_pt116'

Sun Feb 20 21:20:08 2022

Stopping background process MMNL

Sun Feb 20 21:20:09 2022

ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (88, 36) on object 33514955.

Sun Feb 20 21:20:09 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-00376: file 1264 cannot be read at this time

ORA-01110: data file 1264: '/dev/vgora14/rdbs_icdl_pt102'

Sun Feb 20 21:20:09 2022

Stopping background process MMON

Starting background process MMON

Starting background process MMNL

MMON started with pid=46, OS id=1482

Sun Feb 20 21:20:10 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-01578: ORACLE data block corrupted (file # 652, block # 3767844)

ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'

Sun Feb 20 21:20:10 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)

ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'

replication_dependency_tracking turned off (no async multimaster replication found)

Sun Feb 20 21:20:11 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)

ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'

Sun Feb 20 21:20:11 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)

ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'

Sun Feb 20 21:20:11 2022

LOGSTDBY: Validating controlfile with logical metadata

Sun Feb 20 21:20:11 2022

LOGSTDBY: Validation complete

Sun Feb 20 21:20:11 2022

Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:

ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)

ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'

Completed: alter database open

对于异常undo进行处理,数据库正常open

110T oracle故障恢复

110T oracle故障恢复



由于客户短期无法迁移数据,先对于一些坏块进行修复,暂时运行数据库后续有时间窗口进行迁移.

上一篇:ubuntu关机和重启


下一篇:ORA-01843 无效日期,月无效