一.1 BLOG文档结构图
一.2 前言部分
一.2.1 导读和注意事项
各位技术爱好者，看完本文后，你可以掌握如下的技能，也可以学到一些其它你所不知道的知识，~O(∩_∩)O~：
① ipcs的使用
② ipcrm释放oracle内存段
③ sysresv的使用
Tips：
① 若文章代码格式有错乱，推荐使用搜狗、QQ或360浏览器，也可以下载pdf格式的文档来查看，pdf文档下载地址：http://yunpan.cn/cdEQedhCs2kFz （提取码：ed9b）
② 本篇BLOG中命令的输出部分需要特别关注的地方我都用灰色背景和粉红色字体来表示，比如下边的例子中，thread 1的最大归档日志号为33，thread 2的最大归档日志号为43是需要特别关注的地方；而命令一般使用黄色背景和红色字体标注；对代码或代码输出部分的注释一般采用蓝色字体表示。
List of Archived Logs in backup set 11
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- ------------------- ---------- ---------
1 32 1621589 2015-05-29 11:09:52 1625242 2015-05-29 11:15:48
1 33 1625242 2015-05-29 11:15:48 1625293 2015-05-29 11:15:58
2 42 1613951 2015-05-29 10:41:18 1625245 2015-05-29 11:15:49
2 43 1625245 2015-05-29 11:15:49 1625253 2015-05-29 11:15:53
[ZFXXDB1:root]:/>lsvg -o
T_XDESK_APP1_vg
rootvg
[ZFXXDB1:root]:/>
00:27:22 SQL> alter tablespace idxtbs read write;
====》2097152*512/1024/1024/1024=1G
本文如有错误或不完善的地方请大家多多指正，ITPUB留言或QQ皆可，您的批评指正是我写作的最大动力。
一.2.2 相关参考文章链接
http://blog.itpub.net/26736162/viewspace-2112443/
http://blog.itpub.net/26736162/viewspace-2112447/
一.2.3 本文简介
同事搭建DG的时候报错了，然后强制退出会话，结果sqlplus不能进入，且看正文部分。
一.3 相关知识点扫盲
unix/linux下的共享内存、信号量、队列信息管理
在unix/linux下，经常有因为共享内存、信号量，队列等共享信息没有干净地清楚而引起一些问题。
查看共享信息的内存的命令是ipcs [-m|-s|-q]。
默认会列出共享内存、信号量，队列信息，-m列出共享内存，-s列出共享信号量，-q列出共享队列
清除命令是ipcrm [-m|-s|-q] id。
-m 删除共享内存，-s删除共享信号量，-q删除共享队列。
一.4 故障分析及解决过程
一.4.1 故障环境介绍
项目 |
source db |
db 类型 |
rac |
db version |
11.2.0.4 |
db 存储 |
ASM |
ORACLE_SID |
oraDESDB2 |
db_name |
oraDESDB |
OS版本及kernel版本 |
AIX 64位 7.1.0.0 |
OS hostname |
ZFLHRDB2 |
一.4.2 故障发生现象及解决过程
oracle的进程已经关闭，但是执行sqlplus的时候hang住，之后报错ORA-09925: Unable to create audit trail file，加-prelim也不行，主要是oracle的进程已经关掉了，但是sqlplus却进不去。
[ZFLHRDB4:oracle]:/oracle>ps -ef|grep ora_
[ZFLHRDB4:oracle]:/oracle>
[ZFLHRDB4:oracle]:/oracle>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue May 31 14:39:47 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
ERROR:
ORA-09925: Unable to create audit trail file
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 9925
ORA-09925: Unable to create audit trail file
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 9925
[ZFLHRDB4:oracle]:/oracle>
[ZFLHRDB4:oracle]:/oracle>
[ZFLHRDB4:oracle]:/oracle>
[ZFLHRDB4:oracle]:/oracle>
[ZFLHRDB4:oracle]:/oracle>sqlplus -prelim / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue May 31 14:40:26 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
ERROR:
ORA-09925: Unable to create audit trail file
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 9925
ORA-09925: Unable to create audit trail file
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 9925
[ZFLHRDB4:oracle]:/oracle>ipcs
IPC status from /dev/mem as of Tue May 31 14:42:01 BEIST 2016
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x9283a0d2 -Rrw------- root system
q 1 0xffffffff ----------- root system
Shared Memory:
m 3 0x210000ac --rw-rw---- root system
m 395313156 0x0ecaefdc --rw-r----- oracle asmadmin
m 9437189 00000000 --rw-r----- oracle asmadmin
m 276824070 00000000 --rw-r----- oracle asmadmin
m 912261127 0x210000d4 --rw-rw---- grid dba
Semaphores:
s 3145728 0x0100324a --ra-ra-r-- root system
s 1 0x620025b4 --ra-r--r-- root system
s 2 0x02001958 --ra-ra-ra- root system
s 3 0x01001958 --ra-ra-ra- root system
s 9 0x010024be --ra------- root system
s 3145738 0x410000ab --ra-ra---- root system
s 21 0x410000cb --ra-ra---- grid dba
[ZFLHRDB4:oracle]:/oracle>ipcrm -m 395313156
[ZFLHRDB4:oracle]:/oracle>ipcs
IPC status from /dev/mem as of Tue May 31 14:42:23 BEIST 2016
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x9283a0d2 -Rrw------- root system
q 1 0xffffffff ----------- root system
Shared Memory:
m 3 0x210000ac --rw-rw---- root system
m 9437189 00000000 --rw-r----- oracle asmadmin
m 276824070 00000000 --rw-r----- oracle asmadmin
m 912261127 0x210000d4 --rw-rw---- grid dba
Semaphores:
s 3145728 0x0100324a --ra-ra-r-- root system
s 1 0x620025b4 --ra-r--r-- root system
s 2 0x02001958 --ra-ra-ra- root system
s 3 0x01001958 --ra-ra-ra- root system
s 9 0x010024be --ra------- root system
s 3145738 0x410000ab --ra-ra---- root system
s 21 0x410000cb --ra-ra---- grid dba
[ZFLHRDB4:oracle]:/oracle>ipcrm -m 9437189
[ZFLHRDB4:oracle]:/oracle>ipcrm -m 276824070
[ZFLHRDB4:oracle]:/oracle>ipcs
IPC status from /dev/mem as of Tue May 31 14:42:39 BEIST 2016
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x9283a0d2 -Rrw------- root system
q 1 0xffffffff ----------- root system
Shared Memory:
m 3 0x210000ac --rw-rw---- root system
m 912261127 0x210000d4 --rw-rw---- grid dba
Semaphores:
s 3145728 0x0100324a --ra-ra-r-- root system
s 1 0x620025b4 --ra-r--r-- root system
s 2 0x02001958 --ra-ra-ra- root system
s 3 0x01001958 --ra-ra-ra- root system
s 9 0x010024be --ra------- root system
s 3145738 0x410000ab --ra-ra---- root system
s 21 0x410000cb --ra-ra---- grid dba
[ZFLHRDB4:oracle]:/oracle>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue May 31 14:42:46 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SYS@oraDESDB2>
问题解决了，但是为啥会出现这样的错误，我在MOS上搜了以下的文章。
一.4.2.1 MOS上的文件
---Connect /as sysdba or Startup Fails With ORA-09925 When Instance Is Down (文档 ID 392643.1)
In this Document
Symptoms
Cause
Solution
References
APPLIES TO:
Oracle Database - Enterprise Edition - Version 8.1.7.0 to 12.1.0.1 [Release 8.1.7 to 12.1]
Information in this document applies to any platform.
Checked for relevance on 05-Oct-2010
SYMPTOMS
When the database instance is down, trying to connect or startup the database fails with:
ERROR:
ORA-09925: Unable to create audit trail file
SVR4 Error: 13: Permission denied
Additional information: 9925
CAUSE
On Unix systems, the investigation should be done using the OS tracing utility, ie: truss (AIX, Solaris), strace (Linux) or tusc (HP-UX).
truss -aefo sqlplus.trc sqlplus "/ as sysdba"
shows that the problem is caused by a missing directory, insufficient permissions or not enough space on that particular directory:
open("/opt/oracle/admin/test102/adump/ora_31704.aud", O_RDWR|O_CREAT|O_APPEND|O_LARGEFILE, 0660) = -1 EACCES (Permission denied)
From 10gR2, the ORACLEBASE/admin/<SID>/adumpdirectoryisthenewdefaultdirectoryforauditfiles(insteadofpreviousreleaseslocation,ORACLEBASE/admin/<SID>/adumpdirectoryisthenewdefaultdirectoryforauditfiles(insteadofpreviousreleaseslocation,ORACLE_HOME/rdbms/audit).
SOLUTION
1. Starting with 10gR2, check if the
ORACLEBASE/admin/ORACLEBASE/admin/ORACLE_SID/adump
directory exists and create it if not.
If ORACLE_BASE is not set, the default audit directory is still $ORACLE_HOME/rdbms/audit, as in pre 10gR2 releases.
2. If the directory exists (if in previous releases or if ORACLE_BASE not set, check: $ORACLE_HOME/rdbms/audit), make sure that permissions on it are set to 775.
3. If the directory exists and permissions are correctly set, make sure that there is enough space on the file system to accommodate the newly created audit files.
Starting with 10gR2: Note that when the database instance is down and the first connect and the startup command is run, the audit_file_dest parameter is not yet initialized, hence unknown to the shadow process, as such the audit files for these commands can only be written to the default destination. Writing these files is a mandatory requirement for NCSC C2 security evaluation criteria and therefore this behavior cannot be turned off.
4. Generically, ORACLE_BASE is set, as recommended by the documentation. If ORACLE_BASE is not set, then the default audit location becomes: $ORACLE_HOME/rdbms/audit. This is important to mention especially on RAC, where the environment variables are set in the OCR and should be updated, as documented, using srvctl, eg:
srvctl setenv database -d <db name> -t ORACLE_BASE=<Oracle Base directory>
5. A rare scenario was found when ORACLE_PATH is set and the Oracle software transparently appended it to the default value: ORACLEBASE/admin/ORACLEBASE/admin/ORACLE_SID/adump. ORACLE_PATH needs to be unset to workaround the problem. This issue was found to be a regression in the fix to bug 9438890 , and can be fixed by installing patch 14488943.
---ORA-09925 : Error During Database Creation Using Other OS Users. (文档 ID 405885.1)
In this Document
Symptoms
Cause
Solution
This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) process and therefore has not been subject to an independent technical review.
APPLIES TO:
Oracle Server - Enterprise Edition - Version 9.2.0.1 and later
Linux x86
HP-UX PA-RISC (64-bit)
IBM AIX on POWER Systems (64-bit)
Oracle Solaris on SPARC (64-bit)
Linux x86-64
***Checked for relevance on 19-Jun-2012***
SYMPTOMS
Creating database using DBCA logged in as another user belonging to the installation group results in following error
ORA-09925 : Unable to create audit trail file.
If you ignore this error it will error out with following message
ORA-01034 : Oracle not available.
CAUSE
By default Oracle will create directories with permission 755. So the other users belonging to the group will not have write permission which results in this issue.
SOLUTION
Any user other than the software owner belonging to the same group will not be able to create database.
To resolve the issue give write permission to the group on following directories.
1. chmod 775 $ORACLE_HOME/audit
2. chmod 775 $ORACLE_HOME/cfgtoollogs/dbca
3. Give write permission on directories where data files and control files are getting created.
4. Give write permission on $ORACLE_BASE
5. chgrp <oracle group> /etc/oratab
Note: This holds good for other Unix platforms as well.
---OS AUDIT ERROR IN ASM AND ORA-09925 in RDBMS (文档 ID 1921650.1)
In this Document
Symptoms
Cause
Solution
References
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.3 to 11.2.0.4 [Release 11.2]
Information in this document applies to any platform.
SYMPTOMS
1] The following error message is reported in ASM alert.log
OS Audit file could not be created; failing after x retries
2] The following similar message is visible in RDBMS alert.log
Additional information: 9925
ORA-01122: database file 5 failed verification check
ORA-01110: data file 5: '+DATA/asmdb/datafile/rcts.343.849109407'
ORA-01565: error in identifying file '+DATA/asmdb/datafile/rcts.343.849109407'
ORA-17503: ksfdopn:2 Failed to open file +DATA/asmdb/datafile/rcts.343.849109407
ORA-15055: unable to connect to ASM instance
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
NOTE: deferred map free for map id 180
WARNING: ASM communication error: op 18 state 0x40 (9925)
ERROR: slave communication error with ASM
NOTE: Deferred communication with ASM instance
Errors in file /u01/app/oracle/diag/rdbms/asmdb/ASMDB/trace/ASMDB_pmon_3692.trc:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
WARNING: ASM communication error: op 0 state 0x0 (15055)
ERROR: direct connection failure with ASM
3] Logging in ASM instance produces the following error.
$ sqlplus / as sysasm
ERROR:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-01075: you are currently logged on
4] Filesystem for GRID home shows enough space
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/host_root
9.7G 1.8G 7.5G 19% /
/dev/mapper/host_var
4.9G 923M 3.7G 20% /var
/dev/mapper/host_usr
4.9G 3.4G 1.3G 74% /usr
/dev/mapper/host_crs_oracle
54G 23G 28G 46% /crs/oracle <<<<<<<<<------------ GRID HOME filesystem
CAUSE
Permission on GRID_HOME/rdbms/audit was incorrect.
ls−lls−lGRID_HOME/rdbms
drwxr-xr-x. 2 grid oinstall 40960 Jul 31 15:01 admin
dr-xr-xr-x. 2 grid oinstall 12288 Aug 26 08:13 audit <<<<<<------ Here, there is no write privilege by grid OS user.
drwxr-xr-x. 2 grid oinstall 4096 Jul 31 14:59 demo
drwxr-xr-x. 2 grid oinstall 4096 Jul 31 14:59 doc
drwxr-xr-x. 4 grid oinstall 4096 Jul 31 15:01 install
drwxr-xr-x. 2 grid oinstall 4096 Jul 31 14:57 jlib
drwxr-xr-x. 2 grid oinstall 4096 Jul 31 15:01 lib
drwxr-xr-x. 2 grid oinstall 4096 Aug 24 08:05 log
drwxr-xr-x. 2 grid oinstall 4096 Jul 31 14:57 mesg
drwxr-xr-x. 2 grid oinstall 4096 Jul 31 14:57 public
drwxr-xr-x. 5 grid oinstall 4096 Jul 31 14:56 xml
SOLUTION
1] Change the permission so that grid OS user can write on $GRID_HOME/rdbms/audit
As grid OS user:
chmod+wchmod+wGRID_HOME/rdbms/audit
2] Check whether enough free space exists on GRID_HOME filesystem
SQL> show parameter dest
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
audit_file_dest string /oracle/app/11.2.0/grid/rdbms/
audit
background_dump_dest string /oracle/app/grid/diag/asm/+asm
/+ASM2/trace
core_dump_dest string /oracle/app/grid/diag/asm/+asm
/+ASM2/cdump
diagnostic_dest string /oracle/app/grid
user_dump_dest string /oracle/app/grid/diag/asm/+asm
/+ASM2/trace
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[ZFLHRDB2:grid]:/oracle/app/grid>cd /oracle/app/11.2.0/grid/rdbms/audit
[ZFLHRDB2:grid]:/oracle/app/11.2.0/grid/rdbms/audit>l
[ZFLHRDB2:grid]:/oracle/app/11.2.0/grid/rdbms/audit>ls | wc -l
42828
[ZFLHRDB2:grid]:/oracle/app/11.2.0/grid/rdbms/audit>rm -rf *
ksh: /usr/bin/rm: 0403-027 The parameter list is too long.
[ZFLHRDB2:grid]:/oracle/app/11.2.0/grid/rdbms/audit>find . -name "*.aud" | xargs rm
[ZFLHRDB2:grid]:/oracle/app/11.2.0/grid/rdbms/audit>ls
[ZFLHRDB2:root]:/oracle/app/11.2.0/grid/rdbms>chmod 775 audit
[ZFLHRDB2:root]:/oracle/app/11.2.0/grid/rdbms>l
total 176
drwxr-xr-x 2 grid dba 53248 Dec 23 2014 admin
drwxrwxr-x 2 grid dba 256 Jun 01 09:47 audit
drwxr-xr-x 2 grid dba 256 Nov 05 2014 demo
drwxr-xr-x 2 grid dba 256 Nov 05 2014 doc
drwxr-xr-x 4 grid dba 4096 Nov 05 2014 install
drwxr-xr-x 2 grid dba 4096 Nov 05 2014 jlib
drwxr-xr-x 2 grid dba 8192 Dec 23 2014 lib
drwxr-xr-x 2 grid dba 4096 May 31 19:02 log
drwxr-xr-x 2 grid dba 8192 Nov 05 2014 mesg
drwxr-xr-x 2 grid dba 256 Nov 05 2014 public
drwxr-xr-x 5 grid dba 4096 Nov 05 2014 xml
[ZFLHRDB2:root]:/oracle/app/11.2.0/grid/rdbms>
一.4.3 ipcs命令扩展
MOS上的几篇文章：
其中68281.1中提到DETERMINING WHICH INSTANCE OWNS WHICH SHARED MEMORY SEGMENTS & SEMAPHORE SETS，即若是一个主机上有多个oracle实例的话该如何确定哪个共享内存段属于我们该清掉的oracle实例的内存段，下边给个小实验即可，相关的MOS文档可以去小麦苗的云盘下载。
[ZFXDESKDB2:oracle]:/oracle>ps -ef|grep ora_pmon_
oracle 12255344 21626964 0 17:43:01 pts/0 0:00 grep ora_pmon_
oracle 17629238 1 0 18:57:42 - 0:09 ora_pmon_raclhr2
oracle 20250806 1 0 18:57:42 - 0:10 ora_pmon_oraESKDB2
[ZFXDESKDB2:oracle]:/oracle>which sysresv
/oracle/app/oracle/product/11.2.0/db/bin/sysresv
[ZFXDESKDB2:oracle]:/oracle>ORACLE_SID=raclhr2
[ZFXDESKDB2:oracle]:/oracle>sysresv
IPC Resources for ORACLE_SID "raclhr2" :
Shared Memory:
ID KEY
5242886 0xffffffff
5242883 0xffffffff
1048583 0xd92489e0
Oracle Instance alive for sid "raclhr2"
[ZFXDESKDB2:oracle]:/oracle>ipcs
IPC status from /dev/mem as of Wed Jun 1 17:43:47 BEIST 2016
T ID KEY MODE OWNER GROUP
Message Queues:
q 0 0x9283a0d2 -Rrw------- root system
q 1 0xffffffff ----------- root system
Shared Memory:
m 1048576 00000000 --rw-r----- grid dba
m 1048577 00000000 --rw-r----- grid dba
m 1048578 0x210000aa --rw-rw---- root system
m 5242883 00000000 --rw-r----- oracle asmadmin
m 1048580 00000000 --rw-r----- oracle asmadmin
m 1048581 00000000 --rw-r----- oracle asmadmin
m 5242886 00000000 --rw-r----- oracle asmadmin
m 1048583 0xd92489e0 --rw-r----- oracle asmadmin
m 1048584 0xd1a4a5d8 --rw-r----- grid dba
m 8388617 0x3f516768 --rw-r----- oracle asmadmin
m 759169034 0x21000148 --rw-rw---- oracle dba
Semaphores:
s 3145728 0x0100324a --ra-ra-r-- root system
s 1 0x620025b4 --ra-r--r-- root system
s 2 0x02001958 --ra-ra-ra- root system
s 3 0x01001958 --ra-ra-ra- root system
s 9 0x010024be --ra------- root system
s 1048590 0x410000a8 --ra-ra---- root system
s 11534361 0x41000147 --ra-ra---- oracle dba
[ZFXDESKDB2:oracle]:/oracle>ipcs -m
IPC status from /dev/mem as of Wed Jun 1 17:43:56 BEIST 2016
T ID KEY MODE OWNER GROUP
Shared Memory:
m 1048576 00000000 --rw-r----- grid dba
m 1048577 00000000 --rw-r----- grid dba
m 1048578 0x210000aa --rw-rw---- root system
m 5242883 00000000 --rw-r----- oracle asmadmin
m 1048580 00000000 --rw-r----- oracle asmadmin
m 1048581 00000000 --rw-r----- oracle asmadmin
m 5242886 00000000 --rw-r----- oracle asmadmin
m 1048583 0xd92489e0 --rw-r----- oracle asmadmin
m 1048584 0xd1a4a5d8 --rw-r----- grid dba
m 8388617 0x3f516768 --rw-r----- oracle asmadmin
m 759169034 0x21000148 --rw-rw---- oracle dba
[ZFXDESKDB2:oracle]:/oracle>ipcrm -m 5242886
[ZFXDESKDB2:oracle]:/oracle>ipcrm -m 5242883
[ZFXDESKDB2:oracle]:/oracle>ipcrm -m 1048583
[ZFXDESKDB2:oracle]:/oracle>sysresv
IPC Resources for ORACLE_SID "raclhr2" :
Shared Memory
ID KEY
No shared memory segments used
Oracle Instance not alive for sid "raclhr2"
Oracle Instance not alive for sid "raclhr2"
[ZFXDESKDB2:oracle]:/oracle>ps -ef|grep ora_pmon_
oracle 17629238 1 0 18:57:42 - 0:09 ora_pmon_raclhr2
oracle 20250806 1 0 18:57:42 - 0:10 ora_pmon_oraESKDB2
oracle 23330844 21626964 0 17:44:46 pts/0 0:00 grep ora_pmon_
[ZFXDESKDB2:oracle]:/oracle>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Wed Jun 1 17:44:52 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SYS@raclhr2> shutdown abort
ORACLE instance shut down.
SYS@raclhr2> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
一.5 故障处理总结
① ipcs和ipcrm应该学会使用
② sysresv的使用可以查看对应实例的共享内存段
③ 文章中的所有MOS文章在小麦苗的云盘中http://yunpan.cn/cdEQedhCs2kFz （提取码：ed9b）
④ 关于ipcs的使用cuug的老师也讲过公开课，公开课的录像文件参考http://yunpan.cn/cdEQedhCs2kFz （提取码：ed9b）
About Me