ORA-16401: archivelog rejected by RFS
无线出单系统邮件告警10.111.20.1
1. 报错
SYS > ! oerr ora 16041
16041, 00000, "Remote File Server fatal error"
// *Cause: The Remote File Server (RFS) process at a redo transport
// destination encountered a fatal error and was no longer
// available. Further redo transmission to this destination may not be
// possible.
// *Action: Refer to the appropriate RFS trace file located at the redo
// transport destination for details regarding the error encountered
// and take corrective action.
远程传输错误，RFS服务器，与参数fal_server,fal_client参数有关
2. 搜索百度
通常此问题是由于主库的fal_client参数与备库的log_archive_dest_n中的service参数不匹配造成的。
3. 查询参数
Primary
--查询DG成员
SQL> show parameter config
log_archive_config rac,dgwless,wxcdrac
--减少自己本身，可能存在两个DG成员
--查询主库db_unique_name
SQL> show parameter db_unique_name
db_unique_name wxcdrac
--查询主库归档参数
SQL> show parameter archive
NAME VALUE
---------------------------------------------------
log_archive_dest_2 service=dgwless LGWR ASYNC va
lid_for=(online_logfiles,prima
ry_role) db_unique_name=dgwless
log_archive_dest_3 SERVICE=RAC LGWR ASYNC VALID_F
OR=(ONLINE_LOGFILES,PRIMARY_RO
LE) db_unique_name=rac
--查询归档线程参数状态，以上归档线程3对应的参数可以过滤，因为此DG可能已被废弃
--除非是人为或者误操作置为defer，否则基本判断问题，排除此DG
log_archive_dest_state_2 ENABLE
log_archive_dest_state_3 DEFER
--查询主库fal参数
SQL> show parameter fal
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
fal_client string wxcdrac2
fal_server string dgwless
--通过主库参数，查询备库IP地址
SQL> host tnsping dgwless
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.135.101.23)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dgwless)))
OK (90 msec)
--连接备库后，查询验证主备关系
SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON from v$database;
DBID NAME DATABASE_ROL OPEN_MODE FLASHBACK_ON
----------- --------------- ------------ ---------- ------------------
2283108100 RAC PRIMARY READ WRITE NO
SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON from v$database;
DBID NAME DATABASE_ROL OPEN_MODE FLASHBACK_ON
----------- --------------- ------------ ---------- ------------------
2283108100 RAC PHYSICAL STANDBY MOUNTED NO
--standby
SQL> show parameter fal
NAME VALUE
------------- ------------------------------
fal_client dgwless
fal_server wxcdrac --发现与主库fal_client参数不同
--查询备库归档参数
log_archive_dest_1 location=/picclife/app/oracle/archdg/
log_archive_dest_2 SERVICE=WXCDRAC LGWR ASYNC VAL --发现tns也与主库fal_client不同
ID_FOR=(ONLINE_LOGFILES,PRIMAR
Y_ROLE) db_unique_name=wxcdrac
--主库fal_client 在11g已被弃用，按照道理将主库fal_client与备库log_archvie_service不同，无异常，查询版本，
当前10G
Enterprise Edition Release 10.2.0.3.0
--查询主库集群状态
wxcdrac2:/home/oracle$ export ORACLE_SID=+ASM1
wxcdrac2:/home/oracle$ cat .profile
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1
wxcdrac2:/home/oracle$ cd $ORA_CRS_HOME/bin
wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$
wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.wxcdrac.db application ONLINE ONLINE wxcdrac1
ora....c1.inst application ONLINE ONLINE wxcdrac1
ora....c2.inst application ONLINE ONLINE wxcdrac2
ora....SM1.asm application ONLINE ONLINE wxcdrac1
ora....C1.lsnr application ONLINE ONLINE wxcdrac1
ora....ac1.gsd application ONLINE ONLINE wxcdrac1
ora....ac1.ons application ONLINE ONLINE wxcdrac1
ora....ac1.vip application ONLINE ONLINE wxcdrac1
ora....SM2.asm application ONLINE ONLINE wxcdrac2
ora....C2.lsnr application ONLINE ONLINE wxcdrac2
ora....ac2.gsd application ONLINE ONLINE wxcdrac2
ora....ac2.ons application ONLINE ONLINE wxcdrac2
ora....ac2.vip application ONLINE ONLINE wxcdrac2
4. 查询日志
--备库日志
Wed Aug 8 01:59:19 2018
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[2586]: Assigned to RFS process 29818962
RFS[2586]: Identified database type as 'physical standby'
Primary database is in MAXIMUM PERFORMANCE mode
Primary database is in MAXIMUM PERFORMANCE mode
RFS[2586]: Successfully opened standby log 15: '/picclife/app/oracle/oradata/dgwless/std02_1.log'
Wed Aug 8 02:04:42 2018
RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused
Wed Aug 8 02:04:42 2018
Errors in file /picclife/app/oracle/admin/dgwless/udump/dgwless_rfs_13238468.trc:
ORA-16401: archivelog rejected by RFS
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
--查询trace文件
KCRRSNPS
Identified standby redo log 12 for implicit mid-log reconnect
*** 2018-08-08 02:04:42.324
*** 2018-08-08 02:04:42.324 61287 kcrr.c
RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused
This problem can occur as the result of multiple primary and/or
standby database(s) trying to archive to this standby database.
This is not allowed, since successfully completed archivelogs
would be overwritten.
ORA-16401: archivelog rejected by RFS
*** 2018-08-08 02:13:43.562
Dispatching RFS shutdown notification
以上备库查询得到的信息是，归档接受出现网络问题，但不明确
查询主库Alert日志
--查询邮件告警所在的数据库Alert日志
SQL> show parameter background_dump_dest
NAME VALUE
--------------------- ------------------------------
background_dump_dest /picclife/app/oracle/admin/wxcdrac/bdump
Wed Aug 8 01:02:57 2018
LNS: Standby redo logfile selected for thread 1 sequence 3993 for destination LOG_ARCHIVE_DEST_2
Wed Aug 8 01:03:32 2018
Starting control autobackup
Control autobackup written to SBT_TAPE device
comment 'API Version 2.0,MMS Version 10.0.0.116',
media 'V_89405_737884'
handle 'c-2283108100-20180808-01'
Wed Aug 8 01:08:19 2018
Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc2_10682458.trc:
ORA-16401: archivelog rejected by RFS
Wed Aug 8 01:11:58 2018
ARC0: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3135)
ARC0: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Wed Aug 8 01:11:58 2018
Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:
ORA-03135: connection lost contact
FAL[server, ARC0]: Error 3135 creating remote archivelog file 'dgwless'
FAL[server, ARC0]: FAL archive failed, see trace file.
Wed Aug 8 01:11:58 2018
Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Wed Aug 8 01:11:58 2018
ORACLE Instance wxcdrac1 - Archival Error. Archiver continuing.
*** 2018-08-08 01:08:19.693
Error 16401 creating standby archive log file at host 'dgwless'
ORA-16401: archivelog rejected by RFS
--从以上的信息还是未明确发现由于参数问题导致的报错
5. 搜索MOS
1183143.1
ORA-16401: ARCHIVELOG REJECTED BY RFS error occurs when incorrect naming convention for fal_client parameter is set.
Version 10.2.0.4 to 11.1.0.7 [Release 10.2 to 11.1]
配置主备不一致的fal_client时会发生此错误
***** Primary *****
log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)
log_archive_dest_1 location=/u99/ORACLE/p1ncm1/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm1
log_archive_dest_2 service=p1ncm2_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm2
fal_client p1ncm1_fc.csm.fub.com
fal_server p1ncm2_fs.csm.fub.com
********* Standby ********
log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)
log_archive_dest_1 location=/u99/ORACLE/p1ncm2/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm2
log_archive_dest_2 service=p1ncm1_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm1
fal_client p1ncm2_fc.csm.fub.com
fal_server p1ncm1_fs.csm.fub.com
SOLUTION
standby
alter system set fal_client=p1ncm2_dgfal.csm.fub.com scope=both sid='*';
6. 处理
-修改主库的fal_client参数，与备库保持一致
--修改前需要进行验证
SQL> host tnsping wxcdrac2
SQL> host tnsping wxcdrac
SQL> alter system set fal_client=wxcdrac;
--第二次，在其它客户遇到同样报错，问题是tnsping 主库fal_client不通，端口错误。
7. 补充说明
MOS ID 288074.1
Versions 11.2, 12.1 对于11g12c而言，很可能此消息是一个提示作用
Error: ORA-16401 archive log rejected by Remote File Server (RFS)
---------------------------------------------------------------------------
Cause: An attempt was made to re-archive an existing archive log.
Action: See alert log and trace file for more details. No action is necessary.
This is an informational statement provided to record the event for
diagnostic purposes.
Versions 9.2, 10.1, 10.2, 11.1
Error: ORA-16401 archivelog rejected by RFS 对于10g而言，此报错，很可能为fal_client参数配置错误
---------------------------------------------------------------------------
Cause: An attempt was made to re-archive an existing archivelog. This usually
happens because either a multiple primary database or standby
database(s) or both are trying to archive to this standby database.
Action: See alert log and trace file for more details. No action is necessary;
this is an informational statement provided to record the event for
diagnostic purposes.