Oracle数据库非RAC环境出现ORA-600 [2103] 故障的诊断与分析

最近一套数据库出现故障，产生大量的ORA-00600:[2103]报错：CONTROL FILE ENQUEUE等待超时，相关日志如下：

Wed Feb 19 18:57:08 2013Errors in file /oracle/app/oracle/admin/oratest/bdump/oratest_arc1_25671.trc:ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []Thu Feb 20 07:37:23 2013Errors in file /oracle/app/oracle/admin/oratest/udump/oratest_ora_4341.trc:ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []Thu Feb 20 08:03:13 2013Errors in file /oracle/app/oracle/admin/oratest/udump/oratest_ora_5164.trc:ORA-00600: internal error code, arguments: [2103], [1], [0], [1], [900], [], [], []

关于ORA00-00600[]报错的trc内容如下：/oracle/app/oracle/admin/oratest/udump/oratest_ora_4341.trcOracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit ProductionWith the Partitioning, OLAP and Oracle Data Mining optionsJServer Release 9.2.0.7.0 - ProductionORACLE_HOME = /oracle/app/oracle/product/9.2.0System name: HP-UXNode name: oradb1Instance name: oratestRedo thread mounted by this instance: 1Oracle process number: 516Unix process pid: 4341, image: oracle@oradb1 (TNS V1-V3)*** 2013-08-20 07:37:13.333*** SESSION ID:(597.41542) 2013-08-20 07:37:13.305TIMEOUT ON CONTROL FILE ENQUEUEmode=S, type=0, wait=1, eqt=900

根据oracle metalink官方文档[color=navy][Diagnosing ORA-600 [2103] Issues in a NON RAC environment (文档 ID 406191.1)]对这个报错的介绍，这个报错的意思是说：CONTROL FILE ENQUEUE等待超时。
官方文档对这个报错的解释如下：This document explains how to troubleshoot and find out the cause of ORA-600 [2103] in a non RAC environment.An ORA-600 [2103] is signaled when an Oracle process cannot get a CF(Control File) enqueue for 900 seconds.The ORA-600 [2103] could be caused by:•Very slow I/O subsystem where the Control files are stored.（•Frequent log switching •redo logs too small or insufficient logs•Async I/O issue or multiple db_writers, you can't use both of them, back one out•OS / Hardware issues The error in this case points to slow I/O as the CKPT process is blocked on a read of the control file.
The I/O is simply too slow to complete the CKPT process that is holding the CF enqueue.
You can check the OS System log file to see if there are I/O or Async I/O issues reported.
根据官方的介绍，这个报错的出现说明了缓慢的I/O 导致Oracle CKPT进程被阻塞读取控制文件,I/O完成CKPT过程太缓慢,CF排队,以至系统会出现大量的等待。触发这个报错的原因有：1、存储数据库的控制文件的IO系统比较慢2、日志切换频繁,重做日志太小或不足的日志3、异步I/O问题4、OS/硬件问题