本帖最后由 paulyi 于 2014-2-12 11:00 编辑

Oracle Root.sh failed at Failure at final check of Oracle CRS stack 10问题

今天在公司和同事一起测试安装oracle 10g rac for aix6.1+asm安装，在装完
crs集群软件，在第一节点运行root.sh时最后报以下错误:
failed at Failure at final check of Oracle CRS stack
10
查看$ORA_CRS_HOME/log/p520/client下的相关日志报下面错误
[root@p520:/crs/app/oracle/product/crs_1/log/p520/client]#more clsc.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved.
2010-04-16 16:55:05.812: [ COMMCRS][1]clsc_connect: (11068c330) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
2010-04-16 16:55:05.813: [ COMMCRS][1]clsc_connect: (11068b6b0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=SYSTEM.evm.acceptor.auth)
)
2010-04-16 16:55:05.813: [ default][1]Terminating clsd session
2010-04-16 18:41:53.148: [ COMMCRS][1]clsc_connect: (1101bf070) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p520_crs))
2010-04-16 18:41:53.149: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2010-04-16 18:41:53.149: [ default][1]Terminating clsd session
2010-04-16 17:42:17.881: [ COMMCRS][1]clsc_connect: (1101bf070) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p520_crs))
2010-04-16 17:42:17.882: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2010-04-16 17:42:17.882: [ default][1]Terminating clsd session
2010-04-16 17:45:21.416: [ COMMCRS][1]clsc_connect: (1101bf070) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_p520_crs))
2010-04-16 17:45:21.519: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9

crsctl check boot无报错
/etc/init.cssd 无报错
crs进程起不来
没办法，只能查metalink,查到有可能是aix gui 工具没配置，建议在图形窗口运行 install_assist 命令，进行修改时间的配置，配置完成后重新启动操作系统，启动crs,发现有evmd进程没起来。

[oracle@p520:/oracle/app/oracle]$ps -ef|grep crs
root 229546 1 0 17:45:37 - 0:01 /crs/app/oracle/product/crs_1/bin/crsd.bin reboot
oracle 327874 569510 0 19:00:57 pts/1 0:00 grep crs
oracle 389122 360648 0 17:45:41 - 0:00 /bin/sh -c ulimit -c unlimited; cd /crs/app/oracle/product/crs_1/log/p520/cssd; /crs/app/oracle/product/crs_1/bin/ocssd || exit $?
root 409806 1 0 17:46:06 - 0:00 /bin/sh /etc/init.crsd run
oracle 454904 389122 0 17:45:41 - 0:00 /crs/app/oracle/product/crs_1/bin/ocssd.bin
root 524338 418004 0 17:45:40 - 0:00 /crs/app/oracle/product/crs_1/bin/oprocd run -t 1000 -m 500
运行init.crs start 命令 evmd进程正常启动

[root@p520:/etc]#init.crs start
[root@p520:/etc]#init.crs start
Startup will be queued to init within 30 seconds.
[root@p520:/etc]#ps -ef|grep crs
root 229546 1 0 17:45:37 - 0:01 /crs/app/oracle/product/crs_1/bin/crsd.bin reboot
oracle 389122 360648 0 17:45:41 - 0:00 /bin/sh -c ulimit -c unlimited; cd /crs/app/oracle/product/crs_1/log/p520/cssd; /crs/app/oracle/product/crs_1/bin/ocssd || exit $?
root 409806 1 0 17:46:06 - 0:00 /bin/sh /etc/init.crsd run
oracle 454904 389122 0 17:45:41 - 0:00 /crs/app/oracle/product/crs_1/bin/ocssd.bin
root 524338 418004 0 17:45:40 - 0:00 /crs/app/oracle/product/crs_1/bin/oprocd run -t 1000 -m 500
root 569568 421918 0 19:02:19 pts/1 0:00 grep crs
[root@p520:/etc]#ps -ef|grep crs
oracle 208974 323776 0 19:02:22 - 0:00 /crs/app/oracle/product/crs_1/bin/evmlogger.bin -o /crs/app/oracle/product/crs_1/evm/log/evmlogger.info -l /crs/app/oracle/product/crs_1/evm/log/evmlogger.log
root 229546 1 1 17:45:37 - 0:01 /crs/app/oracle/product/crs_1/bin/crsd.bin reboot
oracle 323776 1 0 17:45:53 - 0:00 /crs/app/oracle/product/crs_1/bin/evmd.bin
oracle 389122 360648 0 17:45:41 - 0:00 /bin/sh -c ulimit -c unlimited; cd /crs/app/oracle/product/crs_1/log/p520/cssd; /crs/app/oracle/product/crs_1/bin/ocssd || exit $?
root 409806 1 0 17:46:06 - 0:00 /bin/sh /etc/init.crsd run
oracle 454904 389122 0 17:45:41 - 0:00 /crs/app/oracle/product/crs_1/bin/ocssd.bin
root 524338 418004 0 17:45:40 - 0:00 /crs/app/oracle/product/crs_1/bin/oprocd run -t 1000 -m 500
root 540718 421918 0 19:02:27 pts/1 0:00 grep crs

查看crs状态，都online,正常
[oracle@p520:/oracle/app/oracle]$crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.p520.gsd application ONLINE ONLINE p520
ora.p520.ons application ONLINE ONLINE p520
ora.p520.vip application ONLINE ONLINE p520
ora.p650.gsd application ONLINE ONLINE p650
ora.p650.ons application ONLINE ONLINE p650
ora.p650.vip application ONLINE ONLINE p650

以下为metalink的详细解释
In this Document
[size=-1] https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=725878.1#SYMPTOM]Symptoms
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=725878.1#CHANGE]Changes
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=725878.1#CAUSE]Cause
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=725878.1#FIX]Solution
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=PROBLEM&id=725878.1#REF]References

Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.1
This problem can occur on any platform.

Symptoms
2 node RAC, performing CRS 10.2.0.1 installation, failure at root.sh, CRS stack not started

WARNING: directory '/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: ndb1 ndb1-rac ndb1
node 2: ndb2 ndb2-rac ndb2
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
Now formatting voting device: /dev/rhdisk3
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Failure at final check of Oracle CRS stack.
10

Checking "ps -ef | grep init" during this 600 seconds period, see there is no crs related init processes, eg: init.crsd/init.evmd/init.cssd running. When no processes are detected, then this case apply.
Changes
New installation

Cause
This particular case is caused by the OS init system does not working.

" Failure at final check of Oracle CRS stack.
10"
means CRS daemon did not startup during 600 seconds period.

In root.sh script, it will add CRS related entry in /etc/inittab, run "init q" and expect those 3 CRS related daemon processes to start. With init system problem, none of these daemon processes spawned, this caused CRS process startup failure as they rely on init daemon process to start first.

This can be verified by adding a simple entry in /etc/inittab:
test:2:once:/usr/bin/echo "HELLO TEST" > /tmp/test.log
run "init q" as root user. If the init is working, then there should be a file /tmp/test.log generated.

Solution
Please consult with system administrator for init issue.
e.g. here the solution reference only valid for AIX platform.:
1. Starting the script. install_assist (AIX GUI utility Installation Assistance)
2. Updating for example the date, then exit install_assist properly
3. Reboot the system
After that daemon process in /etc/inittab started, CRS installation completed.

For other platforms, please consult your system admin or vendor for its solution.