本帖最后由 iverycd 于 2018-9-21 14:05 编辑

[size=31px]ORA-03137 TTC协议内部错误问题排查
[size=31px]一、故障背景
某项目页面查询模块打开报错，通信通道的文件结束，生产业务受到影响。[color=#001000]
[size=31px]二、排查过程
1、查看系统其它模块打开正常，受影响的只有1个功能模块，随即查看oracle告警日志，发现其中有大量报错如下：
[font=monospace][size=13px]Wed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]28[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(0, 119, 170)][font=monospace][size=13px]Errors[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]file[font=monospace][size=13px] e:\app\administrator\diag\rdbms\gqdb\gqdb\trace\gqdb_ora_28448[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trc [color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]incident[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]16177[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px]:ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]03137[font=monospace][size=13px]: TTC 协议内部错误: [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]12333[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]64[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]0[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]98[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px]Incident details [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px]: e:\app\administrator\diag\rdbms\gqdb\gqdb\incident\incdir_16177\gqdb_ora_28448_i16177[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trcWed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]28[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]08[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[font=monospace][size=13px]Trace dumping [color=rgb(154, 110, 58)][font=monospace][size=13px]is[font=monospace][size=13px] performing id[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 153, 153)][font=monospace][size=13px][[font=monospace][size=13px]cdmp_20180829072808[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px]Wed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]28[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]38[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(0, 119, 170)][font=monospace][size=13px]Errors[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]file[font=monospace][size=13px] e:\app\administrator\diag\rdbms\gqdb\gqdb\trace\gqdb_ora_45000[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trc [color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]incident[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]16089[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px]:ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]03137[font=monospace][size=13px]: TTC 协议内部错误: [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]12333[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]64[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]0[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]98[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px]Incident details [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px]: e:\app\administrator\diag\rdbms\gqdb\gqdb\incident\incdir_16089\gqdb_ora_45000_i16089[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trcWed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]28[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]41[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[font=monospace][size=13px]Trace dumping [color=rgb(154, 110, 58)][font=monospace][size=13px]is[font=monospace][size=13px] performing id[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 153, 153)][font=monospace][size=13px][[font=monospace][size=13px]cdmp_20180829072841[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px]Wed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]43[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[font=monospace][size=13px]Thread [color=rgb(153, 0, 85)][font=monospace][size=13px]1[font=monospace][size=13px] cannot allocate new log[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] sequence [color=rgb(153, 0, 85)][font=monospace][size=13px]21462[font=monospace][size=13px]Private strand flush [color=rgb(154, 110, 58)][font=monospace][size=13px]not[font=monospace][size=13px] complete [color=rgb(0, 119, 170)][font=monospace][size=13px]Current[font=monospace][size=13px] log[color=rgb(112, 128, 144)][font=monospace][size=13px]# 2 seq# 21461 mem# 0: E:\APP\ADMINISTRATOR\ORADATA\GQDB\REDO02.LOG[font=monospace][size=13px]Thread [color=rgb(153, 0, 85)][font=monospace][size=13px]1[font=monospace][size=13px] advanced [color=rgb(0, 119, 170)][font=monospace][size=13px]to[font=monospace][size=13px] log sequence [color=rgb(153, 0, 85)][font=monospace][size=13px]21462[font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]LGWR switch[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]Current[font=monospace][size=13px] log[color=rgb(112, 128, 144)][font=monospace][size=13px]# 3 seq# 21462 mem# 0: E:\APP\ADMINISTRATOR\ORADATA\GQDB\REDO03.LOG[font=monospace][size=13px]Wed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]30[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]00[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(0, 119, 170)][font=monospace][size=13px]Errors[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]file[font=monospace][size=13px] e:\app\administrator\diag\rdbms\gqdb\gqdb\trace\gqdb_j000_40352[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trc:ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]12012[font=monospace][size=13px]: 自动执行作业 [color=rgb(153, 0, 85)][font=monospace][size=13px]12692[font=monospace][size=13px] 出错ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]06550[font=monospace][size=13px]: 第 [color=rgb(153, 0, 85)][font=monospace][size=13px]1[font=monospace][size=13px] 行[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] 第 [color=rgb(153, 0, 85)][font=monospace][size=13px]729[font=monospace][size=13px] 列: PLS[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]00905[font=monospace][size=13px]: 对象 GQZWFW[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]ASP_REFRESHLEFTTIME 无效ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]06550[font=monospace][size=13px]: 第 [color=rgb(153, 0, 85)][font=monospace][size=13px]1[font=monospace][size=13px] 行[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] 第 [color=rgb(153, 0, 85)][font=monospace][size=13px]729[font=monospace][size=13px] 列: PL[color=rgb(154, 110, 58)][font=monospace][size=13px]/[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[font=monospace][size=13px]: Statement ignoredWed Aug [color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]07[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]33[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]10[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(0, 119, 170)][font=monospace][size=13px]Errors[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]file[font=monospace][size=13px] e:\app\administrator\diag\rdbms\gqdb\gqdb\trace\gqdb_ora_36984[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trc [color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]incident[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]16178[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px]:ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]03137[font=monospace][size=13px]: TTC 协议内部错误: [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]12333[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]64[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]0[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]98[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px]Incident details [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px]: e:\app\administrator\diag\rdbms\gqdb\gqdb\incident\incdir_16178\gqdb_ora_36984_i16178[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trc
2、根据告警提供的trace文件，详细查看报错信息
[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.940[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]SESSION[font=monospace][size=13px] ID:[color=rgb(153, 153, 153)][font=monospace][size=13px]([color=rgb(153, 0, 85)][font=monospace][size=13px]69.59095[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.940[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] CLIENT ID:[color=rgb(153, 153, 153)][font=monospace][size=13px]()[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.940[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] SERVICE NAME:[color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]gqdb[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.940[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] MODULE NAME:[color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]w3wp[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]exe[color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.940[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]ACTION[font=monospace][size=13px] NAME:[color=rgb(153, 153, 153)][font=monospace][size=13px]()[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.940[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]Dump[font=monospace][size=13px] continued [color=rgb(0, 119, 170)][font=monospace][size=13px]from[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]file[font=monospace][size=13px]: e:\app\administrator\diag\rdbms\db\db\trace\db_ora_47708[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]trcORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]03137[font=monospace][size=13px]: TTC 协议内部错误: [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]12333[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]254[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]64[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]0[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][][color=rgb(154, 110, 58)][font=monospace][size=13px]=========[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]Dump[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]for[font=monospace][size=13px] incident [color=rgb(153, 0, 85)][font=monospace][size=13px]16305[font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]ORA [color=rgb(153, 0, 85)][font=monospace][size=13px]3137[font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]12333[color=rgb(153, 153, 153)][font=monospace][size=13px]])[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]========[color=rgb(154, 110, 58)][font=monospace][size=13px]***[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]2018[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]08[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]29[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]14[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]04[font=monospace][size=13px]:[color=rgb(153, 0, 85)][font=monospace][size=13px]34.946[font=monospace][size=13px]dbkedDefDump[color=rgb(153, 153, 153)][font=monospace][size=13px]()[font=monospace][size=13px]: [color=rgb(0, 119, 170)][font=monospace][size=13px]Starting[font=monospace][size=13px] incident [color=rgb(0, 119, 170)][font=monospace][size=13px]default[font=monospace][size=13px] dumps [color=rgb(153, 153, 153)][font=monospace][size=13px]([font=monospace][size=13px]flags[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]0x2[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]level[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]3[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] mask[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]0x0[color=rgb(153, 153, 153)][font=monospace][size=13px])[color=rgb(112, 128, 144)][font=monospace][size=13px]----- Current SQL Statement for this session (sql_id=gy425phpm9wq8) -----[color=rgb(0, 119, 170)][font=monospace][size=13px]SELECT[font=monospace][size=13px] AUDIT_PROJECT[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]RowGuid [color=rgb(0, 119, 170)][font=monospace][size=13px]from[font=monospace][size=13px] AUDIT_PROJECT[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px]AUDIT_TASK [color=rgb(0, 119, 170)][font=monospace][size=13px]where[font=monospace][size=13px] AUDIT_PROJECT[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]TASKGUID[color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px]AUDIT_TASK[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]RowGuid [color=rgb(154, 110, 58)][font=monospace][size=13px]and[font=monospace][size=13px] ITEM_ID [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px]([color=rgb(0, 119, 170)][font=monospace][size=13px]select[font=monospace][size=13px] AUDIT_TASK[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]ITEM_ID [color=rgb(0, 119, 170)][font=monospace][size=13px]from[font=monospace][size=13px] AUDIT_WINDOW_TASK[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px]AUDIT_WINDOW_USER[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px]AUDIT_TASK [color=rgb(0, 119, 170)][font=monospace][size=13px]where[font=monospace][size=13px] AUDIT_WINDOW_TASK[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]WINDOWGUID[color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px]AUDIT_WINDOW_USER[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]WINDOWGUID [color=rgb(154, 110, 58)][font=monospace][size=13px]and[font=monospace][size=13px] AUDIT_WINDOW_TASK[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]TASKGUID[color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px]AUDIT_TASK[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]RowGuid [color=rgb(154, 110, 58)][font=monospace][size=13px]and[font=monospace][size=13px] USERGUID[color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px]:UserGuid [color=rgb(153, 153, 153)][font=monospace][size=13px])[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]and[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]STATUS[color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px]:[color=rgb(0, 119, 170)][font=monospace][size=13px]status
可以看到引起报错的为一条SQL，正好涉及到故障页面查询。
[color=#001000]
三、查阅资料
根据报错代码，查阅MOS文档
[font=monospace][size=13px]Troubleshooting ORA-3137 [12333] Errors Encountered When Using Oracle JDBC Driver (文档 ID 1361107.1)
此报错信息来源于11.2.0.1其中一个bug
[font=monospace][size=13px]Unpublished Bug [color=rgb(153, 0, 85)][font=monospace][size=13px]9703463[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]-[font=monospace][size=13px] ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]3137[font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[color=rgb(153, 0, 85)][font=monospace][size=13px]12333[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]or[font=monospace][size=13px] ORA[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]600[font=monospace][size=13px] [color=rgb(153, 153, 153)][font=monospace][size=13px][[font=monospace][size=13px]kpobav[color=rgb(154, 110, 58)][font=monospace][size=13px]-[color=rgb(153, 0, 85)][font=monospace][size=13px]1[color=rgb(153, 153, 153)][font=monospace][size=13px]][font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]When[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]Using[font=monospace][size=13px] Bind PeekingThis bug affects versions [color=rgb(153, 0, 85)][font=monospace][size=13px]11.1.0.6[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]11.1.0.7[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]and[font=monospace][size=13px] [color=rgb(153, 0, 85)][font=monospace][size=13px]11.2.0.1[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]of[font=monospace][size=13px] the RDBMS[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px] It [color=rgb(154, 110, 58)][font=monospace][size=13px]is[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]fixed[font=monospace][size=13px] [color=rgb(154, 110, 58)][font=monospace][size=13px]in[font=monospace][size=13px] version [color=rgb(153, 0, 85)][font=monospace][size=13px]11.2.0.2[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]of[font=monospace][size=13px] the [color=rgb(0, 119, 170)][font=monospace][size=13px]database[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]It can also occur intermittently[color=rgb(153, 153, 153)][font=monospace][size=13px];[font=monospace][size=13px] similarly [color=rgb(0, 119, 170)][font=monospace][size=13px]to[font=monospace][size=13px] unpublished Bug:[color=rgb(153, 0, 85)][font=monospace][size=13px]8625762[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] this [color=rgb(154, 110, 58)][font=monospace][size=13px]is[font=monospace][size=13px] a bind peeking bug[color=rgb(153, 153, 153)][font=monospace][size=13px].
[font=monospace][color=#003000]
四、解决方案
1、禁用Bind Peeking
[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[color=rgb(154, 110, 58)][font=monospace][size=13px]>[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]alter[font=monospace][size=13px] system [color=rgb(0, 119, 170)][font=monospace][size=13px]set[font=monospace][size=13px] [color=rgb(102, 153, 0)][font=monospace][size=13px]"_optim_peek_user_binds"[color=rgb(154, 110, 58)][font=monospace][size=13px]=[color=rgb(153, 0, 85)][font=monospace][size=13px]false[color=rgb(153, 153, 153)][font=monospace][size=13px];[font=monospace][size=13px]此参数为oracle的隐含参数
2、升级数据库版本
此bug已在11.2.0.3以上版本修复，可升级此版本或更高
[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[color=rgb(154, 110, 58)][font=monospace][size=13px]>[font=monospace][size=13px] col ksppinm [color=rgb(0, 119, 170)][font=monospace][size=13px]for[font=monospace][size=13px] a20[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[color=rgb(154, 110, 58)][font=monospace][size=13px]>[font=monospace][size=13px] col ksppinm [color=rgb(0, 119, 170)][font=monospace][size=13px]for[font=monospace][size=13px] a30[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[color=rgb(154, 110, 58)][font=monospace][size=13px]>[font=monospace][size=13px] col ksppstvl [color=rgb(0, 119, 170)][font=monospace][size=13px]for[font=monospace][size=13px] a30[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[color=rgb(154, 110, 58)][font=monospace][size=13px]>[font=monospace][size=13px] col ksppdesc [color=rgb(0, 119, 170)][font=monospace][size=13px]for[font=monospace][size=13px] a30[color=rgb(0, 119, 170)][font=monospace][size=13px]SQL[color=rgb(154, 110, 58)][font=monospace][size=13px]>[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]SELECT[font=monospace][size=13px] ksppinm[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] ksppstvl[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] ksppdesc [color=rgb(0, 119, 170)][font=monospace][size=13px]FROM[font=monospace][size=13px] x$ksppi x[color=rgb(153, 153, 153)][font=monospace][size=13px],[font=monospace][size=13px] x$ksppcv y [color=rgb(0, 119, 170)][font=monospace][size=13px]WHERE[font=monospace][size=13px] x[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]indx [color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px] y[color=rgb(153, 153, 153)][font=monospace][size=13px].[font=monospace][size=13px]indx [color=rgb(154, 110, 58)][font=monospace][size=13px]AND[font=monospace][size=13px] ksppinm [color=rgb(154, 110, 58)][font=monospace][size=13px]=[font=monospace][size=13px] [color=rgb(102, 153, 0)][font=monospace][size=13px]'_optim_peek_user_binds'[color=rgb(153, 153, 153)][font=monospace][size=13px];[font=monospace][size=13px]KSPPINM KSPPSTVL KSPPDESC[color=rgb(112, 128, 144)][font=monospace][size=13px]------------------------------ ------------------------------ ------------------------------[font=monospace][size=13px]_optim_peek_user_binds [color=rgb(153, 0, 85)][font=monospace][size=13px]TRUE[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]enable[font=monospace][size=13px] peeking [color=rgb(0, 119, 170)][font=monospace][size=13px]of[font=monospace][size=13px] [color=rgb(0, 119, 170)][font=monospace][size=13px]user[font=monospace][size=13px] binds查看隐含参数，此参数为开启状态
最终选择了禁用隐含参数，关闭特性之后，业务系统模块已恢复，告警日志也不再出现报错信息