某通信公司数据库因存储故障导致数据库无法open的恢复记录 -m6米乐安卓版下载

原创肖杰 2022-05-12

1274

背景

早上接到客户反馈，因存储故障，导致cureent redo log损坏，数据库无法open，客户自行多次尝试resetlogs的方式open，但是open失败，请求协助，可接受少量数据丢失。

如下是客户反馈的他们自己的记录，通过查看历史日志，故障发生在凌晨，客户在凌晨2左右开始就一直在反复的通过resetlogs方式open数据库，但均open失败，直到早上上班才请求我们协助：

## 最早报错信息：
thu may 12 02:27:04 2022
abort recovery for domain 0
aborting crash recovery due to error 742
errors in file /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/trace/ycldb1_ora_14075.trc:
ora-00742: log read detects lost write in thread %d sequence %d block %d
ora-00312: online log 6 thread 2: ' datadg/ycldb/onlinelog/group_6.268.837702817'
abort recovery for domain 0
errors in file /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/trace/ycldb1_ora_14075.trc:
ora-00742: log read detects lost write in thread %d sequence %d block %d
ora-00312: online log 6 thread 2: ' datadg/ycldb/onlinelog/group_6.268.837702817'
ora-742 signalled during: alter database open /* db agent *//* {1:44787:239} */...
thu may 12 02:27:04 2022
shutting down instance (abort)
license high water mark = 33
user (ospid: 14199): terminating the instance
instance terminated by user, pid = 14199
thu may 12 02:27:05 2022
instance shutdown complete
## 因为redo丢失，客户从故障时间点开始知道早上上班时间点，反复重试了多次resetlogs，均未open成功，报错信息如下：
sql> alter system set "_allow_resetlogs_corruption"=true scope=spfile;
system altered.
sql> recover database until cancel;
ora-00279: change 17592046373009 generated at 05/11/2022 07:54:52 needed for thread 1
ora-00289: suggestion : /oracle/app/product/11.2.0/db_1/dbs/arch1_133072_837702706.dbf
ora-00280: change 17592046373009 for thread 1 is in sequence #133072
specify log: {=suggested | filename | auto | cancel}
cancel
ora-01547: warning: recover succeeded but open resetlogs would get error below
ora-01194: file 1 needs more recovery to be consistent
ora-01110: data file 1: ' datadg/ycldb/datafile/system.256.837702609'
ora-01112: media recovery not started
sql>     alter database open resetlogs;
    alter database open resetlogs
*
error at line 1:
ora-01092: oracle instance terminated. disconnection forced
ora-00704: bootstrap process failure
ora-00704: bootstrap process failure
ora-00600: internal error code, arguments: [kclchkblk_4], [4096], [163346281], [4095], [4155295895], [], [], [], [], [], [], []
process id: 20666
session id: 8905 serial number: 3
sql>shutdown immedaite;
sql>startup mount;
sql> alter database open;
alter database open
*
error at line 1:
ora-01092: oracle instance terminated. disconnection forced
ora-00704: bootstrap process failure
ora-00704: bootstrap process failure
ora-00600: internal error code, arguments: [kclchkblk_4], [4096], [163346281], [4095], [4155315901], [], [], [], [], [], [], []
process id: 23753
session id: 8905 serial number: 3

ora-600 [kclchkblk_4]错误并没有搜到详细的参数解释，但是根据参数可以猜测应该就是scn不一致，并且也符合cureent redo丢失导致的故障现象，所以接下来尝试推进scn来恢复。

处理过程

重建控制文件

登录到客户数据库，重试了一次resetlogs，发现控制文件已经损坏，但是控制文件并没有备份，只能手工重建：

sql> alter database open resetlogs;
alter database open resetlogs
*
error at line 1:
ora-01139: resetlogs option only valid after an incomplete database recovery
sql> recover database using backup controlfile until cancel;
ora-00283: recovery session canceled due to errors
ora-16433: the database must be opened in read/write mode.

手工构建控制文件：

sql> select name from v$datafile  ;
name
--------------------------------------------------------------------------------
 datadg/ycldb/datafile/system.256.837702609
 datadg/ycldb/datafile/sysaux.257.837702611
 datadg/ycldb/datafile/undotbs1.258.837702611
 datadg/ycldb/datafile/users.259.837702611
 datadg/ycldb/datafile/undotbs2.267.837702781
 datadg/ycldb/datafile/undotbs1.278.837704089
 datadg/ycldb/datafile/undotbs1.279.837704107
 datadg/ycldb/datafile/undotbs1.280.837704127
 datadg/ycldb/datafile/undotbs1.281.837704143
 datadg/ycldb/datafile/undotbs2.282.837704165
 datadg/ycldb/datafile/undotbs2.283.837704183
 datadg/ycldb/datafile/undotbs2.284.837704201
 datadg/ycldb/datafile/undotbs2.285.837704221
 datadg/ycldb/datafile/ts_gd_mbo_01.dbf
 datadg/ycldb/datafile/ts_gd_mbo_02.dbf
 datadg/ycldb/datafile/ts_gd_mbo_03.dbf
 datadg/ycldb/datafile/ts_gd_mbo_04.dbf
 datadg/ycldb/datafile/ts_gd_mbo_05.dbf
 datadg/ycldb/datafile/ts_gd_mbo_06.dbf
 datadg/ycldb/datafile/ts_gd_mbo_07.dbf
 datadg/ycldb/datafile/ts_gd_mbo_08.dbf
 datadg/ycldb/datafile/ts_gd_mbo_09.dbf
 datadg/ycldb/datafile/ts_gd_mbo_10.dbf
 datadg/ycldb/datafile/ts_gd_sap_01.dbf
 datadg/ycldb/datafile/ts_gd_sap_02.dbf
 datadg/ycldb/datafile/ts_gd_sap_03.dbf
 datadg/ycldb/datafile/ts_gd_sap_04.dbf
 datadg/ycldb/datafile/ts_gd_sap_05.dbf
 datadg/ycldb/datafile/ts_gd_sap_06.dbf
 datadg/ycldb/datafile/ts_gd_sap_07.dbf
 datadg/ycldb/datafile/ts_gd_sap_08.dbf
 datadg/ycldb/datafile/ts_gd_sap_09.dbf
 datadg/ycldb/datafile/ts_gd_sap_10.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_01.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_02.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_03.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_04.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_05.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_06.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_07.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_08.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_09.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_10.dbf
 datadg/ycldb/datafile/ts_gd_tel_01.dbf
 datadg/ycldb/datafile/ts_gd_etl_02.dbf
 datadg/ycldb/datafile/ts_gd_etl_03.dbf
 datadg/ycldb/datafile/ts_gd_etl_04.dbf
 datadg/ycldb/datafile/ts_gd_etl_05.dbf
 datadg/ycldb/datafile/ts_gd_etl_06.dbf
 datadg/ycldb/datafile/ts_gd_etl_07.dbf
 datadg/ycldb/datafile/ts_gd_etl_08.dbf
 datadg/ycldb/datafile/ts_gd_etl_09.dbf
 datadg/ycldb/datafile/ts_gd_etl_10.dbf
 datadg/ycldb/datafile/ts_gd_mbo_11.dbf
 datadg/ycldb/datafile/ts_gd_mbo_12.dbf
 datadg/ycldb/datafile/ts_gd_mbo_13.dbf
 datadg/ycldb/datafile/ts_gd_mbo_14.dbf
 datadg/ycldb/datafile/ts_gd_mbo_15.dbf
 datadg/ycldb/datafile/ts_gd_mbo_16.dbf
 datadg/ycldb/datafile/ts_gd_mbo_17.dbf
 datadg/ycldb/datafile/ts_gd_mbo_18.dbf
 datadg/ycldb/datafile/ts_gd_mbo_19.dbf
 datadg/ycldb/datafile/ts_gd_mbo_20.dbf
 datadg/ycldb/datafile/ts_gd_mbo_21.dbf
 datadg/ycldb/datafile/ts_gd_mbo_22.dbf
 datadg/ycldb/datafile/ts_gd_mbo_23.dbf
 datadg/ycldb/datafile/ts_gd_mbo_24.dbf
 datadg/ycldb/datafile/ts_gd_mbo_25.dbf
 datadg/ycldb/datafile/ts_gd_mbo_26.dbf
 datadg/ycldb/datafile/ts_gd_mbo_27.dbf
 datadg/ycldb/datafile/ts_gd_mbo_28.dbf
 datadg/ycldb/datafile/ts_gd_mbo_29.dbf
 datadg/ycldb/datafile/ts_gd_mbo_30.dbf
 datadg/ycldb/datafile/ts_gd_mbo_31.dbf
 datadg/ycldb/datafile/ts_gd_mbo_32.dbf
 datadg/ycldb/datafile/ts_gd_mbo_33.dbf
 datadg/ycldb/datafile/ts_gd_mbo_34.dbf
 datadg/ycldb/datafile/ts_gd_mbo_35.dbf
 datadg/ycldb/datafile/ts_gd_mbo_36.dbf
 datadg/ycldb/datafile/ts_gd_mbo_37.dbf
 datadg/ycldb/datafile/ts_gd_mbo_38.dbf
 datadg/ycldb/datafile/ts_gd_mbo_39.dbf
 datadg/ycldb/datafile/ts_gd_mbo_40.dbf
 datadg/ycldb/datafile/ts_gd_mbo_41.dbf
 datadg/ycldb/datafile/ts_gd_mbo_42.dbf
 datadg/ycldb/datafile/ts_gd_mbo_43.dbf
 datadg/ycldb/datafile/ts_gd_mbo_44.dbf
 datadg/ycldb/datafile/ts_gd_mbo_45.dbf
 datadg/ycldb/datafile/ts_gd_mbo_46.dbf
 datadg/ycldb/datafile/ts_gd_mbo_47.dbf
 datadg/ycldb/datafile/ts_gd_mbo_48.dbf
 datadg/ycldb/datafile/ts_gd_mbo_49.dbf
 datadg/ycldb/datafile/ts_gd_mbo_50.dbf
 datadg/ycldb/datafile/ts_gd_hr_01.dbf
 datadg/ycldb/datafile/ts_gd_hr_02.dbf
 datadg/ycldb/datafile/ts_gd_hr_03.dbf
 datadg/ycldb/datafile/ts_gd_hr_04.dbf
 datadg/ycldb/datafile/ts_gd_hr_05.dbf
 datadg/ycldb/datafile/ts_gd_hr_06.dbf
 datadg/ycldb/datafile/ts_gd_hr_07.dbf
 datadg/ycldb/datafile/ts_gd_hr_08.dbf
 datadg/ycldb/datafile/ts_gd_hr_09.dbf
 datadg/ycldb/datafile/ts_gd_hr_10.dbf
 datadg/ycldb/datafile/user02.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_11.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_12.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_13.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_14.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_15.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_16.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_17.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_18.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_19.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_20.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_21.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_22.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_23.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_24.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_25.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_26.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_27.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_28.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_29.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_30.dbf
 datadg/ycldb/datafile/ts_gd_ods01.dbf
 datadg/ycldb/datafile/ts_gd_sap_11.dbf
 datadg/ycldb/datafile/ts_gd_sap_12.dbf
 datadg/ycldb/datafile/ts_gd_sap_13.dbf
 datadg/ycldb/datafile/ts_gd_sap_14.dbf
 datadg/ycldb/datafile/ts_gd_sap_15.dbf
 datadg/ycldb/datafile/ts_gd_sap_16.dbf
 datadg/ycldb/datafile/ts_gd_sap_17.dbf
 datadg/ycldb/datafile/ts_gd_sap_18.dbf
 datadg/ycldb/datafile/ts_gd_sap_19.dbf
 datadg/ycldb/datafile/ts_gd_sap_20.dbf
 datadg/ycldb/datafile/undotbs106.dbf
 datadg/ycldb/datafile/undotbs107.dbf
 datadg/ycldb/datafile/undotbs108.dbf
 datadg/ycldb/datafile/undotbs109.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_31.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_32.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_33.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_34.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_35.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_36.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_37.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_38.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_39.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_40.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_41.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_42.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_43.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_44.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_45.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_46.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_47.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_48.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_49.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_50.dbf
 datadg/ycldb/datafile/undotbs202.dbf
 datadg/ycldb/datafile/sysaux01.dbf
 datadg/ycldb/datafile/ts_gd_sap_21.dbf
 datadg/ycldb/datafile/ts_gd_sap_22.dbf
 datadg/ycldb/datafile/ts_gd_sap_23.dbf
 datadg/ycldb/datafile/ts_gd_sap_24.dbf
 datadg/ycldb/datafile/ts_gd_sap_25.dbf
 datadg/ycldb/datafile/ts_gd_sap_26.dbf
 datadg/ycldb/datafile/ts_gd_sap_27.dbf
 datadg/ycldb/datafile/ts_gd_sap_28.dbf
 datadg/ycldb/datafile/ts_gd_sap_29.dbf
 datadg/ycldb/datafile/ts_gd_sap_30.dbf
 datadg/ycldb/datafile/ts_gd_sap_31.dbf
 datadg/ycldb/datafile/ts_gd_sap_32.dbf
 datadg/ycldb/datafile/ts_gd_sap_33.dbf
 datadg/ycldb/datafile/ts_gd_sap_34.dbf
 datadg/ycldb/datafile/ts_gd_sap_35.dbf
 datadg/ycldb/datafile/sysaux02.dbf
 datadg/ycldb/datafile/undotbs110.dbf
 datadg/ycldb/datafile/undotbs111.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_51.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_52.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_53.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_54.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_55.dbf
 datadg/ycldb/datafile/sysaux03.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_56.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_57.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_58.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_59.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_60.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_61.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_62.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_63.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_64.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_65.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_66.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_67.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_68.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_69.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_70.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_71.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_72.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_73.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_74.dbf
 datadg/ycldb/datafile/ts_gd_nonsap_75.dbf
 datadg/ycldb/datafile/sysaux04.dbf
 datadg/ycldb/datafile/sysaux05.dbf
 datadg/ycldb/datafile/sysaux06.dbf
 datadg/ycldb/datafile/ts_gd_nonsap.526.1026815149
 datadg/ycldb/datafile/ts_gd_nonsap.527.1026815375
 datadg/ycldb/datafile/ts_gd_nonsap.528.1026815559
 datadg/ycldb/datafile/ts_gd_nonsap.529.1026815685
 datadg/ycldb/datafile/ts_gd_nonsap.530.1026815773
 datadg/ycldb/datafile/ts_gd_nonsap.531.1026815843
 datadg/ycldb/datafile/ts_gd_nonsap.532.1026816071
 datadg/ycldb/datafile/ts_gd_nonsap.533.1026816183
 datadg/ycldb/datafile/ts_gd_nonsap.534.1026816281
 datadg/ycldb/datafile/ts_gd_nonsap.535.1026816361
 datadg/ycldb/datafile/ts_gd_nonsap.536.1026816483
 datadg/ycldb/datafile/ts_gd_nonsap.537.1026816699
 datadg/ycldb/datafile/ts_gd_nonsap.540.1048934637
 datadg/ycldb/datafile/ts_gd_nonsap.541.1048934799
 datadg/ycldb/datafile/ts_gd_nonsap.542.1048934899
 datadg/ycldb/datafile/ts_gd_nonsap.543.1048935011
 datadg/ycldb/datafile/ts_gd_nonsap.544.1049804883
 datadg/ycldb/datafile/ts_gd_nonsap.545.1049804975
 datadg/ycldb/datafile/ts_gd_nonsap.546.1049805049
 datadg/ycldb/datafile/ts_gd_nonsap.547.1049805117
 datadg/ycldb/datafile/ts_gd_nonsap.548.1049859947
 datadg/ycldb/datafile/ts_gd_nonsap.549.1049860031
 datadg/ycldb/datafile/ts_gd_nonsap.550.1049860103
 datadg/ycldb/datafile/ts_gd_nonsap.551.1049860181
 datadg/ycldb/datafile/ts_gd_nonsap.552.1049860267
 datadg/ycldb/datafile/ts_gd_nonsap.553.1049860341
 datadg/ycldb/datafile/ts_gd_nonsap.554.1049860413
 datadg/ycldb/datafile/ts_gd_nonsap.555.1049860709
 datadg/ycldb/datafile/ts_gd_nonsap.556.1049876541
 datadg/ycldb/datafile/ts_gd_nonsap.557.1049876611
 datadg/ycldb/datafile/ts_gd_nonsap.558.1049876745
 datadg/ycldb/datafile/ts_gd_nonsap.559.1055148423
 datadg/ycldb/datafile/ts_gd_nonsap.560.1055148507
 datadg/ycldb/datafile/ts_gd_nonsap.561.1055148581
 datadg/ycldb/datafile/ts_gd_nonsap.562.1055148815
 datadg/ycldb/datafile/ts_gd_nonsap.563.1055148881
 datadg/ycldb/datafile/ts_gd_nonsap.564.1055148967
 datadg/ycldb/datafile/ts_gd_nonsap.565.1055149029
 datadg/ycldb/datafile/ts_gd_nonsap.566.1055149093
 datadg/ycldb/datafile/ts_gd_nonsap.567.1055222485
 datadg/ycldb/datafile/ts_gd_nonsap.568.1055222643
 datadg/ycldb/datafile/ts_gd_nonsap.569.1055222719
 datadg/ycldb/datafile/ts_gd_nonsap.570.1055222825
 datadg/ycldb/datafile/ts_gd_nonsap.571.1055222971
 datadg/ycldb/datafile/ts_gd_nonsap.572.1055223051
 datadg/ycldb/datafile/ts_gd_nonsap.573.1055223181
 datadg/ycldb/datafile/ts_gd_nonsap.574.1055223369
 datadg/ycldb/datafile/ts_gd_nonsap.575.1055223511
 datadg/ycldb/datafile/ts_gd_nonsap.576.1055223573
 datadg/ycldb/datafile/sysaux07.dbf
258 rows selected.
sql> select group#,thread#,members,bytes from v$log;
    group#    thread#    members      bytes
---------- ---------- ---------- ----------
         1          1          1 1073741824
         2          1          1 1073741824
         3          1          1 1073741824
         4          1          1 1073741824
         5          1          1 1073741824
         6          2          1 1073741824
         7          2          1 1073741824
         8          2          1 1073741824
         9          2          1 1073741824
        10          2          1 1073741824
        
sql>  select group#,member from v$logfile order by 1;
    group# member
---------- --------------------------------------------------
         1  datadg/ycldb/onlinelog/group_1.261.1104462159
         2  datadg/ycldb/onlinelog/group_2.262.1104462161
         3  datadg/ycldb/onlinelog/group_3.263.1104462163
         4  datadg/ycldb/onlinelog/group_4.264.1104462167
         5  datadg/ycldb/onlinelog/group_5.265.1104462169
         6  datadg/ycldb/onlinelog/group_6.268.1104462171
         7  datadg/ycldb/onlinelog/group_7.269.1104462173
         8  datadg/ycldb/onlinelog/group_8.270.1104462177
         9  datadg/ycldb/onlinelog/group_9.271.1104462179
        10  datadg/ycldb/onlinelog/group_10.272.1104462181
        
sql> select userenv('language') from dual;
userenv('language')
----------------------------------------------------
american_america.zhs16gbk

重启数据库到nomount并重建控制文件：

sql> startup nomount
sql> create controlfile reuse database "ycldb" resetlogs noarchivelog
  2  maxlogfiles 50
  3  maxlogmembers 5
  4  maxdatafiles 1000
  5  maxinstances 10
  6  maxloghistory 1000
  7  logfile 
  8  group  1 ' datadg/ycldb/onlinelog/group_1.261.1104462159' size 1024m,
  9  group  2 ' datadg/ycldb/onlinelog/group_2.262.1104462161' size 1024m,
 10  group  3 ' datadg/ycldb/onlinelog/group_3.263.1104462163' size 1024m,
 11  group  4 ' datadg/ycldb/onlinelog/group_4.264.1104462167' size 1024m,
 12  group  5 ' datadg/ycldb/onlinelog/group_5.265.1104462169' size 1024m,
 13  group  6 ' datadg/ycldb/onlinelog/group_6.268.1104462171' size 1024m,
 14  group  7 ' datadg/ycldb/onlinelog/group_7.269.1104462173' size 1024m,
 15  group  8 ' datadg/ycldb/onlinelog/group_8.270.1104462177' size 1024m,
 16  group  9 ' datadg/ycldb/onlinelog/group_9.271.1104462179' size 1024m,
 17  group 10 ' datadg/ycldb/onlinelog/group_10.272.1104462181' size 1024m
 18  datafile        
 19  ' datadg/ycldb/datafile/system.256.837702609',
 20  ' datadg/ycldb/datafile/sysaux.257.837702611',
 21  ' datadg/ycldb/datafile/undotbs1.258.837702611',
 22  ' datadg/ycldb/datafile/users.259.837702611',
 23  ' datadg/ycldb/datafile/undotbs2.267.837702781',
 24  ' datadg/ycldb/datafile/undotbs1.278.837704089',
 25  ' datadg/ycldb/datafile/undotbs1.279.837704107',
 26  ' datadg/ycldb/datafile/undotbs1.280.837704127',
 27  ' datadg/ycldb/datafile/undotbs1.281.837704143',
 28  ' datadg/ycldb/datafile/undotbs2.282.837704165',
 29  ' datadg/ycldb/datafile/undotbs2.283.837704183',
 30  ' datadg/ycldb/datafile/undotbs2.284.837704201',
 31  ' datadg/ycldb/datafile/undotbs2.285.837704221',
 32  ' datadg/ycldb/datafile/ts_gd_mbo_01.dbf',
 33  ' datadg/ycldb/datafile/ts_gd_mbo_02.dbf',
 34  ' datadg/ycldb/datafile/ts_gd_mbo_03.dbf',
 35  ' datadg/ycldb/datafile/ts_gd_mbo_04.dbf',
 36  ' datadg/ycldb/datafile/ts_gd_mbo_05.dbf',
 37  ' datadg/ycldb/datafile/ts_gd_mbo_06.dbf',
 38  ' datadg/ycldb/datafile/ts_gd_mbo_07.dbf',
 39  ' datadg/ycldb/datafile/ts_gd_mbo_08.dbf',
 40  ' datadg/ycldb/datafile/ts_gd_mbo_09.dbf',
 41  ' datadg/ycldb/datafile/ts_gd_mbo_10.dbf',
 42  ' datadg/ycldb/datafile/ts_gd_sap_01.dbf',
 43  ' datadg/ycldb/datafile/ts_gd_sap_02.dbf',
 44  ' datadg/ycldb/datafile/ts_gd_sap_03.dbf',
 45  ' datadg/ycldb/datafile/ts_gd_sap_04.dbf',
 46  ' datadg/ycldb/datafile/ts_gd_sap_05.dbf',
 47  ' datadg/ycldb/datafile/ts_gd_sap_06.dbf',
 48  ' datadg/ycldb/datafile/ts_gd_sap_07.dbf',
 49  ' datadg/ycldb/datafile/ts_gd_sap_08.dbf',
 50  ' datadg/ycldb/datafile/ts_gd_sap_09.dbf',
 51  ' datadg/ycldb/datafile/ts_gd_sap_10.dbf',
 52  ' datadg/ycldb/datafile/ts_gd_nonsap_01.dbf',
 53  ' datadg/ycldb/datafile/ts_gd_nonsap_02.dbf',
 54  ' datadg/ycldb/datafile/ts_gd_nonsap_03.dbf',
 55  ' datadg/ycldb/datafile/ts_gd_nonsap_04.dbf',
 56  ' datadg/ycldb/datafile/ts_gd_nonsap_05.dbf',
 57  ' datadg/ycldb/datafile/ts_gd_nonsap_06.dbf',
 58  ' datadg/ycldb/datafile/ts_gd_nonsap_07.dbf',
 59  ' datadg/ycldb/datafile/ts_gd_nonsap_08.dbf',
 60  ' datadg/ycldb/datafile/ts_gd_nonsap_09.dbf',
 61  ' datadg/ycldb/datafile/ts_gd_nonsap_10.dbf',
 62  ' datadg/ycldb/datafile/ts_gd_tel_01.dbf',
 63  ' datadg/ycldb/datafile/ts_gd_etl_02.dbf',
 64  ' datadg/ycldb/datafile/ts_gd_etl_03.dbf',
 65  ' datadg/ycldb/datafile/ts_gd_etl_04.dbf',
 66  ' datadg/ycldb/datafile/ts_gd_etl_05.dbf',
 67  ' datadg/ycldb/datafile/ts_gd_etl_06.dbf',
 68  ' datadg/ycldb/datafile/ts_gd_etl_07.dbf',
 69  ' datadg/ycldb/datafile/ts_gd_etl_08.dbf',
 70  ' datadg/ycldb/datafile/ts_gd_etl_09.dbf',
 71  ' datadg/ycldb/datafile/ts_gd_etl_10.dbf',
 72  ' datadg/ycldb/datafile/ts_gd_mbo_11.dbf',
 73  ' datadg/ycldb/datafile/ts_gd_mbo_12.dbf',
 74  ' datadg/ycldb/datafile/ts_gd_mbo_13.dbf',
 75  ' datadg/ycldb/datafile/ts_gd_mbo_14.dbf',
 76  ' datadg/ycldb/datafile/ts_gd_mbo_15.dbf',
 77  ' datadg/ycldb/datafile/ts_gd_mbo_16.dbf',
 78  ' datadg/ycldb/datafile/ts_gd_mbo_17.dbf',
 79  ' datadg/ycldb/datafile/ts_gd_mbo_18.dbf',
 80  ' datadg/ycldb/datafile/ts_gd_mbo_19.dbf',
 81  ' datadg/ycldb/datafile/ts_gd_mbo_20.dbf',
 82  ' datadg/ycldb/datafile/ts_gd_mbo_21.dbf',
 83  ' datadg/ycldb/datafile/ts_gd_mbo_22.dbf',
 84  ' datadg/ycldb/datafile/ts_gd_mbo_23.dbf',
 85  ' datadg/ycldb/datafile/ts_gd_mbo_24.dbf',
 86  ' datadg/ycldb/datafile/ts_gd_mbo_25.dbf',
 87  ' datadg/ycldb/datafile/ts_gd_mbo_26.dbf',
 88  ' datadg/ycldb/datafile/ts_gd_mbo_27.dbf',
 89  ' datadg/ycldb/datafile/ts_gd_mbo_28.dbf',
 90  ' datadg/ycldb/datafile/ts_gd_mbo_29.dbf',
 91  ' datadg/ycldb/datafile/ts_gd_mbo_30.dbf',
 92  ' datadg/ycldb/datafile/ts_gd_mbo_31.dbf',
 93  ' datadg/ycldb/datafile/ts_gd_mbo_32.dbf',
 94  ' datadg/ycldb/datafile/ts_gd_mbo_33.dbf',
 95  ' datadg/ycldb/datafile/ts_gd_mbo_34.dbf',
 96  ' datadg/ycldb/datafile/ts_gd_mbo_35.dbf',
 97  ' datadg/ycldb/datafile/ts_gd_mbo_36.dbf',
 98  ' datadg/ycldb/datafile/ts_gd_mbo_37.dbf',
 99  ' datadg/ycldb/datafile/ts_gd_mbo_38.dbf',
100  ' datadg/ycldb/datafile/ts_gd_mbo_39.dbf',
101  ' datadg/ycldb/datafile/ts_gd_mbo_40.dbf',
102  ' datadg/ycldb/datafile/ts_gd_mbo_41.dbf',
103  ' datadg/ycldb/datafile/ts_gd_mbo_42.dbf',
104  ' datadg/ycldb/datafile/ts_gd_mbo_43.dbf',
105  ' datadg/ycldb/datafile/ts_gd_mbo_44.dbf',
106  ' datadg/ycldb/datafile/ts_gd_mbo_45.dbf',
107  ' datadg/ycldb/datafile/ts_gd_mbo_46.dbf',
108  ' datadg/ycldb/datafile/ts_gd_mbo_47.dbf',
109  ' datadg/ycldb/datafile/ts_gd_mbo_48.dbf',
110  ' datadg/ycldb/datafile/ts_gd_mbo_49.dbf',
111  ' datadg/ycldb/datafile/ts_gd_mbo_50.dbf',
112  ' datadg/ycldb/datafile/ts_gd_hr_01.dbf',
113  ' datadg/ycldb/datafile/ts_gd_hr_02.dbf',
114  ' datadg/ycldb/datafile/ts_gd_hr_03.dbf',
115  ' datadg/ycldb/datafile/ts_gd_hr_04.dbf',
116  ' datadg/ycldb/datafile/ts_gd_hr_05.dbf',
117  ' datadg/ycldb/datafile/ts_gd_hr_06.dbf',
118  ' datadg/ycldb/datafile/ts_gd_hr_07.dbf',
119  ' datadg/ycldb/datafile/ts_gd_hr_08.dbf',
120  ' datadg/ycldb/datafile/ts_gd_hr_09.dbf',
121  ' datadg/ycldb/datafile/ts_gd_hr_10.dbf',
122  ' datadg/ycldb/datafile/user02.dbf',
123  ' datadg/ycldb/datafile/ts_gd_nonsap_11.dbf',
124  ' datadg/ycldb/datafile/ts_gd_nonsap_12.dbf',
125  ' datadg/ycldb/datafile/ts_gd_nonsap_13.dbf',
126  ' datadg/ycldb/datafile/ts_gd_nonsap_14.dbf',
127  ' datadg/ycldb/datafile/ts_gd_nonsap_15.dbf',
128  ' datadg/ycldb/datafile/ts_gd_nonsap_16.dbf',
129  ' datadg/ycldb/datafile/ts_gd_nonsap_17.dbf',
130  ' datadg/ycldb/datafile/ts_gd_nonsap_18.dbf',
131  ' datadg/ycldb/datafile/ts_gd_nonsap_19.dbf',
132  ' datadg/ycldb/datafile/ts_gd_nonsap_20.dbf',
133  ' datadg/ycldb/datafile/ts_gd_nonsap_21.dbf',
134  ' datadg/ycldb/datafile/ts_gd_nonsap_22.dbf',
135  ' datadg/ycldb/datafile/ts_gd_nonsap_23.dbf',
136  ' datadg/ycldb/datafile/ts_gd_nonsap_24.dbf',
137  ' datadg/ycldb/datafile/ts_gd_nonsap_25.dbf',
138  ' datadg/ycldb/datafile/ts_gd_nonsap_26.dbf',
139  ' datadg/ycldb/datafile/ts_gd_nonsap_27.dbf',
140  ' datadg/ycldb/datafile/ts_gd_nonsap_28.dbf',
141  ' datadg/ycldb/datafile/ts_gd_nonsap_29.dbf',
142  ' datadg/ycldb/datafile/ts_gd_nonsap_30.dbf',
143  ' datadg/ycldb/datafile/ts_gd_ods01.dbf',
144  ' datadg/ycldb/datafile/ts_gd_sap_11.dbf',
145  ' datadg/ycldb/datafile/ts_gd_sap_12.dbf',
146  ' datadg/ycldb/datafile/ts_gd_sap_13.dbf',
147  ' datadg/ycldb/datafile/ts_gd_sap_14.dbf',
148  ' datadg/ycldb/datafile/ts_gd_sap_15.dbf',
149  ' datadg/ycldb/datafile/ts_gd_sap_16.dbf',
150  ' datadg/ycldb/datafile/ts_gd_sap_17.dbf',
151  ' datadg/ycldb/datafile/ts_gd_sap_18.dbf',
152  ' datadg/ycldb/datafile/ts_gd_sap_19.dbf',
153  ' datadg/ycldb/datafile/ts_gd_sap_20.dbf',
154  ' datadg/ycldb/datafile/undotbs106.dbf',
155  ' datadg/ycldb/datafile/undotbs107.dbf',
156  ' datadg/ycldb/datafile/undotbs108.dbf',
157  ' datadg/ycldb/datafile/undotbs109.dbf',
158  ' datadg/ycldb/datafile/ts_gd_nonsap_31.dbf',
159  ' datadg/ycldb/datafile/ts_gd_nonsap_32.dbf',
160  ' datadg/ycldb/datafile/ts_gd_nonsap_33.dbf',
161  ' datadg/ycldb/datafile/ts_gd_nonsap_34.dbf',
162  ' datadg/ycldb/datafile/ts_gd_nonsap_35.dbf',
163  ' datadg/ycldb/datafile/ts_gd_nonsap_36.dbf',
164  ' datadg/ycldb/datafile/ts_gd_nonsap_37.dbf',
165  ' datadg/ycldb/datafile/ts_gd_nonsap_38.dbf',
166  ' datadg/ycldb/datafile/ts_gd_nonsap_39.dbf',
167  ' datadg/ycldb/datafile/ts_gd_nonsap_40.dbf',
168  ' datadg/ycldb/datafile/ts_gd_nonsap_41.dbf',
169  ' datadg/ycldb/datafile/ts_gd_nonsap_42.dbf',
170  ' datadg/ycldb/datafile/ts_gd_nonsap_43.dbf',
171  ' datadg/ycldb/datafile/ts_gd_nonsap_44.dbf',
172  ' datadg/ycldb/datafile/ts_gd_nonsap_45.dbf',
173  ' datadg/ycldb/datafile/ts_gd_nonsap_46.dbf',
174  ' datadg/ycldb/datafile/ts_gd_nonsap_47.dbf',
175  ' datadg/ycldb/datafile/ts_gd_nonsap_48.dbf',
176  ' datadg/ycldb/datafile/ts_gd_nonsap_49.dbf',
177  ' datadg/ycldb/datafile/ts_gd_nonsap_50.dbf',
178  ' datadg/ycldb/datafile/undotbs202.dbf',
179  ' datadg/ycldb/datafile/sysaux01.dbf',
180  ' datadg/ycldb/datafile/ts_gd_sap_21.dbf',
181  ' datadg/ycldb/datafile/ts_gd_sap_22.dbf',
182  ' datadg/ycldb/datafile/ts_gd_sap_23.dbf',
183  ' datadg/ycldb/datafile/ts_gd_sap_24.dbf',
184  ' datadg/ycldb/datafile/ts_gd_sap_25.dbf',
185  ' datadg/ycldb/datafile/ts_gd_sap_26.dbf',
186  ' datadg/ycldb/datafile/ts_gd_sap_27.dbf',
187  ' datadg/ycldb/datafile/ts_gd_sap_28.dbf',
188  ' datadg/ycldb/datafile/ts_gd_sap_29.dbf',
189  ' datadg/ycldb/datafile/ts_gd_sap_30.dbf',
190  ' datadg/ycldb/datafile/ts_gd_sap_31.dbf',
191  ' datadg/ycldb/datafile/ts_gd_sap_32.dbf',
192  ' datadg/ycldb/datafile/ts_gd_sap_33.dbf',
193  ' datadg/ycldb/datafile/ts_gd_sap_34.dbf',
194  ' datadg/ycldb/datafile/ts_gd_sap_35.dbf',
195  ' datadg/ycldb/datafile/sysaux02.dbf',
196  ' datadg/ycldb/datafile/undotbs110.dbf',
197  ' datadg/ycldb/datafile/undotbs111.dbf',
198  ' datadg/ycldb/datafile/ts_gd_nonsap_51.dbf',
199  ' datadg/ycldb/datafile/ts_gd_nonsap_52.dbf',
200  ' datadg/ycldb/datafile/ts_gd_nonsap_53.dbf',
201  ' datadg/ycldb/datafile/ts_gd_nonsap_54.dbf',
202  ' datadg/ycldb/datafile/ts_gd_nonsap_55.dbf',
203  ' datadg/ycldb/datafile/sysaux03.dbf',
204  ' datadg/ycldb/datafile/ts_gd_nonsap_56.dbf',
205  ' datadg/ycldb/datafile/ts_gd_nonsap_57.dbf',
206  ' datadg/ycldb/datafile/ts_gd_nonsap_58.dbf',
207  ' datadg/ycldb/datafile/ts_gd_nonsap_59.dbf',
208  ' datadg/ycldb/datafile/ts_gd_nonsap_60.dbf',
209  ' datadg/ycldb/datafile/ts_gd_nonsap_61.dbf',
210  ' datadg/ycldb/datafile/ts_gd_nonsap_62.dbf',
211  ' datadg/ycldb/datafile/ts_gd_nonsap_63.dbf',
212  ' datadg/ycldb/datafile/ts_gd_nonsap_64.dbf',
213  ' datadg/ycldb/datafile/ts_gd_nonsap_65.dbf',
214  ' datadg/ycldb/datafile/ts_gd_nonsap_66.dbf',
215  ' datadg/ycldb/datafile/ts_gd_nonsap_67.dbf',
216  ' datadg/ycldb/datafile/ts_gd_nonsap_68.dbf',
217  ' datadg/ycldb/datafile/ts_gd_nonsap_69.dbf',
218  ' datadg/ycldb/datafile/ts_gd_nonsap_70.dbf',
219  ' datadg/ycldb/datafile/ts_gd_nonsap_71.dbf',
220  ' datadg/ycldb/datafile/ts_gd_nonsap_72.dbf',
221  ' datadg/ycldb/datafile/ts_gd_nonsap_73.dbf',
222  ' datadg/ycldb/datafile/ts_gd_nonsap_74.dbf',
223  ' datadg/ycldb/datafile/ts_gd_nonsap_75.dbf',
224  ' datadg/ycldb/datafile/sysaux04.dbf',
225  ' datadg/ycldb/datafile/sysaux05.dbf',
226  ' datadg/ycldb/datafile/sysaux06.dbf',
227  ' datadg/ycldb/datafile/ts_gd_nonsap.526.1026815149',
228  ' datadg/ycldb/datafile/ts_gd_nonsap.527.1026815375',
229  ' datadg/ycldb/datafile/ts_gd_nonsap.528.1026815559',
230  ' datadg/ycldb/datafile/ts_gd_nonsap.529.1026815685',
231  ' datadg/ycldb/datafile/ts_gd_nonsap.530.1026815773',
232  ' datadg/ycldb/datafile/ts_gd_nonsap.531.1026815843',
233  ' datadg/ycldb/datafile/ts_gd_nonsap.532.1026816071',
234  ' datadg/ycldb/datafile/ts_gd_nonsap.533.1026816183',
235  ' datadg/ycldb/datafile/ts_gd_nonsap.534.1026816281',
236  ' datadg/ycldb/datafile/ts_gd_nonsap.535.1026816361',
237  ' datadg/ycldb/datafile/ts_gd_nonsap.536.1026816483',
238  ' datadg/ycldb/datafile/ts_gd_nonsap.537.1026816699',
239  ' datadg/ycldb/datafile/ts_gd_nonsap.540.1048934637',
240  ' datadg/ycldb/datafile/ts_gd_nonsap.541.1048934799',
241  ' datadg/ycldb/datafile/ts_gd_nonsap.542.1048934899',
242  ' datadg/ycldb/datafile/ts_gd_nonsap.543.1048935011',
243  ' datadg/ycldb/datafile/ts_gd_nonsap.544.1049804883',
244  ' datadg/ycldb/datafile/ts_gd_nonsap.545.1049804975',
245  ' datadg/ycldb/datafile/ts_gd_nonsap.546.1049805049',
246  ' datadg/ycldb/datafile/ts_gd_nonsap.547.1049805117',
247  ' datadg/ycldb/datafile/ts_gd_nonsap.548.1049859947',
248  ' datadg/ycldb/datafile/ts_gd_nonsap.549.1049860031',
249  ' datadg/ycldb/datafile/ts_gd_nonsap.550.1049860103',
250  ' datadg/ycldb/datafile/ts_gd_nonsap.551.1049860181',
251  ' datadg/ycldb/datafile/ts_gd_nonsap.552.1049860267',
252  ' datadg/ycldb/datafile/ts_gd_nonsap.553.1049860341',
253  ' datadg/ycldb/datafile/ts_gd_nonsap.554.1049860413',
254  ' datadg/ycldb/datafile/ts_gd_nonsap.555.1049860709',
255  ' datadg/ycldb/datafile/ts_gd_nonsap.556.1049876541',
256  ' datadg/ycldb/datafile/ts_gd_nonsap.557.1049876611',
257  ' datadg/ycldb/datafile/ts_gd_nonsap.558.1049876745',
258  ' datadg/ycldb/datafile/ts_gd_nonsap.559.1055148423',
259  ' datadg/ycldb/datafile/ts_gd_nonsap.560.1055148507',
260  ' datadg/ycldb/datafile/ts_gd_nonsap.561.1055148581',
261  ' datadg/ycldb/datafile/ts_gd_nonsap.562.1055148815',
262  ' datadg/ycldb/datafile/ts_gd_nonsap.563.1055148881',
263  ' datadg/ycldb/datafile/ts_gd_nonsap.564.1055148967',
264  ' datadg/ycldb/datafile/ts_gd_nonsap.565.1055149029',
265  ' datadg/ycldb/datafile/ts_gd_nonsap.566.1055149093',
266  ' datadg/ycldb/datafile/ts_gd_nonsap.567.1055222485',
267  ' datadg/ycldb/datafile/ts_gd_nonsap.568.1055222643',
268  ' datadg/ycldb/datafile/ts_gd_nonsap.569.1055222719',
269  ' datadg/ycldb/datafile/ts_gd_nonsap.570.1055222825',
270  ' datadg/ycldb/datafile/ts_gd_nonsap.571.1055222971',
271  ' datadg/ycldb/datafile/ts_gd_nonsap.572.1055223051',
272  ' datadg/ycldb/datafile/ts_gd_nonsap.573.1055223181',
273  ' datadg/ycldb/datafile/ts_gd_nonsap.574.1055223369',
274  ' datadg/ycldb/datafile/ts_gd_nonsap.575.1055223511',
' datadg/ycldb/datafile/ts_gd_nonsap.576.1055223573',
275  276  ' datadg/ycldb/datafile/sysaux07.dbf'
277  character set zhs16gbk;
create controlfile reuse database "ycldb" resetlogs noarchivelog
*
error at line 1:
ora-01503: create controlfile failed
ora-12720: operation requires database is in exclusive mode

报错很明显，由于是集群，只能单节点操作，需要修改一下cluster_database参数：

sql> create pfile ='/oracle/app/ora11g/pfile0512.ora' from spfile;
file created.
sql> shutdown immediate
ora-01507: database not mounted
### 修改pfile
*.cluster_database=false
### 重新从pfile启动
sql> startup nomount pfile ='/oracle/app/ora11g/pfile0512.ora';
oracle instance started.
total system global area 8.5516e 10 bytes
fixed size                  2262656 bytes
variable size            1.8254e 10 bytes
database buffers         6.7109e 10 bytes
redo buffers              151117824 bytes
sql> create controlfile .....
...
..
control file created.
### 控制文件重建完成，重建之后重试了一次resetlogs开库，还是报了scn相关错误：
sql> alter database open resetlogs;
alter database open resetlogs
*
error at line 1:
ora-00603: oracle server session terminated by fatal error
ora-00600: internal error code, arguments: [2662], [4096], [163349674], [4096],
[163435443], [20971648], [], [], [], [], [], []
ora-00600: internal error code, arguments: [2662], [4096], [163349673], [4096],
[163435443], [20971648], [], [], [], [], [], []
ora-01092: oracle instance terminated. disconnection forced
ora-00600: internal error code, arguments: [2662], [4096], [163349671], [4096],
[163435443], [20971648], [], [], [], [], [], []
process id: 44862
session id: 7633 serial number: 3

ora-600 [2662] 参数解释：

arguments:
arg [a] current scn wrap
arg [b] current scn base
arg [c] dependent scn wrap
arg [d] dependent scn base
arg [e] where present this is the dba where the dependent scn came from.

通过参数格式，其实可以发现ora-600[kclchkblk_4]和ora-600[2662]的参数非常一致，并且重建控制文件之后kclchkblk_4消失了，而是变成了2662错误，所以可以推断导致这两个错误的原因应该是差不多的，回头继续深入研究一下kclchkblk_4错误，而2662错误一般都是scn不一致导致，所以接下来推进scn应该就可以解决。

ora-600[2662]处理

根据上面2662报错的相关参数，通过oradebug推进scn:

sql> startup mount pfile ='/oracle/app/ora11g/pfile0512.ora';
oracle instance started.
total system global area 8.5516e 10 bytes
fixed size                  2262656 bytes
variable size            1.8254e 10 bytes
database buffers         6.7109e 10 bytes
redo buffers              151117824 bytes
database mounted.
sql> oradebug setmypid
statement processed.
sql>  oradebug dumpvar sga kcsgscn_
kcslf kcsgscn_ [06001ae70, 06001aea0) = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6001ab50 00000000
sql> select to_char('20971648','xxxxxxxxx') from dual;
to_char('2
----------
   1400080
sql> select to_char('4096','xxxxxxxx') from dual;
to_char('
---------
     1000
sql> oradebug poke 0x06001ae70 4 0x1400090
before: [06001ae70, 06001ae74) = 00000000
after:  [06001ae70, 06001ae74) = 01400090
         
sql> oradebug poke 0x06001ae74 4 0x00001000            
before: [06001ae74, 06001ae78) = 00000000
after:  [06001ae74, 06001ae78) = 00001000
         
sql> oradebug dumpvar sga kcsgscn_
kcslf kcsgscn_ [06001ae70, 06001aea0) = 01400090 00001000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 6001ab50 00000000
sql> alter database open;
alter database open
*
error at line 1:
ora-01113: file 1 needs media recovery
ora-01110: data file 1: ' datadg/ycldb/datafile/system.256.837702609'
sql> recover database using backup controlfile until cancel;
ora-00279: change 17592046473045 generated at 05/12/2022 09:44:54 needed for
thread 1
ora-00289: suggestion :
/oracle/app/product/11.2.0/db_1/dbs/arch1_1_1104486270.dbf
ora-00280: change 17592046473045 for thread 1 is in sequence #1
specify log: {=suggested | filename | auto | cancel}
cancel
ora-01547: warning: recover succeeded but open resetlogs would get error below
ora-01194: file 1 needs more recovery to be consistent
ora-01110: data file 1: ' datadg/ycldb/datafile/system.256.837702609'
ora-01112: media recovery not started
sql> alter database open resetlogs;
alter database open resetlogs
*
error at line 1:
ora-01092: oracle instance terminated. disconnection forced
ora-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [],
[], [], [], [], []
process id: 49001
session id: 7633 serial number: 1

推完scn发现数据库又报了另一个ora-600错误：ora-00600: internal error code, arguments: [kdsgrp1]

ora-600[kdsgrp1]处理

kdsgrp1一般是表示逻辑错误，比如数据，索引坏块，一致性读之类的问题，通过查看trace发现是file# 2 ，即sysaux表空间，此表空间一般都是一些审计，awr等等相关数据，丢失对数据库运行无影响，所以这里通过event 10231 直接给它跳过：

sql> alter system set events '10231 trace name context forever,level 10';
system altered.
sql> recover database using backup controlfile until cancel;
ora-00279: change 17596501983377 generated at 05/12/2022 10:01:32 needed for
thread 1
ora-00289: suggestion :
/oracle/app/product/11.2.0/db_1/dbs/arch1_1_1104487278.dbf
ora-00280: change 17596501983377 for thread 1 is in sequence #1
specify log: {=suggested | filename | auto | cancel}
cancel
ora-01547: warning: recover succeeded but open resetlogs would get error below
ora-01194: file 1 needs more recovery to be consistent
ora-01110: data file 1: ' datadg/ycldb/datafile/system.256.837702609'
ora-01112: media recovery not started
sql> alter database open resetlogs;
database altered.

再次open，数据库已经能open了，但是通过日志查看，发现数据库立马就crash了：

block recovery completed at rba 1.203.16, scn 4097.20971870
dumping diagnostic data in directory=[cdmp_20220512101640], requested by (instance=1, osid=53082 (mmon)), summary=[incident=3168323].
errors in file /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/trace/ycldb1_mmon_53082.trc  (incident=3168324):
ora-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
incident details in: /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/incident/incdir_3168324/ycldb1_mmon_53082_i3168324.trc
use adrci or support workbench to package the incident.
see note 411.1 at my oracle support for error and packaging details.
completed: alter database open resetlogs
block recovery from logseq 1, block 201 to scn 17596501983583
recovery of online redo log: thread 1 group 1 seq 1 reading mem 0
  mem# 0:  datadg/ycldb/onlinelog/group_1.261.1104462159
block recovery completed at rba 1.203.16, scn 4097.20971872
block recovery from logseq 1, block 201 to scn 17596501983585
recovery of online redo log: thread 1 group 1 seq 1 reading mem 0
  mem# 0:  datadg/ycldb/onlinelog/group_1.261.1104462159
block recovery completed at rba 1.205.16, scn 4097.20971874
errors in file /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/trace/ycldb1_smon_53074.trc  (incident=3168292):
ora-00600: internal error code, arguments: [ktpridestroy2], [], [], [], [], [], [], [], [], [], [], []
incident details in: /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/incident/incdir_3168292/ycldb1_smon_53074_i3168292.trc
dumping diagnostic data in directory=[cdmp_20220512101642], requested by (instance=1, osid=53082 (mmon)), summary=[incident=3168324].
use adrci or support workbench to package the incident.
see note 411.1 at my oracle support for error and packaging details.
fatal internal error happened while smon was doing active transaction recovery.
errors in file /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/trace/ycldb1_smon_53074.trc:
ora-00600: internal error code, arguments: [ktpridestroy2], [], [], [], [], [], [], [], [], [], [], []
smon (ospid: 53074): terminating the instance due to error 474
system state dump requested by (instance=1, osid=53074 (smon)), summary=[abnormal instance termination].
system state dumped to trace file /oracle/app/ora11g/diag/rdbms/ycldb/ycldb1/trace/ycldb1_diag_53034_20220512101643.trc
thu may 12 10:16:43 2022
ora-1092 : opitsk aborting process
dumping diagnostic data in directory=[cdmp_20220512101643], requested by (instance=1, osid=53074 (smon)), summary=[abnormal instance termination].
instance terminated by smon, pid = 53074

可以看到日志中报了ora-600 [4194],ora-600[ktpridestroy2]等错误，并且数据库直接crash。

ora-600 [4194]：一般表示检测redo records和rollback（undo）records不一致，这里选择直接将整个undo给屏蔽掉，临时指向system表空间。

ora-600[ktpridestroy2]：一般表示smon在前滚的时候rollback segment出现了不一致，这里可以通过event 10513来禁止smon前滚来解决.

ora-600[4194],ora-600[ktpridestroy2]处理

接下来通过屏蔽undo和通过event 10513事件禁止smon前滚来再次启动数据库：

### 修改参数文件
*.undo_management=manual
*.undo_tablespace=system
### 启动数据库
sql> startup nomount pfile ='/oracle/app/ora11g/pfile0512.ora';
oracle instance started.
total system global area 8.5516e 10 bytes
fixed size                  2262656 bytes
variable size            1.8254e 10 bytes
database buffers         6.7109e 10 bytes
redo buffers              151117824 bytes
...
...
...
sql> alter system set events '10513 trace name context forever,level 2';
system altered.
sql> alter system set events '10231 trace name context forever,level 10';
system altered.
sql> recover database using backup controlfile until cancel;
ora-00279: change 17596502084543 generated at 05/12/2022 10:35:26 needed for
thread 1
ora-00289: suggestion :
/oracle/app/product/11.2.0/db_1/dbs/arch1_2_1104489264.dbf
ora-00280: change 17596502084543 for thread 1 is in sequence #2
specify log: {=suggested | filename | auto | cancel}
cancel
ora-01547: warning: recover succeeded but open resetlogs would get error below
ora-01194: file 1 needs more recovery to be consistent
ora-01110: data file 1: ' datadg/ycldb/datafile/system.256.837702609'
ora-01112: media recovery not started
sql> alter database open resetlogs;
database altered.
sql> 
sql> 
sql> select open_mode from v$database;
open_mode
--------------------
read write
### 数据库open成功，并且没有crash

至此数据库open完成，客户对数据库进行了验证，目前未发现异常，接下来就是一些收尾工作，重建spfile并删除之前配置的一些异常恢复用到的参数，重建undo表空间等等。

重建spfile

略

重建undo及ora-01548处理

sql> create undo tablespace undotbs3 datafile ' datadg/ycldb/datafile/undotbs3.dbf' size 1g autoextend on next 500m;
tablespace created.
sql> create undo tablespace undotbs4 datafile ' datadg/ycldb/datafile/undotbs4.dbf' size 1g autoextend on next 500m;
tablespace created.
sql>  alter system set undo_tablespace=undotbs3 scope=spfile sid='ycldb1';
system altered.
sql> alter system set undo_tablespace=undotbs4 scope=spfile sid='ycldb2';
system altered.

重建完spfile和undo之后，重启数据库发现又继续报错ora-600[ktpridestroy2]，删除原来旧的undo即可：

### 删除旧的undo
sql> drop tablespace undotbs1 including contents and datafiles;
drop tablespace undotbs1 including contents and datafiles
*
error at line 1:
ora-01548: active rollback segment '_syssmu21_301963420$' found, terminate
dropping tablespace
sql> alter rollback segment "_syssmu21_301963420" offline;
rollback segment altered.
sql> drop tablespace undotbs1 including contents and datafiles;
drop tablespace undotbs1 including contents and datafiles
*
error at line 1:
ora-01548: active rollback segment '_syssmu21_301963420$' found, terminate
dropping tablespace
### 发现存在active rollback segment，无法删除旧的undo表空间，offline也不行，最后只能通过隐含参数来offline，重新创建pfile，添加隐含参数，并从pfile启动：
sql> create pfile='/oracle/app/ora11g/pfile.ora' from spfile;
file created.
### 修改pfile：
_offline_rollback_segments=(_syssmu21_301963420$)
_corrupted_rollback_segments=(_syssmu21_301963420$)
### 重新启动
[ora11g@ycldb1 ~]$ srvctl stop instance -d ycldb -i ycldb1
[ora11g@ycldb1 ~]$ sqlplus / as sysdba
sql*plus: release 11.2.0.4.0 production on thu may 12 11:54:35 2022
米乐app官网下载 copyright (c) 1982, 2013, oracle.  all rights reserved.
connected to an idle instance.
sql> startup mount pfile='/oracle/app/ora11g/pfile.ora';
oracle instance started.
total system global area 8.5516e 10 bytes
fixed size                  2262656 bytes
variable size            1.3959e 10 bytes
database buffers         7.1404e 10 bytes
redo buffers              151117824 bytes
database mounted.
sql> alter system set events '10513 trace name context forever,level 2';
system altered.
sql> alter database open;
database altered.
sql> drop tablespace undotbs1 including contents and datafiles;
tablespace dropped.
sql> drop tablespace undotbs2 including contents and datafiles;
drop tablespace undotbs2 including contents and datafiles
*
error at line 1:
ora-01548: active rollback segment '_syssmu59_3908620453$' found, terminate
dropping tablespace
### drop undotbs2的时候同样有一个active的rollback segment，重复上面的步骤删除即可。
sql> drop tablespace undotbs2 including contents and datafiles;
tablespace dropped.

最后将cluster_database参数改回来，否则节点2无法启动：

user (ospid: 35368): terminating the instance due to error 1102
instance terminated by user, pid = 35368

恢复完成

sql> alter system set cluster_database=true scope=spfile sid='*';
system altered.
sql> 
sql> 
sql> 
sql> quit
disconnected from oracle database 11g enterprise edition release 11.2.0.4.0 - 64bit production
with the partitioning, real application clusters, automatic storage management, olap,
data mining and real application testing options
[ora11g@ycldb1 dbs]$ srvctl stop database -d ycldb 
[ora11g@ycldb1 dbs]$ srvctl start database -d ycldb 
[ora11g@ycldb1 dbs]$ exit
logout
[root@ycldb1 ~]# su - grid
[grid@ycldb1 ~]$ crsctl status res -t  
--------------------------------------------------------------------------------
name           target  state        server                   state_details       
--------------------------------------------------------------------------------
local resources
--------------------------------------------------------------------------------
ora.datadg.dg
               online  online       ycldb1                                       
               online  online       ycldb2                                       
ora.datadg02.dg
               online  online       ycldb1                                       
               online  online       ycldb2                                       
ora.listener.lsnr
               online  online       ycldb1                                       
               online  online       ycldb2                                       
ora.ocrdg.dg
               online  online       ycldb1                                       
               online  online       ycldb2                                       
ora.asm
               online  online       ycldb1                   started             
               online  online       ycldb2                   started             
ora.gsd
               offline offline      ycldb1                                       
               offline offline      ycldb2                                       
ora.net1.network
               online  online       ycldb1                                       
               online  online       ycldb2                                       
ora.ons
               online  online       ycldb1                                       
               online  online       ycldb2                                       
--------------------------------------------------------------------------------
cluster resources
--------------------------------------------------------------------------------
ora.listener_scan1.lsnr
      1        online  online       ycldb1                                       
ora.cvu
      1        online  online       ycldb1                                       
ora.iscmdg.db
      1        online  online       ycldb1                   open                
      2        online  online       ycldb2                   open                
ora.jcgldb.db
      1        online  online       ycldb1                   open                
      2        online  online       ycldb2                   open                
ora.oc4j
      1        offline offline                                                   
ora.scan1.vip
      1        online  online       ycldb1                                       
ora.ycldb.db
      1        online  online       ycldb1                   open                
      2        online  online       ycldb2                   open                
ora.ycldb1.vip
      1        online  online       ycldb1                                       
ora.ycldb2.vip
      1        online  online       ycldb2

最后修改时间：2022-05-12 17:38:06

「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」

关注作者

1人已赞赏