之前迁移一个pdb的时候,发现源cdb安装了dv(database vault)组件,但是目标cdb没有安装,因此在datapatch和启动的时候报错,无法正常使用,因此迁移直接回滚了。于是想着测试下用dump的方式迁移,结果又遇到一大堆问题。
1 问题出现
在expdp的时候使用的是下面的命令:
expdp system@pdb_xxxx directory=ext dumpfile=xxxx_expdp%u.dmp logfile=xxxx_expdp.log schemas=username parallel=32 cluster=n compression=all
因为本机可用本地磁盘空间仅有2tb,实际数据量也为2tb左右,为了避免空间不足,因此使用了压缩。但是在执行过程中出现了以下报错:
ora-39126: worker unexpected fatal error in kupw$worker.fetch_xml_objects [ora-22814: attribute or element value is larger than specified in type
ora-06512: at "sys.dbms_sys_error", line 105
ora-06512: at "sys.kupw$worker", line 12630
ora-06512: at "sys.dbms_sys_error", line 105
ora-06512: at "sys.dbms_metadata", line 9571
ora-06512: at "sys.dbms_metadata", line 4250
ora-06512: at "sys.dbms_metadata", line 5008
ora-06512: at "sys.dbms_metadata", line 5327
ora-06512: at "sys.dbms_metadata", line 9552
ora-06512: at "sys.kupw$worker", line 15119
----- pl/sql call stack -----
object line object
handle number name
0x73636f2920 33543 package body sys.kupw$worker.write_error_information
0x73636f2920 12651 package body sys.kupw$worker.determine_fatal_error
0x73636f2920 15452 package body sys.kupw$worker.fetch_xml_objects
0x73636f2920 3917 package body sys.kupw$worker.unload_metadata
0x73636f2920 13746 package body sys.kupw$worker.dispatch_work_items
0x73636f2920 2439 package body sys.kupw$worker.main
0x7be2734c8 2 anonymous block
dbms_metadata.set_filter
dbms_metadata.set_filter
dbms_metadata.set_filter
kupw: in fetch_xml_objects
kupw: end seqno is: 278
kupf$file.open_context
kupf$file.open_context
dbms_metadata.fetch_xml_clob
dbms_metadata.fetch_xml_clob
kupw: in procedure determine_fatal_error with ora-22814: attribute or element value is larger than specified in type
ora-06512: at "sys.dbms_metadata", line 9571
ora-06512: at "sys.dbms_metadata", line 4250
ora-06512: at "sys.dbms_metadata", line 5008
ora-06512: at "sys.dbms_metadata", line 5327
ora-06512: at "sys.dbms_metadata", line 9552
ora-39126: worker unexpected fatal error in kupw$worker.fetch_xml_objects [ora-22814: attribute or element value is larger than specified in type
ora-06512: at "sys.dbms_sys_error", line 105
ora-06512: at "sys.kupw$worker", line 12630
ora-06512: at "sys.dbms_sys_error", line 105
ora-06512: at "sys.dbms_metadata", line 9571
ora-06512: at "sys.dbms_metadata", line 4250
ora-06512: at "sys.dbms_metadata", line 5008
ora-06512: at "sys.dbms_metadata", line 5327
ora-06512: at "sys.dbms_metadata", line 9552
ora-06512: at "sys.kupw$worker", line 15119
----- pl/sql call stack -----
object line object
handle number name
0x73636f2920 33543 package body sys.kupw$worker.write_error_information
0x73636f2920 12651 package body sys.kupw$worker.determine_fatal_error
0x73636f2920 15452 package body sys.kupw$worker.fetch_xml_objects
0x73636f2920 3917 package body sys.kupw$worker.unload_metadata
0x73636f2920 13746 package body sys.kupw$worker.dispatch_work_items
0x73636f2920 2439 package body sys.kupw$worker.main
0x7be2734c8 2 anonymous block
dbms_metadata.set_filter
dbms_metadata.set_filter
dbms_metadata.set_filter
kupw: in fetch_xml_objects
kupw: end seqno is: 278
kupf$file.open_context
kupf$file.open_context
dbms_metadata.fetch_xml_clob
dbms_metadata.fetch_xml_clob
kupw: in procedure determine_fatal_error with ora-22814: attribute or element value is larger than specified in type
ora-06512: at "sys.dbms_metadata", line 9571
ora-06512: at "sys.dbms_metadata", line 4250
ora-06512: at "sys.dbms_metadata", line 5008
ora-06512: at "sys.dbms_metadata", line 5327
ora-06512: at "sys.dbms_metadata", line 9552
因为dump不会对数据库运行造成影响,所以默认情况下数据库不会记录相关的告警和追踪日志。查问题陷入了僵局。
2 找到问题
既然dump本身不会生成告警和追踪日志,没法进一步分析那么就收工开启相关记录:
alter system set events='22814 trace name errorstack level 3';
再次执行dump仍然失败报错,但是在告警日志中出现了以下内容:
pdb_xxxx(6):errors in file /u01/app/oracle/diag/rdbms/xxdbaas/xxdbaas2/trace/xxdbaas2_dw0c_171811.trc: ora-39126: worker unexpected fatal error in kupw$worker.fetch_xml_objects [ora-06512: at "sys.dbms_metadata", line 9571 ora-06512: at "sys.dbms_metadata", line 4250 ora-06512: at "sys.dbms_metadata", line 5008 ora-06512: at "sys.dbms_metadata", line 5327 ora-06512: at "sys.dbms_metadata", line 9552 ora-06512: at "sys.kupw$worker", line 15119 ] ... pdb_xxxx(6):errors in file /u01/app/oracle/diag/rdbms/xxdbaas/xxdbaas2/trace/xxdbaas2_dw0a_171850.trc: ora-39126: worker unexpected fatal error in kupw$worker.fetch_xml_objects [ora-06512: at "sys.dbms_metadata", line 9571 ora-06512: at "sys.dbms_metadata", line 4250 ora-06512: at "sys.dbms_metadata", line 5008 ora-06512: at "sys.dbms_metadata", line 5327 ora-06512: at "sys.dbms_metadata", line 9552 ora-06512: at "sys.kupw$worker", line 15119 ]
在对应的追踪日志中也找到了出现问题的地方:
----- error stack dump -----
at 0x7ffd40eab450 placed dbkda.c@298
ora-39126: worker unexpected fatal error in kupw$worker.fetch_xml_objects [ora-06512: at "sys.dbms_metadata", line 9571
ora-06512: at "sys.dbms_metadata", line 4250
ora-06512: at "sys.dbms_metadata", line 5008
ora-06512: at "sys.dbms_metadata", line 5327
ora-06512: at "sys.dbms_metadata", line 9552
ora-06512: at "sys.kupw$worker", line 15119
]
schema_export/job
at 0x7fb0fe339fb0 placed pfrrun.c@3799
ora-22814: attribute or element value is larger than specified in type
ora-06512: at "sys.dbms_metadata", line 9571
ora-06512: at "sys.dbms_metadata", line 4250
ora-06512: at "sys.dbms_metadata", line 5008
ora-06512: at "sys.dbms_metadata", line 5327
ora-06512: at "sys.dbms_metadata", line 9552
at 0x7ffd40eb17e0 placed kpoodr.c@237
----- current sql statement for this session (sql_id=4q4xc5vstbfq8) -----
begin
sys.kupw$worker.main('sys_export_schema_01', 'system', 0, 0);
end;
这里可以看到是在导出job的时候报的错。
3 处理
既然发现了是job的问题,但是通过告警日志无法排查出事哪个job造成的问题,因此调整了导出命令,将job暂时排除:
expdp system@pdb_xxxx directory=ext dumpfile=xxxx_expdp%u.dmp logfile=xxxx_expdp.log schemas=username parallel=32 cluster=n compression=all exclude=job
导出导入后再通过手动方式去迁移job。
当然最后记得关闭trace:
alter system set events='22814 trace name errorstack off';
当然最后测试完了发现超时严重,也不会采用dump的方式来迁移数据。
总结
本次dump的问题处理,其实主要是通过指定event的方式去抓取对应报错的trace信息并生成追踪日志,然后通过追踪日志去定位问题。
老规矩,知道写了些啥。
「喜欢这篇文章,您的关注和赞赏是给作者最好的鼓励」
关注作者
【米乐app官网下载的版权声明】本文为墨天轮用户原创内容,转载时必须标注文章的来源(墨天轮),文章链接,文章作者等基本信息,否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容,欢迎发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。