沈阳凯文数据恢复中心 服务器数据恢复 各类数据库修复 小型机数据恢复 13386848847 024-31065488 地址:沈阳市和平区三好街同方广场A座10楼1012写字间

EMC光纤非标硬盘数据恢复技术

2014.1.15,沈阳凯文数据恢复呼叫中心13386848847接到一位金融界的客户电话,EMC的存储,八块光纤硬盘组的RAID5,在国家某数据恢复中心进行过数据恢复操作,但遗憾的是被告知数据无法恢复。问其原因,说是所有的硬盘都不能正常识别。凯文数据恢复技术服务工程师立刻就意识到很可能是EMC的光纤硬盘算法不同导致的不认盘,下面我们看看这个案例。图一:EMC存储8块450GB光纤硬盘的RAID5客户类型:某金融财团存储设备:EMC CX500

EMC CX700 MirrorViewS 实施文档 数据恢复

 1.EMC MirrorView介绍MirrorView是基于盘阵之间,以LUN为单位的远程复制软件.适于与CX,CX3系列(CX200,CX300除外),MirrorView分为MirrorView/S(同步),MirrorView/A(异步).enabler分别为MirrorViewEnabler_01.xx.5.yyy.ena, MVAEnabler_01.xx.5.yyy.ena.2.EMC MirrorView/S实施路线图- 确认生产盘阵和镜像盘阵之间FLARE版本是否一致(至少大版本要一致).- 建立生产盘阵和镜像盘阵之间的物理连接,不同型号的盘阵镜像端口(用于做MirrorView的port口)略有不同:      Port 5 for a CX3-20c or CX3-40c SP      Port 3 for a CX600, CX700, or CX3-80 SP      Port 1 for a CX400, CX500, CX3-20, or CX3-40 SP-启用MirrorView enabler启用MirrorView enabler需要盘阵无I/O操作-分配write intent log  EMC官方建议分配两个128MB的LUN做为write intent log使用.如果不分配write intent log,系统会占用SP的MEM做为write intent log.-建立MirrorView connection  由于CX700的SP port口是短波模块,适用与短距离传输,如果要做MirrorView的距离很长,需要通过交换机(交换机之间通过长波模块做级联)用单模光纤线连接(黄色).-创建remote mirror  创建MirrorView的源LUN.-添加secondary image  关联MirrorView的源LUN和镜像LUN.-创建consistency group   如果源LUN之间有关联关系,需要一起添加到consistency group中来保证源LUN间数据的一致性.-测试MirrorView   通过promote来转化primary image和secondary image的角色.   通过fracture来断开primary image和secondary image的同步.

EMC损坏硬盘更换及恢复过程

 更换EMC损坏的磁盘以后,盘阵会自动开始把hot spare备份的数据恢复过来。然后把启用的hot spare盘恢复到初始备用状态。我们可以通过命令来观察这一过程,首先看新换上来的硬盘,注意此时的状态为Equalizing,也就是正在进行平衡的过程:# navicli -h 172.16.9.5 getdisk 1_0_1 Bus 1 Enclosure 0  Disk 1 Vendor Id:               SEAGATE  Product Id:              ST373307 CLAR72  Product Revision:        7A0A Lun:                     6  Type:                    6: RAID5  State:                   EqualizingHot Spare:               6: NO  Prct Rebuilt:            6: 100  Prct Bound:              6: 100  Serial Number:           3HZ6TL7F Sectors:                 139681792 (68204) Capacity:                68238 Private:                 6: 69704  Bind Signature:          0x5cf2, 0, 1 Hard Read Errors:        0 Hard Write Errors:       0 Soft Read Errors:        0 Soft Write Errors:       0 Read Retries:     N/A Write Retries:    N/A Remapped Sectors:        N/A Number of Reads:         980562 Number of Writes:        1107168 Number of Luns:          1 Raid Group ID:           3 Clariion Part Number:    DG118032459   Request Service Time:    N/A Read Requests:           980562 Write Requests:          1107168 Kbytes Read:             99788880 Kbytes Written:          70098518 Stripe Boundary Crossing: 2704586此时hot spare盘的状态为Enabled,其替代的硬盘为:1_0_1

incremental backup恢复错误一例

incremental backup恢复错误一例错误现象:关闭standby数据库,将restore standby controlfile to '/u01/rmanbak/fordg/control01.ctl'恢复出来的控制文件覆盖现有的控制文件,重新启动后出现以下问题:Sat Mar 23 06:42:37 CST 2013alter database recover managed standby database disconnect from session using current logfileSat Mar 23 06:42:37 CST 2013Attempt to start background Managed Standby Recovery process (stdby)MRP0 started with pid=29, OS id=11519Sat Mar 23 06:42:37 CST 2013MRP0: Background Managed Standby Recovery process started (stdby)Sat Mar 23 06:42:37 CST 2013RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9373_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9374_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9375_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9376_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9377_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9378_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9379_789583539.dbf'RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9380_789583539.dbf'Sat Mar 23 06:42:41 CST 2013RFS[1]: Archived Log: '/u01/archivelog/stdby/arc_1_7751_789583539.dbf'Sat Mar 23 06:42:42 CST 2013RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9381_789583539.dbf'Sat Mar 23 06:42:42 CST 2013Managed Standby Recovery starting Real Time ApplySat Mar 23 06:42:42 CST 2013Errors in file /u01/app/oracle/admin/stdby/bdump/stdby_dbw0_11458.trc:ORA-01157: cannot identify/lock data file 24 - see DBWR trace fileORA-01110: data file 24: '+DATA/in_sz_data'ORA-17503: ksfdopn:2 Failed to open file +DATA/in_sz_dataORA-15001: diskgroup "DATA" does not exist or is not mountedORA-15077: could not locate ASM instance serving a required diskgroupORA-29701: unable to connect to Cluster ManagerSat Mar 23 06:42:42 CST 2013Errors in file /u01/app/oracle/admin/stdby/bdump/stdby_dbw0_11458.trc:ORA-01157: cannot identify/lock data file 25 - see DBWR trace fileORA-01110: data file 25: '/u01/oradata/stdby/in_sz_data'ORA-27037: unable to obtain file statusLinux-x86_64 Error: 2: No such file or directoryAdditional information: 3Sat Mar 23 06:42:42 CST 2013Errors in file /u01/app/oracle/admin/stdby/bdump/stdby_dbw0_11458.trc:ORA-01157: cannot identify/lock data file 26 - see DBWR trace fileORA-01110: data file 26: '/u01/oradata/stdby/in_sz_data.9154.810742355'ORA-27037: unable to obtain file statusLinux-x86_64 Error: 2: No such file or directoryAdditional information: 3Sat Mar 23 06:42:42 CST 2013Errors in file /u01/app/oracle/admin/stdby/bdump/stdby_dbw0_11458.trc:ORA-01157: cannot identify/lock data file 27 - see DBWR trace fileORA-01110: data file 27: '/u01/oradata/stdby/in_ac_data.9090.810742665'ORA-27037: unable to obtain file statusLinux-x86_64 Error: 2: No such file or directoryAdditional information: 3MRP0: Background Media Recovery terminated with error 1110Sat Mar 23 06:42:43 CST 2013Errors in file /u01/app/oracle/admin/stdby/bdump/stdby_mrp0_11519.trc:ORA-01110: data file 24: '+DATA/in_sz_data'ORA-01157: cannot identify/lock data file 24 - see DBWR trace fileORA-01110: data file 24: '+DATA/in_sz_data'Managed Standby Recovery not using Real Time ApplySat Mar 23 06:42:43 CST 2013Errors in file /u01/app/oracle/admin/stdby/bdump/stdby_mrp0_11519.trc:ORA-01110: data file 24: '+DATA/in_sz_data'ORA-01157: cannot identify/lock data file 24 - see DBWR trace fileORA-01110: data file 24: '+DATA/in_sz_data'Sat Mar 23 06:42:43 CST 2013MRP0: Background Media Recovery process shutdown (stdby)Sat Mar 23 06:42:43 CST 2013RFS[2]: Archived Log: '/u01/archivelog/stdby/arc_2_9382_789583539.dbf'分析:因为主库新建过数据文件,从主库恢复过来的控制文件中中包含了这些文件,而从库却没有这些文件。解决方法:执行创建文件:SQL> alter database create datafile '+DATA/in_sz_data' as '/u01/oradata/stdby/in_sz_data.312.789659222';Database altered.SQL> alter database create datafile '/u01/oradata/stdby/in_sz_data' as '/u01/oradata/stdby/in_sz_data.312.789659223';Database altered.SQL> alter database create datafile '/u01/oradata/stdby/in_sz_data.9154.810742355' as '/u01/oradata/stdby/in_sz_data.312.789659224';Database altered.SQL> alter database create datafile '/u01/oradata/stdby/in_ac_data.9090.810742665' as '/u01/oradata/stdby/in_ac_data.9090.810742665';再次执行恢复:RMAN> catalog start with '/u01/rmanbak/fordg/';searching for all files that match the pattern /u01/rmanbak/fordg/List of Files Unknown to the Database=====================================File Name: /u01/rmanbak/fordg/standby_STD_20130323_n9o57oao_1_1.bakFile Name: /u01/rmanbak/fordg/control01.ctlFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nbo57oao_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nfo57p2c_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nco57oao_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nao57oao_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_ngo57p2h_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_ndo57ov7_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_neo57p0b_1_1.bakDo you really want to catalog the above files (enter YES or NO)? yescataloging files...cataloging doneList of Cataloged Files=======================File Name: /u01/rmanbak/fordg/standby_STD_20130323_n9o57oao_1_1.bakFile Name: /u01/rmanbak/fordg/control01.ctlFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nbo57oao_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nfo57p2c_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nco57oao_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_nao57oao_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_ngo57p2h_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_ndo57ov7_1_1.bakFile Name: /u01/rmanbak/fordg/standby_STD_20130323_neo57p0b_1_1.bakRMAN>RMAN> RMAN> run {  allocate channel dsk0 type disk;allocate channel dsk1 type disk;allocate channel dsk2 type disk;restore standby controlfile to '/u01/rmanbak/fordg/control01.ctl';recover database noredo;}2> 3> 4> 5> 6> 7>allocated channel: dsk0channel dsk0: sid=1481 devtype=DISKallocated channel: dsk1channel dsk1: sid=1480 devtype=DISKallocated channel: dsk2channel dsk2: sid=1479 devtype=DISKStarting restore at 28-MAR-13channel dsk0: starting datafile backupset restorechannel dsk0: restoring control fileoutput filename=/u01/rmanbak/fordg/control01.ctlchannel dsk0: reading from backup piece /u01/rmanbak/fordg/standby_STD_20130323_nfo57p2c_1_1.bakchannel dsk0: restored backup piece 1piece handle=/u01/rmanbak/fordg/standby_STD_20130323_nfo57p2c_1_1.bak tag=FOR STANDBYchannel dsk0: restore complete, elapsed time: 00:00:05Finished restore at 28-MAR-13Starting recover at 28-MAR-13channel dsk0: starting incremental datafile backupset restorechannel dsk0: specifying datafile(s) to restore from backup setdestination for restore of datafile 00024: /u01/oradata/stdby/in_sz_data.312.789659222destination for restore of datafile 00025: /u01/oradata/stdby/in_sz_data.312.789659223destination for restore of datafile 00026: /u01/oradata/stdby/in_sz_data.312.789659224destination for restore of datafile 00027: /u01/oradata/stdby/in_ac_data.9090.810742665channel dsk0: reading from backup piece /u01/rmanbak/inc2_STD_n2o572go_1_1released channel: dsk0released channel: dsk1released channel: dsk2RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of recover command at 03/28/2013 15:37:14ORA-19870: error reading backup piece /u01/rmanbak/inc2_STD_n2o572go_1_1ORA-19505: failed to identify file "/u01/rmanbak/inc2_STD_n2o572go_1_1"ORA-27037: unable to obtain file statusLinux-x86_64 Error: 2: No such file or directoryAdditional information: 3分析:通过WINDOWS系统FTP上传下载后文件发生了变化,改用NFS传输后恢复正常顺带下NFS的设置服务器端:[root@dgserver stdby]# vi /etc/exports[1]+  Stopped                 vi /etc/exports[root@dgserver stdby]# vi /etc/exports[root@dgserver stdby]# /etc/rc.d/init.d/portmap startStarting portmap:                                          [  OK  ][root@dgserver stdby]# /etc/rc.d/init.d/nfs startStarting NFS services:                                     [  OK  ]Starting NFS quotas:                                       [  OK  ]Starting NFS daemon:                                       [  OK  ]Starting NFS mountd:                                       [  OK  ][root@dgserver stdby]# exportfs -rvexporting *:/u01/rmanbak[root@dgserver stdby]#客户端[root@oracle1 ~]# showmount -e 192.168.13.109Export list for 192.168.13.109:/u01/rmanbak *[root@oracle1 ~]#RMAN> run {  allocate channel dsk0 type disk;allocate channel dsk1 type disk;allocate channel dsk2 type disk;restore standby controlfile to '/u01/rmanbak/fordg/control01.ctl';recover database noredo;}2> 3> 4> 5> 6> 7>allocated channel: dsk0channel dsk0: sid=1481 devtype=DISKallocated channel: dsk1channel dsk1: sid=1480 devtype=DISKallocated channel: dsk2channel dsk2: sid=1479 devtype=DISKStarting restore at 28-MAR-13control file is already restored to file /u01/rmanbak/fordg/control01.ctlrestore not done; all files readonly, offline, or already restoredFinished restore at 28-MAR-13Starting recover at 28-MAR-13channel dsk0: starting incremental datafile backupset restorechannel dsk0: specifying datafile(s) to restore from backup setdestination for restore of datafile 00024: /u01/oradata/stdby/in_sz_data.312.789659222destination for restore of datafile 00025: /u01/oradata/stdby/in_sz_data.312.789659223destination for restore of datafile 00026: /u01/oradata/stdby/in_sz_data.312.789659224destination for restore of datafile 00027: /u01/oradata/stdby/in_ac_data.9090.810742665channel dsk0: reading from backup piece /u01/rmanbak/inc2_STD_n2o572go_1_1released channel: dsk0released channel: dsk1released channel: dsk2RMAN-00571: ===========================================================RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============RMAN-00571: ===========================================================RMAN-03002: failure of recover command at 03/28/2013 16:32:26ORA-19870: error reading backup piece /u01/rmanbak/inc2_STD_n2o572go_1_1ORA-19573: cannot obtain exclusive enqueue for datafile 26查询该FILESQL> select name from v$datafile where file#=26;NAME--------------------------------------------------------------------------------/u01/oradata/stdby/in_sz_data.312.789659224SQL>分析:由于是MOUNT STANDBY方式启动的,决定重启到普通MOUNT状态SQL> shutdown immediate;ORACLE instance shut down.SQL> startup mount;ORACLE instance started.Total System Global Area 1.9327E+10 bytesFixed Size                  2130592 bytesVariable Size            4362079584 bytesDatabase Buffers         1.4948E+10 bytesRedo Buffers               14643200 bytesDatabase mounted.SQL>[oracle@dgserver ~]$ rman target /Recovery Manager: Release 10.2.0.5.0 - Production on Thu Mar 28 16:49:28 2013Copyright (c) 1982, 2007, Oracle.  All rights reserved.connected to target database: STD (DBID=1656746419, not open)RMAN> run {  allocate channel dsk0 type disk;allocate channel dsk1 type disk;allocate channel dsk2 type disk;restore standby controlfile to '/u01/rmanbak/fordg/control01.ctl';recover database noredo;}2> 3> 4> 5> 6> 7>using target database control file instead of recovery catalogallocated channel: dsk0channel dsk0: sid=1481 devtype=DISKallocated channel: dsk1channel dsk1: sid=1480 devtype=DISKallocated channel: dsk2channel dsk2: sid=1479 devtype=DISKStarting restore at 28-MAR-13control file is already restored to file /u01/rmanbak/fordg/control01.ctlrestore not done; all files readonly, offline, or already restoredFinished restore at 28-MAR-13Starting recover at 28-MAR-13channel dsk0: starting incremental datafile backupset restorechannel dsk0: specifying datafile(s) to restore from backup setdestination for restore of datafile 00024: /u01/oradata/stdby/in_sz_data.312.789659222destination for restore of datafile 00025: /u01/oradata/stdby/in_sz_data.312.789659223destination for restore of datafile 00026: /u01/oradata/stdby/in_sz_data.312.789659224destination for restore of datafile 00027: /u01/oradata/stdby/in_ac_data.9090.810742665channel dsk0: reading from backup piece /u01/rmanbak/inc2_STD_n2o572go_1_1channel dsk0: restored backup piece 1piece handle=/u01/rmanbak/inc2_STD_n2o572go_1_1 tag=INC2channel dsk0: restore complete, elapsed time: 00:24:27channel dsk0: starting incremental datafile backupset restorechannel dsk0: specifying datafile(s) to restore from backup setdestination for restore of datafile 00024: /u01/oradata/stdby/in_sz_data.312.789659222destination for restore of datafile 00025: /u01/oradata/stdby/in_sz_data.312.789659223channel dsk0: reading from backup piece /u01/rmanbak/fordg/standby_STD_20130323_nbo57oao_1_1.bakchannel dsk1: starting incremental datafile backupset restorechannel dsk1: specifying datafile(s) to restore from backup setdestination for restore of datafile 00026: /u01/oradata/stdby/in_sz_data.312.789659224channel dsk1: reading from backup piece /u01/rmanbak/fordg/standby_STD_20130323_nco57oao_1_1.bakchannel dsk2: starting incremental datafile backupset restorechannel dsk2: specifying datafile(s) to restore from backup setdestination for restore of datafile 00027: /u01/oradata/stdby/in_ac_data.9090.810742665channel dsk2: reading from backup piece /u01/rmanbak/fordg/standby_STD_20130323_neo57p0b_1_1.bakchannel dsk0: restored backup piece 1piece handle=/u01/rmanbak/fordg/standby_STD_20130323_nbo57oao_1_1.bak tag=FOR STANDBYchannel dsk0: restore complete, elapsed time: 00:00:03channel dsk1: restored backup piece 1piece handle=/u01/rmanbak/fordg/standby_STD_20130323_nco57oao_1_1.bak tag=FOR STANDBYchannel dsk1: restore complete, elapsed time: 00:00:10channel dsk2: restored backup piece 1piece handle=/u01/rmanbak/fordg/standby_STD_20130323_neo57p0b_1_1.bak tag=FOR STANDBYchannel dsk2: restore complete, elapsed time: 00:10:35Finished recover at 28-MAR-13released channel: dsk0released channel: dsk1released channel: dsk2RMAN>

CLASSPATH没有设置Notes客户端无法连接到Domino服务器

   一些Lotus Notes® 7.x 客户端突然无法连接到Lotus® Domino® server,尽管您重新安装并配置客户端时,这些客户端可以连接到服务器并完成配置。您可能会看到类似下面的错误:"Unable to find path to server. To trace this connection, use File - Preferences - User Preferences - Ports - Trace (Notes client) or Trace command (Domino server)""The remote server is not a known TCP/IP host.""The TCP/IP protocol stack reported that it ran out of memory. Consult your network documentation to increase configured memory, or reduce Notes connections by limiting clients (see SERVER_MAXSESSIONS parameter in Notes Admin Guide)."

IBM DS4000/DS5000添加新硬盘的步骤与注意事项

    添加前的准备工作:    1.对DS4000或DS5000系统中的数据进行完全备份    2.确认备份已经成功    3.确认当前的系统的控制器微码,NVSRAM和ESM的微码(参考11)    注意:在升级微码前,请查看微码中的README文件,确认微码的升级步骤    4.确认硬盘微码为最新版本    5.确认当前DS4000或DS5000中的硬盘处于最佳(optimal)状态    6.如果连接了扩展柜,通过Storage Manager的Read_Link_Status功能和Major Event Log(MEL)功能,确认扩展柜环路(drive loop)处于最佳(optimal)状态    7.查看并解决Major Event Log(MEL)中的错误问题    8.保存当前DS4000或DS5000的配置信息    9.如果混插FC和SATA硬盘,请购买并激活混插许可    10.添加硬盘前,请检查硬盘与当前DS4000或DS5000的系统的兼容性(参考11)    11.更多详细工作请参考IBM System Storage DS4000 and DS5000 Hard Drive and Storage Expansion Enclosure Installation and Migration Guide文档中的第二章内容

IBM DS4000 DS5000电池保护原理与时间

   DS4000系列存储Cache电池保护时间标称为72小时。由于DS4000的Cache是一种易失性存储介质,需要持续供电才能存储数据,一旦失去供电Cache中的数据将丢失。如果DS4000打开Cache功能(尤其是write cache),应用端数据写到Cache中就会返回写成功,但此时数据并没有写到磁盘中(一般需要一定条件触发从cache写到磁盘,如:10秒以后或Cache容量达到80%等,该过程称作flush),若意外掉电,电池就会保护Cache中的数据,该电池为充电电池,可以保护72小时。

Tags:

发布: admin 分类: 技术文章 评论: 0 浏览: 69

IBM DS4000 DS5000热备盘配置方法

  对于Stoarage Manager 10,定义热备盘有两种方法:自动分配(Automatic assign)和手工分配(Manual assign)。    注意:    1)从Storage Manager 10.10开始,全局热备盘不再有数量限制。    2)热备盘的容量要大于或等于存储子系统中其他磁盘的容量    3)在包含大量磁盘的配置中,由于重建时间较长,有必要定义多个热备盘    4)如果DS4000中包含FC和SATA硬盘,需要考虑定义FC和SATA作为热备盘,因为FC硬盘不能保护SATA盘创建的Raid组,反之亦然。

ServerGuide 引导安装指南

  使用ServerGuide 设置和安装CD 时,您不需要安装软盘。可以使用CD 配置任何支持的IBM 服务器型号。安装程序提供了安装服务器型号所需要的任务列表。在装有ServeRAID 适配器或具有RAID 能力的集成SCSI 控制器的服务器上,您可以运行SCSIRAID 配置程序来创建逻辑驱动器。  注:ServerGuide 程序的特征和功能可能随版本的不同而略有不同。要了解您现有版本的更多信息,请启动ServerGuide 设置和安装CD 并查看联机概述。并非所有的服务器型号都支持所有特点。ServerGuide 程序需要一台具有已启用的可启动(可引导)CD-ROM 驱动器的、受支持的IBM 服务器。除ServerGuide 设置和安装CD 外,您还必须准备好操作系统CD 以安装操作系统。

IBM刀片服务器宕机故障现象巧排除

   此排除IBM刀片服务器宕机故障所适用的机型仅限于BladeCenter所有8677机型; BladeCenter HS20所有8678机型; BladeCenter HS20所有8832机型; BladeCenter HS40所有8839机型; BladeCenter JS20所有8842机型; BladeCenter T所有8720机型; BladeCenter T所有8730机型。  IBM刀片服务器宕机故障现象