APPLIES TO:
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database - Enterprise Edition - Version 10.2.0.1 to 12.1.0.1 [Release 10.2 to 12.1]
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Generic (Platform Independent)
SYMPTOMS
1) ASM diskgroup cannot be mounted due to the next error:
SQL> alter diskgroup <DGNAME> mount;
alter diskgroup <DGNAME> mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "2"
2) ASM reports block corruption:
Wed Apr 09 01:26:05 2014
NOTE: SMON starting instance recovery for group <DGNAME> domain 2 (mounted)
NOTE: F1X0 found on disk 0 au 2 fcn 0.469914
NOTE: SMON skipping disk 1 - no header
NOTE: starting recovery of thread=2 ckpt=2.10251 group=2 (<DGNAME>)
WARNING: ASM recovery read a corrupted ACD block 21004
NOTE: a corrupted block was dumped to the trace file
ORA-15196: invalid ASM block header [kfr.c:8098] [endian_kfbh] [3] [21004] [0 != 1]
ERROR: ASM recovery failed to read ACD block 21004
NOTE: cache initiating offline of disk 1 group <DGNAME>
NOTE: process _smon_+asm1 (26726) initiating offline of disk 1.3915939526 (<DGNAME>_0001) with mask 0x7e in group 2
CAUSE
1) “/dev/<DISK #1>” (<DGNAME>_0001) disk was overlapped by an OS volume, it shows OS metadata associated to the “LVM2 001” logical volume (all the ASM metadata was wiped out):
$ kfed read <DGNAME>_0001_<DISK #1>.dump | head -25
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
2ABD671E9400 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
2ABD671E9600 4542414C 454E4F4C 00000001 00000000 [LABELONE........]
2ABD671E9610 E4E1DDB1 00000020 324D564C 31303020 [.... ...LVM2 001] 2ABD671E9620 50365A77 71327874 34303156 4B4E6136 [wZ6Ptx2qV1046aNK]
2ABD671E9630 35395159 5147634C 487A5A38 63575A37 [YQ95LcGQ8ZzH7ZWc]
2ABD671E9640 00000000 00000019 00030000 00000000 [................]
2ABD671E9650 00000000 00000000 00000000 00000000 [................]
2ABD671E9660 00000000 00000000 00001000 00000000 [................]
2ABD671E9670 0002F000 00000000 00000000 00000000 [................]
2ABD671E9680 00000000 00000000 00000000 00000000 [................]
Repeat 215 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
2) The ““/dev/<DISK #1>” disk was used to create the next logical OS volume while it was already assigned to an ASM diskgroup.
3) This overlapping corrupted the "/dev/<DISK #1>" (<DGNAME>_0001) disk.
SOLUTION
The <DGNAME> diskgroup needs to be recreated and database files restored from backup due to the <DGNAME> diskgroup was overlapped by the OS, in other words the corruption occurred and came outside Oracle, it cannot be repaired since the OS volume overlapped the data in the “/dev/<DISK #1>” disk.