If you see something like this on your monitor or in your logs, you should worry about your important data.
And immediately change your HDD!
1 2 3 4 5 6 7 8 9 10 | ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen ata1.00: failed command: FLUSH CACHE EXT ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 res 40/00:0c:b9:68:55/00:00:05:00:00/40 Emask 0x4 (timeout) ata1.00: status: { DRDY } ata1: hard resetting link ata1: link is slow to respond, please be patient (ready=0) ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata1.00: configured for UDMA/133 ata1: EH complete |
Step 1:
Install SMART-Tolls to check your HDDs and find our which of them makes trouble (you need the serial number of broken to remove it)
Debian way installation (works for Ubuntu too):
1 | apt-get install smartmontools |
Check both HDDs and diff the output:
1 2 3 | smartctl -d ata -a /dev/sda > sda.txt smartctl -d ata -a /dev/sdb > sdb.txt vimdiff sda.txt sdb.txt |
In my case the sda HDD was broken, which must be replaced.
Legend:
md0 is my boot partition
md1 is my LVM
Step 2:
Set your broken HDD to faulty state
1 2 | mdadm --manage /dev/md1 --fail /dev/sda2 mdadm --manage /dev/md0 --fail /dev/sda1 |
Step 3:
Remove broken sda from array:
1 2 | mdadm /dev/md0 --remove /dev/sda1 mdadm /dev/md1 --remove /dev/sda2 |
Step 4:
Replace the physical HDD now!
Step 5:
Prepare the new HDD for RAID (copy the MBR to it):
1 | dd if=/dev/sdb of=/dev/sda count=1 bs=512 |
Step 6:
Tell your kernel to read the partition table of the new HDD:
1 | sfdisk -R /dev/sda |
Step 7:
Add the new HDD to RAID:
1 2 | mdadm --manage /dev/md0 -a /dev/sda1 mdadm --manage /dev/md1 -a /dev/sda2 |
Now wait until your RAID make a resync:
1 2 3 4 5 | cat /proc/mdstat Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid1 sda2[2] sdb2[1] 732306880 blocks [2/1] [_U] [>............] recovery = 2.4% (18162048/732306880) finish=172.3min speed=69073K/sec |
thx man! saved my day ;)
Great tutorial. Not many people know how to rebuild RAID.