Good writeup! <br><br><snipage><br><br><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
This all seems fine and dandy. However, I had no Internet access at<br>
home at the time so I couldn't use my computer SSH in. I tried from the<br>
phone, but it wasn't letting me in, either. I figured the cell network<br>
was being crappy, so I went to the office. When I arrived, I found that<br>
the system was utterly unresponsive, like I'd expect to find a Windows<br>
box with its head shoved up its ass. The access light for the bad drive<br>
was on, and nothing else. The only thing the system would respond to<br>
was the Magic SysRq key. So, I did an emergency unmount, sync for<br>
several seconds, and then rebooted. When it came back up the kernel<br>
said "Oh, I was in the middle of scrubbing, lemmie get right back to<br>
that", with all five members in the array active.<br>
<br>
Waitaminute. It should have kept the data from before that said that<br>
the disk at /dev/sdb failed. It didn't.<br></blockquote><div><br>Actually, it did. It had already marked the sdb drive as dirty and for scrubbing to continue as long as needed.<br> <br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Fine, so I tried to take /dev/sdb out, which succeeded, and then I<br>
rebooted again. Nope, it didn't remember, and it started scrubbing again.<br>
<br>
Fine, I said. So the next morning (nothing was open at this point) I<br>
went and got a 1 TB disk (ouch---NOT what I wanted, but nobody seems to<br>
have any 750 GB disks anywhere). Fine. I went in, and the server was<br>
again unresponsive, it would only listen to the magic sysrq.<br>
<br>
I rebooted, and dropped to a shell. At this point, the kernel wouldn't<br>
do anything with the array. It said "failed to start dirty degraded<br>
array". I thought I was surely screwed.<br></blockquote><div><br>The linguistic connotations here are hilarious! <br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
At this point it said that /dev/sd{c,d1,e1,f1} were fine and in-sync.<br>
It said that /dev/sdb was a "spare" (what? It failed! No sane person,<br>
no idiot even, would use that as a spare for anything!).<br></blockquote><div><br>Yep. sdb is totally hosed. Most likely the tiny 1-2 block reserved section used by RAID systems for storing data about the drive.<br></div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Anyway, so I tried to remove /dev/sdb. It said that it was busy.<br>
<br>
I said, "fine, I'll swap it out." So I did. Then it said "unable to<br>
add disk to array." The other one was gone finally, but now I couldn't<br>
add a disk. Alright.<br>
<br>
So I put /dev/sdb back and was back at the dirty degraded place I was<br>
before. Finally, I thought to boot with a Live CD in the hopes that it<br>
wouldn't autostart the array. Then I realized I'd forgotten about mdadm<br>
--stop /dev/md0 --- so I tried that, and it worked (though not the first<br>
time, the first time it said it was busy). Alright, cool. So I then<br>
reassembled the array with /dev/sd{c,d1,e1,f1} and it came up. I<br>
exited, the system resumed booting, and it didn't attempt to scrub.<br>
Awesome.<br></blockquote><div>Yes. You found the magic incantation. Stop the array, slaughter the dead drive, do admin things, start the array.<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
At this point I powered down, replaced the drive, partitioned the new<br>
drive and made it a member of the array. Fantastic. It's working<br>
again, rebuilding the new drive (partition).<br>
<br>
And then this morning, as it finished rebuilding that drive it found<br>
another bad one. At least this time swapping out the disk was a simple,<br>
quick matter (10, maybe 15 minutes). It seems that if the kernel finds<br>
a bad disk while scrubbing, it doesn't handle it all that well. But<br>
having found the new dead drive without it trying to scrub the array was<br>
much easier, as it did what I asked without putting up a fight.<br></blockquote><div><br>computers suck sometimes. alcohol helps. If you drink enough, you don't care about pouring a fifth of grain into a drive array and setting it on fire! <br>
</div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Sigh.<br>
<br>
Happy new year!<br>
<div class="HOEnZb"><div class="h5"><br>
--<br>
A man who reasons deliberately, manages it better after studying Logic<br>
than he could before, if he is sincere about it and has common sense.<br>
--- Carveth Read, “Logic”<br>
<br>
</div></div><br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote><br></div><br><br clear="all"><br>-- <br>-- <br>James P. Kinney III<br><br>As long as the general population is passive, apathetic, diverted to
consumerism or hatred of the vulnerable, then the powerful can do as
they please, and those who survive will be left to contemplate the
outcome.<br>- <i><i><i><i>2011 Noam Chomsky<br><br><a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a><br></i></i></i></i><br>