<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Jan 20, 2009, at 8:58 AM, Jeff Lightner wrote:</div><div><br></div><blockquote type="cite"><div>Seeing two mirrors drop out in RAID10 shouldn't be an issue so long as<br>you have two spares because it should be able to recreate both mirrors<br>as it is both mirrored and parity striped. (Losing two in RAID1 would<br>of course be an issue [assuming you didn't have a 3rd mirror]). What<br>hardware array was being used that dropping two mirrors caused a problem<br>for RAID10?</div></blockquote><div><br></div>Maybe I didn't say that clearly enough - I've seen a RAID 10 lose both drives of a single mirror (in other words, the stripe set was broken, total data loss, chaos, screaming, and finally a restore from backup!) multiple times. And yes, I felt like God hated me, along with a few other supernatural entities. </div><div><br><blockquote type="cite"><div>I'll have to say that even if having two mirrors drop in RAID10 were a<br>problem it must be a sign that God hates the person that had it if the<br>two drives that failed (assuming it was only two) happened to be the<br>ones that were mirrors of each other. We've been using RAID10 for the<br>last 4 years since our last major RAID5 fiasco and have not seen such a<br>failure.</div></blockquote><div><br></div>Hehe, I've noticed that RAID level preference seems to be alot like OS preference. You are shaped by your experiences and react based on them. I don't trust RAID 10 (though I'll use it when I can't use RAID5, ie for a database server) and I've never had that much of a problem with RAID5. The closest thing was a 6TB array in which a drive died and another one was showing DRIVE-ERROR, but it made it through the first rebuild to replace the dead drive, and then through the second to replace the one showing error like a champ.</div><div><br></div><div>If I had my way, I'd yank the drives out of the servers entirely, just build a nice sexy SAN and export disks to the servers via iSCSI. But that's.... expensive.</div><div><br></div><div><blockquote type="cite"><div>-----Original Message-----<br>From: <a href="mailto:ale-bounces@ale.org">ale-bounces@ale.org</a> [<a href="mailto:ale-bounces@ale.org">mailto:ale-bounces@ale.org</a>] On Behalf Of Pat<br>Regan<br>Sent: Tuesday, January 20, 2009 2:56 AM<br>To: <a href="mailto:ale@ale.org">ale@ale.org</a><br>Subject: Re: [ale] ATL Colocation and file server suggestions<br><br>Ken Ratliff wrote:<br><blockquote type="cite"><blockquote type="cite">However, software RAID 1, 10 is excellent and performance compatible<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">with a hardware card.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I still prefer to do RAID 10 on hardware. I've found software raid to<br></blockquote><br><blockquote type="cite">be pretty finicky, drives dropping out of the array for no good <br></blockquote><blockquote type="cite">reason, and you don't notice it until the slight performance hit for <br></blockquote><blockquote type="cite">the rebuild makes you go 'hm.'<br></blockquote><br>If drives are randomly dropping out of Linux software RAID something is<br>wrong. I can only recall having two machine that had random drives<br>dropping out of MD devices. Both showed errors when running memtest86.<br> One was a bad CPU, I believe the other was bad RAM (IIRC).<br><br><blockquote type="cite">I actually don't like RAID 10 at all. I'd rather toss the 4 drives <br></blockquote><blockquote type="cite">into a RAID5 and get more space. Sure, a RAID 10 will allow you to <br></blockquote><blockquote type="cite">survive 2 dead drives, as long as it's the right 2 drives. I've seen <br></blockquote><blockquote type="cite">both drives of one mirror fail in a RAID 10 a few times, and that has<br></blockquote><br><blockquote type="cite">pretty much the same result as 2 dead drives in a RAID 5.<br></blockquote><br>Redundancy isn't the first reason to choose RAID 10 over RAID 5. If it<br>were, everyone would just choose RAID 6 since that would let you lose<br>any two drives.<br><br>RAID 5 has a terribly write performance problem. Doing a random<br>uncached write to a RAID 5 involves a read and a write to one stripe on<br>every drive.<br><br><blockquote type="cite">Software RAID1 I have no problem with though. It's quick, easy, the <br></blockquote><blockquote type="cite">performance hit is negligible unless you have something that's really<br></blockquote><br><blockquote type="cite">pounding the disk i/o and as someone else mentioned, being able to <br></blockquote><blockquote type="cite">split the mirror and use them as fully functional drives does <br></blockquote><blockquote type="cite">occasionally have it's uses.<br></blockquote><br>Hardware RAID 1 shouldn't have a write performance penalty. Software<br>RAID 1 (or 10) requires double the bus bandwidth for writes. I can't<br>speak for all implementations, but Linux MD RAID 1 spreads reads out<br>over all drives in the raid set.<br><br><blockquote type="cite">Yeah, we found out the hard way that software RAID5 is a very very bad<br></blockquote><br><blockquote type="cite">idea, especially if you're running it on a high activity web server. <br></blockquote><blockquote type="cite">After enough times of having a drive in software raid5 die before <br></blockquote><blockquote type="cite">you're done rebuilding from the previous drive failure, you kind of <br></blockquote><blockquote type="cite">learn that maybe this isn't such a good idea (or you tell the night <br></blockquote><blockquote type="cite">crew to turn apache off so that the array can rebuild in peace, but <br></blockquote><blockquote type="cite">that's not something properly spoken of in public!). The performance <br></blockquote><blockquote type="cite">hit for software RAID5 just isn't worth implementing it.<br></blockquote><br>Your slow rebuilds likely had nothing to do with the performance of<br>software RAID 5. I would imagine you needed to tweak<br>'/proc/sys/dev/raid/speed_limit_min' up from the default of 1MB/sec.<br><br>There is very little reason for a hardware controller to beat Linux MD<br>at RAID 5, especially on modern hardware. It only requires one more<br>drive worth of bus bandwidth than a hardware controller would require.<br>Processors have always been able to compute parity faster than current<br>hardware cards. dmesg on my laptop tells me that I can compute RAID 6<br>parity at 2870 megabytes per second.<br><br>I am not saying software RAID is for everyone. It has other advantages<br>besides cost, but if you have a large budget those advantages aren't as<br>useful :).<br><br><blockquote type="cite">Now with that being said, no form of RAID is truly safe. I had a <br></blockquote><blockquote type="cite">server today drop both drives in one of it's RAID1's. They were older<br></blockquote><br><blockquote type="cite">36 gig SCSI's, so it was about time anyway, but losing both of them <br></blockquote><blockquote type="cite">meant I got to spend time flattening the box and reinstalling it. This<br></blockquote><br><blockquote type="cite">is also why I try to avoid using drives from the same manufacturer and<br></blockquote><br><blockquote type="cite">batch when building arrays, as well. If you don't, you better pray to<br></blockquote><br><blockquote type="cite">god that the rebuild completes before the next one dies. It's said <br></blockquote><blockquote type="cite">that RAID is no substitute for a proper backup, and that's true. (And<br></blockquote><br><blockquote type="cite">my life being somewhat of an essay in irony, the box that dropped both<br></blockquote><br><blockquote type="cite">drives in the mirror today was being used as a backup server.)<br></blockquote><br>This paragraph reminded me of three(!) things. RAID is not a backup.<br>You know this, but lots of people don't.<br><br>Second, have you ever had the annoyance of replacing a failed drive with<br>another of the same make/model and the replacement drive is in fact<br>smaller than the failed drive? Whenever I use software RAID I sacrifice<br>a few percent off the end of the drive just to keep this from happening.<br><br>Which reminds me of another aspect of software RAID that has been<br>helpful for me in the past. Software RAID is managed at the partition<br>level. On smaller boxes I've often set up a relatively small RAID 1 or<br>10 at the front of the drives and RAID 5 on the rest.<br><br>My home media server is set up like this, and so is my person Xen host<br>that I have in a colo. I'm very budget conscious when I spend my own<br>dollars. The Xen server has an uptime of 300 days right now with no<br>RAID failures :).<br><br><blockquote type="cite">(Also, I'm not preaching at you, Jim, I'm sure you know all this crap,<br></blockquote><br><blockquote type="cite">I'm just making conversation!)<br></blockquote><br>I like conversation, and I could just as easily be on your side of this<br>one. :)<br><br><blockquote type="cite"><blockquote type="cite">RAID 1 recovery is substantially quicker and drives<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">are low cost enough to not need the N-1 space of RAID 5.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">All depends on your storage needs. We have customers with 4 TB arrays,<br></blockquote><br><blockquote type="cite">6 TB arrays, and one with an 8.1 TB array (which presents some <br></blockquote><blockquote type="cite">interesting challenges when you need to fsck the volume.... why we <br></blockquote><blockquote type="cite">used reiser for the filesystem on that array, I have no idea). Those <br></blockquote><blockquote type="cite">are a little hard to do in RAID 1 :)<br></blockquote><br>I'm up near 4 TB at home. That isn't even a big number anymore! :)<br><br>I had 4 TB back in 2001. That was mighty expensive, though, and it sure<br>wasn't a single volume. I only mention this so you don't just think I'm<br>some punk with a RAID 5 on his TV blowing smoke :)<br><br>Pat<br>----------------------------------<br>CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential information and is for the sole use of the intended recipient(s). If you are not the intended recipient, any disclosure, copying, distribution, or use of the contents of this information is prohibited and may be unlawful. If you have received this electronic transmission in error, please reply immediately to the sender that you have received the message in error, and delete it. Thank you.<br>----------------------------------<br><br>_______________________________________________<br>Ale mailing list<br><a href="mailto:Ale@ale.org">Ale@ale.org</a><br><a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br></div></blockquote></div><br></body></html>