[ale] ATL Colocation and file server suggestions

[Pat Regan]

Ken Ratliff wrote:
>> Hardware RAID 1 shouldn't have a write performance penalty.  Software
>> RAID 1 (or 10) requires double the bus bandwidth for writes.  I can't
>> speak for all implementations, but Linux MD RAID 1 spreads reads out
>> over all drives in the raid set.
> 
> I think you meant software RAID 1 shouldn't have a write performance  
> penalty?

I probably combined bits of my thought up there.  What I meant to
express was that RAID 1, even in software, shouldn't measurably degrade
write performance.  Linux MD RAID 1 (and since it uses the same driver,
RAID 10) can potentially double random read performance by spreading out
read requests between drives.  Good hardware controllers should do the
same for you.

I'm just trying to say that not only does a mirror not hurt your
performance, it can actually improve it.

>> Your slow rebuilds likely had nothing to do with the performance of
>> software RAID 5.  I would imagine you needed to tweak
>> '/proc/sys/dev/raid/speed_limit_min' up from the default of 1MB/sec.
>> There is very little reason for a hardware controller to beat Linux MD
>> at RAID 5, especially on modern hardware.  It only requires one more
>> drive worth of bus bandwidth than a hardware controller would require.
>> Processors have always been able to compute parity faster than current
>> hardware cards.  dmesg on my laptop tells me that I can compute RAID 6
>> parity at 2870 megabytes per second.
> 
> No the slow rebuilds usually have to do with the fact that the server  
> is live and doing well over 100mb/s of traffic, when the server is  
> under load and the CPU is pegged, the parity calc for an array  
> rebuilding doesn't help things. Of course, neither does the iowait.

This is incorrect.

If you set the speed limit cpu load won't matter.  The kernel will take
as big of a timeslice as it has to in an attempt to make sure the array
rebuilds at speed_limit_min.  It must default to 1MB/sec for historical
reasons, when disks were slower.

Of course, it can't use up any more CPU than you have disk i/o
bandwidth, so there is a ceiling on how much CPU it can soak.

>> I'm up near 4 TB at home.  That isn't even a big number anymore! :)
> 
>> I had 4 TB back in 2001.  That was mighty expensive, though, and it  
>> sure
>> wasn't a single volume.  I only mention this so you don't just think  
>> I'm
>> some punk with a RAID 5 on his TV blowing smoke :)
> 
> You've got me beat, I think I've got something in the area of 3.5 TB  
> total of local storage, with 2 of that being in an array on my file  
> server, I didn't build that until '04. Sure was fun walking into work  
> and seeing someones eyes bug out when I mentioned I had a 2 TB array  
> though. Now, even I'm disdainful of it! 'Only 2 TB? God, I'm such a  
> loser!'. As soon as I feel like  I can trust the Seagate 1.5 TB  
> drives, I'll probably be upgrading my big array though.

Back then our 4TB was a half rack of 72 gig fibre channel disks.  It
wasn't a very dense rack, though.  It allowed for a lot more cooling
space between drives than our Compaq drive enclosures did.  IIRC, the
Compaq held 15 drives in 3u.  I don't remember how much less our EMC
rack held per u.

Pat

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: OpenPGP digital signature
Url : http://mail.ale.org/pipermail/ale/attachments/20090120/a5572a74/attachment.bin