[ale] Giant storage system suggestions

Thu Jul 12 00:22:45 EDT 2012

Remember those massive ibm tape arrays with the robotic arm retriever?
They're a bit before my time, but there's a marketing video on youtube
for them.

Is it possible to go the backblaze route and rsync between them two of them?

Here's how backblaze built their own 4u 67tb appliance for was 7867$,
which comes to 0.117 dollars per gigabyte:

http://archiveteam.org/index.php?title=Cheap_storage

On 7/11/12, Alex Carver <agcarver+ale at acarver.net> wrote:
> Hey Jeff,
>
> For work $0.50/GB isn't too bad.  It's a bit higher than what I priced
> out the hardware to be if I built it myself.  For home that's still a
> bit steep. :)
>
> What kind of initial scaling is available in that first investment?  The
> last time I got a commercial solution, the box was physically so tiny
> that I had to fork over another significant chunk of change if I wanted
> to add a modest increment capacity because the drive bays were full.  In
> this case the drives were 500GB drives and it was a 12 bay enclosure.
> The total net storage was only 3-4 TB.  Additionally, it turned out that
> the solution scaled very poorly.  The second unit could not be merged
> with the first without erasing the first.
> Spending another several thousand for an additional 3 TB is just not
> going to fly (especially one that ended up not working right).  Some of
> the data sets generated at work are 50-100 GB each.
>
> This is part of the reason why I was focused on rolling my own.  I could
> decide on everything from the ground up and ensure it was going to work
> without breaking the bank each time it was scaled.  I may reinvestigate
> commercial style storage again since it's been a little while since
> being burned by that last attempt but I still want to consider a
> self-assembled solution.
>
> -Alex
>
> On 7/11/2012 18:40, Jeff Layton wrote:
>> Alex,
>>
>> I work for a major vendor and we have solutions that scale larger
>> than this but I'm not going to give you a commercial or anything,
>> just some advice.
>>
>> I have friends and customers who have tried to go the homemade
>> route ala' Backblaze (sorry for those who love BB but I can tell
>> you true horror stories about it) and have lived to regret it. Just
>> grabbing a few RAID cards and some drives and slapping them
>> together doesn't really work (believe me - I've tried it myself as
>> have others). I recommend buying enterprise grade hardware, but
>> that doesn't mean it has to be expensive. You can get well under
>> $0.50/GB with 3 years of full support all the way to the file system.
>> Now sure if this meets your budget or not - it may be a bit higher
>> than you want.
>>
>> I can also point you to documentation we publish that explains
>> in gory detail how we build our solutions. All the commands and
>> configurations are published including the tuning we do. But
>> as part of this, I highly recommend XFS. We scale it to 250TB's
>> with no issue and we have a customer who's gone to 576TB's
>> for a lower performance file system.
>>
>> I also recommend getting a server with a reasonable amount
>> of memory in case you need to do an fsck. Memory always
>> helps. I would also think about getting a couple of small 15K
>> drives and running them as RAID-0 for a swap space. If the
>> file system starts and fsck and swaps (which can easily do
>> for larger file systems) you will be grateful - fsck performance
>> is much, much better and takes less time.
>>
>> If you want to go a bit cheaper, then I recommend going the
>> Gluster route. You can get it for free and it only takes a bunch
>> of servers. However, if the data is important, then build two
>> copies of the hardware and rsync between them - at least you
>> have a backup copy at some point.
>>
>> Good luck!
>>
>> Jeff
>>
>>
>>
>>
>>
>> ________________________________
>> From: Alex Carver <agcarver+ale at acarver.net>
>> To: Atlanta Linux Enthusiasts <ale at ale.org>
>> Sent: Wed, July 11, 2012 5:21:08 PM
>> Subject: Re: [ale] Giant storage system suggestions
>>
>> No, performance is not the issue, cost and scalability are the main
>> drivers.  There will be very few users of the storage (at home it would
>> just be me and a handful of computers) and at work it would be maybe
>> five to ten people at most that just want to archive large data files to
>> be recalled as needed.
>>
>> Safety is certainly important but I don't want to burn too many disks to
>> redundancy and lose storage space in the array.  I didn't plan to have
>> one monolithic RAID5 array either since that would get really slow which
>> is why I first thought of small arrays (4-8 disks per array) merged with
>> each other into a single logical volume.
>>
>> On 7/11/2012 14:12, Lightner, Jeff wrote:
>>> If you're looking at stuff on that scale is performance not an issue?
>>> There
>>> are disk arrays that can go over fibre and if it were me I'd probably be
>>> looking  at those especially if performance was a concern.
>>>
>>> RAID5 is begging for trouble - losing 2 disks in a RAID5 means the whole
>>> RAID
>>> set is kaput.  I'd recommend at least RAID6 and even better (for
>>> performance)
>>> RAID10.
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of Alex
>> Carver
>>> Sent: Wednesday, July 11, 2012 5:04 PM
>>> To: Atlanta Linux Enthusiasts
>>> Subject: [ale] Giant storage system suggestions
>>>
>>> I'm trying to design a storage system for some of my data in a way that
>>> will be
>>> useful to duplicate the design for a project at work.
>>>
>>> Digging around online it seems that a common suggestion has been a good
>>> motherboard, a SATA/SAS card, a SATA/SAS expander, and then a huge
>>> chassis to
>>> support all of the SATA drives.
>>>
>>> It looks like one of the recommended SATA/SAS cards is an LSI 9200 series
>>> card
>>> connected to an Intel RES2SV240 expander.
>>>
>>> What I'm trying to achieve is continually expandable storage space.  As
>>> more
>>> storage is required, I just keep slipping drives into the system.
>>> If I max out a case, I just add a SATA/SAS card, use external SATA/SAS
>>> cables
>>> (do those exist to go from SFF-8087 to SFF-8088?), another expander and
>>> then
>>> stretch into a new case.
>>>
>>> It's obviously going to run linux or I wouldn't be asking here. :)  The
>>> entire
>>> storage system will probably start somewhere around 10-16 TB and grow
>>> from
>>> there.  The first question would be suggestions for an optimal
>>> configuration of the disks.   For example, should the drives be grouped
>>> into say RAID-5 arrays with four devices per array and then logically
>>> combine
>>> them in software into a single storage volume?  If so, what file system
>>> will
>>> support something that could potentially reach beyond 100 TB (not that
>>> I'd reach
>>> 100 TB anytime soon but it can happen)?
>>>
>>> Thanks,
>>> _______________________________________________
>>> Ale mailing list
>>> Ale at ale.org
>>> http://mail.ale.org/mailman/listinfo/ale
>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>> http://mail.ale.org/mailman/listinfo
>>>
>>>
>>>
>>>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>

-- 
Sent from my mobile device

Looking for better conference calls? Try Uberconference:

http://uberconference.com/zevcxTpX