[ale] SSD speeds

[Jim Kinney]

On Mon, Jul 7, 2014 at 8:53 AM, Derek Atkins <warlord at mit.edu> wrote:

> Jim Kinney <jim.kinney at gmail.com> writes:
>
> > I think all the major have a package for it but it's not an install
> option
> > for any that I use.
> > ZFS is pretty bada$$ as long are you have a boatload of RAM and lots of
> > spindles. Putting it on a 1 or 2 drive system seems rather pointless.
>  But
> > I'm still expecting to have it mainstream Linux in a year or so. The
> > de-dupe would be useful for my work but it would require 512GB RAM for
> the
> > storage size.
>
> Regular ZFS requires about 1GB RAM per TB of data.
> Adding in DeDup ups the requirement to about 5GB/TB.  :-/
>
> Do you really need DeDup?  Personally I feel I can live without it.
>

80TB of research data with much know duplication as people unzip a 3TB data
file or make copies of image files for offline work in alternate folders.
Currently testing for duplication with a home-grown tool that does a name
check plus an MD5SUM and stores in a database. If both match from elsewhere
in the NAS, flag the newest creation date as delete with soft link to
original. If the MD5SUM matches notify owner and flag to delete and replace
with a soft link. If name matches and size is similar, skip first X bytes
and rerun MD5SUM as name change can alter metadata.

So, yeah, dedupe is useful for my group but rather costly at 400GB plus OS
RAM. As such, I'm keeping the batch script as detailed above until I can
hack up a solution using block-by-block checking as in rsync. Training is
cheaper than drive space :-)

>
> -derek
> --
>        Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory
>        Member, MIT Student Information Processing Board  (SIPB)
>        URL: http://web.mit.edu/warlord/    PP-ASEL-IA     N1NWH
>        warlord at MIT.EDU                        PGP key available
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>



-- 
-- 
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you gain
at one end you lose at the other. It's like feeding a dog on his own tail.
It won't fatten the dog.
- Speech 11/23/1900 Mark Twain


*http://heretothereideas.blogspot.com/
<http://heretothereideas.blogspot.com/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20140707/97db73da/attachment.html>