<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 7, 2014 at 8:53 AM, Derek Atkins <span dir="ltr"><<a href="mailto:warlord@mit.edu" target="_blank">warlord@mit.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">Jim Kinney <<a href="mailto:jim.kinney@gmail.com">jim.kinney@gmail.com</a>> writes:<br>
<br>
> I think all the major have a package for it but it's not an install option<br>
> for any that I use.<br>
> ZFS is pretty bada$$ as long are you have a boatload of RAM and lots of<br>
> spindles. Putting it on a 1 or 2 drive system seems rather pointless. But<br>
> I'm still expecting to have it mainstream Linux in a year or so. The<br>
> de-dupe would be useful for my work but it would require 512GB RAM for the<br>
> storage size.<br>
<br>
</div>Regular ZFS requires about 1GB RAM per TB of data.<br>
Adding in DeDup ups the requirement to about 5GB/TB. :-/<br>
<br>
Do you really need DeDup? Personally I feel I can live without it.<br></blockquote><div><br></div><div>80TB of research data with much know duplication as people unzip a 3TB data file or make copies of image files for offline work in alternate folders. Currently testing for duplication with a home-grown tool that does a name check plus an MD5SUM and stores in a database. If both match from elsewhere in the NAS, flag the newest creation date as delete with soft link to original. If the MD5SUM matches notify owner and flag to delete and replace with a soft link. If name matches and size is similar, skip first X bytes and rerun MD5SUM as name change can alter metadata.<br>
<br></div><div>So, yeah, dedupe is useful for my group but rather costly at 400GB plus OS RAM. As such, I'm keeping the batch script as detailed above until I can hack up a solution using block-by-block checking as in rsync. Training is cheaper than drive space :-)<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im HOEnZb"><br>
-derek<br>
--<br>
Derek Atkins, SB '93 MIT EE, SM '95 MIT Media Laboratory<br>
Member, MIT Student Information Processing Board (SIPB)<br>
URL: <a href="http://web.mit.edu/warlord/" target="_blank">http://web.mit.edu/warlord/</a> PP-ASEL-IA N1NWH<br>
<a href="mailto:warlord@MIT.EDU">warlord@MIT.EDU</a> PGP key available<br>
</div><div class="HOEnZb"><div class="h5">_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr">-- <br>James P. Kinney III<br><i><i><i><i><br></i></i></i></i>Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.<br>
- Speech 11/23/1900 Mark Twain<br><i><i><i><i><br><a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a><br></i></i></i></i></div>
</div></div>