<div dir="ltr"><div>That sucks!<br><br></div>I will need to look closely at the filesystem tool chain. Maybe there's a new flag to disable the cache to unstick a file delete.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Mon, Apr 1, 2013 at 7:14 AM, Wolf Halton <span dir="ltr"><<a href="mailto:wolf.halton@gmail.com" target="_blank">wolf.halton@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>A gotcha on ZFS:<br></div>If it ever fills up a drive array, for instance the backup process starts writing recursively, you can't remove anything, since it uses disk space to write a cache of what you delete. You have to copy the files to a larger array and reformat your stuffed ZFS array. Ask me how I know?<br>
</div><div class="gmail_extra"><div><div class="h5"><br><br><div class="gmail_quote">On Sat, Mar 30, 2013 at 1:49 AM, gcs8 <span dir="ltr"><<a href="mailto:gcsviii@gmail.com" target="_blank">gcsviii@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">David, my whole rack takes under 6 amps to power, I am pretty good at hardware design..</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Mar 30, 2013 at 12:11 AM, Jim Kinney <span dir="ltr"><<a href="mailto:jim.kinney@gmail.com" target="_blank">jim.kinney@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Heh, heh.<br><br>Informatics research eats drive space like it's free.<br><br>You should see the nifty new tape system that UPS delivered today.<br>
<br>dual LTO6 with 48 slots. It will be managed by a head end connected over 10G ethernet to second server with 2 arrays (1 has 15 2TB drive and the other is 12 2TB drives.)<br>
<br>The 16 4TB drive system actually has dual 2.5 SAS drives for OS drives plus the main 16 and a honkin' LTO6 tape system with 24 slots. This connected over a dedicated 10G ethernet to the main number cruncher with serious CPU and RAM and x16 2TB drives. There's an option to add an additional 12 drives in the back of the server.<br>
<br><a href="http://www.supermicro.com/products/chassis/3U/837/SC837E16-RJBOD1.cfm" target="_blank">http://www.supermicro.com/products/chassis/3U/837/SC837E16-RJBOD1.cfm</a><div><div><br><br><div class="gmail_quote">
On Fri, Mar 29, 2013 at 10:41 PM, dev null zero two <span dir="ltr"><<a href="mailto:dev.null.02@gmail.com" target="_blank">dev.null.02@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">rainbow tables ofc :-) </p>
<p dir="ltr">what sort of case are you using for that many hdds? </p>
<p dir="ltr">Sent from my mobile. Please excuse the brevity, spelling, and punctuation. </p><div><div>
<div class="gmail_quote">On Mar 29, 2013 10:36 PM, "David Tomaschik" <<a href="mailto:david@systemoverlord.com" target="_blank">david@systemoverlord.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I can only imagine the power bill for 20-30 hard drives & all the controllers and CPU needed to spin that all up. (And unfortunately, I have good guesses as to what its used for.)</div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Mar 29, 2013 at 3:13 PM, gcs8 <span dir="ltr"><<a href="mailto:gcsviii@gmail.com" target="_blank">gcsviii@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">i run raidz2, if i could chang anything it would be adding SSD caching and dual SSD ZIL.</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 29, 2013 at 5:07 PM, Jim Kinney <span dir="ltr"><<a href="mailto:jim.kinney@gmail.com" target="_blank">jim.kinney@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">That sounds encouraging. I'll have probably 2 arrays. One with 15 1TB data and a 2nd with 16 4TB SAS2. thinking raidz with a spare drive for each.</p>
<div><div>
<div class="gmail_quote">On Mar 29, 2013 4:43 PM, "gcs8" <<a href="mailto:gcsviii@gmail.com" target="_blank">gcsviii@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I run a 20 2TB disk array at home and have rather fond of ZFS, its been stable for me for ~3 years now.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 29, 2013 at 3:10 PM, Ed Cashin <span dir="ltr"><<a href="mailto:ecashin@noserose.net" target="_blank">ecashin@noserose.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Thanks in advance for the notes you take. I'm interested to hear what you find.<br><br></div>
By the way, work sent me to a Nexenta course in 2010 where I learned ZFS stuff from Richard Elling:<br>
<br> <a href="http://www.richardelling.com/" target="_blank">http://www.richardelling.com/</a><br><br></div>... and one thing that kept coming up is how users usually reach for the RAID-Z options when stripes of mirrors would perform better and offer more flexibility. I thought that was notable.<br>
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Fri, Mar 29, 2013 at 2:44 PM, Jim Kinney <span dir="ltr"><<a href="mailto:jim.kinney@gmail.com" target="_blank">jim.kinney@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>There was an update to this that just hit /.<br><br>WooHoo!<br><br>I have not used ZFS (minimal solaris experience) before but I have a situation where raidz looks like a very good choice. An external 16 bay JBOD box hanging off a SAS HBA looks like a prime candidate. The big issue for me is the support for native NFSv4 ACLs which solves some issues.<br>
<br>As I go through this exercise, I'll keep some notes and maybe look at a presentation on doing this.<br><br>hints, gotcha's, are very welcome.<br><br>I did find a pdf of the zfsadmin.pdf <a href="http://pdfhome.org/zfsadmin.pdf" target="_blank">http://pdfhome.org/zfsadmin.pdf</a><span><font color="#888888"><br clear="all">
<br>-- <br>-- <br>James P. Kinney III<br><i><i><i><i><br></i></i></i></i>Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.<br>
- Speech 11/23/1900 Mark Twain<br><i><i><i><i><br><a href="http://electjimkinney.org" target="_blank">http://electjimkinney.org</a><br><a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a><br>
</i></i></i></i>
</font></span><br></div></div>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><br>-- <br> Ed Cashin <<a href="mailto:ecashin@noserose.net" target="_blank">ecashin@noserose.net</a>><br> <a href="http://noserose.net/e/" target="_blank">http://noserose.net/e/</a><br>
<a href="http://www.coraid.com/" target="_blank">http://www.coraid.com/</a>
</font></span></div>
<br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Charles Selfridge<br><br>PBYC IT director<div><br></div><div><a href="tel:%28404%29%20910-3409" value="+14049103409" target="_blank">(404) 910-3409</a></div>
</div>
<br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div>
</div></div><br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Charles Selfridge<br><br>PBYC IT director<div><br></div><div><a href="tel:%28404%29%20910-3409" value="+14049103409" target="_blank">(404) 910-3409</a></div>
</div>
</div></div><br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>David Tomaschik<br>OpenPGP: 0x5DEA789B<br><a href="http://systemoverlord.com" target="_blank">http://systemoverlord.com</a><br><a href="mailto:david@systemoverlord.com" target="_blank">david@systemoverlord.com</a>
</div>
<br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div>
</div></div><br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>-- <br>James P. Kinney III<br><i><i><i><i><br></i></i></i></i>Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.<br>
- Speech 11/23/1900 Mark Twain<br><i><i><i><i><br><a href="http://electjimkinney.org" target="_blank">http://electjimkinney.org</a><br><a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a><br>
</i></i></i></i>
</div></div><br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Charles Selfridge<br><br>PBYC IT director<div><br></div><div><a href="tel:%28404%29%20910-3409" value="+14049103409" target="_blank">(404) 910-3409</a></div>
</div>
</div></div><br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br></div></div><span class="HOEnZb"><font color="#888888">Wolf Halton<br>This Apt Has Super Cow Powers - <a href="http://sourcefreedom.com" target="_blank">http://sourcefreedom.com</a><br>
Open-Source Software in Libraries - <a href="http://FOSS4Lib.org" target="_blank">http://FOSS4Lib.org</a><br>
Advancing Libraries Together - <a href="http://LYRASIS.org" target="_blank">http://LYRASIS.org</a><br>Apache Open Office Developer <a href="mailto:wolfhalton@apache.org" target="_blank">wolfhalton@apache.org</a><br>
</font></span></div>
<br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>-- <br>James P. Kinney III<br><i><i><i><i><br></i></i></i></i>Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.<br>
- Speech 11/23/1900 Mark Twain<br><i><i><i><i><br><a href="http://electjimkinney.org" target="_blank">http://electjimkinney.org</a><br><a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a><br>
</i></i></i></i>
</div>