<html><head><base href="x-msg://23/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div>This is a quick a dirty /bin/sh hacked up script..<div><br></div><div><div>#!/bin/sh</div><div>#</div><div># you must hack this up to work for you.</div><div>#</div><div># data drive = /data (source of the backups)</div><div># backup drive = /backups (where the backup copy goes)</div><div>#</div><div># there is only five folders in /data and no files directly in that folder.</div><div># the folder names are "data", "apps", "contacts", "images", and "twitter_backups".</div><div>#</div><div># make sure we have destination folders....</div><div>#</div><div>if [ ! -d /backups/data ]; then</div><div> mkdir -p /backups/data</div><div>fi</div><div><br></div><div>if [ ! -d /backups/apps ]; then</div><div> mkdir -p /backups/apps</div><div>fi</div><div><br></div><div>if [ ! -d /backups/contacts ]; then</div><div> mkdir -p /backups/contacts</div><div>fi</div><div><br></div><div>if [ ! -d /backups/images ]; then</div><div> mkdir -p /backups/images</div><div>fi</div><div><br></div><div>if [ ! -d /backups/twitter_backups ]; then</div><div> mkdir -p /backups/twitter_backups</div><div>fi</div><div>#</div><div># now to do the copies. I am using cp in the example. you can use cp, tar, cpio, rsync, etc.</div><div># screen -d -m makes a temporary "daemon" out of the process. Once the process finishes, the screen daemon stops for that particular copy.</div><div>#</div><div>screen -d -m cp -r /data/data/* /backups/data/</div><div>screen -d -m cp -r /data/apps/* /backups/apps/</div><div>screen -d -m cp -r /data/contacts/* /backups/contacts/</div><div>screen -d -m cp -r /data/images/* /backups/images/</div><div>screen -d -m cp -r /data/twitter_backups/* /backups/twitter_backups/</div><div><br></div><div><br></div><div>personally I would write it in perl (I love perl) and put lots more error checking and setup checking (like to make sure the source and destination drives are mounted, etc).</div><div><br></div><div>but this gives you an idea.</div><div><br></div><div><div>On Oct 29, 2009, at 6:01 PM, <a href="mailto:mmillard1@comcast.net">mmillard1@comcast.net</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><div style="font-family: Arial; font-size: 12pt; color: rgb(0, 0, 0); ">How do I copy the files multithreaded? I've honestly never tried it.<div><br>----- Original Message -----<br>From: "scott" <<a href="mailto:scott@sboss.net">scott@sboss.net</a>><br>To: "Atlanta Linux Enthusiasts - Yes! We run Linux!" <<a href="mailto:ale@ale.org">ale@ale.org</a>><br>Sent: Thursday, October 29, 2009 5:47:47 PM GMT -05:00 US/Canada Eastern<br>Subject: Re: [ale] Disk IO Question<br><br><div><br></div>There is a few bottlenecks.. (or potential ones):<div>* PCI bus speed (the card AND the mobo bus)</div><div>* speed rates/transfer rates/seek times on the drives</div><div>* the enclosures might have a max throughput that is a bottleneck.</div><div>* cpu usage (getting cpu bound is easy to do).</div><div>* memory usage (running out of physical ram, and swapping is killer (in a bad way) in this situation).</div><div><br></div><div>also are you copying one (or very few) large files, or lots of medium sized ones? if you are copying lots of files, you can run multithreaded on the copy process to use more bandwith of the system.</div><div><br></div><div>hard to give you the smoking gun.</div><div><br></div><div>Sorry</div><div><br></div><div><div><div>On Oct 29, 2009, at 5:40 PM, Greg Clifton wrote:</div><br class="Apple-interchange-newline"><blockquote>What about bus contention. I'm not familiar with the IBM model you mentioned, but if you are running PCI cards (32bit, 33MHz) then you do have a bit of a bottleneck there, plus if there are other expansion cards in the system they can slow things down.<br>GC<br><br><br><div class="gmail_quote">On Thu, Oct 29, 2009 at 5:10 PM, James Taylor<span dir="ltr"><<a href="mailto:James.Taylor@eastcobbgroup.com" target="_blank">James.Taylor@eastcobbgroup.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); margin-top: 0pt; margin-right: 0pt; margin-bottom: 0pt; margin-left: 0.8ex; padding-left: 1ex; ">Make that 15k RPM *SAS* drives...<br><br>>>> "James Taylor" <<a href="mailto:James.Taylor@eastcobbgroup.com" target="_blank">James.Taylor@eastcobbgroup.com</a>> 10/29/2009 05:04 PM >>><br><div><div></div><div class="h5">I would suggest that your datapaths are not going to be the bottleneck with SATA drives.<br>I had a client running SATA for several years on an iSCSI appliance, and one day we hit a brick wall with the mail performance. We spent weeks trying to tune the I/O paths to deal with it, and when we looked at the actual SATA drive transfer capacity, we realized the drives were the problem.<br>We replaced the drives with 15k RPM SATA drives, and life has been good ever since.<br>Check your drive specs and see if that's where the limitation really is.<br>-jt<br><br><br>James Taylor<br>The East Cobb Group, Inc.<br>678-697-9420<br><a href="mailto:james.taylor@eastcobbgroup.com" target="_blank">james.taylor@eastcobbgroup.com</a><br><a href="http://www.eastcobbgroup.com/" target="_blank">http://www.eastcobbgroup.com</a><br><br><br><br><br>>>> <<a href="mailto:mmillard1@comcast.net" target="_blank">mmillard1@comcast.net</a>> 10/29/2009 04:49 PM >>><br><br><br>I have a Suse OES server running on an IBM 346with 2 gig of RAM. I've installed two Addonics Multilane SATA cards connect with Multilane cables to external SATA enclosures housing 4 WD Caviar Green 1.5 TB Drives.<br><br><br><br>I have about 1.8 TB of Data on one Enclosure which changes daily. My plan was to copy the 1.8 TB to the second enclosure daily and send the drives in that unit off site.<br><br><br><br>My expectation was that by having 4 independant SATA paths per unit would give me substatial performance when moving data between these units.<br><br><br><br>I'm seeing a constant speed of about 60 gig per hour. This is much slower than I expected. Have any of you done anything like this? Is this the kind of performance I should expect? Hopefull some of you bright people can share some wisdom with me.<br><br><br><br>_______________________________________________<br>Ale mailing list<br><a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br><a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>See JOBS, ANNOUNCE and SCHOOLS lists at<br><a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br><br><br><br>_______________________________________________<br>Ale mailing list<br><a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br><a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>See JOBS, ANNOUNCE and SCHOOLS lists at<br><a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br></div></div></blockquote></div><br><div id="refHTML"></div>_______________________________________________<br>Ale mailing list<br><a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br><a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br>See JOBS, ANNOUNCE and SCHOOLS lists at<br><a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br></blockquote></div><br></div><br>_______________________________________________ Ale mailing list<span class="Apple-converted-space"> </span><a href="mailto:Ale@ale.org">Ale@ale.org</a><a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><span class="Apple-converted-space"> </span>See JOBS, ANNOUNCE and SCHOOLS lists at<a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a></div></div>_______________________________________________<br>Ale mailing list<br><a href="mailto:Ale@ale.org">Ale@ale.org</a><br><a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a><br>See JOBS, ANNOUNCE and SCHOOLS lists at<br><a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a><br></div></blockquote></div><br></div></body></html>