[mirror-admin] Fastest way to copy data between two servers?

[James A. Peltier]

----- Original Message -----
| On 11/22/11 11:09 AM, Scott Baker wrote:
| > My old Fedora mirror is going to be retired and replaced newer
| > hardware.
| > I need to copy data between the two and I'm wondering what the
| > fastest
| > way is going to be? Usually I'd just rsync + SSH between them but
| > that's
| > probably not the most efficient for 3TB of data. Then I started
| > thinking, what if I NFS mounted the old one and used rsync + NFS
| > instead.
| >
| > I suppose the absolute fastest would NFS + "cp -a"? That doesn't
| > give
| > me any recovery if it crashes. The transfer is going to take a long
| > time, so I'm pretty sure I want rsync for recoverability. Any
| > suggestions on best practices to move a large amount of data like
| > this?
| 
| I tend to stick with rsync + SSH but in parallel directories. There is
| a perl script we use which allows for easy parallel forks of rsync +
| SSH
| and keeps track of the children, etc.
| 
| I find that if I'm syncing over a WAN connection (i.e RTP to San
| Jose),
| it's must faster to sync 10 directories simultaneously than the single
| parent directory. Has to do with TCP window size, WAN latency, etc.
| 
| I've not used tar + netcat or tar pipe over rsh in many years, but it
| was always the fastest.
| 
| /Brian/
| --
| Brian Long | |
| Corporate Security Programs Org . | | | . | | | .
| ' '
| C I S C O
| 
| --

Because tar over netcat is a stream of data it tends to utilize the bandwidth of the network much better.  Once the original tar over netcat is completed I'll follow up with a couple of rsync's just to make sure all data was copied.  The fact that rsync copies individual files means that it tends to waste bandwidth more.  At least that's my findings. ;)

-- 
James A. Peltier
IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier at sfu.ca
Website : http://www.sfu.ca/itservices
          http://blogs.sfu.ca/people/jpeltier
I will do the best I can with the talent I have

--