[mirror-admin] Fastest way to copy data between two servers?

[Brian Long]

On 11/22/11 11:09 AM, Scott Baker wrote:
> My old Fedora mirror is going to be retired and replaced newer hardware.
> I need to copy data between the two and I'm wondering what the fastest
> way is going to be? Usually I'd just rsync + SSH between them but that's
> probably not the most efficient for 3TB of data. Then I started
> thinking, what if I NFS mounted the old one and used rsync + NFS instead.
> 
> I suppose the absolute fastest would NFS + "cp -a"?  That doesn't give
> me any recovery if it crashes. The transfer is going to take a long
> time, so I'm pretty sure I want rsync for recoverability. Any
> suggestions on best practices to move a large amount of data like this?

I tend to stick with rsync + SSH but in parallel directories.  There is
a perl script we use which allows for easy parallel forks of rsync + SSH
and keeps track of the children, etc.

I find that if I'm syncing over a WAN connection (i.e RTP to San Jose),
it's must faster to sync 10 directories simultaneously than the single
parent directory.  Has to do with TCP window size, WAN latency, etc.

I've not used tar + netcat or tar pipe over rsh in many years, but it
was always the fastest.

/Brian/
-- 
       Brian Long                             |       |
       Corporate Security Programs Org    . | | | . | | | .
                                              '       '
                                              C I S C O

--