[mirror-admin] Fastest way to copy data between two servers?

[Chris Schanzle]

On 11/22/2011 11:25 AM, James Peltier wrote:
> Tar over a netcat pipe is very fast and I use it all the time

If the files are small, such as drpms, the netcat pipe ends up underutilized, particularly on the extracting side.  I like to use mbuffer and use a good chunk of RAM to buffer the ends of the pipe to weather the small file storms.  E.g.,

on destmachine, set up listener/extractor on port 37960 (don't forget firewall):

mbuffer -I :37960 -m 2G -s 1M | tar -C /destdir  --totals -xpSf - -b 128


on sourcemachine:

tar --totals -cSb 128 --one-file-system -f - -C /srcdir . | mbuffer -m 2G -s 1M -O destmachine:37960


Mbuffer also gives status output of in/out rates, total transferred, and buffer utilization.

Using '-b 128' tells tar to use 64KB read/write sizes (default=10KB) to be more friendly to common RAID block sizes and reduce system calls.  Season to taste.

--