[mirror-admin] Fastest way to copy data between two servers?

[James Peltier]

Neat!


--
James A. Peltier
IT Services - Research Computing Group
Simon Fraser University


On 2011-11-22, at 12:06 PM, Chris Schanzle <schanzle at nist.gov> wrote:

> On 11/22/2011 11:25 AM, James Peltier wrote:
>> Tar over a netcat pipe is very fast and I use it all the time
> 
> If the files are small, such as drpms, the netcat pipe ends up underutilized, particularly on the extracting side.  I like to use mbuffer and use a good chunk of RAM to buffer the ends of the pipe to weather the small file storms.  E.g.,
> 
> on destmachine, set up listener/extractor on port 37960 (don't forget firewall):
> 
> mbuffer -I :37960 -m 2G -s 1M | tar -C /destdir  --totals -xpSf - -b 128
> 
> 
> on sourcemachine:
> 
> tar --totals -cSb 128 --one-file-system -f - -C /srcdir . | mbuffer -m 2G -s 1M -O destmachine:37960
> 
> 
> Mbuffer also gives status output of in/out rates, total transferred, and buffer utilization.
> 
> Using '-b 128' tells tar to use 64KB read/write sizes (default=10KB) to be more friendly to common RAID block sizes and reduce system calls.  Season to taste.
> 
> --

--