[mirror-admin] Rsync/SSH/Tar file transfers

[Pieter De Wit]

On Wed, 7 Dec 2011, Scott Baker wrote:

<snip>

> #3) Tar + Netcat
>
> Started out SMOKING fast. The disk write actually peaked ABOVE 1000Mb/s
> at the beginning! Transfer isn't as bursty, and it has averaged out
> around 90Mb/s. Transferred 15409176735 bytes in 1010 seconds, which is
> ~122Mb/s.
>
> time tar cf - 2011-12-06/ | nc client.com 7000

To avoid the bursty times, what about using "mbuffer" in the transfer. I 
use dd,nc,mbuffer when P2V'ing machines to VMWare.

Here is the command I would use for test:

Client:

time tar cf - 2011-12-06/ | mbuffer -s <tar_block_size> -m 1G | nc client.com 7000

Server:

nc -l 7000 | mbuffer -s <as_above> -m 1G > file.tar_or_where_ever

You can adjust the buffer sizes. For the P2V example, I stuck to 
RAM-256meg, since it was a SysResCD boot.

Cheers,

Pieter

--