[ale] best way to copy 3Tb of data

[Jim Kinney]

The slowdown is the encypt/decrypt. The non-server method relies on ssh for
transport. You can also use rsh for no security at all and it will be
faster. By using the rsync server, you drop the ssh security so if the user
must enter credentials to access the NAS, you might want to double check if
the rsync credentials are sent as plain text in the same way as ftp.
On Oct 27, 2015 10:49 AM, "Todor Fassl" <fassl.tod at gmail.com> wrote:

> I know you don't need to have an rsync server to copy files via rsync but
> from what I've read, rsync protocol is way fater than ssh. And you have to
> have an rsync server to use the rsync protocol, right?
>
> On 10/27/2015 08:46 AM, Jim Kinney wrote:
>
>> Rsync doesn't require an rsync server. It provides a solid backup. Rsync
>> it back and it's all golden.
>>
>> Tarball will need enough space to be built or will need to be built
>> 'over the wire' using a tar|<transport method>|tar process. Second
>> optional.
>>
>> Tar is faster but rsync is easier.
>>
>> A 4TB external hard drive and sneaker net also works and provides
>> verifiable copies. Rsync to a local drive is fast especially with an
>> external sata port.
>>
>> On Oct 27, 2015 9:37 AM, "Todor Fassl" <fassl.tod at gmail.com
>> <mailto:fassl.tod at gmail.com>> wrote:
>>
>>     One of the researchers I support wants to backup 3T of data to his
>>     space on our NAS. The data is on an HPC cluster on another network.
>>     It's not an on-going backup. He just needs to save it to our NAS
>>     while the HPC cluster is rebuilt. Then he'll need to copy it right
>> back.
>>
>>     There is a very stable 1G connection between the 2 networks. We have
>>     plenty of space on our NAS. What is the best way to do the caopy?
>>     Ideally, it seems we'd want to have boththe ability to restart the
>>     copy if it fails part way through and to end up with a compressed
>>     archive like a tarball. Googling around tends to suggest that it's
>>     eitehr rsync or tar. But with rsync, you wouldn't end up with a
>>     tarball. And with tar, you can't restart it in the middle. Any other
>>     ideas?
>>     Since the network connection is very stable, I am thinking of
>>     suggesting tar.
>>
>>     tar zcvf - /datadirectory | ssh user at backup.server "cat >
>>     backupfile.tgz"
>>
>>     If the researcher would prefer his data to be copied to our NAS as
>>     regular files, just use rsync with compression. We don't have an
>>     rsync server that is accessible to the outside world. He could use
>>     ssh with rsync but I could set up rsync if it would be worthwhile.
>>
>>     Ideas? Suggestions?
>>
>>
>>
>>     on at the far end.
>>
>>     He is going to need to copy the data back in a few weeks. It might
>>     even be worthwhile to send it via tar without
>>     uncompressing/unarchiving it on receiving end.
>>
>>
>>
>>     _______________________________________________
>>     Ale mailing list
>>     Ale at ale.org <mailto:Ale at ale.org>
>>     http://mail.ale.org/mailman/listinfo/ale
>>     See JOBS, ANNOUNCE and SCHOOLS lists at
>>     http://mail.ale.org/mailman/listinfo
>>
>>
>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>>
>>
> --
> Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20151027/77f01c79/attachment.html>