[ale] best way to copy 3Tb of data

[Jim Kinney]

Dumb question: Why is data _stored_ on an HPC cluster? The storage for
an HPC should be a separate entity entirely. It's a High Performance
cluster, not a Large Storage cluster. Ideally, a complete teardown and
rebuild of an HPC should have exactly zero impact on the HPC users'
data. Any data kept on the local space of an HPC is purely scratch/temp
data and is disposable with the possible exception of checkpoint data
and that should be written back to the main storage and deleted once
the full run is completed.
On Tue, 2015-10-27 at 08:33 -0500, Todor Fassl wrote:
> One of the researchers I support wants to backup 3T of data to his
> space 
> on our NAS. The data is on an HPC cluster on another network. It's
> not 
> an on-going backup. He just needs to save it to our NAS while the HPC
> cluster is rebuilt. Then he'll need to copy it right back.
> 
> There is a very stable 1G connection between the 2 networks. We have 
> plenty of space on our NAS. What is the best way to do the caopy? 
> Ideally, it seems we'd want to have boththe ability to restart the
> copy 
> if it fails part way through and to end up with a compressed archive 
> like a tarball. Googling around tends to suggest that it's eitehr
> rsync 
> or tar. But with rsync, you wouldn't end up with a tarball. And with 
> tar, you can't restart it in the middle. Any other ideas?
> Since the network connection is very stable, I am thinking of
> suggesting 
> tar.
> 
> tar zcvf - /datadirectory | ssh user at backup.server "cat >
> backupfile.tgz"
> 
> If the researcher would prefer his data to be copied to our NAS as 
> regular files, just use rsync with compression. We don't have an
> rsync 
> server that is accessible to the outside world. He could use ssh with
> rsync but I could set up rsync if it would be worthwhile.
> 
> Ideas? Suggestions?
> 
> 
> 
> on at the far end.
> 
> He is going to need to copy the data back in a few weeks. It might
> even 
> be worthwhile to send it via tar without uncompressing/unarchiving it
> on 
> receiving end.
> 
> 
> 
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
-- 
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/ale/attachments/20151027/746ae364/attachment.html>