<html><head></head><body><div>Dumb question: Why is data _stored_ on an HPC cluster? The storage for an HPC should be a separate entity entirely. It's a High Performance cluster, not a Large Storage cluster. Ideally, a complete teardown and rebuild of an HPC should have exactly zero impact on the HPC users' data. Any data kept on the local space of an HPC is purely scratch/temp data and is disposable with the possible exception of checkpoint data and that should be written back to the main storage and deleted once the full run is completed.</div><div><br></div><div>On Tue, 2015-10-27 at 08:33 -0500, Todor Fassl wrote:</div><blockquote type="cite"><pre>One of the researchers I support wants to backup 3T of data to his space
on our NAS. The data is on an HPC cluster on another network. It's not
an on-going backup. He just needs to save it to our NAS while the HPC
cluster is rebuilt. Then he'll need to copy it right back.
There is a very stable 1G connection between the 2 networks. We have
plenty of space on our NAS. What is the best way to do the caopy?
Ideally, it seems we'd want to have boththe ability to restart the copy
if it fails part way through and to end up with a compressed archive
like a tarball. Googling around tends to suggest that it's eitehr rsync
or tar. But with rsync, you wouldn't end up with a tarball. And with
tar, you can't restart it in the middle. Any other ideas?
Since the network connection is very stable, I am thinking of suggesting
tar.
tar zcvf - /datadirectory | ssh <a href="mailto:user@backup.server">user@backup.server</a> "cat > backupfile.tgz"
If the researcher would prefer his data to be copied to our NAS as
regular files, just use rsync with compression. We don't have an rsync
server that is accessible to the outside world. He could use ssh with
rsync but I could set up rsync if it would be worthwhile.
Ideas? Suggestions?
on at the far end.
He is going to need to copy the data back in a few weeks. It might even
be worthwhile to send it via tar without uncompressing/unarchiving it on
receiving end.
_______________________________________________
Ale mailing list
<a href="mailto:Ale@ale.org">Ale@ale.org</a>
<a href="http://mail.ale.org/mailman/listinfo/ale">http://mail.ale.org/mailman/listinfo/ale</a>
See JOBS, ANNOUNCE and SCHOOLS lists at
<a href="http://mail.ale.org/mailman/listinfo">http://mail.ale.org/mailman/listinfo</a>
</pre></blockquote><div class="-x-evo-signature-wrapper"><span><pre>--
James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
</pre></span></div></body></html>