<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000'><font face="arial, helvetica, sans-serif"><span style="font-size: 12pt;">You didn't say what you liked about the tarball. Is it the compression or the one file to deal with? In case it's the compression, here are some ideas.</span></font><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;"><br></div><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;">There is a FUSE compression filesystem called fusecompress. I can just install it on my desktop (openSUSE) from my regular OS repos. You may be able to do the same, or otherwise you can get the source and some instructions here:</div><div><font face="arial, helvetica, sans-serif">https://code.google.com/p/fusecompress/wiki/Usage</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Basically, you could just create a directory on your NAS (say, mkdir /storage/datadirectory) and then type:</font></div><div><font face="arial, helvetica, sans-serif"># fusecompress /storage/datadirectory</font></div><div><font face="arial, helvetica, sans-serif">That makes a compressed filesystem (which would contain the contents of /storage/datadirectory, if there were any) and mounts it back over that spot. You could then use rsync to copy to /storage/datadirectory and get the compression advantage of tarball along with the restart advantage of rsync.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Similarly, if your underlying filesystem is already btrfs by chance, it has compression built in and you can enable it for a directory using chattr. I haven't really played with this though.</font></div><div><font face="arial, helvetica, sans-serif">https://btrfs.wiki.kernel.org/index.php/Compression</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">By the way, I've been using rsync for many years and never set up an rsync server. I always use it via ssh. </font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Scott<br></font><br><hr id="zwchr" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 12pt;"><div style="color: rgb(0, 0, 0); font-family: Helvetica, Arial, sans-serif; font-size: 12pt; font-weight: normal; font-style: normal; text-decoration: none;"><b>From: </b>"Todor Fassl" <fassl.tod@gmail.com><br><b>To: </b>"Atlanta Linux Enthusiasts" <ale@ale.org><br><b>Sent: </b>Tuesday, October 27, 2015 9:33:37 AM<br><b>Subject: </b>[ale] best way to copy 3Tb of data<br><br>One of the researchers I support wants to backup 3T of data to his space <br>on our NAS. The data is on an HPC cluster on another network. It's not <br>an on-going backup. He just needs to save it to our NAS while the HPC <br>cluster is rebuilt. Then he'll need to copy it right back.<br><br>There is a very stable 1G connection between the 2 networks. We have <br>plenty of space on our NAS. What is the best way to do the caopy? <br>Ideally, it seems we'd want to have boththe ability to restart the copy <br>if it fails part way through and to end up with a compressed archive <br>like a tarball. Googling around tends to suggest that it's eitehr rsync <br>or tar. But with rsync, you wouldn't end up with a tarball. And with <br>tar, you can't restart it in the middle. Any other ideas?<br>Since the network connection is very stable, I am thinking of suggesting <br>tar.<br><br>tar zcvf - /datadirectory | ssh user@backup.server "cat > backupfile.tgz"<br><br>If the researcher would prefer his data to be copied to our NAS as <br>regular files, just use rsync with compression. We don't have an rsync <br>server that is accessible to the outside world. He could use ssh with <br>rsync but I could set up rsync if it would be worthwhile.<br><br>Ideas? Suggestions?<br><br><br><br>on at the far end.<br><br>He is going to need to copy the data back in a few weeks. It might even <br>be worthwhile to send it via tar without uncompressing/unarchiving it on <br>receiving end.<br><br><br><br>_______________________________________________<br>Ale mailing list<br>Ale@ale.org<br>http://mail.ale.org/mailman/listinfo/ale<br>See JOBS, ANNOUNCE and SCHOOLS lists at<br>http://mail.ale.org/mailman/listinfo<br></div><br></div></div></body></html>