<html><head></head><body><div>James and Ed,</div><div><br></div><div>One thing I've found is a HUGE number of bad symlinks - most are simply pointing back to a non-existent source file (user deletion likely) while a second batch has a source file tree that appears to have been moved after the links were made. Additionally, links were made (attempted) from <old dir>/foo* to ../<new dir>/foo* (literal * in the names!) in what looks like an attempt to mass link to a collection of similar named files in folders. Yeah. _that_ worked :-) NOT!</div><div><br></div><div><sigh></div><div><br></div><div>find is being quite helpful. I didn't think about the old tar | tar process. Tar may be a better way to move the actual data and the links (that are real) as it's blissfully ignorant and low level enough. Thanks!</div><div><br></div><div>Some further background: The destination is a gluster storage cluster mounted on a machine that also has 4 4TB drives attached (RAID5 backup) as the source. The glusterfs reports a zillion or so issues and all see to involve symlinks. The rsync from backup raid to new storage space reported a zillion issues with symlinks and IO errors. Of course sysadmin panic sets in with IO errors > 0 and especially > 7000! </div><div><br></div><div>As soon as I get the error numbers to 0, I can reconfig to support a third machine with the checksum bricks and add more storage overall. This is the last of the repairs from the RAID6 3-drive crash that trashed the one node (all 100+TB) last August.</div><div><br></div><div>On Thu, 2018-01-11 at 20:23 +0000, Putnam, James M. wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre> Tar (with some combination of switches) may be able to do all this for you. A
quick test would tell.
Upping the block size to some multiple of the native file system block size may
let the OS DMA directly/from to user space (at least it did in SunOS/Solaris/BSD*,
not sure if Linux does that these days) which would kill some of the tar overhead.
--
James M. Putnam
Visiting Professor of Computer Science
The air was soft, the stars so fine,
the promise of every cobbled alley so great,
that I thought I was in a dream.
________________________________________
From: Ale [<a href="mailto:ale-bounces@ale.org">ale-bounces@ale.org</a>] on behalf of Jim Kinney via Ale [<a href="mailto:ale@ale.org">ale@ale.org</a>]
Sent: Thursday, January 11, 2018 3:04 PM
To: Atlanta User Group (E-mail)
Subject: [ale] interesting problem
Imagine a giant collection of files, several TB, of unknown directory names and unknown directory depths at any point. From the top of that tree, you need to cd into EVERY directory, find the symlinks in each directory and remake them in a parallel tree on the same system but in a different starting point. Rsync is not happy with the relative links so that fails as each link looks to be relative to the location of the process running rsync.
It is possible given the source of this data tree that recursive, looping symlinks exist. That must be recreated in the new location.
It looks like a find to list all symlinks in the entire tree then cd to each final location to recreate is best. That can be sped up with running multiple processes splitting the link list into sections.
Better ideas?
--
James P. Kinney III Every time you stop a school, you will have to build a jail. What you gain at one end you lose at the other. It's like feeding a dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark Twain <a href="http://heretothereideas.blogspot.com/">http://heretothereideas.blogspot.com/</a>
</pre></blockquote><div><span><pre><pre>-- <br></pre>James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
</pre></span></div></body></html>