[mirror-admin] Please use --delay-updates

[J.H.]

On 04/14/2010 07:44 PM, Carlos Carvalho wrote:
> J.H. (warthog9 at kernel.org) wrote on 14 April 2010 11:24:
>  >On 04/14/2010 07:18 AM, Matt_Domsch at Dell.com wrote:
>  >> I received a report last night from a Fedora user in Germany who experienced a weird problem.  MirrorManager would direct him to a mirror, yum would download the repomd.xml file from it and verify it was current and correct.  Yum would then go to download the repodata/*-prestodata* file noted in repomd.xml, and would fail with a HTTP 404 (not found), and then fail over to the next mirror, only to have the same thing happen, until it hit a complete mirror.
>  >> 
>  >> So two requests:
>  >> 
>  >> 1) please use the --delay-updates option to rsync.  I know it requires more disk space, but it reduces the window in which mirrors are inconsistent like this.
>  >
>  >While I'll disagree that this makes sense from the mirror perspective
> 
> I think everybody agrees it's for the benefit of the user, not the
> mirror :-)
> 
>  >one thing that should be imperative in using --delay-updates is the
>  >use of --partial-dir=DIR.
> 
> --delay-updates implies --partial-dir=.~tmp~

Yes but the problem is that in entities who are syncing from you should
not be getting those .~tmp~ directories, and realistically they
shouldn't be generally accessible from any mechanism, http, rsync, etc.

> 
>  >Specifically use a directory that isn't even visible inside your
>  >fedora directory,
> 
> Not sure what you mean. rsync will add an exclude rule for the partial
> dir automatically. If you don't want it to appear in any way (even in
> listings) you have to put it outside the repo. This means you must use
> an absolute path, otherwise it'll be taken relative to the repo. Using
> an absolute path is not feasible. All downloads would go to a single
> directory, however there are files in Fedora with the same name in
> different directories, such as bootimg.iso.

No my implication was that basically what we are after is an atomic
sync, and that while things are in flight they are never put before a
user.  I could have sworn I've used --partial-dir in this manor before
(and that it built the tree up to match the destination).

However what I'm implicating can be accomplished in the same way with a
slightly modified version of the atomic-rsync where it would do a three
pass sync, the first doing a link-dest into the temporary directory,
rsyncing from fedora, and link-desting back into place.

- John 'Warthog9' Hawley

--