[mirror-admin] Please use --delay-updates

[J.H.]

On 04/15/2010 07:40 AM, Carlos Carvalho wrote:
> J.H. (warthog19 at eaglescrag.net) wrote on 15 April 2010 00:32:
>  >On 04/14/2010 07:44 PM, Carlos Carvalho wrote:
>  >> J.H. (warthog9 at kernel.org) wrote on 14 April 2010 11:24:
>  >>  >While I'll disagree that this makes sense from the mirror perspective
>  >> 
>  >> I think everybody agrees it's for the benefit of the user, not the
>  >> mirror :-)
>  >> 
>  >>  >one thing that should be imperative in using --delay-updates is the
>  >>  >use of --partial-dir=DIR.
>  >> 
>  >> --delay-updates implies --partial-dir=.~tmp~
>  >
>  >Yes but the problem is that in entities who are syncing from you should
>  >not be getting those .~tmp~ directories, and realistically they
>  >shouldn't be generally accessible from any mechanism, http, rsync, etc.
> 
> This is not a reason to not use delay-updates. The benefit is bigger
> than the loss. Besides, the temporary file (not the partial one) is
> always visible, no matter what options you use.

I'm not saying anything against using --delay-updates explicitly, beyond
that it's an I/O thrash.  The bigger problem with --delay-updates is the
problems it causes with having the .~tmp~ directories in the base trees.

<snip>
>  >However what I'm implicating can be accomplished in the same way with a
>  >slightly modified version of the atomic-rsync where it would do a three
>  >pass sync, the first doing a link-dest into the temporary directory,
>  >rsyncing from fedora, and link-desting back into place.
> 
> Better use the script recommended in the rsync man page...
> 
> However this process is extremely expensive compared to delay-updates.
> Then the benefit is no longer worth the cost to the mirrors.

Well there are the problems, as I continue to watch them for a lot of
other distros, when you have the .~tmp~ directories in the main path,
specifically if an rsync is running you can end up with a large number
of .~tmp~ directories in your own tree.  While this is obviously
expected when you have, mirrors or other entities syncing from you (and
keep in mind there are entities that use things like FTP to sync their
trees) you can cause three separate and problematic occurrences:
	1) You can end up with a nasty nesting of .~tmp~ directories if
	   you have multiple servers in the chain syncing simultaneously
	   (again keep in mind while the mirrors are using rsync, end
	   users aren't all doing that)
	2) You can end up in situations where files in .~tmp~ cannot be
	   deleted from your mirror.  Rsync has issues with this, why I
	   have no idea, but I can point out numerous instances where
	   if it gets synced to a client it has to be gone and manually
	   deleted
	3) You get an inconsistent view across rsync, ftp and http

So yes, some of this can be fixed with setting up appropriate excludes
in your rsyncd config, but this still leaves the inconsistent views
across ftp and http (where ftp is the only real concern).  Since Matt is
already suggesting people add things to their rsync scripts, I'm
advocating we should do full and proper atomic syncs (or as atomic as
makes sense).

- John 'Warthog9' Hawley

--