[mirror-admin] redundant transfers testing->updates

[Chris Schanzle]

On 03/09/2012 03:57 PM, Kevin Fenzi wrote:
> Note that this has not yet been implemented in bodhi (our update
> application).
>
> See:
>
> https://fedorahosted.org/bodhi/ticket/624
>
> bodhi 2.0 is being worked on now, and hopefully this will be one of the
> things in it. ;)
>
> kevin

Thank you - I didn't know that ticket existed.  It seems to define the problem/solution well.

I find it surprising that from all appearances, I'm standing fairly alone my support for this traffic reduction optimization.  This isn't the first time I'm out there by myself.  :-)

In the interim, it might be possible for high-tier mirrors (or even fedora infrastructure) to delay the deletion via:

1. get new files via rsync without --delete

2. hardlink the destination dir

3. schedule deletes to occur in the future, something like:
    rsync -n -v --delete ... | awk '/^deleting / {print $2}' | xargs rm -f -- | (cd $destdir; at now + 1 day)

Now this doesn't reduce transfers for the site *doing* the above, but anyone downstream will benefit.

I'll admit that's a lot more effort than just doing a single rsync every few hours.  And it requires two rsync passes on each run, which stinks now that fedora is over 500k files[1].  But I bet most admins have scripts that at least implement locking to avoid multiple simultaneous rsyncs and other useful things like excluding development and testing trees during prime-time.

Thanks for the thought,
Chris

[1] reducing /proc/sys/vm/vfs_cache_pressure to 10 or so made a big difference for me to prefer caching inode and dirents (random i/o) rather than the file data (mostly sequential i/o).

--