[mirror-admin] rsync and updates-testing -> updates-released

[Jesse Keating]

On Wed, 2009-07-15 at 12:50 +0300, Axel Thimm wrote:
> Hi,
> 
> currently rsync does not detect file relocations/renames like the
> updates-testing to updates(-released) moves and while there are a
> couple of patches floating around (for local renames only) it is
> unlikely that there will soon be an rsync that does.
> 
> So when a package moves from updates-testing to updates, rsync
> considers this to be a new file and pulls it again, every package is
> therefore pulled by mirrors twice.
> 
> If instead of moving the files they were linked, then rsync -h would
> notice and wouldn't pull it again. The drawback is that the old file
> would have to remain in updates-testing.
> 
> Could we use a link-and-expire method instead of directly moving from
> testing->updates? If we assume the mirrors sync at least once a day,
> an expiry of 1 1/2 days for example would save half the bandwidth (of
> packages that go testing first, not security updates or
> rawhide). Since the "old" files are hardlinked they won't be consuming
> more than an extra inode.
> 
> Alternatively if this is unlikely to happen there could be a way in
> between:
> 
>    Hardlink the contents of updates-testing and updates-released into
>    a common dot-named folder. When packages move from updates-testing
>    to updates-released, rsyncs that include this folder will identify
>    that the "new" package is a hardlink to a package in this
> dot-folder
>    and simply hardlink it.
> 
> This method avoids having to set up expiry methods and would work out
> of the box with most mirrors already. It will consume inodes only but
> will cut the bandwidth of all packages moving through testing by two.

None of this will work with our current updates, as we use different GPG
signatures on the -testing updates vs the normal updates, so the files
aren't the same and cannot be hardlinked.

That said there is a proposal out to reduce this to one key per release.
It is really sad that upstream rsync can't get a feature in to handle
local copies.

Once we're using the same key for every package release we can think
about some what to make the transition from updates-testing to updates
easier.

Crazy idea, there is no reason why the packages have to live in specific
directories, only the repodata has to live there.  We could use relative
paths to a common dir of packages and just create repodata for specific
package sets for updates-testing vs updates, then the packages would
never move.  Would get around having a odd . dir with hidden files or
expiration issues.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20090715/62548fff/attachment.bin 
-------------- next part --------------
--