[mirror-admin] master server sync stats and recommendations

[Carlos Carvalho]

Josh Boyer (jwboyer at gmail.com) wrote on 22 April 2009 07:51:
 >On Wed, Apr 22, 2009 at 2:01 AM, Axel Thimm <Axel.Thimm at atrpms.net> wrote:
 >> Hi,
 >>
 >> I think two items would improve the situation a lot:
 >>
 >> a) Introduce a file checking method for mirrors to check whether a
 >>   full rsync is needed (If it were just packages, the repomd file
 >>   could be used).
 >
 >Matt described fullfilelist earlier.  This is updated on any rawhide,
 >updates, or epel push.  Is there a reason that isn't sufficient for
 >what you describe here?

This file is not enough for updating without hammering upstream. As it
is now it only shows removed and new files but doesn't give info on
existing files that changed. If it were completed with the necessary
info it'd be possible for each mirror to download it and decide
exactly what it needs to pull.

Axel Thimm (Axel.Thimm at ATrpms.net) wrote on 22 April 2009 15:50:
 >This starts with
 >fullfilelist
 >linux
 >linux/extras
 >linux/extras/3
 >linux/extras/3/SRPMS
 >linux/extras/3/SRPMS/fslint-2.18-1.fc3.src.rpm
 >linux/extras/3/SRPMS/QuantLib-0.3.11-4.fc3.src.rpm
 >...
 >
 >and my browser is still downloading after several minutes.

You shouldn't use the browser to do this!!!

fullfilelist should be pulled by rsync, so that only changes will be
transmitted. If all mirrors used this method the file could be kept in
cache. It could also be compressed, but only with a gzip that has the
--rsyncable capability (which doesn't seem to be the case in fedora).

It'd be very useful if there was a directory where each mirror puts a
timestamp when it finishes updating. This timestamp could be used by
downstream to see if it can update and also to judge if its upstream
is up-to-date. This is used by other distros.

--