[mirror-admin] master server sync stats and recommendations

[Axel Thimm]

On Wed, Apr 22, 2009 at 07:51:59AM -0400, Josh Boyer wrote:
> On Wed, Apr 22, 2009 at 2:01 AM, Axel Thimm <Axel.Thimm at atrpms.net> wrote:
> > Hi,
> >
> > I think two items would improve the situation a lot:
> >
> > a) Introduce a file checking method for mirrors to check whether a
> >   full rsync is needed (If it were just packages, the repomd file
> >   could be used).
> 
> Matt described fullfilelist earlier.  This is updated on any rawhide,
> updates, or epel push.  Is there a reason that isn't sufficient for
> what you describe here?  It can be found at:
> 
> http://download.fedora.redhat.com/pub/fedora/fullfilelist

This starts with
fullfilelist
linux
linux/extras
linux/extras/3
linux/extras/3/SRPMS
linux/extras/3/SRPMS/fslint-2.18-1.fc3.src.rpm
linux/extras/3/SRPMS/QuantLib-0.3.11-4.fc3.src.rpm
...

and my browser is still downloading after several minutes.

No, I was thinking more of small and simple files placed at the top of
the rsync modules like

# cat .../fedora/linux/updates/.lastupdated
Wed Apr 22 12:38:12 UTC 2009

This should reflect the date this specific module changed, no need to
mix release and updates

An rsync script could rsync just this one 29 bytes file, compare it to
the version on the mirror and only proceed when it is changed/newer
etc.

Or even compare the tier-0/1/2 master mirror's content to
download.fedora.redhat.com's to decide whether the mirror is current
or not, and use the lowest tier.

For example if the rsync script has an ordered preferred list of
master mirrors like tier-2.de.dl.fedoraproject.org,
tier-1.de.dl.fedoraproject.org, tier-0.eu.dl.fedoraproject.org, then
the script would
- get download.fedora.redhat.com's updates/.lastupdated file and
- start comparing with the mirrors from left to right, and
- the first matching one is picked for the download, if it is newer
  than what the mirror already carries.
-- 
Axel.Thimm at ATrpms.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20090422/e78704f5/attachment.bin 
-------------- next part --------------
--