[mirror-admin] Having to throttle back rsync on download servers

[Stephen John Smoogen]

On 27 February 2014 06:44, Carlos Carvalho <carlos at fisica.ufpr.br> wrote:

> If you want to experiment with what gives best performance, fine. If
> you just want to make sure updates work well it's better to limit
> access to tier 1 mirrors only. That's the whole point of tiering. I
> prefer this option.
>
> We update every hour but check the timestamp of fullfilelist and only
> do the dreadful scan when it's changed, so there's no impact on the
> server. All mirrors that deserve access to the tier 0's should do the
> same...
>
>
Could you share your scripts on how you do this? Every mirror has a
different way of doing things because they mirror so many different servers
but showing best practices might help others to do similar things.



> Stephen John Smoogen (smooge at gmail.com) wrote on 26 February 2014 09:11:
>  >When trying to do the mirrors on local disks we got better in some
>  >stats but were running into problems with keeping up with failed
>  >disks
>
> The number of disks in the handful tier-0 mirrors is small enough
> (<~20) to be perfectly feasible to have them local.
>
>
We aren't just serving the tier 0 servers. We are the default http access
for everyone who can't find a mirror or hard codes to using us. We also
serve packages for various engineering tasks internally which makes local
storage non-scalable. We could drop rsync to only tier 0/1 mirrors.
However, from the large number of mirrors that hit us directly it would
break a lot of people and I want to make sure I announce it, double
announce it, slashdot it, etc so that when the inevitable "Fedora is
dropping mirrors" stories come out it is clear why and how mirrors not
following the tier process should work now.



>  >keeping stuff in sync (box x out of n would always not get y package
> when the
>  >others and that would be the one server every dns picks).
>
> You have to trigger them from the master to avoid updates while the
> repository is changing. If you monitor the update results from each
> mirror you can, even automatically, pull out of the dns list the one
> that didn't complete well.
>
> It IS feasible to have mirrors with local disks. Other distros do it.
>
>
I didn't say it wasn't feasible and I know other distros do it.. I worked
with the CentOS guys for a while. It just isn't how we have been set up for
8+ years so we have a lot of assumptions in place around it.



-- 
Stephen J Smoogen.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/mirror-admin/attachments/20140227/bec20745/attachment.html>
-------------- next part --------------
--