[mirror-admin] rsync targets for Fedora content please

[Matt_Domsch at Dell.com]

As the number of directories with Fedora content has exploded, in large part due to splitting the 20k files into separate [a]/ [b]/ [c]/ directories, the MirrorManager crawler has gotten slower too.  90 minutes and even 2 hours isn't sufficient anymore, especially for sites that are complete mirrors.  To try to alleviate this, I've bumped up the MM crawler timeout to 3 hours, but that's not a great solution.

If your mirror offers an rsync target, and you list your rsync URL in MirrorManager for each category, the crawler will now use that (thanks to Adrian Reber).  This can take crawls down from >2 hours to ~25 minutes, most of which time is spent hammering the MM database instead of hitting your mirror with a ton of HTTP HEAD requests.

Please consider adding an rsync module for each Category of content you carry, and add the URLs into MirrorManager.

Thanks,
Matt

--
Matt Domsch
Distinguished Engineer, Technology Strategist
Dell | Office of the CTO


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.ale.org/pipermail/mirror-admin/attachments/20130712/3a34de25/attachment.html>
-------------- next part --------------
--