[mirror-admin] MM crawler now every 4-5 hours

[Matt Domsch]

On Fri, Mar 28, 2008 at 12:55:12PM +0100, Uwe Kiewel wrote:
> Matt Domsch schrieb:
> >I made some improvements to the MirrorManager crawler's launcher
> >yesterday, so now it constantly keeps 15 hosts being crawled in
> >parallel.  This has reduced the crawl time from about 12 hours to cover
> >everyone, to 4-5 hours.  This should mean faster update times for the
> >mirrorlist and publiclist pages - if the crawler thinks a directory
> >isn't up-to-date, wait a few hours (instead of 12) and it should
> >self-correct.  Of course, please continue to run report_mirror too.
> >
> >  
> 
> What is the difference for MM between crawling and using data from 
> report_mirror?

Trust, but verify! :-)

Without the crawler, if you've checked in, but then drop offline for
whatever reasons (temporarily or permanently), we have no way to know
this, and would still try to direct users at you.  If the crawler
can't get at you, with it's limited bandwidth and disk access (just
HTTP HEAD or FTP DIR calls, which do cause disk stat()s, but no data
transfer), then users can't get at you either.

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--