[mirror-admin] Mirror comes and goes on the mirrorlist

[Matt Domsch]

On Mon, Mar 17, 2008 at 09:05:19AM -0500, Chris Adams wrote:
> Once upon a time, Matt Domsch <Matt_Domsch at dell.com> said:
> > I just ran the crawler against your host, and for some reason it
> > stopped after scanning quite a few directories.  It made it as far as:
> > 
> > http://mirror.hiwaay.net/pub/fedora/linux/extras/6/x86_64/repoview/layout
> > 
> > but didn't complete.  I'll investigate further.
> 
> Thanks for looking at this.  Checking my log, it made some odd-looking
> requests (all of which were 404) after hitting the above directory:
> 
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/development/SRPMS/synce-serial-0.9.1-2.fc7.src.rpm HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/development/i386/yum-versionlock-1.1.3-1.fc7.noarch.rpm HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/development/ppc/yum-versionlock-1.1.3-1.fc7.noarch.rpm HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/development/x86_64/yum-versionlock-1.1.3-1.fc7.noarch.rpm HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/internet2/i2.c HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrorlists/publiclist/index.html HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrorlists/publiclist/EPEL/5/index.html HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:05 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrorlists/publiclist/Fedora/7/x86_64/index.html HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrorlists/publiclist/Fedora/9-Alpha/x86_64/index.html HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrorlists/static/css/fedora.css HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrorlists/static/images/fedora/images/line-bottom.png HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrors/samples/custom-xmlrpc-calls.py HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrors/sqlobject-history/2006-11-24/Config_postgres.sql HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/mirrors/static/images/fedora/images/line-bottom.png HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"
> 209.132.177.200 mirror.hiwaay.net - [17/Mar/2008:08:40:06 -0500] "HEAD http://mirror.hiwaay.net/pub/fedora/linux/extras/server-scripts/test/server_tester.py HTTP/1.1" 404 0 "-" "mirrormanager-crawler/0.1 (+http://hosted.fedoraproject.org/projects/mirrormanager)"


Thanks this was most helpful.

on the Infrastructure servers, the crawler moved from server app3 to
server app4 last week. However, app4 didn't have the master mirror NFS
directories moutned, like app3 did, so it was getting the list of
files in the mirrormanager application, instead of the list of files
in on the master mirrors.  Doh.  I fixed this now, and have restarted
the sync and crawler.  Should correct itself over the next few hours.

Thanks,
Matt

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--