[mirror-admin] What am I doing wrong?

[Matt Domsch]

On Tue, Oct 07, 2008 at 12:02:53PM -0700, C.M. Connelly wrote:
> "MD" == Matt Domsch <Matt_Domsch at dell.com>
> 
>     MD> I manually ran the crawler to see what it sees.  Today's
>     MD> rawhide pacakges aren't found on your server, e.g.
> 
>     MD> send: u'HEAD
>     MD> /fedora/linux/development/x86_64/os/Packages/sim-0.9.5-0.11.20080923svn2261rev.fc10.i386.rpm
>     MD> reply: 'HTTP/1.1 404 Not Found\r\n'
> 
>     MD> send: u'HEAD
>     MD> /fedora/linux/development/i386/os/Packages/xorg-x11-server-Xorg-1.5.1-6.fc10.i386.rpm
>     MD> reply: 'HTTP/1.1 404 Not Found\r\n'
> 
>     MD> So the crawler can't find these files, and marks that
>     MD> directory as not up-to-date, so you don't get returned in
>     MD> the mirrorlist.
> 
> Hmm.  We're syncing twice a day (at 6:00 AM and PM, PST), so it
> sounds like either we're syncing (from archive.linux.duke.edu)
> just before they sync, and thus missing the newer packages, or
> else that the crawler is expecting a much more frequent sync
> schedule.
> 
> We can certainly increase the frequency of our syncs or adjust
> when they happen if that would help get everything, um, in
> sync....

rawhide pushes happen between 4am and 5am Eastern US time generally.
Not sure when Duke mirrors thereafter, but 6:00 AM for you is 3-4
hours after rawhide is pushed.  And this morning you were out of
date, so it's likely Duke didn't get updated before you pulled from
them.

FWIW, ibiblio mirrors every 6 hours (0, 6, 12, 18) Eastern time.  It
generally takes them about 30 minutes to do an rsync run.

> On the web UI, I didn't mean to come off as supercritical -- I
> found things to be a bit confusing when I was initially setting
> our mirror up and when I was looking to see if there was something
> that I could tweak to make our development mirrors show up.

no offense taken - I'm hoping my rant will serve to attract people
with better web UI skills than I.
 
> The history is really helpful (and might be a nice thing to
> include in the wiki, although probably not on the ``how to set up
> a mirror'' page itself!).  We had done some mirroring earlier,
> before I had a dedicated machine, so I had used some of the other
> modules in the past.  And, yeah, I'm not sure that we didn't have
> a nonstandard layout on our mirror.  So I can totally see how the
> UI pieces would work with the more chaotic world of the
> not-so-distant past, and also how they can be confusing now that
> things have become more organized.
> 
> Finally, *thank you*, Matt -- I think you've done a great job of
> getting things working in a sensible way, and you've certainly
> made my experience of providing an official mirror a lot easier.
> As ever, there are a few bits leftover that need to be tidied up,
> but it seems like you're doing a good job with that, too, and I
> hope that my confusion and questions help you with that process
> rather than make things worse.

indeed, there's always room for improvement.  Adrian Reber has done an
outstanding job over the past few months helping new mirrors (both
public and private) get set up.  And now he's taking on coding bits of
MirrorManager too.  Anyone else I can entice?  Fame, no fortune (but
I'll buy you a beer or three at a FUDCon) await those who accept the
challenge!


-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--