[mirror-admin] What am I doing wrong?

[Domsch, Matt]

On Mon, Oct 06, 2008 at 06:22:28PM -0500, C.M. Connelly wrote:
> "MD" == Matt Domsch <Matt_Domsch at dell.com>
> 
>     MD> https://admin.fedoraproject.org/mirrormanager/host_category/2626
>     MD> is your Fedora Linux category.  On this page you can add
>     MD> URLs to the pub/fedora/linux content.  Likewise for the
>     MD> other Categories you are carrying.
> 
> On a related note, our local mirror seems to be top choice on
> campus for most things, but not rawhide/development.  I don't see
> a rawhide or development option in the drop-down list on the
> add category page.

rawhide lives under the 'Fedora Linux' category.

https://admin.fedoraproject.org/mirrormanager/host_category/1983
shows that the database does not think you have very many directories
under pub/fedora/linux/development, mostly a few ppc directories.
 
> We're doing a fedora-enchilada sync, so I would have expected that
> the rawhide material would have been included in the Fedora Linux
> category.

It is part of fedora-enchilada, and I see from browsing your site that
you have the development/ directory and content.


Then:
a) report_mirror isn't reporting such.  Having fixed several things in
   this area server-side last week, I'm going to trust this is working
   fine for now...
b) the crawler can't see it  (you're a public mirror)
c) it's "yet another bug" in my code. :-)  Been finding a few of those
   this week.

I manually ran the crawler to see what it sees.  Today's rawhide
pacakges aren't found on your server, e.g.

send: u'HEAD /fedora/linux/development/x86_64/os/Packages/sim-0.9.5-0.11.20080923svn2261rev.fc10.i386.rpm
reply: 'HTTP/1.1 404 Not Found\r\n'

send: u'HEAD /fedora/linux/development/i386/os/Packages/xorg-x11-server-Xorg-1.5.1-6.fc10.i386.rpm
reply: 'HTTP/1.1 404 Not Found\r\n'

So the crawler can't find these files, and marks that directory as not
up-to-date, so you don't get returned in the mirrorlist.


> If it's not, can we maybe get a separate category and a
> note to remind folks to add the actual path to their materials?
> Or add some more information to the host page to give better
> guidance on what sorts of categories should be added?[*]
> (Even cooler would be some code that looked at what was crawled
> and suggested other categories, but I'll take what I can get.)
> 
>    Claire
> 
> [*] IMHO, the way you set up categories is confusing.  First you
>     choose a category, then you're supposed to provide the URL for
>     the upstream source you're syncing from.  Only after you click
>     the Save Host Category button do you get an option to add your
>     local URLs.

Yes, I'm not the world's best web app designer.  This was my first
turbogears app.  I'd be happy for any one to redo all of
mirrormanager/controllers.py to not suck.  Adrian Reber has been
exorcizing all the Fedoraisms from MM into separate config options, so
he can use it for rpmfusion.org.  I'd welcome further assistance fixing up
the web UI.


A little history.  For at least 8 years, there were a big list of 
rsync modules.  fedora-linux-core, fedora-linux-extras,
fedora-linux-updates fedora-linux-updates-testing, ... and so on.
There was no fedora-enchilada, so individual mirrors would create
their own directory structures, and then rsync the content from the
above modules into their directory structure willy-nilly.  Categories
were originally intended to map those lower-level rsync modules,
because then we knew if you had that Category/module, your directory
structure below that matched that of the master mirror.  But we had no
idea where that whole directory structure was being put.  Some people
would put it at /pub/fedora/linux/{core,extras}.  Others at
/fedora/{1,2,3} which was really core/.  Others at yet more random
places.  Figuring out what arbitrary subtrees on a mirror matched up
with subtrees on the master was difficult - so I punted.  I created
the Categories concept, and left it up to each mirror admin to fill it
in with their arbitrary paths to that content.

Around Fedora 7 IIRC, we added the fedora-enchilada rsync module,
which many people are using now.  This is good.  This matches
/pub/fedora and all the content below it.  The categories line up as:

'Fedora Linux' == /pub/fedora/linux  == rsync:fedora-enchilada/linux
'Fedora Web' == /pub/fedora/web      == rsync:fedora-enchilada/web

Oh, then we added EPEL, but not inside the /pub/fedora (hence
fedora-enchilada) module.  Need another Category for that then.

'Fedora EPEL' == /pub/epel 

Now we've started adding other content, not staged on the master, but
that we want to make available to mirrors.  Secondary Architectures,
OLPC builds, netbook builds and the like.  But for our purposes we
need them all to look like a uniform directory namespace, so
download.fp.o/path/to/url always works.

'Fedora Secondary Arches' == /pub/fedora-secondary
'Fedora Other' == /pub/alt


So Categories came in handy after all.


>     If you're using one of the grouped syncs (like
>     fedora-enchilada), are you supposed to use the group rsync URL
>     or are you supposed to figure out the actual upstream
>     directory and use that?  Crazy!

I should probably make that field disappear.  When I put it in, I had
hoped that people would use it, so we could create a map of:

Mirror A rsyncs from Mirror B
Mirror C rsyncs from Mirror B
...

but that hasn't really happened.  So for now just ignore this field.

Thanks,
Matt

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--