[mirror-admin] metalinks (was: MirrorManager upgrades)

[Matt Domsch]

On Sun, Oct 05, 2008 at 08:24:56PM +0300, Axel Thimm wrote:
> On Sat, Oct 04, 2008 at 04:37:12PM -0500, Matt Domsch wrote:
> > FWIW, I've been working on an upgrade to mirrormanager, to allow it to
> > expose the mirrorlist as a metalink file.  Longer-term, yum is growing
> > the capability to use a metalink file for the mirrorlist, which brings
> > with it the ability to check checksums and signatures of the
> > repository files (whats in the repodata/ directories).  It's not
> > looking like that feature will make Fedora 10, but at least the MM
> > code server-side will be in place.
> > 
> > One feature of metalinks is the ability for the user's client app to
> > download multiple chunks of a large file from multiple servers in
> > parallel.  For the moment I've disabled this feature in the metalink
> > files being published, by setting maxconnections=1, and by not
> > publishing info about smaller "chunks" of the ISO files.  I know
> > "download accelerators" have been problematic for our mirrors, and I
> > don't want to exacerbate the problems that adding metalinks might
> > bring.
> 
> It is a different setup than the typical download accelerator that
> allocated several slots for the *same* peers, so we shouldn't have the
> "client monopolizes mirror" scenario anymore.
> 
> What is the plan with the preference attribute? How is this going to
> play out? For example some networks have registered their IP ranges,
> so they don't generate external traffic. Now I see that the first
> mirror has a preference of 100 and the others start at 99 dropping.

This winds up being client-specific.  Yum mirrorlists have been
prioritized for a couple releases now: it tries the first mirror
listed, then the second, in order...  The same list that had been returned
in text/plain format to mirrorlist queries will be returned in
metalink format, with preference=100 for the first listed,
preference=99 for the second, etc.  If >100 are available, the lowest N-99
on the list all wind up with preference=1.  Mirrors inside a specified
netblock will still appear at the top of the list with higher
preference than those not in the netblock.

> i wouldn't want to see this network using concurrent connections to
> the outside world to (try to) speed up things. otoh i'd love the
> feature to have stand-ins for a broken master mirror.

Each of the mirrors shows up as a <resource> in the metalink list.  I
put <resource maxconnections=1> exactly to tell the clients _not_ to
do concurrent connections.  If the clients choose to ignore that,
well, not sure what can be done.  We provide >1 mirror on the list so
there's a chance to fallback, but that same information also give the
ability for a client to ignore maxconnections=1 and try parallel
downloads.

If it winds up being seriously bad in practice, we can remove this for
ISOs again.  I'm counting on our mirrors to help provide guidance on
this.

Thanks,
Matt

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--