[mirror-admin] metalinks

[Axel Thimm]

Hi,

On Sun, Oct 05, 2008 at 12:42:27PM -0500, Matt Domsch wrote:
> On Sun, Oct 05, 2008 at 08:24:56PM +0300, Axel Thimm wrote:
> > On Sat, Oct 04, 2008 at 04:37:12PM -0500, Matt Domsch wrote:
> > > FWIW, I've been working on an upgrade to mirrormanager, to allow it to
> > > expose the mirrorlist as a metalink file.  Longer-term, yum is growing
> > > the capability to use a metalink file for the mirrorlist, which brings
> > > with it the ability to check checksums and signatures of the
> > > repository files (whats in the repodata/ directories).  It's not
> > > looking like that feature will make Fedora 10, but at least the MM
> > > code server-side will be in place.
> > > 
> > > One feature of metalinks is the ability for the user's client app to
> > > download multiple chunks of a large file from multiple servers in
> > > parallel.  For the moment I've disabled this feature in the metalink
> > > files being published, by setting maxconnections=1, and by not
> > > publishing info about smaller "chunks" of the ISO files.  I know
> > > "download accelerators" have been problematic for our mirrors, and I
> > > don't want to exacerbate the problems that adding metalinks might
> > > bring.
> > 
> > It is a different setup than the typical download accelerator that
> > allocated several slots for the *same* peers, so we shouldn't have the
> > "client monopolizes mirror" scenario anymore.
> > 
> > What is the plan with the preference attribute? How is this going to
> > play out? For example some networks have registered their IP ranges,
> > so they don't generate external traffic. Now I see that the first
> > mirror has a preference of 100 and the others start at 99 dropping.
> 
> This winds up being client-specific.  Yum mirrorlists have been
> prioritized for a couple releases now: it tries the first mirror
> listed, then the second, in order...  The same list that had been returned
> in text/plain format to mirrorlist queries will be returned in
> metalink format, with preference=100 for the first listed,
> preference=99 for the second, etc.  If >100 are available, the lowest N-99
> on the list all wind up with preference=1.  Mirrors inside a specified
> netblock will still appear at the top of the list with higher
> preference than those not in the netblock.
> 
> > i wouldn't want to see this network using concurrent connections to
> > the outside world to (try to) speed up things. otoh i'd love the
> > feature to have stand-ins for a broken master mirror.
> 
> Each of the mirrors shows up as a <resource> in the metalink list.  I
> put <resource maxconnections=1> exactly to tell the clients _not_ to
> do concurrent connections.

I understand all of the above, but the long term intention is to not
have maxconnections=1, and in principle this is a good thing.

> If the clients choose to ignore that, well, not sure what can be
> done.  We provide >1 mirror on the list so there's a chance to
> fallback, but that same information also give the ability for a
> client to ignore maxconnections=1 and try parallel downloads.
> 
> If it winds up being seriously bad in practice, we can remove this for
> ISOs again.  I'm counting on our mirrors to help provide guidance on
> this.

Well, just what happens (or is planned to happen once the bits are all
there) if you set maxconnections=5. Will it not probe the second/third
etc. mirror as well? Who (and how) is going to decide for per client
resource settings? For example net A will want just to use the traffic
cheap mirrors B and C (but allow for the failover). Net D will want to
use B and E etc. The settings need to come from MM if these are
supposed to be large nets.

What I want to point out is that there needs to be some mechanism to
tell MM what value to set for maxconnections, and that this is not
anymore a pure mirror side decision like the netblocks (actually the
netblocks weren't either).

Could MM be exanded to allow mirror admins to setup maxconnections
values depending on country/netblocks?

And, of course at the end of the day clients can override everything
like they can today as well. But I'm after the transparent no-touch
setup in my netblocks.
-- 
Axel.Thimm at ATrpms.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20081005/fc5bf9a1/attachment.bin 
-------------- next part --------------
--