[mirror-admin] Breaking the 1TB limit?

[Matt Domsch]

On Tue, Feb 23, 2010 at 12:05:39PM -0800, J.H. wrote:
> Ok I'll admit I haven't read through this thread completely yet, but
> here's some things I've noticed:
> 
> http://userweb.kernel.org/~warthog9/ring-o-shame.png
> 
> (I'll apologize for the miss-alignment and such but it's the quickest
> easiest way to generate the graph)
> 
> As it stands Fedora is chewing up 16% of my primary disk space, or about
> 880G of space.

Yep.
 
> Something to consider might be pushing things to archive quicker, which
> would generally help with the disk usage as well.

There is a limiting factor there, which is we want to keep the
"current" releases on the mirrors.  We have been tardy in getting the
obsolete releases (most recently, 9 and 10) moved off to archive.
Part of that has been just people time, part of that has been getting
the storage for archive.fp.o configured properly.  9 and 10 are
getting set to get moved, to make space to land 13-Alpha, so you'll
see that shortly.

Archive content can definitely be handled by fewer mirrors.  Right now
I'm seeing 6 mirrors (really 7 as us.kernel.org is 2) carrying "Fedora
Archive" content

http://mirrors.fedoraproject.org/publiclist/Fedora/8/

but Fedora 8 is only getting about 73k unique IP addresses requesting
content a week, and even fewer requesting F8 and F9 content.  So yes,
obsolete releases can be handled on slower disks and by fewer mirrors,
without too much worry.

Alternative Spins (e.g. not the Fedora Live Desktop spin) account for
about 25% of our torrent downloads; the KDE spin alone is nearly 14%
and that's already on the master mirrors.  So it's really just the
downloads for that additional 11% (though it may increase if it's
easier to download - that's the thought at least).  I'm sure we're
going to start placing those on alt.fp.o (the "Fedora Other" category
in MM, secondary01::alt/ right now), but if there's sufficient
download traffic, and few mirrors, we may wind up moving those to the
master mirror too, which would be a step function increase in space
consumption.


-- 
Matt Domsch
Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--