[mirror-admin] Breaking the 1TB limit?

[J.H.]

Ok I'll admit I haven't read through this thread completely yet, but
here's some things I've noticed:

http://userweb.kernel.org/~warthog9/ring-o-shame.png

(I'll apologize for the miss-alignment and such but it's the quickest
easiest way to generate the graph)

As it stands Fedora is chewing up 16% of my primary disk space, or about
880G of space.  (Mind you Mandriva is chewing up more, and that's the
next e-mail I'm sending)

On 02/22/2010 11:08 AM, Matt Domsch wrote:
> Since the dawn of Fedora, we have, through informal agreement, limited
> the amount of content that Fedora provides for mirrors to carry, to
> under 1TB.  As obsolete releases expire, they get removed (more or
> less), and as new releases are available, they get added, yet we've
> managed to stay below 1TB (though sometimes dangerously close to it).
> 
> For some content, such as spins, videos, and the like, we have set up
> a second set of content, that is not widely mirrored, but is available
> should you wish to mirror it.  This is the "alt" content on alt.fp.o.
> Likewise, as we migrate obsolete releases from the master mirror, it's
> moved to archive.fp.o, and some mirrors also carry that for us.

Something to consider might be pushing things to archive quicker, which
would generally help with the disk usage as well.

> We have over 9000 packages in the Fedora 13 repository, growing at
> over 1000/year.  We have quite a few Fedora Spins that would
> appreciate more widespread mirroring of the work they do.

How much deviation are the spins looking at?  Is jigdo good enough for
their ISO needs?  Has it been considered if jigdo is good enough for the
general case of creating the CD/DVD images? (Curious more than anything,
not explicitly recommending it but would love to hear commentary on it
if Fedora is interested in commenting)

> My question for the mirrors: at what point can we increase beyond 1TB?
> Is that a meaningful number anymore?

It's still a meaningful number, disk is cheaper but fast disk is still
expensive.  Increasing the total repository size also adds a lot more
memory pressure on systems as well, which generally means that the
hardware that's capable of keeping up with mirroring Fedora also goes up.

> In my mind, the question is two-fold:
> 
> 1) do many mirrors have more than 1TB they would be willing to use to
> host Fedora content?  If so, how many, and how much more?

Depends on how far your talking about taking this?  1.5T? 2T?  Does all
of this content needs "fast" disk?  If this is content that's similar to
the spins and what's on alt it gives the mirrors the ability to accept
that the big content can go on fast and expensive disks, while the other
content can go on things like 1-2T 7200rpm drives that are a lot cheaper.

Personally I can probably go up to about 1.5T on the "fast" disk, beyond
that I'm likely going to need to start looking at the infrastructure on
how to make things all work.

> 2) do we have the bandwidth, collectively, to mirror that much more
> content?  Due to our use of Tiering, we have done a decent job of
> getting content out to our mirrors faster than in the past.  If we are
> pushing that much more content, can we handle it?

Since we have tiering we have solved a *LOT* of our problems with
bandwidth, but your right that much more content means that much more
bandwidth load on the mirrors.  Personally I think we can handle it
collectively, assuming your not expecting 500G churn on a daily basis.

It might help us better understand the issue if you could give us a
better idea of what additional content we are talking about?  Is it just
the growth of packages?  Is there other content that will cause this
increase?

> I welcome your feedback on this.  There is no formal decision being
> made, I'm just trying to gauge the abilities of our fantastic mirrors,
> without whom we'd be sunk.

It takes a good wrangler to herd all these cats ;-)

- John 'Warthog9' Hawley

--