[mirror-admin] Fedora 9 Preview syncing now

[David Timms]

Matt Domsch wrote:
> On Mon, Apr 21, 2008 at 04:26:21PM -0400, Jesse Keating wrote:
>> On Sat, 2008-04-19 at 17:11 -0400, Jesse Keating wrote:
...
> As this is the first time we've tried the Tier 0 -> 1 -> 2 mirroring,
> please pay attention to how well this works for you (delay between
> when Jesse made the bits available and when you finally had them; ease
> of getting the bits; bandwidth you were able to use in getting the
> bits).  I'd like to know that all went well for you, as well as any
> particular problems you may have encountered, so we can address them
> before the full release of F9 on 13-May.
> 
> For the iBiblio Tier 0 mirror, it was able to pull the content from
> Red Hat directly at about 100Mbit/sec, total download time was right
> at 2 hours; of the 47GB total, 33GB were new ISOs, 14GB was able to be
> hardlinked from the development/ tree so not transferred again.

Compare to this an enduser downloader - and it did take a bit of effort 
first up:
I have a rawhide server that has a heap of packages installed, probably 
1:1 within dvd.iso:everything packages. This gets yum updated everyday 
{think rawhide rsync that the mirror's do}.

0h:00 -> 4h:00 {0->3.1%} : ran the torrent for a 4 hours, and requested 
someone who has the full iso run iso-info to find the structure of the 
dvd iso. This information is stored within about the first 1MB of the iso.

4h:00 -> 4h:07 {->29.8%} : since bittorrent tries to distribute 
downloads, I was able to get only a partial directory listing. Used a 
python script I wrote last year to partially build the recently released 
Preview iso by inserting just some of the rpms I had downloaded already. 
This took about 7mins to effectively insert the correct data at the 
correct part of the iso.

4h:07 -> 8h:00 {->37.28%} : recheck and continue the torrent overnite 
{getting 10kB/sec dl, 40kB/sec ul}.

8h:00 -> 8h:10 {->62.1%} : a fedora user send me a complete iso-info 
dump {about 200kB}. inserted the already downloaded rpms into the iso image.

8h:10 -> 15h:00 {->66.7%} : recheck and continue the torrent

15h:00 -> 15h:15 {->93.2%} : wrote a script to extract the file names 
from the dvd iso-info, and convert it to an rsync include file. rsynced 
from rawhide {nearest rawhide mirror} only the items included on the 
iso. inserted those files into the image. {actually the time taken to 
run the insert is bigger each time because it reinserts all the 
pre-downloaded parts again}.

15h:15 -> 20h:00 {->100%} : recheck and continue the torrent 'til complete.

Originally, bittorrent was saying about 7days to complete the download. 
  By inserting about 71% of the size of iso {not live media} that Matt 
mentions, is it worth developing such techniques to be used at the 
mirror level, and hence save serious bandwidth for mirrors who mirror 
rawhide {I don't know the percentage who do}, or the pre-release iso 
series ?
While jigdo does this, it's metadata file is huge in comparison 250KB -> 
250 MB or more, and in my experience has been slow and sometimes fails 
to complete at all {pyjigdo}.

Could it be worked into the mirror sites normal automated processes in 
the future ?

One thing I hadn't considered is how different is an unsigned rawhide 
rpm to it's other wise identical signed release ? Does rsync make good 
use of the similarity ?

--