[mirror-admin] where are all the indices of the repository?

[Axel Thimm]

On Thu, Jul 09, 2009 at 08:23:07PM -0700, J.H. wrote:
> Carlos Carvalho wrote:
>>  >Why don't you want to use --delay-updates?
>>
>> Because of the disk hit. Fedora updates very often involve more than
>> 10,000 files, and all these renames in sequence hit the disk hard. A
>> few days ago an update of about 12,700 files took about 20min of
>> renaming, and another a few days earlier of >20,000 took more than
>> 33min.

That is just 10 renames per second! There seems to be a filesystem or
configuration problem on your server, it should scale much higher than
that.

>> During these periods the number of transactions in the disks was
>> around 98% of the maximum. Distributing the renames during the much
>> longer download time avoids these peaks.
>
> If it helps any, kernel.org doesn't use --delay-updates and really I've  
> never heard much in the way of complaints or issues with this.  I would  
> agree with the spike of disk activity, and how that can be very bad as 
> well.

A way around --delay-updates is to have a multi-pass rsync which first
transfers rpms only w/o --delete*, then transfers everything w/o
--delete* (repodata including rpms to avoid any racing between data
and metadata) and finally does a full rsync w/ --delete* options
(again full for avoiding racing problems). That's the way I used it
before rsync (on my mirror) had the delay options.

If you add the disk and network costs on master and slave you will
find that --delay-updates is much cheaper than a manual
multi-rsync. And single-pass rsyncing w/o taking into account the
racing situation will lead to unhappy users with errors like md5
mismatches or 404s of packages.

OTOH yum has become so tolerant on broken mirrors these days that the
problem of missynced mirrors is not always surfacing up.
-- 
Axel.Thimm at ATrpms.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
Url : http://mail.ale.org/pipermail/mirror-admin/attachments/20090710/3c75c432/attachment-0001.bin 
-------------- next part --------------
--