[mirror-admin] dl.fedoraproject.org - max connections reached

[Kevin Fenzi]

On Wed, 8 Mar 2017 11:37:29 -0500
Chuck Anderson <cra at WPI.EDU> wrote:

> On Wed, Mar 08, 2017 at 08:50:24AM -0700, Kevin Fenzi wrote:
> > On Wed, 8 Mar 2017 10:17:19 -0500 Chuck Anderson <cra at WPI.EDU>
> > wrote:  
> > > Perhaps normal, but that means q-f-m can't properly distinguish
> > > this case from an unexpected error 23, and so it gets stuck in an
> > > infinite loop.  
> > 
> > Huh, so you are seeing this just loop and loop? That is indeed
> > unexpected. 
> > 
> > Can you file a issue on it?
> > https://pagure.io/quick-fedora-mirror/issues/  
> 
> I filed a couple issues.
> 
> But there is still some other problem with the master mirror content.
> fullfiletimelist is not updated, but the files are.
> 
> Here is one example.  fullfiletimelist-fedora has these lines in it:
> 
> 1488902746      f       564
> linux/development/26/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170307.n.0.iso.manifest
> 1488902740      f       201
> linux/development/26/Everything/x86_64/iso/Fedora-Everything-26-x86_64-20170307.n.0-CHECKSUM
> 1488902802      f       465567744
> linux/development/26/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170307.n.0.iso
> 
> But rsync is saying those files don't exist:
> 
> rsync: link_stat
> "/fedora/linux/development/26/Everything/x86_64/iso/Fedora-Everything-26-x86_64-20170307.n.0-CHECKSUM" (in
> fedora-buffet0) failed: No such file or directory (2) rsync:
> link_stat
> "/fedora/linux/development/26/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170307.n.0.iso" (in
> fedora-buffet0) failed: No such file or directory (2) rsync:
> link_stat
> "/fedora/linux/development/26/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-26-20170307.n.0.iso.manifest" (in
> fedora-buffet0) failed: No such file or directory (2)
> 
> When I go to
> http://dl.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/iso/
> I see newer files:
> 
> [   ] Fedora-Everything-26-x86_64-20170308.n.0-CHECKSUM
> 2017-03-08 15:25  201 [   ]
> Fedora-Everything-netinst-x86_64-26-20170308.n.0.iso
> 2017-03-08 09:33  445M [TXT]
> Fedora-Everything-netinst-x86_64-26-20170308.n.0.iso.manifest
> 2017-03-08 12:24  564 
> 
> Why wasn't fullfiletimelist updated

It's not / cannot be 100% atomic. There's multiple things that update
content. The best we can do is update after each one, but if say: 

a fedora 25 updates push syncs out and 
a rawhide compose finishes and starts syncing out
and fedora 25 finishes and updates the fullfiletimelist, it will have
some stuff thats in rawhide, but not other things after rawhide is
still syncing. 

So, when the fullfiletimelist is out of sync like this it usually gets
corrected reasonably soon after all the things syncing finish and
update it. 

If we were doing this over from the ground up I would have seperate
modules and targets for each of these things so only one process could
be updating a tree at a time. Sadly, thats not at all what we have
currently. :( 

kevin




-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://mail.ale.org/pipermail/mirror-admin/attachments/20170308/c728c6fe/attachment.sig>
-------------- next part --------------
--