[mirror-admin] dl.fedoraproject.org - max connections reached

[Chuck Anderson]

The q-f-m retries are due to:

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1505) [generator=3.0.6]

which are in turn caused by errors like these:

rsync: link_stat "/fedora/linux/updates/testing/25/x86_64/w/whois-5.2.15-1.fc25.x86_64.rpm" (in fedora-buffet0) failed: No such file or directory (2)
...
removing duplicate name fedora/linux/updates/testing/26/armhfp/repodata from file list (867520)
removing duplicate name fedora/linux/updates/testing/26/x86_64 from file list (867529)
removing duplicate name fedora/linux/updates/testing/26/x86_64 from file list (867530)
removing duplicate name fedora/linux/updates/testing/26/x86_64/repodata from file list (867532)
delta-transmission enabled
...

On Wed, Mar 08, 2017 at 09:26:06AM -0500, Chuck Anderson wrote:
> It looks like q-f-m is in a loop of retries.  The first two tries got
> most of the files (754GB and 14GB) with respectable transfer rates
> (7.8MB/sec and 2.6MB/sec).  Then things went south.  I'm not sure if I
> should kill q-f-m and restart it or what.
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 900595689 bytes  received 753237293272 bytes  7830276.96 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 14450699 bytes  received 14232793225 bytes  2599150.58 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 2869733 bytes  received 41227635 bytes  12907.18 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 2869733 bytes  received 41227635 bytes  5020.48 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 2869733 bytes  received 41227635 bytes  4047.30 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 2869733 bytes  received 41227635 bytes  7598.41 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 2869733 bytes  received 41235139 bytes  11356.99 bytes/sec
> 
> >> Calling /usr/bin/rsync -aSH -f R .~tmp~ --stats --delay-updates --out-format=@ %i  %n%L -v -v --files-from=master-transferlist.sorted rsync://dl05.fedoraproject.org/fedora-buffet0/ /srv/ftp/pub/fedoraproject.org
> sent 3329550 bytes  received 54057222 bytes  17964.24 bytes/sec
> 
> On Wed, Mar 08, 2017 at 09:03:41AM -0500, Chuck Anderson wrote:
> > Due to the Fedora 26 branching last week, many mirrors are having
> > trouble completing a sync.  It was taking my mirror over 24 hours to
> > even get a file list before file transfers could begin , so I had to
> > give up with the regular rsync method and use quick-fedora-mirror.
> > That is working now, but even so it is still taking over 24 hours to
> > complete the file transfers.  I suspect that the trees are not
> > hardlinked completely and it is downloading way more than necessary.
> > 
> > On Wed, Mar 08, 2017 at 11:01:46AM +0000, HEAnet Mirror Admin wrote:
> > > Hi,
> > > 
> > > We've had trouble connecting to dl.fedoraproject.org for the last few
> > > days - it appears to be hitting the maximum number of rsync
> > > connections quite frequently.
> > > 
> > > rg

--