[mirror-admin] Issues with mirror

[Chris Schanzle]

On 07/06/2015 08:16 PM, Matthew Taylor wrote:
> Hi all,
>
> Looking for some advice here. We have been running our Fedora Linux mirror (fedora.mirror.digitalpacific.com.au) for several months without any problems.. serving anywhere from 3 to 6TB per month, however as of late.. our mirror has been failing the automated crawler checks.
>
> We see this in the tail of crawler log:
>> 2015-07-06 17:01:34,080 - WARNING - Host 1997 marked not up2date: Crawler timed out before completing.  Host is likely overloaded.
>> 2015-07-06 17:01:35,373 - INFO - Ending crawl of <Host(1997 - fedora.mirror.digitalpacific.com.au)> with status 2
> Crawler log is here - http://pastebin.com/LtmRLbrd
>
> The thing is, the mirror server isn't overloaded. The server has ample resources at hand, disk IO and bandwidth is fine, and we've also tuned Apache too. The crawler sees to come from 209.132.181.102, which is only 155ms~ away (AU<-->US). Our transit and peering services are no where near their capacity.
>
> MTR is here - http://pastebin.com/yi0aeP1i (source: 101.0.101.66).
>
> We first saw the mirrormanager in our logs here:
>> 209.132.181.102 - - [07/Jul/2015:00:00:18 +1000] "HEAD /linux/ HTTP/1.1" 200 - "-" "mirrormanager-crawler/0.1 (+http://fedorahosted.org/mirrormanager)"
> And last touch here:
>> 209.132.181.102 - - [07/Jul/2015:03:00:17 +1000] "HEAD /linux/atomic/rawhide/objects/5e/54f962dcddd631e25e230db3928ad3986391c2c91a03001b0760254af19aa4.filez HTTP/1.1" 200 - "-" "mirrormanager-crawler/0.1 (+http://fedorahosted.org/mirrormanager)"
> Is it normal for the mirror to take 3 hours to fail?
>
> We are rsyncing from a tier1 mirror (rsync://mirrors.kernel.org/fedora/), and they're completing without any problems. report_mirror is also functioning fine too.
>
> Anyone who is able to shed some light on this.. it would be greatly appreciated.
>
> Thanks!
>
> Regards,
> Matthew.
>
>
> --

Perhaps related, but mirrors.kernel.org has to addresses to different machines and one of the mirror servers is out of date.  I emailed ftpadmin at kernel.org Sunday but have not gotten a response yet.

Specifically:

$ host mirrors.kernel.org
mirrors.kernel.org is an alias for mirrors.us.kernel.org.
mirrors.us.kernel.org has address 149.20.37.36
mirrors.us.kernel.org has address 198.145.20.143
mirrors.us.kernel.org has IPv6 address 2620:3:c000:a:0:1994:3:14
mirrors.us.kernel.org has IPv6 address 2001:4f8:4:6f:0:1994:3:14


Just look at the content in:

http://198.145.20.143/fedora/updates/20/i386/repodata
http://149.20.37.36/fedora/updates/20/i386/repodata

[feel free to check /22/ as well -- similar issue with 149.x being dated 11-Jun-2015.]

So basically, you have a 50/50 chance of deleting lots of current content vs re-downloading GB's of content.

I worked around this by temporarily hardcoding the 198.x address into my sync script.

Hope this helps!

--