[mirror-admin] [Fedora Infrastructure] #2878: download-i2 netapp is out of date

[Matt_Domsch at Dell.com]

download-i2 is again in sync with the master mirror and can be used by mirrors connected via Internet2 or similar.  We apologize for the inconvenience.
-Matt

--
Matt Domsch
Technology Strategist
Social Media and Community Professional
Dell | Office of the CTO

-----Original Message-----
From: mirror-list-d-bounces at redhat.com [mailto:mirror-list-d-bounces at redhat.com] On Behalf Of Domsch, Matt
Sent: Friday, July 15, 2011 1:56 PM
To: mirror-list-d at redhat.com
Subject: FW: [Fedora Infrastructure] #2878: download-i2 netapp is out of date

Be advised that download-i2 is running about 48 hours "behind" as a master mirror due to some changes on the back-end storage that is forcing a resync.  All mirrors with access to download-i2 may instead use dl.fedoraproject.org for the short term, with just a change in the host name to be accessed.

We apologize for the inconvenience.  

--
Matt Domsch
Technology Strategist
Social Media and Community Professional
Dell | Office of the CTO

-----Original Message-----
From: Will Foster
Sent: Friday, July 15, 2011 11:29 AM
To: sysadmin-web-members at fedoraproject.org
Subject: Re: [Fedora Infrastructure] #2878: download-i2 netapp is out of date

On 15/07/11 15:35 -0000, Fedora Infrastructure wrote:
>#2878: download-i2 netapp is out of date
>----------------------+------------------------------------------------
>----------------------+-----
> Reporter:  mdomsch   |       Owner:  mdomsch,sysadmin-web-members
>     Type:  bug       |      Status:  new
> Priority:  critical  |   Milestone:  Fedora 15
>Component:  Mirrors   |     Version:
> Severity:  High      |    Keywords:
>----------------------+------------------------------------------------
>----------------------+-----
> = phenomenon =
> -----Original Message-----
> From: Bill Pemberton
> Sent: Friday, July 15, 2011 8:29 AM
> To: Domsch, Matt
> Subject: Re: New I2-connected master mirrors
>
>
> It looks like download-i2.fedoraproject.org isn't getting updated.  I 
> changed my mirror to sync from it and I noticed the crawler complained 
> that I'm not up to date.  Sure enough, I can see many files in updates 
> that are on dl.fedoraproject.org but are not on 
> download-i2.fedoraproject.org.
>
> --
> Bill
>
> = reason =
>
> = recommendation =
>
>--
>Ticket URL: 
><https://fedorahosted.org/fedora-infrastructure/ticket/2878>
>Fedora Infrastructure <http://fedoraproject.org/wiki/Infrastructure>
>Fedora Infrastructure Project for Bugs, feature requests and access to our source code.

Hey guys,

There is quite a large transfer in progress from PHX2, once that completes it will be up to date.

Source                    Destination           State          Lag        Status
ntap-fedora-a:fedora_ftp  n3ardu-3:fedora_ftp       Source         44:57:03   Transferring  (441 GB done)

I believe the reason for the majority of the transfer is a 'wafl reallocate' that was being run on the source, it's basically distributing the data more evenly across the blocks - this block re-allocation is probably then included in any additional mirror/file replication to the RDU source and something I did not anticipate.  

I've paused the block re-allocation on the source so it is no longer included in the replication changeset from PHX2 --> RDU but it still looks like there is around ~500g (estimating 48hrs or less) until it's caught up.

This was something that we fired off earlier in the week to address slow rsync times on a few of the RDU nodes there.  Looking at some statistics in RDU, the disks in the new tray we added several months ago weren't sharing the read IOPS, this would alleviate that and we ran it on the source (PHX2) to provide consistency.

We can still continue the RDU block re-allocation on the RDU side, however.

-will

--