[mirror-admin] master server sync stats and recommendations

[Chris Adams]

Once upon a time, Matt Domsch <Matt_Domsch at dell.com> said:
> On Tue, Apr 21, 2009 at 06:31:57PM -0500, Chris Adams wrote:
> > A simple shell bit to compare your timestamp against a master and exit
> > if they are the same:
> > 
> > ########################################################################
> > MASTER=download3.fedora.redhat.com
> > LOCAL=/data/mirror
> > FILE=/pub/fedora/fullfilelist
> > 
> > mstamp=$(curl -I http://$MASTER$FILE 2> /dev/null | grep '^Last-Modified:' | cut -d' ' -f2- | tr -d '\r')
> > lstamp=$(LANG=C date -u '+%a, %d %b %Y %T GMT' -d @$(stat -c '%Y' $LOCAL$FILE))
> > [ "$mstamp" = "$lstamp" ] && exit 0
> > ########################################################################
> 
> Thanks for this!

Just as a test, I put this at the top of my enchilada and EPEL sync
scripts.  I saw the EPEL fullfilelist date change with no apparent other
file changes, so you might want to check on the generation (is there
some condition where it would put a new copy in place despite no files
being changed?).

> Would it help if I made a set of DNS entries such as:
> mirror-tier1-us-1.fedoraproject.org
> mirror-tier1-us-2.fedoraproject.org
> mirror-tier1-de-1.fedoraproject.org
> ...
> 
> and caused them to be CNAMEs to the actual Tier 1 mirrors?  Only, I
> suppose, if they all use the same rsync module definitions.

Well, they'd have to use the same rsync modules and ACLs.  Given the
"fun" we used to have with the download.fedora.redhat.com alias (where
flipping between not-quite-in-sync servers caused files to be downloaded
and deleted, back and forth), I'd probably shy away from using a CNAME
alias again.

> The fact that the master mirrors don't use the MM rsync_acl is in part
> due to the fact that it's so easy to put yourself on the list.

Yeah, I didn't think about that.

> I'm open to suggestions on how to improve this

I think you are on the right track.  I think improving the "convergence"
time is the key.  If the tier 1 mirrors are synced shortly after the
master, and tier 2 mirrors can tell when their chosen tier 1 is synced,
being a tier 2 isn't a big issue.

-- 
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

--