[mirror-admin] master server sync stats and recommendations

[Chris Adams]

Once upon a time, Axel Thimm <Axel.Thimm at ATrpms.net> said:
> > As far as fetching the timestamp via HTTP HEAD: well, we are HTTP
> > mirrors after all.
> 
> Which doesn't mean that the have any http clients on them, in fact
> hardening an exposed server sometimes means removing all http/ftp
> clients.

If you have a RHEL/CentOS/Fedora/etc. server and you maintain it with
yum, you have python and python-urlgrabber installed.  The report_mirror
script is python and uses sockets.

Besides, if you don't like curl, and you have bash, you can just:

(echo -e 'HEAD /pub/epel/fullfilelist HTTP/1.0\n' 1>&0; cat -) <>/dev/tcp/mirror.hiwaay.net/80

So, an updated bash bit that first checks the timestamp and then the
contents (and updates the local timestamp if the contents didn't
change), only using bash, grep, date, cut, tr, and cat:

########################################################################
#!/bin/bash

MASTER=download3.fedora.redhat.com
LOCAL=/data/mirror
FILE=/pub/fedora/fullfilelist

wwwreq() {
	host=$1
	shift
	(echo -e "$@ HTTP/1.0\n" 1>&0; tr -d '\r') <>/dev/tcp/$host/80
}

# Check the timestamp
mstamp=$(wwwreq $MASTER "HEAD $FILE" | grep '^Last-Modified:' | cut -d' ' -f2-)
lstamp=$(LANG=C date -u '+%a, %d %b %Y %T GMT' -d @$(stat -c '%Y' $LOCAL$FILE))
[ "$mstamp" = "$lstamp" ] && exit 0

# Check the contents
headskip=0
same=0
wwwreq $MASTER "GET $FILE" | while read l; do
	if [ $headskip = 0 ]; then
		[ "$l" = "" ] && headskip=1
	else
		echo "$l"
	fi
done | diff $LOCAL$FILE - | grep -q . || same=1
if [ $same = 1 ]; then
	touch -m -d "$mstamp" $LOCAL$FILE
	exit 0
fi

# Do rsync
# ...
########################################################################

-- 
Chris Adams <cmadams at hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

--