[mirror-admin] rsync filtering to reduce master mirror load

[Matt Domsch]

On Fri, Apr 10, 2009 at 07:31:56AM +0200, J??n ONDREJ (SAL) wrote:
> On Tue, Apr 07, 2009 at 08:31:12AM -0500, Matt Domsch wrote:
> > For example, this script:
> > 
> > #!/bin/sh
> > now=$(date -u +%s)
> > yesterday=$((now - (24 * 60 * 60)))
> > wget -O - \
> >   "https://admin.fedoraproject.org/mirrormanager/rsyncFilter?categories=Fedora%20Linux&since=$yesterday&stripprefix=pub/fedora" \
> >   2>/dev/null
> 
> Your version works too, but I have a bit updated script:
> 
> LAST_UPDATE=~/.fedora.update
> 
> # mirrormanager_exclude /tmp/rh_exclude /linux/updates
> mirrormanager_exclude() {
>   if [ -f "$LAST_UPDATE" ]; then
>     t0=`stat -c '%Y' "$LAST_UPDATE"`
>   else  
>     t0=0
>   fi
>   touch "$LAST_UPDATE"
>   URL="https://admin.fedoraproject.org/mirrormanager/rsyncFilter"
>   wget -q -O "$1" \
>     "$URL?categories=Fedora%20Linux&since=$t0&stripprefix=pub/fedora$2"
> }
> 
> This update stores last update time in a file, and do not require to set,
> how often was update started. Also URL is stripped into 2 parts to be <80
> chars long.
> 
> First parameter is an exclude file, second is which part to be excluded
> (for me fedora updates).

this is indeed an improvement, thanks.  However, the timestamp
calculation isn't quite right unless your system clock is UTC. Care
to finish that aspect? :-)

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--