[mirror-admin] Having to throttle back rsync on download servers

[Carlos Carvalho]

Stephen John Smoogen (smooge at gmail.com) wrote on 27 February 2014 10:58:
 >
 >
 >
 >On 27 February 2014 06:44, Carlos Carvalho <carlos at fisica.ufpr.br> wrote:
 >
 >    If you want to experiment with what gives best performance, fine. If
 >    you just want to make sure updates work well it's better to limit
 >    access to tier 1 mirrors only. That's the whole point of tiering. I
 >    prefer this option.
 >
 >    We update every hour but check the timestamp of fullfilelist and only
 >    do the dreadful scan when it's changed, so there's no impact on the
 >    server. All mirrors that deserve access to the tier 0's should do the
 >    same...
 >
 >
 >
 >Could you share your scripts on how you do this? Every mirror has a different
 >way of doing things because they mirror so many different servers but showing
 >best practices might help others to do similar things.

Here's the skeleton of a portion of our update script, which uses
rsync exclusively.

There must be a file in the repository that's modified whenever the
repository changes. Fedora doesn't have a specific file for this like
other distros but fullfilelist can be used.

TIME_STAMP=name of the file, starting from the root of the repository
FROM=upstream site
TO=path of local repository

In bash:

cd /to/scratch/directory
export TZ=UTC

if [ $TIME_STAMP ] && [ -e $TO/$TIME_STAMP ]; then
    # use rsync without a destination to get listing only
    rsync [options] $FROM/$TIME_STAMP > time-stamp
    # parse first line to get info from time_stamp file; use date to
    # convert format to seconds from Epoch
    upstream_timestamp=$(date -d "$(awk 'BEGIN {getline; print $3 " " $4}' time-stamp)" +%s)
    # same for local version
    local_timestamp=$(stat -c %Y $TO/$TIME_STAMP)
    if (($upstream_timestamp <= $local_timestamp + 5)); then
        echo "Upstream timestamp is not more recent than here"
        echo "Aborting"
        exit 0
    fi
fi

--