[mirror-admin] Having to throttle back rsync on download servers

[Fabrice Bellet]

Hi!

On Thu, Feb 27, 2014 at 10:34:32PM -0300, Carlos Carvalho wrote:
> Stephen John Smoogen (smooge at gmail.com) wrote on 27 February 2014 10:58:
>  >
>  >
>  >
>  >On 27 February 2014 06:44, Carlos Carvalho <carlos at fisica.ufpr.br> wrote:
>  >
>  >    If you want to experiment with what gives best performance, fine. If
>  >    you just want to make sure updates work well it's better to limit
>  >    access to tier 1 mirrors only. That's the whole point of tiering. I
>  >    prefer this option.
>  >
>  >    We update every hour but check the timestamp of fullfilelist and only
>  >    do the dreadful scan when it's changed, so there's no impact on the
>  >    server. All mirrors that deserve access to the tier 0's should do the
>  >    same...
>  >
>  >
>  >
>  >Could you share your scripts on how you do this? Every mirror has a different
>  >way of doing things because they mirror so many different servers but showing
>  >best practices might help others to do similar things.
> 
> Here's the skeleton of a portion of our update script, which uses
> rsync exclusively.
> 
> There must be a file in the repository that's modified whenever the
> repository changes. Fedora doesn't have a specific file for this like
> other distros but fullfilelist can be used.
> 
> TIME_STAMP=name of the file, starting from the root of the repository
> FROM=upstream site
> TO=path of local repository
> 
> In bash:
> 
> cd /to/scratch/directory
> export TZ=UTC
> 
> if [ $TIME_STAMP ] && [ -e $TO/$TIME_STAMP ]; then
>     # use rsync without a destination to get listing only
>     rsync [options] $FROM/$TIME_STAMP > time-stamp
>     # parse first line to get info from time_stamp file; use date to
>     # convert format to seconds from Epoch
>     upstream_timestamp=$(date -d "$(awk 'BEGIN {getline; print $3 " " $4}' time-stamp)" +%s)
>     # same for local version
>     local_timestamp=$(stat -c %Y $TO/$TIME_STAMP)
>     if (($upstream_timestamp <= $local_timestamp + 5)); then
>         echo "Upstream timestamp is not more recent than here"
>         echo "Aborting"
>         exit 0
>     fi
> fi

Thanks for this script, it's very useful. By the way, it would be
great if the fedora-secondary repository also could provide a
fullfilelist timestamp.

best wishes,
-- 
fabrice

--