[mirror-admin] [mirrors #578169] Mirroring scripts?

[Ondrej Sury via RT]

I use something like this:

The launchtool is from: http://www.enricozini.org/sw/launchtool/

~~~~~~~~~~~~~~

$ crontab -l | grep fedora
# Fedora -- https://fedoraproject.org/wiki/Infrastructure/Mirroring/Tiering#Tier_1_Mirrors
0 */6 * * * runonce fedora-enchilada smartrsync dl.fedoraproject.org::fedora-enchilada /srv/mirrors/fedora/ && report_mirror --config etc/report_mirror.conf >/dev/null
0 */6 * * * runonce fedora-epel smartrsync dl.fedoraproject.org::fedora-epel /srv/mirrors/epel/ && report_mirror --config etc/report_mirror.conf > /dev/null >/dev/null
0 */6 * * * runonce fedora-archive smartrsync dl.fedoraproject.org::fedora-archive /srv/mirrors/fedora-archive/ && report_mirror --config etc/report_mirror.conf >/dev/null
0 */6 * * * runonce fedora-secondary smartrsync dl.fedoraproject.org::fedora-secondary /srv/mirrors/fedora-secondary/ && report_mirror --config etc/report_mirror.conf >/dev/null
0 */6 * * * runonce fedora-alt smartrsync dl.fedoraproject.org::fedora-alt /srv/mirrors/fedora-alt/ && report_mirror --config etc/report_mirror.conf > /dev/null

~~~~~~~~~~~~~~

$ cat /home/mirror/bin/runonce
#!/bin/bash

BASEDIR=${BASEDIR:-"${HOME}"}

# How many logfiles to keep
LOGROTATE=${LOGROTATE:-14}

# Source our common functions
. "${BASEDIR}/etc/common"

tag="$1"
shift

launchtool="launchtool --tag $tag --pidfile --piddir /home/mirror/.launchtool/ --log-child-output=file:/home/mirror/log/launchtool-$tag.log --log-child-errors=file:/home/mirror/log/launchtool-$tag.err"

$launchtool --check --log-launchtool-output=ignore --log-launchtool-errors=ignore >/dev/null
status=$?

if [ $status -eq 1 ]; then
    $launchtool -- $@
fi

savelog "/home/mirror/log/launchtool-$tag.err"
savelog "/home/mirror/log/launchtool-$tag.log"

exit $status

~~~~~~~~~~~~~~~~~~~~~~

$ cat /home/mirror/bin/smartrsync
#!/bin/bash

umask 022
cd $HOME

fatal() {
    log "$@"
    exit 1
}

warn() {
    log "$@"
}

log() {
    echo "$(date -R) smartrsync $@"
}

usage() {
    fatal "smartsync <RSYNCSOURCE> <BASEDIR>"
}

[ -z "$1" -o -z "$2" ] && usage

RSYNCSOURCE=$1
BASEDIR=$2

if [ -z "$3" ]; then
    perl -le 'sleep rand 1800'
    QUIET=--quiet
else
    shift
    shift
    QUIET=$@
fi

if [ ! -d ${BASEDIR} ]; then
    warn "${BASEDIR} does not exist yet, trying to create it..."
    mkdir -p ${BASEDIR} || fatal "Creation of ${BASEDIR} failed."
fi

failures=0

log "Sync $RSYNCSOURCE starting."

TMPDIR=$(mktemp --directory "${BASEDIR}/.~tmp~.XXXXXXXXXX")

while [ "$complete" != "true" ]; do

    if [ "$failures" -gt 0 ]; then
        sleep 5m
        if [ "$failures" -gt 3 ]; then
            fatal "Sync $RSYNCSOURCE failed."
        fi
    fi

    rsync --recursive --times --links --hard-links --perms \
        --stats ${QUIET} \
        --temp-dir="${TMPDIR}" \
        --delay-updates --numeric-ids \
        --delete --delete-delay \
        --exclude ".~tmp~*" \
        ${RSYNCSOURCE} ${BASEDIR}

    status=$?

    if [ "$status" -ne 0 ]; then
        complete="false"
        failures=$(($failures+1))
    else
        complete="true"
    fi
done

rm -rf "${TMPDIR}"

# Hardlink the new files
hardlink --respect-name --exclude '\.~tmp~.*' "${BASEDIR}"

log "Sync $RSYNCSOURCE finished."

Cheers,
--
 Ondřej Surý -- Chief Science Officer
 --------------------------------------------
 CZ.NIC, z.s.p.o.    --     Laboratoře CZ.NIC
 Milesovska 5, 130 00 Praha 3, Czech Republic
 mailto:ondrej.sury at nic.cz    https://nic.cz/
 --------------------------------------------

----- Original Message -----
> From: "mirrors" <mirrors at nic.cz>
> Sent: Wednesday, April 8, 2015 5:58:04 PM
> Subject: [mirrors #578169] Mirroring scripts?

> Wed Apr 08 17:58:03 2015: Request 578169 was acted upon.
> Transaction: Ticket created by mirror-list-d at redhat.com
>       Queue: mirrors
>     Subject: Mirroring scripts?
>       Owner: Nobody
>  Requestors: mirror-list-d at redhat.com
>      Status: new
> Ticket <URL: https://tracker.nic.cz/Ticket/Display.html?id=578169 >
> 
> 
> I'm bringing up a couple of mirrors.  They're large enough to hold all
> of fedora-buffet0 plus some other stuff, and after several days of
> downloading I have all of that content copied over.  (The machines are
> already in the ACL for that rsync module.)
> 
> I was wondering if someone could share their mirroring scripts,
> particularly how you deal with the accumulation of those .~tmp~
> directories when using the various delay options to rsync.  I can just
> cobble things together from what's on
> https://fedoraproject.org/wiki/Infrastructure/Mirroring but I'd hate to
> get it wrong.
> 
> Also, does anyone who mirrors fedora-buffet0 separately pull subtrees at
> a different rate, or do you always pull the whole thing?  It takes a
> _long_ time to generate the entire file list (> 3M files).
> 
> - J<
> 
> --

--