[mirror-admin] rsync filtering to reduce master mirror load

[Matt Domsch]

(finishing the post)

One of the things that's bothered me for a while is that each mirror
syncs itself to it's upstream mirror (either a master, Tier 0, or Tier
1).  But in general, content on the master mirrors changes only a
few times a day (generally one rawhide push, one updates/ push, one
pub/epel/updates/testing push).  Most of the content doesn't change.
And running rsync to discover that nothing has changed is expensive on
the upstream server - millions of stat() calls.

I call these the "null rsyncs".

In the next version of MirrorManager to roll out (hopefully today if I
finish working the bugs out), the MM database now keeps track of the
"last changed time" of each directory.  Using this, it can generate an
rsync FILTER RULES file (rsync --exclude-file=<somefile>), which rsync
then uses to reduce the full directory tree traversal, and limits it
only to those directory paths that have changed.

For example, this script:

#!/bin/sh
now=$(date -u +%s)
yesterday=$((now - (24 * 60 * 60)))
wget -O - \
  "https://admin.fedoraproject.org/mirrormanager/rsyncFilter?categories=Fedora%20Linux&since=$yesterday&stripprefix=pub/fedora" \
  2>/dev/null


will return an rsync filter rules file (appended) for the
fedora-enchilada rsync module on the master mirrors.  This lists each
directory, and each of its parents specifically, which have changed
after <since> (seconds-since-the-epoch-UTC).

This wget takes about 0.4 seconds, instead of many minutes for a full
null rsync.  The goal is to run this before you kick off your actual
rsync run, using as <since> the value of the timestamp at the time you
last ran rsync.  If nothing has changed, you wind up with a filter
rules file that looks like:

# rsync --exclude-from=(this file)
# includes
# excludes
- *
# end of list

(e.g. include nothing, exclude everything).  You can test for this,
and not run rsync, or go ahead and run rsync but it exits immediately.

There are a couple points to be aware of.

1) this doesn't track status of the Tier N mirrors, only the master.
Would you find it worthwhile to have it report for the  Tier N mirrors
also?  If so, those would have to run report_mirror.

2) you could now increase your polling interval (to say every hour),
but only call rsync a couple times a day.

3) the database is updated every 15 minutes or so.  Polling more
frequently won't help.

4) the 'Fedora EPEL' category, top directory, is /pub/epel. The
infrastructure for that is on a different NetApp which takes regular
snapshots.  The snapshots are not exposed via rsync, but they are
created as subdirectories /pub/epel/.snapshot, which modifies the
ctime on /pub/epel, so it appears that /pub/epel (and only that dir)
has changed all the time.  I don't have a good solution to this right
now.

5) this could make update distribution load spike shortly after the
new content hits the masters, rather than being randomly distributed
over the day.  We'll have to watch how this works.  I'm hoping the
reduced load for the null rsyncs will in fact speed up distribution of
the changed content.

Thoughts?

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

# rsync --exclude-from=(this file)
# includes
+ linux/
+ linux/*
+ linux/development/
+ linux/development/*
+ linux/development/i386/
+ linux/development/i386/*
+ linux/development/i386/debug/
+ linux/development/i386/debug/*
+ linux/development/i386/debug/repodata/
+ linux/development/i386/debug/repodata/*
+ linux/development/i386/os/
+ linux/development/i386/os/*
+ linux/development/i386/os/EFI/
+ linux/development/i386/os/EFI/*
+ linux/development/i386/os/EFI/BOOT/
+ linux/development/i386/os/EFI/BOOT/*
+ linux/development/i386/os/Packages/
+ linux/development/i386/os/Packages/*
+ linux/development/i386/os/images/
+ linux/development/i386/os/images/*
+ linux/development/i386/os/images/pxeboot/
+ linux/development/i386/os/images/pxeboot/*
+ linux/development/i386/os/isolinux/
+ linux/development/i386/os/isolinux/*
+ linux/development/i386/os/repodata/
+ linux/development/i386/os/repodata/*
+ linux/development/ppc/
+ linux/development/ppc/*
+ linux/development/ppc/debug/
+ linux/development/ppc/debug/*
+ linux/development/ppc/debug/repodata/
+ linux/development/ppc/debug/repodata/*
+ linux/development/ppc/os/
+ linux/development/ppc/os/*
+ linux/development/ppc/os/Packages/
+ linux/development/ppc/os/Packages/*
+ linux/development/ppc/os/repodata/
+ linux/development/ppc/os/repodata/*
+ linux/development/ppc64/
+ linux/development/ppc64/*
+ linux/development/ppc64/debug/
+ linux/development/ppc64/debug/*
+ linux/development/ppc64/debug/repodata/
+ linux/development/ppc64/debug/repodata/*
+ linux/development/ppc64/os/
+ linux/development/ppc64/os/*
+ linux/development/ppc64/os/Packages/
+ linux/development/ppc64/os/Packages/*
+ linux/development/ppc64/os/repodata/
+ linux/development/ppc64/os/repodata/*
+ linux/development/source/
+ linux/development/source/*
+ linux/development/source/SRPMS/
+ linux/development/source/SRPMS/*
+ linux/development/source/SRPMS/repodata/
+ linux/development/source/SRPMS/repodata/*
+ linux/development/x86_64/
+ linux/development/x86_64/*
+ linux/development/x86_64/debug/
+ linux/development/x86_64/debug/*
+ linux/development/x86_64/debug/repodata/
+ linux/development/x86_64/debug/repodata/*
+ linux/development/x86_64/os/
+ linux/development/x86_64/os/*
+ linux/development/x86_64/os/EFI/
+ linux/development/x86_64/os/EFI/*
+ linux/development/x86_64/os/EFI/BOOT/
+ linux/development/x86_64/os/EFI/BOOT/*
+ linux/development/x86_64/os/Packages/
+ linux/development/x86_64/os/Packages/*
+ linux/development/x86_64/os/images/
+ linux/development/x86_64/os/images/*
+ linux/development/x86_64/os/images/pxeboot/
+ linux/development/x86_64/os/images/pxeboot/*
+ linux/development/x86_64/os/isolinux/
+ linux/development/x86_64/os/isolinux/*
+ linux/development/x86_64/os/repodata/
+ linux/development/x86_64/os/repodata/*
+ linux/releases/
+ linux/releases/*
+ linux/releases/test/
+ linux/releases/test/*
+ linux/releases/test/11-Beta/
+ linux/releases/test/11-Beta/*
+ linux/releases/test/11-Beta/Live/
+ linux/releases/test/11-Beta/Live/*
+ linux/releases/test/11-Beta/Live/x86_64/
+ linux/releases/test/11-Beta/Live/x86_64/*
+ linux/updates/
+ linux/updates/*
+ linux/updates/10/
+ linux/updates/10/*
+ linux/updates/10/SRPMS/
+ linux/updates/10/SRPMS/*
+ linux/updates/10/SRPMS/repodata/
+ linux/updates/10/SRPMS/repodata/*
+ linux/updates/10/i386/
+ linux/updates/10/i386/*
+ linux/updates/10/i386/debug/
+ linux/updates/10/i386/debug/*
+ linux/updates/10/i386/debug/repodata/
+ linux/updates/10/i386/debug/repodata/*
+ linux/updates/10/i386/repodata/
+ linux/updates/10/i386/repodata/*
+ linux/updates/10/ppc/
+ linux/updates/10/ppc/*
+ linux/updates/10/ppc/debug/
+ linux/updates/10/ppc/debug/*
+ linux/updates/10/ppc/debug/repodata/
+ linux/updates/10/ppc/debug/repodata/*
+ linux/updates/10/ppc/repodata/
+ linux/updates/10/ppc/repodata/*
+ linux/updates/10/ppc64/
+ linux/updates/10/ppc64/*
+ linux/updates/10/ppc64/debug/
+ linux/updates/10/ppc64/debug/*
+ linux/updates/10/ppc64/debug/repodata/
+ linux/updates/10/ppc64/debug/repodata/*
+ linux/updates/10/ppc64/repodata/
+ linux/updates/10/ppc64/repodata/*
+ linux/updates/10/x86_64/
+ linux/updates/10/x86_64/*
+ linux/updates/10/x86_64/debug/
+ linux/updates/10/x86_64/debug/*
+ linux/updates/10/x86_64/debug/repodata/
+ linux/updates/10/x86_64/debug/repodata/*
+ linux/updates/10/x86_64/repodata/
+ linux/updates/10/x86_64/repodata/*
+ linux/updates/9/
+ linux/updates/9/*
+ linux/updates/9/SRPMS.newkey/
+ linux/updates/9/SRPMS.newkey/*
+ linux/updates/9/SRPMS.newkey/repodata/
+ linux/updates/9/SRPMS.newkey/repodata/*
+ linux/updates/9/i386.newkey/
+ linux/updates/9/i386.newkey/*
+ linux/updates/9/i386.newkey/debug/
+ linux/updates/9/i386.newkey/debug/*
+ linux/updates/9/i386.newkey/debug/repodata/
+ linux/updates/9/i386.newkey/debug/repodata/*
+ linux/updates/9/i386.newkey/repodata/
+ linux/updates/9/i386.newkey/repodata/*
+ linux/updates/9/ppc.newkey/
+ linux/updates/9/ppc.newkey/*
+ linux/updates/9/ppc.newkey/debug/
+ linux/updates/9/ppc.newkey/debug/*
+ linux/updates/9/ppc.newkey/debug/repodata/
+ linux/updates/9/ppc.newkey/debug/repodata/*
+ linux/updates/9/ppc.newkey/repodata/
+ linux/updates/9/ppc.newkey/repodata/*
+ linux/updates/9/ppc64.newkey/
+ linux/updates/9/ppc64.newkey/*
+ linux/updates/9/ppc64.newkey/debug/
+ linux/updates/9/ppc64.newkey/debug/*
+ linux/updates/9/ppc64.newkey/debug/repodata/
+ linux/updates/9/ppc64.newkey/debug/repodata/*
+ linux/updates/9/ppc64.newkey/repodata/
+ linux/updates/9/ppc64.newkey/repodata/*
+ linux/updates/9/x86_64.newkey/
+ linux/updates/9/x86_64.newkey/*
+ linux/updates/9/x86_64.newkey/debug/
+ linux/updates/9/x86_64.newkey/debug/*
+ linux/updates/9/x86_64.newkey/debug/repodata/
+ linux/updates/9/x86_64.newkey/debug/repodata/*
+ linux/updates/9/x86_64.newkey/repodata/
+ linux/updates/9/x86_64.newkey/repodata/*
+ linux/updates/testing/
+ linux/updates/testing/*
+ linux/updates/testing/10/
+ linux/updates/testing/10/*
+ linux/updates/testing/10/SRPMS/
+ linux/updates/testing/10/SRPMS/*
+ linux/updates/testing/10/SRPMS/repodata/
+ linux/updates/testing/10/SRPMS/repodata/*
+ linux/updates/testing/10/i386/
+ linux/updates/testing/10/i386/*
+ linux/updates/testing/10/i386/debug/
+ linux/updates/testing/10/i386/debug/*
+ linux/updates/testing/10/i386/debug/repodata/
+ linux/updates/testing/10/i386/debug/repodata/*
+ linux/updates/testing/10/i386/repodata/
+ linux/updates/testing/10/i386/repodata/*
+ linux/updates/testing/10/ppc/
+ linux/updates/testing/10/ppc/*
+ linux/updates/testing/10/ppc/debug/
+ linux/updates/testing/10/ppc/debug/*
+ linux/updates/testing/10/ppc/debug/repodata/
+ linux/updates/testing/10/ppc/debug/repodata/*
+ linux/updates/testing/10/ppc/repodata/
+ linux/updates/testing/10/ppc/repodata/*
+ linux/updates/testing/10/ppc64/
+ linux/updates/testing/10/ppc64/*
+ linux/updates/testing/10/ppc64/debug/
+ linux/updates/testing/10/ppc64/debug/*
+ linux/updates/testing/10/ppc64/debug/repodata/
+ linux/updates/testing/10/ppc64/debug/repodata/*
+ linux/updates/testing/10/ppc64/repodata/
+ linux/updates/testing/10/ppc64/repodata/*
+ linux/updates/testing/10/x86_64/
+ linux/updates/testing/10/x86_64/*
+ linux/updates/testing/10/x86_64/debug/
+ linux/updates/testing/10/x86_64/debug/*
+ linux/updates/testing/10/x86_64/debug/repodata/
+ linux/updates/testing/10/x86_64/debug/repodata/*
+ linux/updates/testing/10/x86_64/repodata/
+ linux/updates/testing/10/x86_64/repodata/*
+ linux/updates/testing/9/
+ linux/updates/testing/9/*
+ linux/updates/testing/9/SRPMS.newkey/
+ linux/updates/testing/9/SRPMS.newkey/*
+ linux/updates/testing/9/SRPMS.newkey/repodata/
+ linux/updates/testing/9/SRPMS.newkey/repodata/*
+ linux/updates/testing/9/i386.newkey/
+ linux/updates/testing/9/i386.newkey/*
+ linux/updates/testing/9/i386.newkey/debug/
+ linux/updates/testing/9/i386.newkey/debug/*
+ linux/updates/testing/9/i386.newkey/debug/repodata/
+ linux/updates/testing/9/i386.newkey/debug/repodata/*
+ linux/updates/testing/9/i386.newkey/repodata/
+ linux/updates/testing/9/i386.newkey/repodata/*
+ linux/updates/testing/9/ppc.newkey/
+ linux/updates/testing/9/ppc.newkey/*
+ linux/updates/testing/9/ppc.newkey/debug/
+ linux/updates/testing/9/ppc.newkey/debug/*
+ linux/updates/testing/9/ppc.newkey/debug/repodata/
+ linux/updates/testing/9/ppc.newkey/debug/repodata/*
+ linux/updates/testing/9/ppc.newkey/repodata/
+ linux/updates/testing/9/ppc.newkey/repodata/*
+ linux/updates/testing/9/ppc64.newkey/
+ linux/updates/testing/9/ppc64.newkey/*
+ linux/updates/testing/9/ppc64.newkey/debug/
+ linux/updates/testing/9/ppc64.newkey/debug/*
+ linux/updates/testing/9/ppc64.newkey/debug/repodata/
+ linux/updates/testing/9/ppc64.newkey/debug/repodata/*
+ linux/updates/testing/9/ppc64.newkey/repodata/
+ linux/updates/testing/9/ppc64.newkey/repodata/*
+ linux/updates/testing/9/x86_64.newkey/
+ linux/updates/testing/9/x86_64.newkey/*
+ linux/updates/testing/9/x86_64.newkey/debug/
+ linux/updates/testing/9/x86_64.newkey/debug/*
+ linux/updates/testing/9/x86_64.newkey/debug/repodata/
+ linux/updates/testing/9/x86_64.newkey/debug/repodata/*
+ linux/updates/testing/9/x86_64.newkey/repodata/
+ linux/updates/testing/9/x86_64.newkey/repodata/*

# excludes
- *

# end of list

--