[mirror-admin] ideas for recognizing bitflips / dir changess

[Matt Domsch]

Problem: it can take 6+ hours for sufficient mirrors to pick up the
bitflip on release day.

Possible solutions:
1) manually bitflip mirrors.  Drawback: requires human intervention at
   the right time.

2) automatically bitflip mirrors using rsync (normal process).
   Drawback: only happens after a mirror rsyncs, which can take an
   indeterminate amount of time.

3) bitflip master mirror much earlier (e.g. 6 hours ahead of public
   announcement).  This gives most mirrors sufficient time to get an
   rsync run in.  Drawback: bit can be flowing ahead of the
   announcement.  This moves the "point of no return" back 6 hours.

4) push mirroring.  Drawback: not yet implemented.  Somewhat complex.

5) Give mirrors a way to know when a directory has changed.
5a) the rsyncFilter method exposed by MirrorManager.
5b) possible new MirrorManager method (not presently exposed, but could be fairly quickly):
  an RSS feed that lists the times the directories in a given Category have changed.  Something like:

$ wget -O - 'https://admin.fedoraproject.org/mirrormanager/directorychanges/rss2_0?since=1239015514?categories=Fedora%20Linux'


<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
<channel>
  <title>Fedora Master Mirror Directory Changes</title>
  <link>http://mirrors.fedoraproject.org</link>
  <item>
    <title>pub/fedora/linux/development/x86_64/os/repodata at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/x86_64/os/repodata ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/source/SRPMS/repodata at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/source/SRPMS/repodata ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/source/SRPMS at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/source/SRPMS ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/ppc64/os/repodata at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/ppc64/os/repodata ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/x86_64/debug/repodata at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/x86_64/debug/repodata ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/x86_64/os at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/x86_64/os ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/x86_64/debug at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/x86_64/debug ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/ppc64/os/Packages at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/ppc64/os/Packages ctime=1239015515</description>
  </item><item>
    <title>pub/fedora/linux/development/x86_64/os/Packages at 1239015515</title>
    <link />
    <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
    <description>directory=pub/fedora/linux/development/x86_64/os/Packages ctime=1239015515</description>
  </item>
</channel>
</rss>



the question becomes: would this, or something like it, be valuable?
Would you use it?  Would you prefer a different form than an RSS feed
for such?  We would need a client-side parser application, which ran
rsync against the specific recently-changed directories, so similar to
the rsyncFilter (in fact, in my quick hack it uses the same SQL
query).  But you could execute it every 30 minutes whereas you
couldn't do the same for rsync sanely.

Thoughts?

Thanks,
Matt

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--