[mirror-admin] ideas for recognizing bitflips / dir changess
cralin at gmail.com
cralin at gmail.com
Sun May 3 06:38:20 EDT 2009
On Sun, 2009-05-03 at 00:34 -0500, Matt Domsch wrote:
> Problem: it can take 6+ hours for sufficient mirrors to pick up the
> bitflip on release day.
>
> Possible solutions:
> 1) manually bitflip mirrors. Drawback: requires human intervention at
> the right time.
>
> 2) automatically bitflip mirrors using rsync (normal process).
> Drawback: only happens after a mirror rsyncs, which can take an
> indeterminate amount of time.
>
> 3) bitflip master mirror much earlier (e.g. 6 hours ahead of public
> announcement). This gives most mirrors sufficient time to get an
> rsync run in. Drawback: bit can be flowing ahead of the
> announcement. This moves the "point of no return" back 6 hours.
>
> 4) push mirroring. Drawback: not yet implemented. Somewhat complex.
>
> 5) Give mirrors a way to know when a directory has changed.
> 5a) the rsyncFilter method exposed by MirrorManager.
> 5b) possible new MirrorManager method (not presently exposed, but could be fairly quickly):
> an RSS feed that lists the times the directories in a given Category have changed. Something like:
>
> $ wget -O - 'https://admin.fedoraproject.org/mirrormanager/directorychanges/rss2_0?since=1239015514?categories=Fedora%20Linux'
>
>
> <?xml version="1.0" encoding="utf-8"?>
> <rss version="2.0">
> <channel>
> <title>Fedora Master Mirror Directory Changes</title>
> <link>http://mirrors.fedoraproject.org</link>
> <item>
> <title>pub/fedora/linux/development/x86_64/os/repodata at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/x86_64/os/repodata ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/source/SRPMS/repodata at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/source/SRPMS/repodata ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/source/SRPMS at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/source/SRPMS ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/ppc64/os/repodata at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/ppc64/os/repodata ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/x86_64/debug/repodata at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/x86_64/debug/repodata ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/x86_64/os at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/x86_64/os ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/x86_64/debug at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/x86_64/debug ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/ppc64/os/Packages at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/ppc64/os/Packages ctime=1239015515</description>
> </item><item>
> <title>pub/fedora/linux/development/x86_64/os/Packages at 1239015515</title>
> <link />
> <pubDate>Mon, 06 Apr 2009 10:58:35 GMT</pubDate>
> <description>directory=pub/fedora/linux/development/x86_64/os/Packages ctime=1239015515</description>
> </item>
> </channel>
> </rss>
>
>
>
> the question becomes: would this, or something like it, be valuable?
> Would you use it? Would you prefer a different form than an RSS feed
> for such? We would need a client-side parser application, which ran
> rsync against the specific recently-changed directories, so similar to
> the rsyncFilter (in fact, in my quick hack it uses the same SQL
> query). But you could execute it every 30 minutes whereas you
> couldn't do the same for rsync sanely.
>
> Thoughts?
>
> Thanks,
> Matt
>
Looking at the "man tree" comand I can generate a list of dirs with
their permission and the date of last modification like this:
# tree -fdDpix --noreport linux
linux
[drwxr-xr-x Mar 19 18:36] linux/releases
[drwxr-xr-x Nov 20 19:37] linux/releases/10
[drwxr-xr-x Nov 19 20:11] linux/releases/10/Everything
[drwxr-xr-x Nov 19 20:11] linux/releases/10/Everything/i386
[drwxr-xr-x Nov 19 23:03] linux/releases/10/Everything/i386/os
[drwxr-xr-x Feb 1 19:06] linux/releases/10/Everything/i386/os/Packages
[drwxr-xr-x Jan 31 22:57] linux/releases/10/Everything/i386/os/repodata
[drwxr-xr-x Nov 19 20:11] linux/releases/10/Everything/x86_64
[drwxr-xr-x Nov 20 0:38] linux/releases/10/Everything/x86_64/os
[drwxr-xr-x Jan 31 22:51]
linux/releases/10/Everything/x86_64/os/Packages
[drwxr-xr-x Nov 20 19:15]
linux/releases/10/Everything/x86_64/os/repodata
[drwxr-xr-x Nov 18 23:31] linux/releases/10/Fedora
[drwxr-xr-x Nov 20 2:53] linux/releases/10/Fedora/i386
[drwxr-xr-x Nov 20 2:56] linux/releases/10/Fedora/i386/iso
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/i386/os
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/i386/os/Packages
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/i386/os/images
[drwxr-xr-x Jan 31 22:50]
linux/releases/10/Fedora/i386/os/images/pxeboot
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/i386/os/isolinux
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/i386/os/repodata
[drwxr-xr-x Nov 20 1:35] linux/releases/10/Fedora/i386/os/repoview
[drwxr-xr-x Jan 31 22:50]
linux/releases/10/Fedora/i386/os/repoview/layout
[drwxr-xr-x Nov 20 4:04] linux/releases/10/Fedora/x86_64
[drwxr-xr-x Nov 20 4:04] linux/releases/10/Fedora/x86_64/iso
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/x86_64/os
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/x86_64/os/Packages
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/x86_64/os/images
[drwxr-xr-x Jan 31 22:50]
linux/releases/10/Fedora/x86_64/os/images/pxeboot
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/x86_64/os/isolinux
[drwxr-xr-x Jan 31 22:50] linux/releases/10/Fedora/x86_64/os/repodata
[drwxr-xr-x Nov 20 3:03] linux/releases/10/Fedora/x86_64/os/repoview
[drwxr-xr-x Jan 31 22:50]
linux/releases/10/Fedora/x86_64/os/repoview/layout
[drwxrwsr-x Mar 19 18:43] linux/updates
[drwxr-xr-x May 2 15:58] linux/updates/10
[drwxr-xr-x May 2 16:22] linux/updates/10/i386
[drwxr-xr-x May 2 16:05] linux/updates/10/i386/debug
[drwxr-xr-x May 2 16:22] linux/updates/10/i386/repodata
[drwxr-xr-x May 2 16:51] linux/updates/10/x86_64
[drwxr-xr-x May 2 16:06] linux/updates/10/x86_64/debug
[drwxr-xr-x May 2 16:51] linux/updates/10/x86_64/repodata
This is similar with fullfilelist.txt from master but it does contain
additional infos about permissions and date of last change of each dir
in the tree.
I would assume it's not a very difficult task to compare this list
(rsync-ed from master) and compare it against the same command executed
on the local data in order to detect what dirs have been changed on the
master.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.ale.org/pipermail/mirror-admin/attachments/20090503/17e1f839/attachment-0001.html
-------------- next part --------------
--
More information about the Mirror-admin
mailing list