[mirror-admin] push mirroring plans

[Chuck Anderson]

On Wed, Dec 17, 2008 at 01:13:07AM -0500, Ken Tossell wrote:
> What would you all think of a system that works like this:
>  - On master updates, the master server sends an HTTP query (like  
> report_mirror) to admin.fp.org, telling MM to set all the tier-1  
> mirrors' "needs to pull" flags.
>  - Every k minutes (where k is something small, like 5-10), each tier-1 
> mirror queries MM, and if its flag is set, it syncs with the master. Once 
> it's synced, it tells MM, which resets the tier-1 flag and sets the flags 
> for that mirror's tier-2 clients.
>  - Every 3k minutes, the tier-2 mirrors follow the same process.
>
> Small updates, then, should propagate throughout tier 2 in less than an 
> hour, without MM or upstream mirrors having any access to downstream 
> hosts. The "Do I need to pull?" queries should be really easy to handle, 
> as they wouldn't require any (expensive) database access.

I like this "pull/pull" method.  I was going to suggest something 
similar using a TIMESTAMP file instead of MM, but using MM seems 
better because it avoids a situation where an upstream mirror updates 
the TIEMSTAMP before it is actually done syncing, and eliminates the 
need to treat this file specially during syncing, etc.

Getting a bit off-topic, but MM could later be extended with more 
flags than "needs to pull", such as "hardlink from src to dst path", 
"rename this directory" and other similar sets of instructions for the 
downstream mirror to follow.  This is similar to an idea I brought up 
a while ago with "sync scripts", but safer since there would be a very 
limited sets of "hints" given by MM that mirrors could choose to 
follow and how to implement on their own.

--