[mirror-admin] push mirroring - who owns the SSH keys?

[Matt Domsch]

I'm starting to think again about push mirroring, with an eye to
having something in place for general use by Fedora 12.  Anyone who
would care to help would be greatly appreciated.

In the grand scheme, I envision:

1.  rel-eng posts new content to the master
2.  rel-eng waits for new content to finish replicating to all the
    masters (for the curious, that's currently done via a NetApp
    SnapMirror).
3.  rel-eng informs MirrorManager that new content is ready.
    Hopefully with directory paths included (so as to not need to
    rescan the whole server)

For N in [0 1 2]:

  4.  MM informs each Tier N mirror that new content is ready.
  5.  Each Tier N mirrors download the new content.
  6.  Each Tier N mirror runs report_mirror to inform MM that it has the
      new content.

For this to work, MM is going to need:
a) To know the tiering hierarchy (who pulls from whom).  MM has a
   field for this today (the "upstream" field on each HostCategory
   page) though it's not used for anything at present, and any values
   set presently are probably meaningless.
b) A trigger method to inform each mirror that their upstream has
changed.

For b), Debian uses an 'ssh push' trigger method.  The upstream mirror
does an outbound SSH to each downstream mirror, using a per-pair SSH
key, the private half is only known to the upstream mirror.  The
script itself executed on the downstream mirror simply sets up an
rsync pull to occur, then exits; the actual pull happens
independently.

Other triggers could be email (signed) sent out from MM to the
robot_email address on the Host page; a message on an AMQP bus (which
would require such users to have an open connection to the AMQP server
running in Fedora Infrastructure), or [insert your favorite trigger
method here].  I'm open to several, it's just a small matter of code.

Thinking about the SSH push method, and particularly, key management.
Should MM create the keypairs and maintain them?  This would give a
lot of flexibility to downstream mirrors, being able to change their
upstream "at will" (edit the upstream field in MM, and you immediately
start to get notifications when your upstream changes; no need to have
your new upstream mirror admin get involved).  But would people feel
comfortable with this?

Can we use one keypair per downstream mirror, or do we need one
keypair per (upstream, downstream) pair?  The upstream's (private)
half of the keypair is only known to MM.

Should a security breach happen to MM, the private half of the keypairs could
become known.  This can be mitigated by ensuring the keypairs can only
run one command on the downstream mirror, one that would be relatively
safe for anyone to run at any time.  But would it be better for MM to
have all those keypairs, or for each (upstream, downstream) mirroring
arrangement to have their own keypairs for this purpose, and MM has
nothing to do with it?  When the upstream runs report_mirror, it then
runs the ssh push triggers to its downstreams itself...

Looking for ideas, input, and coders.

Thanks,
Matt
Fedora Mirror Wrangler

-- 
Matt Domsch
Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux

--