[mirror-admin] master server sync stats and recommendations

[Simon Valiquette]

Axel Thimm un jour écrivit:
> On Thu, Apr 23, 2009 at 06:34:39AM -0400, Simon Valiquette wrote:
>>> If you can't make sure that the server has
>>> always enough resources for all projects to do a push mirroring
>>> simultaneously, then you get issues with traffic and high CPU loads.
>>   When pushing other mirror, if you push a high number of mirrors, it is  
>> quite easy not to send the PUSH signal to every mirrors at the same time, 
>> or to push let say 5 or 10 mirrors maximum at once if that is what your  
>> hardware/bandwidth allows you.
> 
> No, it's from the client perspective. As long as only one project like
> Debian uses pushing it may not create peaks, but if you have a mirror
> onto which you would allow Debian, Fedora, FreeBSD and so on to push
> at random intervalls, you can have the servers push simultaneously.

   Which is not a problem for the mirror that get pushed.  If the load 
start to be too high, just delay furter rsync until the load goes down and 
voilà!  It is your server, so you can control that, and it is what I 
already do.

   There is also other simple way to handle this problem such as limiting 
the number of concurent rsyncing to N and then queue the other requests 
(and not forget to pass a reasonnable timeout to the rsync command).

   AFAIK, the official script from Debian doesn't do that yet on the 
client side, but should this mecanism get more popular someone will care 
enough to standardise a way to limit the number of concurent rsync or the 
load created by them.

> IOW push mirroring is not scaling beyond one or two projects, and most
> probably Debian and Ubuntu are coordinating their push intervalls to
> avoid the above mentioned scenario.

   First time I ever heard that Ubuntu and Debian are coordinating their 
push, and my logs prove me that they don't.  Actually, Debian is normaly 
making 4 updates a day on a very regular schedule while Ubuntu seems to 
makes at least between 5 and 7 updates a day on a very random schedule, 
sometime almost exactly at the same time than Debian.


>>   Well, even with that, it couldn't compromise a mirror that properly  
>> tied the ssh key with a specific command as it should when using the push 
>> mecanism
> 
> You would have an important (or the most important) security layer
> stripped off. Now the attecker just needs to find vulnerabilities in
> your script.

   How will he be able to exploit them? Without the help of a very serious 
security bug in OpenSSL, whatever command he will pass to SSH will get 
ignored and the script you choose will still get called.

   He can still pass parameters to the script, but it is useless if the 
script just ignore them (which is my case).  I have quite a bit of 
knowledge about how to crack a server, and a lot of imagination, but if I 
find a bug in OpenSSH I think it is unlikely that I would need to also 
exploit a bug in the trigerred script as well.


   Personnaly, I am way more worried about the security bugs still laying 
in rsync than the security implication of running OpenSSH.  If my mirror 
was really a mission critical system, I wouldn't be running rsync at all.

> If it is written by every admin, he will surely find a
> couple of mirrors to crack. If it is a central script he will even
> have the source to check it for vulnerabilities.

   Which is quite futile if the script just ignore (ie. don't use) any 
parameter that you could pass to it, because that's about the only way you 
could influence the script.

   And since the parameters that could validly be passed are usualy known 
in advance and very simple, validating them properly is not terribly 
difficult (please, I don't want to move the discussion on how 
easy/difficult/tricky it can be).

> 
>> (at most, it could create unnecessary load on the server, but not
>> DoS it if the script is properly implemented).
> 
> I wonder what is easier, have the simple polling method as Chris and I
> outlined or writing security hardened scripts? Chris' method already
> works with a couple of lines.

   Well, why both mecanisms couldn't be offered and just let sysadmins 
decide for themselves? I have no problem with that, tough I prefer to get 
pushed because that way I know that I am not rsyncing in a middle of an 
update on the master or the tier-1 side.

   For tier-1 mirror, unless they pool every 10 or 15 minutes, they will 
never be as much up to date than by using the push mecanism, and the 
master can choose to push them during a window when other updates in the 
middle of the rsyncing process will be unlikely, which is a good thing.


Simon Valiquette


--