[ale] stunnel fails and emits SYN flooding messages

[Michael H. Warfield]

On Tue, 2010-02-09 at 11:51 -0500, Allan Metts wrote:
> We're experiencing intermittent failures with our servers running
> stunnel on Amazon EC2. When the failure occurs, stunnel continues
> attempts to authenticate the certificate, but no longer allows
> connections.

Not quite sure what you mean by this.  You say it continues to "attempt
to authenticate", which requires a connection, but then you say it "no
longer allows connections"?  That doesn't make sense to me, where a
connection is a tcp connection.  You're apparently getting a failure
somewhere.  What is the actually error your getting?

What are you using stunnel for?  I use stunnel for very little any more.
I use to use it for a VPN under certain circumstances and have used it
for frontends to pop and imap servers which were not SSL enabled in
order to enable pop3s and imaps.  Is this a server side instance of
stunnel or a client side instance of stunnel?

> A review of /var/log/messages shows the following messages once each
> minute, beginning at the time of failure (these message occur less
> frequently at other times as well):

> kernel: possible SYN flooding on port 443. Sending cookies.

SYN flooding is from incoming SYN packets.  That's a common part of DDoS
attacks.  Do you have stunnel listening on port 443?  If so, why not a
real SSL web server on that port?  Or is this an SSL VPN you have set up
to get through sites that block everything except http and https?  I
deal with one of these now, but it's not stunnel based.

Recently, there has been quite a bit of activity like this.  Pushdo (a
DDoS capable botnet) has been reportedly attacking a number of sites,
including one which I'm associated with.  The result was thousands of
httpd processes and thousands of sockets waiting in "WAIT TIMEOUT" state
(closed but waiting for final timeout) along with thousands in normal
"CONNECTED" state.  The action is not really a SYN flood, though,
because the connections are being completed (a true SYN flood floods
your machine with SYN packets but then never responds to the SYN-ACK
leaving the initial handshake in a half open state while it times out)
and it is entering a full SSL negotiation and https request.

> Also notable is the fact that stunnel seems to be consuming most of
> the system memory at this point (over 2GB, 78%). Sockstat shows
> several thousand activate connections (which is in-line with our
> typical usage), but does not seem to be showing any that are stranded.

SSL negotiation is rather processor intensive.  That tells me that what
ever is hitting you is going through a full SSL negotiation and that's
what is driving your load average so high.

> Can anyone suggest what might be happening? Any recommended remedies?

You're going to have to explain more about your setup.  This is the best
I can do with what you've provided.

> Thanks in advance,
> Allan

Regards,
Mike
-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/ale/attachments/20100210/1c719210/attachment.bin