[ale] SSH Cisco Networking Issue

[Michael H. Warfield]

On Fri, 2010-09-17 at 15:40 +0000, Watson, Keith wrote: 
> > -----Original Message-----
> > From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
> > Omar Chanouha
> > Sent: Friday, September 17, 2010 01:02
> > To: Atlanta Linux Enthusiasts - Yes! We run Linux!
> > Subject: Re: [ale] SSH Cisco Networking Issue
> > 
> > > Most SSH
> > >packets are much smaller than your MTU, but a large amount of data
> > >could well exceed this.  If the firewall is dropping fragments, you
> > >would get a behavior similar to what you've described.
> > 
> > AMAAAAAAAAAAZINGGGGGGGG!!!!!!!!!!!!!
> > 
> > David, you sir are nothing shy of the man!!!!!!!!!
> > 
> > I throttled the MTU on my server, and now I am able to send/recieve as
> > much data as I want!
> > 
> > This brings up another question about MTU size, but I'll make another thread
> > for that.
> > 
> > Thanks again!
> > 

> Omar,

> Sorry for arriving at the party late. The symptoms are classic of an MTU mismatch.

I would beg to differ.

You're close but, if everything is working PROPERLY, there is no such
thing as an MTU mismatch.  If you think you have an MTU miss-match, it's
almost certain that you're really seeing something different and just
masking the real problem by reducing the MTU.

What it sounds like is a PMTU (Path MTU) discovery problem.  PMTU
discovery, RFC 1191 from 20 years ago, automatically adjusts to maximize
packet size from the TCP layer across variable size MTU's in the
transmission path.  The primary cause of its failure is misinformed,
misguided, over zealous firewall administrators that block ICMP.  PMTU
discovery depends on ICMP / ICMP_DEST_UNREACH / ICMP_FRAG_NEEDED to
work.  The way it works is the TCP layer sends out a packet at the
maximum interface MTU size with the DF (Don't Fragment) bit set.  If it
gets an error of ICMP_FRAG_NEEDED back, it reduces the packet size until
it's clear end to end.  If it doesn't get that error back, it doesn't
back the packet size down but the packet is dropped by that router with
the smaller MTU and the connection hangs until it times out.

By reducing the MTU size at your end, you're reducing the size of that
initial packet until you're down to the PMTU size and then, of course,
it works.  You're really just artificially masking the PMTU discovery
problem.  The fact that reducing the MTU works tells me that you've
probably really got a PMTU discovery problem.

Another test is to disable PMTU discovery in the Linux kernel and bump
the MTU size back up.  But you may have to do this at both ends to make
it work.  Then retest.  If you can only set one end, do it and send a
large transmission from that side to the other.  Then it never sets the
DF flag and will fragment on the IP layer at that intermediate hop.
That results in sub-optimal performance but should also work.  It will
confirm that some hop along the path is blocking ICMP and it's the real
guilty party.  My bet would be on that Cisco firewall.

You can disable PMTU discovery like this on Linux:

echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc

More information here:

http://www.netheaven.com/pmtu.html

This is an old, well known, problem but inexperienced firewall people
with an overblown sense of security still insist on disabling all ICMP
and damn the consequences.

Regards,
Mike

> Are any of the machines sitting behind a NAT router? The reason I ask is that consumer grade NAT routers are sometimes a bit deficient in properly negotiating MTU for encrypted connections, which results in symptoms like the ones you are seeing. Setting the MTU of your machine to 1438 or less should solve the problem.
> 
> When Linksys first came out with their routers we had a rash of bizarre https, vpn, and ssh behavior just like you described. I was sent home for a day with a Linksys to find the problem. I found that MTU was not be negotiated properly.
> 
> Most consumer router vendors eventually solved the problem with a firmware upgrade. If you are not using NAT it doesn't mean you still don't have an MTU problem. There is some networking device between you and them that is not negotiating properly.
> 
> Here is the conclusion from the report:
> 
>    A larger header is required for HTTPS over a VPN.
>    When the MTU is set to 1500 the packets are fragmented
>    whenever the data size to be transmitted is >1K.
> 
>       header size + data size > 1500 bytes
> 
> 
>    Setting MTU to 1438 prevents the packets
>    from fragmenting by limiting the amount of data in
>    the packet.
> 
>       header size + data size = 1500 bytes
> 
>    Since not all possible scenarios were tested
>    an MTU value of less than 1438 should be used
>    to provide a safety margin. However, as MTU
>    decreases more packets are needed to send the
>    same amount of data. Therefore a safe compromise
>    would be to set MTU to 1400.
> 
> Keith
> 

-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/ale/attachments/20100925/7d46c210/attachment-0001.bin