[ale] SSH Cisco Networking Issue

Sun Sep 26 00:10:39 EDT 2010

On Sat, 2010-09-25 at 17:17 -0400, Omar Chanouha wrote: 
> Wow, nicely done!

> Would that explain why I cannot ping or traceroute the ip?

Wow!  Bingo!  That was the question I was loath to ask because it was
possible that they would allow that but STILL break PMTU discovery.  If
you can't ping the IP I can now just about guarantee that this is the
problem.  Hell yes.

Mike

> Thanks,
> 
> -O
> 
> On Sat, Sep 25, 2010 at 3:36 PM, Michael H. Warfield <mhw at wittsend.com> wrote:
> > On Fri, 2010-09-17 at 15:40 +0000, Watson, Keith wrote:
> >> > -----Original Message-----
> >> > From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
> >> > Omar Chanouha
> >> > Sent: Friday, September 17, 2010 01:02
> >> > To: Atlanta Linux Enthusiasts - Yes! We run Linux!
> >> > Subject: Re: [ale] SSH Cisco Networking Issue
> >> >
> >> > > Most SSH
> >> > >packets are much smaller than your MTU, but a large amount of data
> >> > >could well exceed this.  If the firewall is dropping fragments, you
> >> > >would get a behavior similar to what you've described.
> >> >
> >> > AMAAAAAAAAAAZINGGGGGGGG!!!!!!!!!!!!!
> >> >
> >> > David, you sir are nothing shy of the man!!!!!!!!!
> >> >
> >> > I throttled the MTU on my server, and now I am able to send/recieve as
> >> > much data as I want!
> >> >
> >> > This brings up another question about MTU size, but I'll make another thread
> >> > for that.
> >> >
> >> > Thanks again!
> >> >
> >
> >> Omar,
> >
> >> Sorry for arriving at the party late. The symptoms are classic of an MTU mismatch.
> >
> > I would beg to differ.
> >
> > You're close but, if everything is working PROPERLY, there is no such
> > thing as an MTU mismatch.  If you think you have an MTU miss-match, it's
> > almost certain that you're really seeing something different and just
> > masking the real problem by reducing the MTU.
> >
> > What it sounds like is a PMTU (Path MTU) discovery problem.  PMTU
> > discovery, RFC 1191 from 20 years ago, automatically adjusts to maximize
> > packet size from the TCP layer across variable size MTU's in the
> > transmission path.  The primary cause of its failure is misinformed,
> > misguided, over zealous firewall administrators that block ICMP.  PMTU
> > discovery depends on ICMP / ICMP_DEST_UNREACH / ICMP_FRAG_NEEDED to
> > work.  The way it works is the TCP layer sends out a packet at the
> > maximum interface MTU size with the DF (Don't Fragment) bit set.  If it
> > gets an error of ICMP_FRAG_NEEDED back, it reduces the packet size until
> > it's clear end to end.  If it doesn't get that error back, it doesn't
> > back the packet size down but the packet is dropped by that router with
> > the smaller MTU and the connection hangs until it times out.
> >
> > By reducing the MTU size at your end, you're reducing the size of that
> > initial packet until you're down to the PMTU size and then, of course,
> > it works.  You're really just artificially masking the PMTU discovery
> > problem.  The fact that reducing the MTU works tells me that you've
> > probably really got a PMTU discovery problem.
> >
> > Another test is to disable PMTU discovery in the Linux kernel and bump
> > the MTU size back up.  But you may have to do this at both ends to make
> > it work.  Then retest.  If you can only set one end, do it and send a
> > large transmission from that side to the other.  Then it never sets the
> > DF flag and will fragment on the IP layer at that intermediate hop.
> > That results in sub-optimal performance but should also work.  It will
> > confirm that some hop along the path is blocking ICMP and it's the real
> > guilty party.  My bet would be on that Cisco firewall.
> >
> > You can disable PMTU discovery like this on Linux:
> >
> > echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc
> >
> > More information here:
> >
> > http://www.netheaven.com/pmtu.html
> >
> > This is an old, well known, problem but inexperienced firewall people
> > with an overblown sense of security still insist on disabling all ICMP
> > and damn the consequences.
> >
> > Regards,
> > Mike
> >
> >> Are any of the machines sitting behind a NAT router? The reason I ask is that consumer grade NAT routers are sometimes a bit deficient in properly negotiating MTU for encrypted connections, which results in symptoms like the ones you are seeing. Setting the MTU of your machine to 1438 or less should solve the problem.
> >>
> >> When Linksys first came out with their routers we had a rash of bizarre https, vpn, and ssh behavior just like you described. I was sent home for a day with a Linksys to find the problem. I found that MTU was not be negotiated properly.
> >>
> >> Most consumer router vendors eventually solved the problem with a firmware upgrade. If you are not using NAT it doesn't mean you still don't have an MTU problem. There is some networking device between you and them that is not negotiating properly.
> >>
> >> Here is the conclusion from the report:
> >>
> >>    A larger header is required for HTTPS over a VPN.
> >>    When the MTU is set to 1500 the packets are fragmented
> >>    whenever the data size to be transmitted is >1K.
> >>
> >>       header size + data size > 1500 bytes
> >>
> >>
> >>    Setting MTU to 1438 prevents the packets
> >>    from fragmenting by limiting the amount of data in
> >>    the packet.
> >>
> >>       header size + data size = 1500 bytes
> >>
> >>    Since not all possible scenarios were tested
> >>    an MTU value of less than 1438 should be used
> >>    to provide a safety margin. However, as MTU
> >>    decreases more packets are needed to send the
> >>    same amount of data. Therefore a safe compromise
> >>    would be to set MTU to 1400.
> >>
> >> Keith
> >>
> >
> > --
> > Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
> >   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
> >   NIC whois: MHW9          | An optimist believes we live in the best of all
> >  PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
> >
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://mail.ale.org/mailman/listinfo/ale
> > See JOBS, ANNOUNCE and SCHOOLS lists at
> > http://mail.ale.org/mailman/listinfo
> >
> >
> 

-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/ale/attachments/20100926/8f996976/attachment.bin 