[ale] SSH Cisco Networking Issue
Michael H. Warfield
mhw at WittsEnd.com
Mon Sep 27 22:15:52 EDT 2010
On Mon, 2010-09-27 at 17:07 -0600, Michael Hirsch wrote:
> On Sat, Sep 25, 2010 at 10:10 PM, Michael H. Warfield <mhw at wittsend.com> wrote:
> > On Sat, 2010-09-25 at 17:17 -0400, Omar Chanouha wrote:
> >> Wow, nicely done!
> >
> >> Would that explain why I cannot ping or traceroute the ip?
> >
> > Wow! Bingo! That was the question I was loath to ask because it was
> > possible that they would allow that but STILL break PMTU discovery. If
> > you can't ping the IP I can now just about guarantee that this is the
> > problem. Hell yes.
>
> Coincidentally, I had a problem just last week that I solved by
> reducing my MTU. In my case it is Outlook running in Wine (from
> codeweavers.com--great product). When syncing my mail it would hang
> after a while. Googling led me to try reducing my MTU and it started
> working. Sweet!
>
> But now I would guess that this is really just a symptom, as Mike
> described, of broken PMTU discovery.
>
> I'd love to get my IT guys to fix it, but in my case pinging the
> mailserver succeeds, as does traceroute. I'm guessing that if I talk
> to our overburdened IT dept they will not believe me that there is a
> problem.
That's part of the problem and why I didn't raise that question
initially. It is possible (and I've seen them do it) where they allow
ICMP ECHO and ICMP ECHO_REPLY but then block the other ICMP types they
don't understand. Hence, ping works and large packets don't.
You MIGHT try "ping -M do -s 1500 host" and see if it breaks. The "-M
do" says do prohibit fragmentation (don't ask - I don't know why it's
that way) and the -s 1500 sets the packet size. Back it down till it
works. If it does, you have your smoking gun. Still, I'm not sure I
can guarantee the test.
> Is there another I can do to demonstrate a problem? If I give them
> the real complaint (Outlook in Wine on Linux not working) they'll
> probably say it isn't supported and hang up.
> Thanks,
> Michael
Regards,
Mike
> >
> > Mike
> >
> >> Thanks,
> >>
> >> -O
> >>
> >> On Sat, Sep 25, 2010 at 3:36 PM, Michael H. Warfield <mhw at wittsend.com> wrote:
> >> > On Fri, 2010-09-17 at 15:40 +0000, Watson, Keith wrote:
> >> >> > -----Original Message-----
> >> >> > From: ale-bounces at ale.org [mailto:ale-bounces at ale.org] On Behalf Of
> >> >> > Omar Chanouha
> >> >> > Sent: Friday, September 17, 2010 01:02
> >> >> > To: Atlanta Linux Enthusiasts - Yes! We run Linux!
> >> >> > Subject: Re: [ale] SSH Cisco Networking Issue
> >> >> >
> >> >> > > Most SSH
> >> >> > >packets are much smaller than your MTU, but a large amount of data
> >> >> > >could well exceed this. If the firewall is dropping fragments, you
> >> >> > >would get a behavior similar to what you've described.
> >> >> >
> >> >> > AMAAAAAAAAAAZINGGGGGGGG!!!!!!!!!!!!!
> >> >> >
> >> >> > David, you sir are nothing shy of the man!!!!!!!!!
> >> >> >
> >> >> > I throttled the MTU on my server, and now I am able to send/recieve as
> >> >> > much data as I want!
> >> >> >
> >> >> > This brings up another question about MTU size, but I'll make another thread
> >> >> > for that.
> >> >> >
> >> >> > Thanks again!
> >> >> >
> >> >
> >> >> Omar,
> >> >
> >> >> Sorry for arriving at the party late. The symptoms are classic of an MTU mismatch.
> >> >
> >> > I would beg to differ.
> >> >
> >> > You're close but, if everything is working PROPERLY, there is no such
> >> > thing as an MTU mismatch. If you think you have an MTU miss-match, it's
> >> > almost certain that you're really seeing something different and just
> >> > masking the real problem by reducing the MTU.
> >> >
> >> > What it sounds like is a PMTU (Path MTU) discovery problem. PMTU
> >> > discovery, RFC 1191 from 20 years ago, automatically adjusts to maximize
> >> > packet size from the TCP layer across variable size MTU's in the
> >> > transmission path. The primary cause of its failure is misinformed,
> >> > misguided, over zealous firewall administrators that block ICMP. PMTU
> >> > discovery depends on ICMP / ICMP_DEST_UNREACH / ICMP_FRAG_NEEDED to
> >> > work. The way it works is the TCP layer sends out a packet at the
> >> > maximum interface MTU size with the DF (Don't Fragment) bit set. If it
> >> > gets an error of ICMP_FRAG_NEEDED back, it reduces the packet size until
> >> > it's clear end to end. If it doesn't get that error back, it doesn't
> >> > back the packet size down but the packet is dropped by that router with
> >> > the smaller MTU and the connection hangs until it times out.
> >> >
> >> > By reducing the MTU size at your end, you're reducing the size of that
> >> > initial packet until you're down to the PMTU size and then, of course,
> >> > it works. You're really just artificially masking the PMTU discovery
> >> > problem. The fact that reducing the MTU works tells me that you've
> >> > probably really got a PMTU discovery problem.
> >> >
> >> > Another test is to disable PMTU discovery in the Linux kernel and bump
> >> > the MTU size back up. But you may have to do this at both ends to make
> >> > it work. Then retest. If you can only set one end, do it and send a
> >> > large transmission from that side to the other. Then it never sets the
> >> > DF flag and will fragment on the IP layer at that intermediate hop.
> >> > That results in sub-optimal performance but should also work. It will
> >> > confirm that some hop along the path is blocking ICMP and it's the real
> >> > guilty party. My bet would be on that Cisco firewall.
> >> >
> >> > You can disable PMTU discovery like this on Linux:
> >> >
> >> > echo 1 > /proc/sys/net/ipv4/ip_no_pmtu_disc
> >> >
> >> > More information here:
> >> >
> >> > http://www.netheaven.com/pmtu.html
> >> >
> >> > This is an old, well known, problem but inexperienced firewall people
> >> > with an overblown sense of security still insist on disabling all ICMP
> >> > and damn the consequences.
> >> >
> >> > Regards,
> >> > Mike
> >> >
> >> >> Are any of the machines sitting behind a NAT router? The reason I ask is that consumer grade NAT routers are sometimes a bit deficient in properly negotiating MTU for encrypted connections, which results in symptoms like the ones you are seeing. Setting the MTU of your machine to 1438 or less should solve the problem.
> >> >>
> >> >> When Linksys first came out with their routers we had a rash of bizarre https, vpn, and ssh behavior just like you described. I was sent home for a day with a Linksys to find the problem. I found that MTU was not be negotiated properly.
> >> >>
> >> >> Most consumer router vendors eventually solved the problem with a firmware upgrade. If you are not using NAT it doesn't mean you still don't have an MTU problem. There is some networking device between you and them that is not negotiating properly.
> >> >>
> >> >> Here is the conclusion from the report:
> >> >>
> >> >> A larger header is required for HTTPS over a VPN.
> >> >> When the MTU is set to 1500 the packets are fragmented
> >> >> whenever the data size to be transmitted is >1K.
> >> >>
> >> >> header size + data size > 1500 bytes
> >> >>
> >> >>
> >> >> Setting MTU to 1438 prevents the packets
> >> >> from fragmenting by limiting the amount of data in
> >> >> the packet.
> >> >>
> >> >> header size + data size = 1500 bytes
> >> >>
> >> >> Since not all possible scenarios were tested
> >> >> an MTU value of less than 1438 should be used
> >> >> to provide a safety margin. However, as MTU
> >> >> decreases more packets are needed to send the
> >> >> same amount of data. Therefore a safe compromise
> >> >> would be to set MTU to 1400.
> >> >>
> >> >> Keith
> >> >>
> >> >
> >> > --
> >> > Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
> >> > /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
> >> > NIC whois: MHW9 | An optimist believes we live in the best of all
> >> > PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
> >> >
> >> > _______________________________________________
> >> > Ale mailing list
> >> > Ale at ale.org
> >> > http://mail.ale.org/mailman/listinfo/ale
> >> > See JOBS, ANNOUNCE and SCHOOLS lists at
> >> > http://mail.ale.org/mailman/listinfo
> >> >
> >> >
> >>
> >
> > --
> > Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
> > /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
> > NIC whois: MHW9 | An optimist believes we live in the best of all
> > PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
> >
> > _______________________________________________
> > Ale mailing list
> > Ale at ale.org
> > http://mail.ale.org/mailman/listinfo/ale
> > See JOBS, ANNOUNCE and SCHOOLS lists at
> > http://mail.ale.org/mailman/listinfo
> >
> >
>
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
Url : http://mail.ale.org/pipermail/ale/attachments/20100927/b4f7bc2d/attachment.bin
More information about the Ale
mailing list