<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">As noted below it appears the issue was the invalid gateway. We’ve not seen a recurrence since last Thursday when I modified the gateway. We had seen the issue every day last week until we changed the gateway.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> ale-bounces@ale.org [mailto:ale-bounces@ale.org]
<b>On Behalf Of </b>Lightner, Jeff<br>
<b>Sent:</b> Friday, July 10, 2015 1:53 PM<br>
<b>To:</b> Atlanta Linux Enthusiasts<br>
<b>Subject:</b> Re: [ale] Odd network issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">We’ve now run more than 24 hours since changing the gateway to what it should be rather than the server’s IP and haven’t seen the issue come back. Given that it had come back every day this week (and more than
once on one day) this may be resolved.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The only question now is, given that this server was installed in 2011 why was this suddenly a problem this week? This past weekend they updated our core network switches and I suspect something in that triggered
this new behavior.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">This was one of our original RHEL6 installations so I suspect the gateway configuration was a holdover from the way the hated NetworkManager originally configured things. We now turn that off on RHEL6 in favor
of the older network service (and had at some point turned off on this one).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
</span><a href="mailto:ale-bounces@ale.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">ale-bounces@ale.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> [</span><a href="mailto:ale-bounces@ale.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">mailto:ale-bounces@ale.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">]
<b>On Behalf Of </b>Lightner, Jeff<br>
<b>Sent:</b> Thursday, July 09, 2015 11:03 AM<br>
<b>To:</b> Atlanta Linux Enthusiasts<br>
<b>Subject:</b> Re: [ale] Odd network issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">By “machines” if you mean the servers we are testing FROM where one is able to ping the printer but the affected server can not the answer is yes they’re in the same subnet/VLAN.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><br>
If you mean the affected IPs that the server can’t ping no as I noted in original post their in our WAN at various locations in the US so are not in the same subnet as the servers.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">On checking the gateway just now I saw it was using the interfaces IP as the default gateway instead of the default gateway we normally use for the VLAN. I don’t really think that is the issue because as noted
we can reach things after bouncing the interface so it was using that both before and after bounce. I went ahead and changed it just now to rule it out.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">
</span><a href="mailto:ale-bounces@ale.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">ale-bounces@ale.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> [</span><a href="mailto:ale-bounces@ale.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">mailto:ale-bounces@ale.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">]
<b>On Behalf Of </b>Jeremy T. Bouse<br>
<b>Sent:</b> Thursday, July 09, 2015 9:34 AM<br>
<b>To:</b> </span><a href="mailto:ale@ale.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">ale@ale.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"><br>
<b>Subject:</b> Re: [ale] Odd network issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Are both machines on the same subnet? If not are their default routes correct and default gateways know how to get to each subnet? This sounds like behavior I've seen before when default gateway route was incorrect
and trying to communicate with server not on the same subnet.<o:p></o:p></p>
<div>
<p class="MsoNormal">On 7/9/2015 9:17 AM, Lightner, Jeff wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:5.0pt;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
We have a server that appears to be unable to connect to various IPs (mostly printers) in our WAN at times.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
The odd thing is that the interface is not down on either the server or the printer. The server can reach multiple other IPs when this occurs but for some reason can’t reach a few. Similarly when this occurs we can reach these other IPs from other servers
in the same VLAN.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
There are no errors shown in statistics on the interface (eth0) on the server nor on the port it attached to on the switch. There are no errors shown in /var/log/messages or dmesg.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
The IPs are not always the same ones. In fact yesterday when we saw the issue on IPs I tested the IPs that it couldn’t reach the day before that I’d resolved and they were still working.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
We’ve tried:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Killing the lp process that is hung at the start of this issue on server side.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Clearing arp on the server.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Clearing arp on the switch.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Bouncing cups (note issue is NOT just cups – when this occurs we can not ping the affected IPs nor can we telnet in on port 9100 as we would normally be able to do).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
The only thing that seems to resolve the issue (and does each time) is having the interface bounce on the server.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
We’’ve done that by:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Rebooting the server<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Ifup/ifdown on the server interface<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Removing and reseating the cable.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
Resetting the switch port.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
In each of those we see the port go down then come back up and after that the previously unreachable IPs are again reachable.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
I’m suspecting a bad interface port on either the server or the switch but in the absence of actual errors can’t prove one way or the other.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
This is different than any other network issue I’ve seen. I’m wondering if anyone has run into this and can shed any light on it?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:1.0in;margin-bottom:0in;margin-left:1.0in;margin-bottom:.0001pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:.5in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt">
<o:p></o:p></p>
</blockquote>
</div>
</body>
</html>