Doesn't MII Polling Interval have to have a non-0 value? A 0 value means no polling so no notice of failure (I think)<br><br><div class="gmail_quote">On Thu, Sep 16, 2010 at 11:36 AM, Lightner, Jeff <span dir="ltr"><<a href="mailto:jlightner@water.com">jlightner@water.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div lang="EN-US" link="blue" vlink="blue" style="word-wrap: break-word;">
<div>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">cat /proc/net/bonding/bond0 ouput:</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Ethernet Channel Bonding Driver: v3.0.3
(March 23, 2006)</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Bonding Mode: load balancing (round-robin)</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">MII Status: up</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">MII Polling Interval (ms): 0</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Up Delay (ms): 0</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Down Delay (ms): 0</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Slave Interface: eth2</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">MII Status: up</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Link Failure Count: 0</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Permanent HW addr: 00:04:23:ba:f1:20</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Slave Interface: eth3</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">MII Status: up</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Link Failure Count: 0</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">Permanent HW addr: 00:04:23:ba:f1:21</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">The switch log was not helpful – it simply
shows the links going up and down and doesn’t even tell us WHEN it saw that
because its time field had something like 2+ years in it. The network admin
reset the time so if it occurs again we’ll have better time. There is no
other detail than the links going down and up. </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;">I don’t think its an issue with the
bonding setup or the switch’s recognition of that because we have another RAC
environment like this one that does the same bonding setup and uses the same
switch since it was first put in place over 2 years ago. In fact the one that
went down is a Test environment built modeled on the other once which is our
Production environment. This test environment has been running since around
April of this year. If flapping were an issue I’d expect to have seen it long
before now.</span></font></p>
<p class="MsoNormal"><font size="2" face="Arial" color="navy"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font></p>
<div>
<div align="center" class="MsoNormal" style="text-align: center;"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">
<hr width="100%" size="2" align="center">
</span></font></div>
<p class="MsoNormal"><b><font size="2" face="Tahoma"><span style="font-size: 10pt; font-family: Tahoma; font-weight: bold;">From:</span></font></b><font size="2" face="Tahoma"><span style="font-size: 10pt; font-family: Tahoma;">
<a href="mailto:ale-bounces@ale.org" target="_blank">ale-bounces@ale.org</a> [mailto:<a href="mailto:ale-bounces@ale.org" target="_blank">ale-bounces@ale.org</a>] <b><span style="font-weight: bold;">On Behalf Of </span></b>Joey Rutledge<br>
<b><span style="font-weight: bold;">Sent:</span></b> Thursday, September 16, 2010
11:05 AM<br>
<b><span style="font-weight: bold;">To:</span></b> Atlanta Linux Enthusiasts -
Yes! We run Linux!<br>
<b><span style="font-weight: bold;">Subject:</span></b> Re: [ale] bond0 went down</span></font></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"> </span></font></p>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">A few questions I have:</span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">What type of bond method are you using? round robin, active
passive, etc cat /proc/net/bonding/bond0</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">What is the uplink switch and do you have logs on it that you can check
for when the interfaces went down?</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">I've seen in our environment that round-robin simply doesn't work with
the switch configuration and causes interfaces to flap. We use
active-passive bonding for all of our servers.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">Joey</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"> </span></font></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;">On Sep 15, 2010, at 5:11 PM, Lightner, Jeff wrote:</span></font></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"><br>
<br>
</span></font></p>
<span style="word-spacing: 0px;">
<div link="blue" vlink="purple">
<div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Can anyone tell me what the below messages mean?
I didn’t find many hits on the web:</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Interface
bond0.IPv6 no longer relevant for mDNS.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Leaving mDNS
multicast group on interface bond0.IPv6 with address fe80::204:23ff:feba:f120.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Interface
bond0.IPv4 no longer relevant for mDNS.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Leaving mDNS
multicast group on interface bond0.IPv4 with address 192.168.8.73.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Withdrawing
address record for fe80::204:23ff:feba:f120 on bond0.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Withdrawing
address record for 192.168.8.73 on bond0.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: New relevant
interface bond0.IPv4 for mDNS.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Joining mDNS
multicast group on interface bond0.IPv4 with address 192.168.8.73.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Registering new
address record for 192.168.8.73 on bond0.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Interface
eth2.IPv6 no longer relevant for mDNS.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Leaving mDNS
multicast group on interface eth2.IPv6 with address fe80::204:23ff:feba:f120.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Withdrawing
address record for fe80::204:23ff:feba:f120 on eth2.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 kernel: bonding: bond0: Interface
eth2 is already enslaved!</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Interface
eth3.IPv6 no longer relevant for mDNS.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Leaving mDNS
multicast group on interface eth3.IPv6 with address fe80::204:23ff:feba:f120.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 avahi-daemon[6709]: Withdrawing
address record for fe80::204:23ff:feba:f120 on eth3.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:45 atlrdtd1 kernel: bonding: bond0: Interface
eth3 is already enslaved!</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:47 atlrdtd1 avahi-daemon[6709]: New relevant
interface bond0.IPv6 for mDNS.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:47 atlrdtd1 avahi-daemon[6709]: Joining mDNS
multicast group on interface bond0.IPv6 with address fe80::204:23ff:feba:f120.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Sep 14 13:15:47 atlrdtd1 avahi-daemon[6709]: Registering new
address record for fe80::204:23ff:feba:f120 on bond0.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Background: </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">We have an Oracle RAC cluster of 2 nodes.
Yesterday one of the nodes rebooted and its log indicates that Oracle forced
the reboot to preserve cluster integrity. There were no other
messages in that node’s /var/log/messages near the time of this message and
reboot. </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">We use a private lan setup on 2 bonded NICs on each side for
the Oracle Cluster Ready Services to communicate with each
other. That is bond0 and is using 2 Intel GigE NIC ports on
both sides (eth2 and eth3 are the NICs). We found that the
connectivity on the private lan had gone away and on checking found that both
eth2 and eth3 on the node that got these messages was showing no
link. Running “ifdown bond0” followed by “ifup bond0”
re-established links on both eth2 and eth3.</span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">The above messages occurred on the node where bond0’s links
were down less than 2 minutes before the node that rebooted issued the message
about shutting down to preserve cluster integrity. It seems fairly
clear the cause of the reboot was the loss of connectivity but I can’t really
determine from the above log entries WHY bond0 went down. So was hoping
someone had seen something like this and could give me a clue. </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">P.S. We don’t actually use the ipv6 – the relevant
addresses are the ipv4 ones. Apparently the guy who set this up
didn’t disable ipv6 on these NICs but I don’t believe that is the issue as they
have been up for a few months with this configuration.</span></font></p>
</div>
</div>
<div>
<p class="MsoNormal"><font size="4" face="Helvetica"><span style="font-size: 13.5pt; font-family: Helvetica;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">Proud partner. Susan G. Komen for the Cure.</span></font><font size="4" face="Helvetica"><span style="font-size: 13.5pt; font-family: Helvetica;"></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="4" face="Helvetica"><span style="font-size: 13.5pt; font-family: Helvetica;"> </span></font></p>
</div>
<div>
<p class="MsoNormal"><i><i><font size="1" face="Arial" color="green"><span style="font-size: 7.5pt; font-family: Arial; color: green;">Please consider our
environment before printing this e-mail or attachments.</span></font></i></i><font size="4" face="Helvetica"><span style="font-size: 13.5pt; font-family: Helvetica;"></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="2" face="Arial"><span style="font-size: 10pt; font-family: Arial;">----------------------------------<br>
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential
information and is for the sole use of the intended recipient(s). If you are
not the intended recipient, any disclosure, copying, distribution, or use of
the contents of this information is prohibited and may be unlawful. If you have
received this electronic transmission in error, please reply immediately to the
sender that you have received the message in error, and delete it. Thank you.<br>
----------------------------------</span></font><font size="1" face="Courier New"><span style="font-size: 9pt; font-family: "Courier New";"></span></font></p>
</div>
<p class="MsoNormal"><font size="4" face="Helvetica"><span style="font-size: 13.5pt; font-family: Helvetica;">_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a></span></font></p>
</div>
</span></div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size: 12pt;"></span> </font></p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>-- <br>James P. Kinney III<br>I would rather stumble along in freedom than walk effortlessly in chains.<br><br><br>