<div dir="ltr">As I mentioned, I had to get this stuff into production so I ended up settling for making the bond the only interface available. At the southwest meetup last night we discussed this and I said I would post my scripts for this configuration. There's nothing too special in them, here they are (default RHEL `network` service is disabled; some formatting is off due to Ansible templating):<div><br></div><div>/etc/systemd/system/bond.service:</div><div>```</div><div><div>[Unit]</div><div>Description=manual bond routing</div><div>Before=network.target</div><div><br></div><div>[Service]</div><div>Type=oneshot</div><div>RemainAfterExit=yes</div><div>ExecStartPre=/usr/local/bin/bond-down</div><div>ExecStart=/usr/local/bin/bond-up</div><div>ExecStop=/usr/local/bin/bond-down</div><div><br></div><div>[Install]</div><div>WantedBy=multi-user.target</div><div>```</div><div><br></div><div>/usr/local/bin/bond-up:</div><div>```</div><div><div>#!/bin/bash</div><div><br></div><div>if [ "$(whoami)" != "root" ]; then</div><div> echo "must be root"</div><div> exit 1</div><div>fi</div><div><br></div><div>echo "checking for existing bond device"</div><div>ip link show dev bond1 2>/dev/null</div><div>bond_added=$?</div><div><br></div><div>if [ ${bond_added} -eq 0 ]; then</div><div> echo "bond device already added, nothing to do"</div><div> exit 0</div><div>fi</div><div><br></div><div>echo "adding bond device"</div><div>ip link add dev bond1 address 00:c5:4c:29:11:a7 type bond</div><div><br></div><div>if [ $? -eq 0 ]; then</div><div> echo "setting bond mode and monitoring config"</div><div> echo balance-alb > /sys/class/net/bond1/bonding/mode && \</div><div> echo 100 > /sys/class/net/bond1/bonding/miimon</div><div><br></div><div> if [ ! $? -eq 0 ]; then</div><div> echo "failed to set bond mode/monitor config, aborting"</div><div> ip link del dev bond1 type bond</div><div> exit 1</div><div> fi</div><div><br></div><div> echo "enslaving nic em1 to bond bond1"</div><div> ip link set em1 down</div><div> ip link set em1 master bond1</div><div> if [ ! $? -eq 0 ]; then</div><div> echo "could not enslave em1"</div><div> ip link del dev bond1 type bond</div><div> exit 1</div><div> fi</div><div> echo "enslaving nic em2 to bond bond1"</div><div> ip link set em2 down</div><div> ip link set em2 master bond1</div><div> if [ ! $? -eq 0 ]; then</div><div> echo "could not enslave em2"</div><div> ip link del dev bond1 type bond</div><div> exit 1</div><div> fi</div><div> fi</div><div><br></div><div>echo "bringing bond1 up"</div><div>ip link set bond1 up</div></div><div><br></div><div><div>echo "adding bond ip address"</div><div># This shouldn't be possible, but let's be thorough</div><div>ip addr show dev bond1 | grep 10.0.1.5</div><div>bond_ip_up=$?</div><div>if [ ${bond_ip_up} -eq 0 ]; then</div><div> echo "bond ip already added, nothing to do"</div><div>else</div><div> ip addr add <a href="http://10.0.1.5/24">10.0.1.5/24</a> dev bond1</div><div>fi</div><div><br></div><div>echo "establishing bond routing"</div><div>ip route add default via 10.0.1.1 dev bond1</div><div>ip route flush cache</div><div><br></div><div>echo "bond bond1 created"</div><div>exit 0</div></div><div>```</div><div><br></div><div>/usr/local/bin/bond-down:</div><div>```</div><div><div>#!/bin/bash</div><div><br></div><div>if [ "$(whoami)" != "root" ]; then</div><div> echo "must be root"</div><div> exit 1</div><div>fi</div><div><br></div><div>echo "checking for bond device bond1"</div><div>ip link show dev bond1 2>/dev/null</div><div>bond_added=$?</div><div><br></div><div># could not find the bond device so nothing to do</div><div>if [ ${bond_added} -eq 1 ]; then</div><div> echo "no device to bring down"</div><div> exit 0</div><div>fi</div><div><br></div><div>echo "freeing enslaved nic em1 from bond bond1"</div><div>ip link set em1 nomaster</div><div>echo "freeing enslaved nic em2 from bond bond1"</div><div>ip link set em2 nomaster</div><div><br></div><div>echo "removing bond device bond1"</div><div>ip link del dev bond1 type bond</div><div>exit $?</div></div><div>```</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 11, 2017 at 4:06 PM, James Sumners <span dir="ltr"><<a href="mailto:james.sumners@gmail.com" target="_blank">james.sumners@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I lied. I'm not having fun.<div><br></div><div>I have a system with three NICs: eth0, eth1, eth2. This system is to be a "load balancer," or, more accurately, a reverse proxy for many end points. My desire is to make eth0 a "maintenance" NIC and bond eth1 & eth2 into the primary service interface, bond0. I have three subnets in play: <a href="http://10.0.1.0/24" target="_blank">10.0.1.0/24</a>, <a href="http://10.0.2.0/24" target="_blank">10.0.2.0/24</a>, and <a href="http://10.0.3.0/24" target="_blank">10.0.3.0/24</a>. Pretend that 10.0.2/24 and 10.0.3/24 are public, Internet accessible, subnets and 10.0.1/24 is private. The proxy will serve end points on all three of these subnets.</div><div><br></div><div>Okay, so let's setup the interfaces:</div><div><br></div><div>```</div><div>$ echo -e "1 service\n2 bond" >> /etc/iproute2/rt_tables</div><div><br></div><div>$ ip link set eth0 up</div><div>$ ip addr add <a href="http://10.0.1.2/24" target="_blank">10.0.1.2/24</a> dev eth0</div><div>$ ip rule add iif eth0 prio 0 table service</div><div>$ ip route add to <a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a> via 10.0.1.1 dev eth0 prio 10000 table main</div><div>$ ip route add default via 10.0.1.1 dev eth0 table service</div><div>$ ip route flush cache</div><div><br></div><div>$ ip link add dev bond0 address 00:00:00:aa:bb:cc type bond</div><div>$ echo balance-alb > /sys/class/net/bond0/bonding/<wbr>mode</div><div>$ echo 100 > /sys/class/net/bond0/bonding/<wbr>miimon</div><div>$ ip link set eth1 master bond0</div><div>$ ip link set eth2 master bond0</div><div>$ ip link set bond0 up</div><div>$ ip addr add <a href="http://10.0.2.2/24" target="_blank">10.0.2.2/24</a> dev bond0 # see note 1 below</div><div>$ ip rule add iif bond0 prio 0 table bond</div><div>$ ip route add default via 10.0.2.1 dev bond0 table bond</div><div>$ ip route flush cache</div><div>```</div><div><br></div><div>Cool. Now let's add an end point:</div><div><br></div><div>```</div><div>$ ip addr add <a href="http://10.0.3.15/32" target="_blank">10.0.3.15/32</a> dev bond0</div><div>```</div><div><br></div><div>So, what's the problem? The switches see 10.0.3.15 as being associated with eth0. Thus, things don't work correctly. I can use tcpdump to monitor the traffic on bond0, ping 10.0.3.15 and see the traffic come in, but the pinger never gets a pong.</div><div><br></div><div>At this point I'm probably just going to say to hell with the maintenance interface and have all traffic on the bond and routed with the main table. But I figured I'd see if anyone has any guesses about why this configuration isn't working. To the best of my knowledge, the following should be true:</div><div><br></div><div>1. Traffic originating on the system will be routed through 10.0.1.1 via the eth0 interface as per the "main" routing table.</div><div>2. Traffic originating remotely via 10.0.1.2 will route through 10.0.1.1 via the eth0 interface as per the "service" routing table.</div><div>3. Traffic originating remotely via 10.0.2.2 or 10.0.3.15 will route through 10.0.2.1 via the bond0 interface as per the "bond" routing table.</div><div><br></div><div>Note 1: this is actually a pair of systems configured for failover with Ucarp as provided by <a href="https://github.com/jsumners/ucarp-rhel7" target="_blank">https://github.com/<wbr>jsumners/ucarp-rhel7</a> . Ucarp needs a "master IP" to tie the VIPs to.</div></div></blockquote></div><br clear="all"><div><br></div>-- <br><div class="gmail_signature">James Sumners<br><a href="http://james.sumners.info/" target="_blank">http://james.sumners.info/</a> (technical profile)<br><a href="http://jrfom.com/" target="_blank">http://jrfom.com/</a> (personal site)<br><a href="http://haplo.bandcamp.com/" target="_blank">http://haplo.bandcamp.com/</a> (music)</div>
</div></div></div>