two nics to one lan

I have two NICs, one LAN. eth0: 172.16.59.1, eth1: 172.16.59.17. This
afternoon something odd happened (I'm not sure what), I see in the logs:

May 20 13:08:37 x440 kernel: NETDEV WATCHDOG: eth1: transmit timed out
May 20 13:08:40 x440 kernel: e1000: eth1 NIC Link is Up 1000 Mbps Full
Duplex

The link may have come back up, but all connectivity was hosed. No ping
response on 172.16.59.17 _or_ 172.16.59.1. ifconfig claimed things were
normal, and the routing table was also normal, which I believe is the
problem. Whatever happened caused the eth1 route to be non-functional,
but all traffic bound to 172.16.59.0/24 was being routed through eth1.

Unfortunately I don't have the routing table from then, because I was
frantically bringing interfaces down and up in hopes that connectivity
would be restored (it was). But here is my current route config:

# ip route show
172.16.59.0/24 dev eth1 scope link
172.16.59.0/24 dev eth1 proto kernel scope link src 172.16.59.17
127.0.0.0/8 dev lo scope link
default via 172.16.59.15 dev eth0

# tcpdump -n -i eth0 src 172.16.59.1 and dst net 172.16.59.0/24

That command gives no output at all, but s/eth0/eth1/ gives lots. All
the traffic is being routed through eth1, as you would expect from that
routing configuration. But I didn't ask for two 172.16.59.0/24 routes
via eth1, and before I did ifup eth1 (after doing some forensics and it
was down) there was one well-configured route via eth0 for that subnet.
(This is RHEL AS 2.1)

So the crux of my question is, is there a way to configure the routing
such that the preferred interface for frames from 172.16.59.1 is eth0,
and the preferred interface for frames from 172.16.59.17 is eth1?
Preferably one that wouldn't preclude the alternate route if one route
goes down, but the route isn't likely to go down unless the whole
interface does and there's not likely going to be traffic to
172.16.59.17 if eth1 is down...

At least I'd like to have a route over eth0 if eth1 goes away, and the
routing config above doesn't look like there is (although I could add
one, right?) Note I'm not asking about load balancing, although that
wouldn't be a bad side effect either.

I'm just starting to feel very comfortable with routing stuff, so it's
possible (likely) that I've made some bad assumptions in the above
dialog; I'd appreciate it if you'd point them out to me!

Hans Fugal <hfugal@wencor.com> wrote in message news:<c8jc71$hbi$1@terabinaries.xmission.com>...
> I have two NICs, one LAN. eth0: 172.16.59.1, eth1: 172.16.59.17. This
> afternoon something odd happened (I'm not sure what), I see in the logs:
>
> May 20 13:08:37 x440 kernel: NETDEV WATCHDOG: eth1: transmit timed out
> May 20 13:08:40 x440 kernel: e1000: eth1 NIC Link is Up 1000 Mbps Full
> Duplex
>
> The link may have come back up, but all connectivity was hosed. No ping
> response on 172.16.59.17 _or_ 172.16.59.1. ifconfig claimed things were
> normal, and the routing table was also normal, which I believe is the
> problem. Whatever happened caused the eth1 route to be non-functional,
> but all traffic bound to 172.16.59.0/24 was being routed through eth1.
>
> Unfortunately I don't have the routing table from then, because I was
> frantically bringing interfaces down and up in hopes that connectivity
> would be restored (it was). But here is my current route config:
>
> # ip route show
> 172.16.59.0/24 dev eth1 scope link
> 172.16.59.0/24 dev eth1 proto kernel scope link src 172.16.59.17
> 127.0.0.0/8 dev lo scope link
> default via 172.16.59.15 dev eth0
>
> # tcpdump -n -i eth0 src 172.16.59.1 and dst net 172.16.59.0/24
>
> That command gives no output at all, but s/eth0/eth1/ gives lots. All
> the traffic is being routed through eth1, as you would expect from that
> routing configuration. But I didn't ask for two 172.16.59.0/24 routes
> via eth1, and before I did ifup eth1 (after doing some forensics and it
> was down) there was one well-configured route via eth0 for that subnet.
> (This is RHEL AS 2.1)
>
> So the crux of my question is, is there a way to configure the routing
> such that the preferred interface for frames from 172.16.59.1 is eth0,
> and the preferred interface for frames from 172.16.59.17 is eth1?
> Preferably one that wouldn't preclude the alternate route if one route
> goes down, but the route isn't likely to go down unless the whole
> interface does and there's not likely going to be traffic to
> 172.16.59.17 if eth1 is down...

Linux with two nics on same subnet on one host is not intuitive ;-)

> At least I'd like to have a route over eth0 if eth1 goes away, and the
> routing config above doesn't look like there is (although I could add
> one, right?) Note I'm not asking about load balancing, although that
> wouldn't be a bad side effect either.

Not clear to me what you want to accomplish here -- a backup link?

> I'm just starting to feel very comfortable with routing stuff, so it's
> possible (likely) that I've made some bad assumptions in the above
> dialog; I'd appreciate it if you'd point them out to me!

Don't feel too bad about your assumptions -- Linux has a quite unusal
and very flexible stack implementation.

Start with these:
http://lartc.org/howto/
http://linux-ip.net/html/
http://www.policyrouting.org/iproute2-toc.html
http://www.policyrouting.org/PolicyR...NLINE/TOC.html

You should already have:
/usr/share/doc/iproute?X???/ip?cref.ps

It will also be handy to have all the howtos from tldp for background
and to answer specifics that will likely crop up.
http://www.tldp.org/docs.html#howto

As is, Linux uses a single, main routing table -- and any nic will
answer/accept frames for any IP used by the host. You will need
multiple routing tables -- ie., basic policy routing -- configured to
get firmer control.

I think you'll find all your answers in the above docs -- the online
book is a very good background/primer on policy routing. You will
likely need to adjust your FW rules if you want each nic to "see"
different traffic.

hth,
prg
email above disabled

P Gentry wrote:

> Linux with two nics on same subnet on one host is not intuitive ;-)
So I've discovered. ;-)

>>At least I'd like to have a route over eth0 if eth1 goes away, and the
>>routing config above doesn't look like there is (although I could add
>>one, right?) Note I'm not asking about load balancing, although that
>>wouldn't be a bad side effect either.
>
>
> Not clear to me what you want to accomplish here -- a backup link?
When the box was set up I'm sure he thought 172.16.59.1 traffic was
going out eth0 and 172.16.59.17 traffic was going out eth1. As it is it
all goes out one or the other, i.e. eth1. So backup, yes, in the sense
that if one card goes haywire the other card and IP address function
normally. Or if all traffic were load balanced and/or failover to the
other card that would be ok too. The problem was that the stack
apparently didn't know anything was wrong and so the interface stayed
"up" and the routes didn't change, so everything went out eth1 which
mean nothing went anywhere.

> Start with these:
> http://lartc.org/howto/
> http://linux-ip.net/html/
> http://www.policyrouting.org/iproute2-toc.html
> http://www.policyrouting.org/PolicyR...NLINE/TOC.html
>
> You should already have:
> /usr/share/doc/iproute?X???/ip?cref.ps
>
> It will also be handy to have all the howtos from tldp for background
> and to answer specifics that will likely crop up.
> http://www.tldp.org/docs.html#howto
>
> As is, Linux uses a single, main routing table -- and any nic will
> answer/accept frames for any IP used by the host. You will need
> multiple routing tables -- ie., basic policy routing -- configured to
> get firmer control.
>
> I think you'll find all your answers in the above docs -- the online
> book is a very good background/primer on policy routing. You will
> likely need to adjust your FW rules if you want each nic to "see"
> different traffic.

Thanks for the references, I'll read them. I'm sure they'll be very
helpful and I may have follow-up questions later.

Hans Fugal <hfugal@wencor.com> said:
>So backup, yes, in the sense that if one card goes haywire the other
>card and IP address function normally. Or if all traffic were load
>balanced and/or failover to the other card that would be ok too.

What is the next link from the Linux machine? Are these two cards
connected to a single switch? If so, _if_ the switch supports
channel bonding (aka Cisco EtherChannel, aka Sun trunking), you could
configure the two interfaces to form one virtual interface, which would
load-balance the traffic across the two cards in normal situations,
and upon failure of one link would use the remaining one. And have
just one IP address (unless you specifically need more, and create
virtual interfaces on the bonded interface.
--
Wolf a.k.a. Juha Laiho Espoo, Finland
(GC 3.0) GIT d- s+: a C++ ULSH++++$ P++@ L+++ E- W+$@ N++ !K w !O !M V
PS(+) PE Y+ PGP(+) t- 5 !X R !tv b+ !DI D G e+ h---- r+++ y++++
"...cancel my subscription to the resurrection!" (Jim Morrison)

Juha Laiho wrote:
> Hans Fugal <hfugal@wencor.com> said:
>
>>So backup, yes, in the sense that if one card goes haywire the other
>>card and IP address function normally. Or if all traffic were load
>>balanced and/or failover to the other card that would be ok too.
>
>
> What is the next link from the Linux machine? Are these two cards
> connected to a single switch?
Yes, they're connected to the same switch.

> If so, _if_ the switch supports
> channel bonding (aka Cisco EtherChannel, aka Sun trunking), you could
> configure the two interfaces to form one virtual interface, which would
> load-balance the traffic across the two cards in normal situations,
> and upon failure of one link would use the remaining one.
That's the view of incoming packets, and is less perfectly implemented
already in the form of the two IP addresses. However linux would
continue to route outbound packets through a single interface.

> And have
> just one IP address (unless you specifically need more, and create
> virtual interfaces on the bonded interface.

That's an interesting proposal. I'm fairly new here so I don't know the
historical reason why there's two interfaces with two IP addresses on
the same subnet. If load balancing was the primary reason this might be
a direction to look at.

Hans Fugal <hfugal@wencor.com> wrote in message news:<c8ljq7$ogd$1@terabinaries.xmission.com>...
> Juha Laiho wrote:
> > Hans Fugal <hfugal@wencor.com> said:
> >
> >>So backup, yes, in the sense that if one card goes haywire the other
> >>card and IP address function normally. Or if all traffic were load
> >>balanced and/or failover to the other card that would be ok too.
> >
> >
> > What is the next link from the Linux machine? Are these two cards
> > connected to a single switch?
> Yes, they're connected to the same switch.
>
> > If so, _if_ the switch supports
> > channel bonding (aka Cisco EtherChannel, aka Sun trunking), you could
> > configure the two interfaces to form one virtual interface, which would
> > load-balance the traffic across the two cards in normal situations,
> > and upon failure of one link would use the remaining one.
> That's the view of incoming packets, and is less perfectly implemented
> already in the form of the two IP addresses. However linux would
> continue to route outbound packets through a single interface.
>
> > And have
> > just one IP address (unless you specifically need more, and create
> > virtual interfaces on the bonded interface.
>
> That's an interesting proposal. I'm fairly new here so I don't know the
> historical reason why there's two interfaces with two IP addresses on
> the same subnet. If load balancing was the primary reason this might be
> a direction to look at.

I think J.L. has the most likely "best" proposal if the upstream links
will support it.

regards,
prg
email above disabled