multiple interfaces to Internet issue

Hello,

I have an issue with multiple connections to the Internet. I tried
following the steps described in [1] but things are not working
properly. I would like the network setup as follows:

______
| |- ppp0 -- Dynamic IP (PPPoE on eth2)
Internal---- eth0 | GW |
|____|- eth1 -- Static IP -> Static's GW


From [1], the steps I did were:

a. ip route flush table 4
b. ip route show table main | grep -Ev ^default \
| while read ROUTE ; do
ip route add table 4 $ROUTE
done
c. ip route add table 4 default via <Static IP>
d. iptables -t mangle -A PREROUTING -p tcp --dport 22 -s \
<Internal Net>/24 -j MARK --set-mark 4
e. iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
f. iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source \
<Static IP>
g. ip rule add fwmark 4 table 4
h. ip route flush cache

Now if I try to connect to (say) a web server, everything is fine: it
goes out the PPPoE connection just fine. When I try to SSH to the
machine (the same box as the web query) I never get the password
prompt.

Using tcpdump I get the following results. This is listening on eth1
as I try to SSH to the destination from an internal box (using lynx
to connect to the same destination results in a web page):

tcpdump: listening on eth1
07:13:12.614674 <Static IP>.37662 > <Dest IP>.ssh: S \
2808907073:2808907073(0) win 5840 <mss1460,sackOK,timestamp \
611570059 0,nop,wscale 0> (DF)
07:13:12.649772 <Dest IP>.ssh > <Static IP>.37662: S \
2414052745:2414052745(0) \
ack 2808907074 win 65535 <mss 1400,nop,wscale \
0,nop,nop,timestamp 2742813 611570059> (DF)
07:13:15.609403 <Static IP>.37662 > <Dest IP>.ssh: S \
2808907073:2808907073(0) win 5840 <mss 1460,sackOK,timestamp 61 \
1570359 0,nop,wscale 0> (DF)
07:13:15.643437 <Dest IP>.ssh > <Static IP>.37662: S \
2414052745:2414052745(0) \
ack 2808907074 win 65535 <mss 1400,nop,wscale \
0,nop,nop,timestamp 2743112 611570359> (DF)
07:13:18.634659 <Dest IP>.ssh > <Static IP>.37662: S \
2414052745:2414052745(0) ack \
2808907074 win 65535 <mss 1400,nop,wscale 0,nop,nop,timestamp \
2743412 611570359> (DF)

This is what the destination sees (not the same transaction):

tcpdump: listening on fxp0 (fxp is a FreeBSD interface)
07:15:59.917179 <Static IP>.37663 > <Dest IP>.ssh: S 30 \
01400670:3001400670(0) win 5840 <mss 1400,sackOK,timestamp
6115867860,nop,wscale 0> (DF)
07:15:59.917319 <Dest IP>.ssh > <Static IP>.37663: S 65 \
5604264:655604264(0) ack 3001400671 win 65535 <mss \
1452,nop,wscale0,nop,nop,timestamp 2759543 611586786> (DF)
07:16:02.911250 <Static IP>.37663 > <Dest IP>.ssh: S 30 \
01400670:3001400670(0) win 5840 <mss 1400,sackOK,timestamp \
6115870860,nop,wscale 0> (DF)
07:16:02.911369 <Dest IP>.ssh > <Static IP>.37663: S 65 \
5604264:655604264(0) ack 3001400671 win 65535 <mss \
1452,nop,wscale0,nop,nop,timestamp 2759842 611587086> (DF)
07:16:05.905034 <Dest IP>.ssh > <Static IP>.37663: S 65 \
5604264:655604264(0) ack 3001400671 win 65535 <mss \
1452,nop,wscale0,nop,nop,timestamp 2760142 611587086> (DF)

Also, I don't get a echo response back from the static IP. If I ping
the static's GW I get answers, but not the actual static IP. The echo
requests are gettng there though:

07:35:41.966769 <Dest IP> > <Static IP>: icmp: echo request
07:35:42.977156 <Dest IP> > <Static IP>: icmp: echo request
07:35:43.992579 <Dest IP> > <Static IP>: icmp: echo request
07:35:44.997944 <Dest IP> > <Static IP>: icmp: echo request
07:35:46.003377 <Dest IP> > <Static IP>: icmp: echo request

No responses come back though.

Any suggestions?

[1] http://linux-ip.net/html/adv-multi-internet.html

--
David Magda <dmagda at ee.ryerson.ca>, http://www.magda.ca/
Because the innovator has for enemies all those who have done well under
the old conditions, and lukewarm defenders in those who may do well
under the new. -- Niccolo Machiavelli, _The Prince_, Chapter VI

[comp.protocols.tcp-ip added in a cross-post. Original article was
only in comp.os.linux.networking.]

David Magda <dmagda+trace040423@ee.ryerson.ca> writes:
[...]
> ______
> | |- ppp0 -- Dynamic IP (PPPoE on eth2)
> Internal---- eth0 | GW |
> |____|- eth1 -- Static IP -> Static's GW
>
[...]
> Using tcpdump I get the following results. This is listening on
> eth1 as I try to SSH to the destination from an internal box (using
> lynx to connect to the same destination results in a web page):
[...]

Examing the output of tcpdump a bit more closely, it seems that the
host where the SSH client is trying to connect from never gets the
ACK in the TCP setup handshake. It's being sent by the server, it's
received on the external interface of the the gateway, but it never
makes it to the internal network.

The client machine keeps trying to setup a TCP connection, but never
receives the ACK. This is the interface (the client keeps trying to
setup the TCP connection):

tcpdump: listening on eth0
02:26:10.873080 [SSH client].37705 > [SSH server].22: S \
769441999:769441999(0) win 5840 <mss 1460,sackOK,timestamp \
6184875090,nop,wscale 0> (DF) [tos 0x10]
02:26:13.866409 [SSH client].37705 > [SSH server].22: S \
769441999:769441999(0) win 5840 <mss 1460,sackOK,timestamp \
6184878090,nop,wscale 0> (DF) [tos 0x10]

The external interface is getting the ACK (not from the same session,
but gets the point accross):

02:26:11.527294 [GW Ext. IP].ssh > [SSH server].49161: P \
224:336(112) ack 1 win 10944 <nop,nop,timestamp 557609690 \
1169951> (DF) [tos 0x10]

The ACK for the TCP connection setup is being sent by the server:

tcpdump: listening on fxp0
02:26:10.933176 [SSH server NATed].37705 > [SSH server].22: S \
769441999:769441999(0) win 5840 <mss 1400,sackOK,timestamp \
6184875090,nop,wscale 0> (DF) [tos 0x10]
02:26:10.933226 [SSH server].22 > [SSH server NATed].37705: S \
1054657654:1054657654(0) ack 769442000 win 65535 \
<mss 1452,nop,wscale0,nop,nop,timestamp 1071666 618487509> (DF)
02:26:13.923678 [SSH server].22 > [SSH server NATed].37705: S \
1054657654:1054657654(0) ack 769442000 win 65535 \
<mss 1452,nop,wscale0,nop,nop,timestamp 1071966 618487509> (DF)
02:26:13.926659 [SSH server NATed].37705 > [SSH server].22: S \
769441999:769441999(0) win 5840 <mss 1400,sackOK,timestamp \
6184878090,nop,wscale 0> (DF) [tos 0x10]
02:26:13.926712 [SSH server].22 > [SSH server NATed].37705: S \
1054657654:1054657654(0) ack 769442000 win 65535 \
<mss 1452,nop,wscale0,nop,nop,timestamp 1071966 618487809> (DF)
02:26:19.923038 [SSH server].22 > [SSH server NATed].37705: S \
1054657654:1054657654(0) ack 769442000 win 65535 \
<mss 1452,nop,wscale0,nop,nop,timestamp 1072566 618487809> (DF)

I've tried doing an SSH connection to multiple hosts and it's always
the same thing.

Here are my iptable rules:

gw2:~# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- anywhere anywhere
SNAT all -- anywhere anywhere to:<Static IP>

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

gw2:~# iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- 192.168.108.0/24 anywhere tcp \
dpt:ssh MARK set 0x4

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

IP rule list:

gw2:~# ip rule list
0: from all lookup local
32765: from all fwmark 4 lookup 4
32766: from all lookup main
32767: from all lookup default

Routing tables:

gw2:/home/mpathix# ip route show table main
<PPPoE peer> dev ppp0 proto kernel scope link src 69.158.104.154
63.250.109.128/29 dev eth1 proto kernel scope link src
<Static IP>
192.168.108.0/24 dev eth0 proto kernel scope link src
<GW's Internal IP>
default via <PPPoE peer> dev ppp0

gw2:/home/mpathix# ip route show table 4
<PPPoE peer> dev ppp0 proto kernel scope link src 69.158.104.154
63.250.109.128/29 dev eth1 proto kernel scope link src
<Static IP>
192.168.108.0/24 dev eth0 proto kernel scope link src
<Static IP>
default via <Static's GW> dev eth1

So basically packets are getting out, but they're not getting back
in.

Any suggestions?

[I'm also posting this to the LARTC mailing list.]