iptables rule causes delay in password prompt

Granted, I am far from an iptables expert, so assume nothing about my
expertise here. My iptables rules has about 340 lines (some comments).
I'm trying to block all access to an off-site server (i.e.
192.168.20.121 below), only allowing connection from one on-site server
(i.e. 192.168.1.20 below). The allowed server is our backup server,
being allowed to rsync itself to the off-site server via ssh. When I
add this rule:

-A FORWARD -s ! 192.168.1.20 -d 192.168.20.121 -j DROP

I accomplish what I want. SSH (or any other connection) is blocked
except from the allowed server. However, with this rule in place, there
is consistently a 10-15 second delay before receiving a password prompt
in ssh connections. Without the rule, immediate prompting takes place.

I've checked all logs on both servers, with no indication of a problem.
I've straced the PID (using top to try to identify what process is
running that may be holding it up), with no noticable PID to trace. The
only thing noticable is after ssh connnection is established.

I have tried reordering the rule to the top and bottom of this FORWARD
chain, as well as placing it at the very top of the rules. I've also
tried switching the -d and -s listing order in the rule itself. No
luck.

Any ideas on where I can look to solve this? Or, anyone have similar
symptoms and solutions?

fmeeusen@gmail.com wrote:
> Granted, I am far from an iptables expert, so assume nothing about my
> expertise here. My iptables rules has about 340 lines (some comments).
> I'm trying to block all access to an off-site server (i.e.
> 192.168.20.121 below), only allowing connection from one on-site server
> (i.e. 192.168.1.20 below). The allowed server is our backup server,
> being allowed to rsync itself to the off-site server via ssh. When I
> add this rule:
>
> -A FORWARD -s ! 192.168.1.20 -d 192.168.20.121 -j DROP
>
> I accomplish what I want. SSH (or any other connection) is blocked
> except from the allowed server. However, with this rule in place, there
> is consistently a 10-15 second delay before receiving a password prompt
> in ssh connections. Without the rule, immediate prompting takes place.
>
> I've checked all logs on both servers, with no indication of a problem.
> I've straced the PID (using top to try to identify what process is
> running that may be holding it up), with no noticable PID to trace. The
> only thing noticable is after ssh connnection is established.
>
> I have tried reordering the rule to the top and bottom of this FORWARD
> chain, as well as placing it at the very top of the rules. I've also
> tried switching the -d and -s listing order in the rule itself. No
> luck.
>
> Any ideas on where I can look to solve this? Or, anyone have similar
> symptoms and solutions?


Check if your target is attempting an IDENT for the attempted
SSH connection. It is a TCP connection from the target to the
source (in reverse direcrtion) using TCP port 113.

A way to dig further is to install Ethereal or tcpdump and
take traces of the SSH connection set-up. You'll see from
the timestamps who is delaying.

--

Tauno Voipio
tauno voipio (at) iki fi

Hi -

On 18 Nov 2005 12:29:20 -0800, fmeeusen@gmail.com wrote:

>Any ideas on where I can look to solve this? Or, anyone have similar
>symptoms and solutions?

As Tauno has indicated, it could be an IDENT problem or similar.
Trying change DROP to REJECT (so the connector gets a rejection packet
back instead of nothing) and see if that takes care of the problem.

--
Ken
http://www.ke9nr.net/

Really appreciate the replies. I'm tossed between too many projects
now, so sorry for the delay. I ran tcpdump -vv on the server I was
trying to connect from, and received the following output. It makes
very little sense to me, except that there is a 10 second delay between
the attempt to connect to "target", and the actual permission being
granted (13:17:33.405452 - 13:17:43.367647). (I've obviously changed
domain info for security. I've also renamed the servers to descriptive
names; i.e. "target", "source", "firewall", etc.).

13:17:31.621153 IP source.mydomain.com.ssh >
firewall.mydomain.com.47524: P 3130197734:3130197862(128) ack
1299811027 win 8704 <nop,nop,timestamp 103876948 2679965792>
13:17:31.621616 IP firewall.mydomain.com.47524 >
source.mydomain.com.ssh: . ack 128 win 3874 <nop,nop,timestamp
2679965795 103876948>
13:17:32.160926 arp who-has 192.168.1.62 tell firewall.mydomain.com
13:17:32.166798 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:33.166715 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:33.298013 IP firewall.mydomain.com.60623 >
source.mydomain.com.ssh: P 1173061681:1173061729(48) ack 2084941618 win
3068 <nop,nop,timestamp 2679967472 103876676>
13:17:33.298289 IP source.mydomain.com.ssh >
firewall.mydomain.com.60623: P 1:49(48) ack 48 win 8704
<nop,nop,timestamp 103877116 2679967472>
13:17:33.298390 IP firewall.mydomain.com.60623 >
source.mydomain.com.ssh: . ack 49 win 3068 <nop,nop,timestamp
2679967472 103877116>
13:17:33.302244 IP source.mydomain.com.32772 > dns.mydomain.com.domain:
6105+ AAAA? target.mydomain.com. (42)
13:17:33.302388 IP dns.mydomain.com.domain > source.mydomain.com.32772:
6105 0/0/0 (42)
13:17:33.302436 IP source.mydomain.com.32772 > dns.mydomain.com.domain:
6106+ AAAA? target. (26)
13:17:33.302636 IP dns.mydomain.com.domain > source.mydomain.com.32772:
6106 NXDomain* 0/0/0 (26)
13:17:33.302661 IP source.mydomain.com.32772 > dns.mydomain.com.domain:
6107+ A? target.mydomain.com. (42)
13:17:33.302761 IP dns.mydomain.com.domain > source.mydomain.com.32772:
6107 1/0/0 A[|domain]
13:17:33.302830 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
S 2927703394:2927703394(0) win 5840 <mss 1460,sackOK,timestamp
103877116 0,nop,wscale 0>
13:17:33.303260 IP target.mydomain.com.ssh > source.mydomain.com.32879:
S 2834760207:2834760207(0) ack 2927703395 win 5792 <mss
1460,sackOK,timestamp 95777510 103877116,nop,wscale 0>
13:17:33.303272 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
.. ack 1 win 5840 <nop,nop,timestamp 103877116 95777510>
13:17:33.304510 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1:42(41) ack 1 win 5792 <nop,nop,timestamp 95777512 103877116>
13:17:33.304519 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
.. ack 42 win 5840 <nop,nop,timestamp 103877116 95777512>
13:17:33.304582 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 1:42(41) ack 42 win 5840 <nop,nop,timestamp 103877116 95777512>
13:17:33.304884 IP target.mydomain.com.ssh > source.mydomain.com.32879:
.. ack 42 win 5792 <nop,nop,timestamp 95777512 103877116>
13:17:33.304888 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 42:650(608) ack 42 win 5840 <nop,nop,timestamp 103877116 95777512>
13:17:33.305259 IP target.mydomain.com.ssh > source.mydomain.com.32879:
.. ack 650 win 6688 <nop,nop,timestamp 95777512 103877116>
13:17:33.305884 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 42:650(608) ack 650 win 6688 <nop,nop,timestamp 95777513 103877116>
13:17:33.305948 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 650:674(24) ack 650 win 6688 <nop,nop,timestamp 103877117 95777513>
13:17:33.309132 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 650:802(152) ack 674 win 6688 <nop,nop,timestamp 95777516 103877117>
13:17:33.311748 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 674:818(144) ack 802 win 6688 <nop,nop,timestamp 103877117 95777516>
13:17:33.318502 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 802:1266(464) ack 818 win 6688 <nop,nop,timestamp 95777526 103877117>
13:17:33.322105 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 818:834(16) ack 1266 win 7904 <nop,nop,timestamp 103877118 95777526>
13:17:33.362352 IP target.mydomain.com.ssh > source.mydomain.com.32879:
.. ack 834 win 6688 <nop,nop,timestamp 95777570 103877118>
13:17:33.362356 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 834:882(48) ack 1266 win 7904 <nop,nop,timestamp 103877122 95777570>
13:17:33.362726 IP target.mydomain.com.ssh > source.mydomain.com.32879:
.. ack 882 win 6688 <nop,nop,timestamp 95777570 103877122>
13:17:33.362728 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1266:1314(48) ack 882 win 6688 <nop,nop,timestamp 95777570 103877122>
13:17:33.362779 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 882:946(64) ack 1314 win 7904 <nop,nop,timestamp 103877122 95777570>
13:17:33.365600 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1314:1378(64) ack 946 win 6688 <nop,nop,timestamp 95777573 103877122>
13:17:33.365651 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 946:1186(240) ack 1378 win 7904 <nop,nop,timestamp 103877123
95777573>
13:17:33.405452 IP target.mydomain.com.ssh > source.mydomain.com.32879:
.. ack 1186 win 7904 <nop,nop,timestamp 95777613 103877123>
13:17:34.166632 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:34.345013 arp who-has server4.mydomain.com tell
source.mydomain.com
13:17:34.345154 arp reply server4.mydomain.com is-at 00:03:93:bb:e9:78
13:17:34.722933 arp who-has dns.mydomain.com tell server6.mydomain.com
13:17:34.826498 IP server5.mydomain.com.netbios-ns >
192.168.1.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST;
BROADCAST
13:17:35.166675 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:36.273156 arp who-has nonexistant.mydomain.com
(00:c0:b7:65:87:00) tell apc.mydomain.com
13:17:37.886215 IP mail.mydomain.com.52889 > source.mydomain.com.rsync:
.. ack 3880256534 win 0 <nop,nop,timestamp 267497628 103865572>
13:17:37.886220 IP source.mydomain.com.rsync > mail.mydomain.com.52889:
.. ack 1 win 0 <nop,nop,timestamp 103877575 267492315>
13:17:38.295012 arp who-has dns.mydomain.com tell source.mydomain.com
13:17:38.295103 arp reply dns.mydomain.com is-at 00:e0:81:20:e9:78
13:17:39.156475 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:40.156394 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:41.156311 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:41.323338 arp who-has nonexistant.mydomain.com
(00:c0:b7:65:87:00) tell apc.mydomain.com
13:17:42.156353 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:43.156270 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:43.367647 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1378:1570(192) ack 1186 win 7904 <nop,nop,timestamp 95787575
103877123>
13:17:43.372668 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 1186:1570(384) ack 1570 win 7904 <nop,nop,timestamp 103878123
95787575>
13:17:43.373144 IP target.mydomain.com.ssh > source.mydomain.com.32879:
.. ack 1570 win 9120 <nop,nop,timestamp 95787581 103878123>
13:17:43.374018 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1570:1602(32) ack 1570 win 9120 <nop,nop,timestamp 95787581
103878123>
13:17:43.374125 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 1570:1634(64) ack 1602 win 7904 <nop,nop,timestamp 103878123
95787581>
13:17:43.374892 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1602:1650(48) ack 1634 win 9120 <nop,nop,timestamp 95787582
103878123>
13:17:43.374983 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
P 1634:2018(384) ack 1650 win 7904 <nop,nop,timestamp 103878123
95787582>
13:17:43.385262 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1650:1698(48) ack 2018 win 10336 <nop,nop,timestamp 95787593
103878123>
13:17:43.385266 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1698:1794(96) ack 2018 win 10336 <nop,nop,timestamp 95787593
103878123>
13:17:43.385358 IP source.mydomain.com.ssh >
firewall.mydomain.com.60623: P 49:145(96) ack 48 win 8704
<nop,nop,timestamp 103878125 2679967472>
13:17:43.385513 IP firewall.mydomain.com.60623 >
source.mydomain.com.ssh: . ack 145 win 3068 <nop,nop,timestamp
2679977560 103878125>
13:17:43.425012 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
.. ack 1794 win 7904 <nop,nop,timestamp 103878129 95787593>
13:17:43.425613 IP target.mydomain.com.ssh > source.mydomain.com.32879:
P 1794:1842(48) ack 2018 win 10336 <nop,nop,timestamp 95787633
103878129>
13:17:43.425617 IP source.mydomain.com.32879 > target.mydomain.com.ssh:
.. ack 1842 win 7904 <nop,nop,timestamp 103878129 95787633>
13:17:43.425677 IP source.mydomain.com.ssh >
firewall.mydomain.com.60623: P 145:193(48) ack 48 win 8704
<nop,nop,timestamp 103878129 2679977560>
13:17:43.425988 IP firewall.mydomain.com.60623 >
source.mydomain.com.ssh: . ack 193 win 3068 <nop,nop,timestamp
2679977600 103878129>
13:17:43.490326 arp who-has 192.168.1.253 tell firewall.mydomain.com
13:17:44.156189 arp who-has 192.168.1.65 tell dialup.mydomain.com
13:17:44.490243 arp who-has 192.168.1.253 tell firewall.mydomain.com
13:17:45.490161 arp who-has 192.168.1.253 tell firewall.mydomain.com
13:17:45.758380 IP firewall.mydomain.com.47524 >
source.mydomain.com.ssh: P 1:49(48) ack 128 win 3874 <nop,nop,timestamp
2679979933 103876948>

By the way, I did change DROP to REJECT, with no notable difference. I
also added a rule to allow tcp 113 from target to source, but that was
kind of a half-hearted shot in the dark experiment.

add on 192.168.1.20 /etc/hosts file:
192.168.20.121 server_name

enjoy...

On Fri, 18 Nov 2005 21:29:20 +0100, <fmeeusen@gmail.com> wrote:

> Granted, I am far from an iptables expert, so assume nothing about my
> expertise here. My iptables rules has about 340 lines (some comments).
> I'm trying to block all access to an off-site server (i.e.
> 192.168.20.121 below), only allowing connection from one on-site server
> (i.e. 192.168.1.20 below). The allowed server is our backup server,
> being allowed to rsync itself to the off-site server via ssh. When I
> add this rule:
>
> -A FORWARD -s ! 192.168.1.20 -d 192.168.20.121 -j DROP
>
> I accomplish what I want. SSH (or any other connection) is blocked
> except from the allowed server. However, with this rule in place, there
> is consistently a 10-15 second delay before receiving a password prompt
> in ssh connections. Without the rule, immediate prompting takes place.
>
> I've checked all logs on both servers, with no indication of a problem.
> I've straced the PID (using top to try to identify what process is
> running that may be holding it up), with no noticable PID to trace. The
> only thing noticable is after ssh connnection is established.
>
> I have tried reordering the rule to the top and bottom of this FORWARD
> chain, as well as placing it at the very top of the rules. I've also
> tried switching the -d and -s listing order in the rule itself. No
> luck.
>
> Any ideas on where I can look to solve this? Or, anyone have similar
> symptoms and solutions?
>

Thank you for the suggestion. I see the reasoning behind it, but it did
not have any effect to the problem. Considering the problem goes away
when I disengage the firewall rule, I'm pretty sure the problem is in
my omission of some rule that will allow the type of communication the
server is asking for... very probably what Tauno has suggested. I have
just not yet made the appropriate firewall rule change to incorporate
this. tcpdump has not revealed anything meaningful to me, though it is
possible I have not read it properly, or not incorporated the
appropriate switches to get the info I need. The installation and
dependencies of Ethereal seems rather large, but perhaps I should
aptitude install it to give it a go.

Solved: created a firewall rule allowing DNS inquiries between this
remote server and the DNS server. No delays now when connecting. Not
sure if it's the ideal solution, but it works with no obvious security
risks. Thank you for the suggestions.