Is it a black hole, or just my ruleset? Mystery!

Some users cannot access my web site. Here's what I know:

- Running j-archive.com on an OpenBSD 3.1 server behind an ADSL line.
The pf ruleset allows for HTTP and TCP traffic.
- Works fine for most users, but some users cannot access the site.
- The users who cannot access the site *can* access smaller web pages
hosted from the same server.
- With my pf ruleset unloaded, the site becomes accessible to those who
could not access it.
- With the pf ruleset loaded but the "block out log all" line
commented, the site becomes accessible to all.
- Adding the line "scrub out all no-df" to the ruleset does not make
the site accessible to those who could not access it.
- Adding the line "scrub out all max-mss 1470" to the ruleset (on the
theory that this will clear up a Path MTU black hole) causes a syntax
error on that line. (Why?)
- One user who cannot normally access the site reports that he can ping
my server up to 8104 bytes, but a packet size of 8105 bytes never goes
through.

Is there anything else I can try to fix this for my users? Or should I
just inform the unlucky few that their networks are improperly
configured and there's nothing I can change on my end to make the site
accessible other than bringing my firewall down completely?

More on Path MTU Black Holes:
"Over-Zealous Security Administrators Are Breaking the Internet"
http://tinyurl.com/9qmmh

All the best,
Robert K S

My pf ruleset:

# Variable expansion
networksegment="192.168.254.0/24"
myserver="192.168.254.101/32"
timekeeper="128.9.176.30/32"
# /24 indicates only the first 24 bits of the address are to be used;
# /32 signifies the entire address is to be used.

# Clean up fragmented and abnormal packets
scrub in all

# Block everything by default.
block in log all
block out log all

# Allow loopback packets
pass in quick on lo0 all
pass out quick on lo0 all

# Start by allowing the normal remote management access. Port 22 for
SSH.
pass in quick on rl0 proto tcp from any to $myserver port = 22 flags
S keep state

# Allow local NTP (port 123) and ping (ICMP) traffic in both
directions.
pass in quick on rl0 proto udp from $timekeeper port = 123 to
$networksegment port = 123
pass out quick on rl0 proto udp from $networksegment port = 123 to
$timekeeper port = 123
pass in quick on rl0 proto icmp from any to $networksegment
pass out quick on rl0 proto icmp from $networksegment to any keep state

# Allow all web traffic (80 for http and 443 for https).
pass in quick on rl0 proto tcp from any to $myserver port = 80 flags
S/SA modulate state
pass out quick on rl0 proto tcp from $myserver to any port = 80 flags
S/SA modulate state
pass in quick on rl0 proto tcp from any to $myserver port = 443 flags
S/SA modulate state

On 25 Apr 2005 12:56:28 -0700, Robert K S <robertks@hotmail.com> wrote:
> Some users cannot access my web site. Here's what I know:
>
> - Running j-archive.com on an OpenBSD 3.1 server behind an ADSL line.
> The pf ruleset allows for HTTP and TCP traffic.
> - Works fine for most users, but some users cannot access the site.
> - The users who cannot access the site *can* access smaller web pages
> hosted from the same server.
> - With my pf ruleset unloaded, the site becomes accessible to those who
> could not access it.
> - With the pf ruleset loaded but the "block out log all" line
> commented, the site becomes accessible to all.
> - Adding the line "scrub out all no-df" to the ruleset does not make
> the site accessible to those who could not access it.
> - Adding the line "scrub out all max-mss 1470" to the ruleset (on the
> theory that this will clear up a Path MTU black hole) causes a syntax
> error on that line. (Why?)

What is intended function of that (I am not familiar with OpenBSD)?
Ethernet mtu 1500 would have mss 1460, PPPoE max mtu 1492 max would have
mss 1452, although, I hear XP PPPoE uses mtu 1480 which would be mss
1440. Dialup PPP _might_ have mtu 1524, although, that should not be a
problem for receiving packets smaller than that. Your max-mss 1470 would
be an oddball mtu 1510 (maybe an error because it exceeds your ethernet).

> - One user who cannot normally access the site reports that he can ping
> my server up to 8104 bytes, but a packet size of 8105 bytes never goes
> through.
>
> Is there anything else I can try to fix this for my users? Or should I
> just inform the unlucky few that their networks are improperly
> configured and there's nothing I can change on my end to make the site
> accessible other than bringing my firewall down completely?
>
> More on Path MTU Black Holes:
> "Over-Zealous Security Administrators Are Breaking the Internet"
> http://tinyurl.com/9qmmh

If your private IP server is behind PPPoE, you might try setting LAN nic
of server to mtu 1492 (or whatever mtu of PPPoE). Then at least your end
would hopefully use a suitable mss.

I had a similar problem, except I couldn't get to certain sites.

Make sure in ppp.conf:
set mtu 1452
enable mssfixup


And in pf.conf:
scrub in all max-mss 1452
scrub out all max-mss 1452

This may help as well:
http://groups-beta.google.com/group/...+nichol&rnum=5

Has something to do with what pppoe does to the packets (I think?).
Anyway, hope I helped.

best regards,
Reid Nichol

As I mentioned, I cannot set max-mss in my pf.conf since it gives a
syntax error.

No ideas why?

Also, I don't have a ppp.conf. My OpenBSD box is connected to a
Speedstream DSL modem/router.

> As I mentioned, I cannot set max-mss in my pf.conf since it gives a syntax
> error.
>
> No ideas why?
Please note that I changed the if in what I sub'd. The below changes are
clean on my system (OBSD 3.5).

The error is on this line (*without* scrub in the file):
pass in quick on rl0 proto tcp from any to $myserver port = 22 flags S
keep state

Changed to this and error goes away:
pass in quick on xl0 proto tcp from any to $myserver port ssh flags S/SA
keep state

Add in this (right where you specify) and still no sytax error:
scrub in all max-mss 1452
scrub out all max-mss 1452


You also shouldn't explicitly state you if in the rule. You should set a
var and use that. ie what if you got a new if? Do you want to change x
number of lines, or just the one var?

On a last note, where I come from, my last post contained some ideas. So,
I'm just wondering, is this post enough of an idea for you?

> Also, I don't have a ppp.conf. My OpenBSD box is connected to a
> Speedstream DSL modem/router.
Then you should've either:
1) ignored that part of the post as stating this now makes you look like
an prick.
or
2) mentioned it before as mostly (in my experience at least) ADSL = PPPoE
and it may change something. I don't think that it does, but then that
would just make this immaterial to the discussion (see (1)).

Thanks for the suggestions, Reid, but it is definitely the "max-mss"
that is giving me the syntax error on my OpenBSD 3.1 system (and I
thought 3.1 supported this directive). Making the modification to the
SSH line that you suggest only gives me a further error as it says xl0
in an "unknown interface."

But you're right of course that variable expansion would be the proper
way to denote addresses, devices, etc.

Robert K S wrote:

> Also, I don't have a ppp.conf. My OpenBSD box is connected to a
> Speedstream DSL modem/router.


In that case try knocking the MTU on your outside NIC down some. Might even
be a good idea to set it very low (say 500) then if everyone can get
through slowly up it until they can't.

Reid Nichol wrote:

>> Also, I don't have a ppp.conf. My OpenBSD box is connected to a
>> Speedstream DSL modem/router.
> Then you should've either:
> 1) ignored that part of the post as stating this now makes you look like
> an prick.
> or
> 2) mentioned it before as mostly (in my experience at least) ADSL = PPPoE
>

This seems to be national issue. The UK standard is PPPoA although some
experiments are being made with PPPoE.

I have no idea which other countries (if any) use PPPoA.