forbidden vs not found with tilde

I have very little influence over the people who run the Apache server
I use. It always gives 403 (forbidden) when it fails to access a URL
with a tilde. That is, it never gives 404 (not found) for directories
that do not exist. Clearly this avoids giving anything away to
crackers. However, it also means it takes ages for the google.bot to
drop dead pages automatically.

Is this the default behaviour, or has the web master set it up? Can it
be changed if I work on the web master?

TIA,
Peter

On Mon, 11 Feb 2008, Peter Scott wrote:

> I have very little influence over the people who run the Apache server
> I use.

First you need to find out whether you can use Apache's .htaccess
files for directives. Create a file named
.htaccess
in your www directory and write some garbage into it.
If you get an "Internal server error", then we can proceed further.
If you do not get such an error, speak to the server administrator.

--
Solipsists of all countries - unite!

Thanks Andreas,

> First you need to find out whether you can use Apache's .htaccess
> files for directives. Create a file named
> .htaccess
> in your www directory and write some garbage into it.
> If you get an "Internal server error", then we can proceed further.
> If you do not get such an error, speak to the server administrator.

I can do that and I am already using it as a work-round, generating
404 errors for the removed pages.

I'd still like to know if there is a better way.

Peter

On Wed, 13 Feb 2008, Peter Scott wrote:

> I can do that and I am already using it as a work-round, generating
> 404 errors for the removed pages.

Fine then.

> I'd still like to know if there is a better way.

Sorry, but I don't understand what your problem is. Could you
please explain in more detail what you want to achieve?
[ Use the address http://www.example.com/ as sample. ]

--
Solipsists of all countries - unite!

"Peter Scott" <p.scott@shu.ac.uk> wrote in message
news:e08dba0f-0f87-471c-9318-28ac435d1de9@e6g2000prf.googlegroups.com...
>I have very little influence over the people who run the Apache server
> I use. It always gives 403 (forbidden) when it fails to access a URL
> with a tilde. That is, it never gives 404 (not found) for directories
> that do not exist. Clearly this avoids giving anything away to
> crackers. However, it also means it takes ages for the google.bot to
> drop dead pages automatically.
>
> Is this the default behaviour, or has the web master set it up? Can it
> be changed if I work on the web master?
>

It does appear to give things away to crackers, at least I'm seeing the
behaviour 403 for an existing user (or 200 if they have a web page in place)
and 404 for non-existing users. I'll assume this is what you are seeing...

So this would mean that the users specified after the tilde exist on the
server -
One thing that can be done is to explicitly disable the users that do not
have pages, with two lines like this:

UserDir public_html
UserDir disabled user1 user2

(the first line should already exist, so just add the second one underneath)
another would be to remove the users from the server (unless they still need
to be there for some other reason)
You could also use rewrite rules like these:

RewriteRule ^/~user1(/.*|)$ xxx [G,L]

RewriteRule ^/~user2(/.*|)$ xxx [G,L]

although these will produce '410 gone' rather than 404.

I have solved the problem! It is nothing to do with the Apache
configuration.

I want:

www.example.com/~fred/nonesuch

to give 404 (not found) if nonesuch does not exist. I want:

www.example.com/~fred/dir/nonesuch

to give 404 if dir or nonesuch does not exist.

It turns out they both give 403 (forbidden) when the lowest level
directory is unreadable. That is, when fred/public_html and
fred/public_html/dir are rwx--x--x. Making those directories readable
(rwxr-xr-x) makes Apache return 404 (not found).

Thinking about it, this is actually the optimum behaviour by Apache;
it
only reveals to the world as much as the owner of fred/public_html has
already given away by his or her choice of permissions.