Apache 2.0: lack of listeners

Hello everyone,

I have a problem with Apache 2.0.46, packaged by RedHat for its EL3,
and the number of requests it can handle, or at least I think the
problem lies there. The server runs a webapp with Tomcat5, we use
mod_jk 1.2.6 as a glue between both.

We have our site monitored with ip-label (www.ip-label.net), which
makes the same scenario run from 6 different sites, every 15 minutes.
In its reports, it tells that a few objects couldn't be downloaded,
most of the time static ones (there was an issue with tomcat
configuration but this is a separate issue from this one.

I have made a simple test: my firewall, based on iptables, can tell me
how many connections there were to port 443 during a period of time, so
I reset the counter, counted the lines in the access_log, slept for 15
minutes and then displayed the iptables counter and the number of lines
in the access_log again: each time, there are more connections to port
443 than there are new lines in the access_log (approx. 30 to 50).
Which is why I believe Apache is misconfigured. The trouble is, nothing
at all shows in the error logs...

The trouble is, my Apache skills are average at best and therefore I'm
looking for help... Here are the relevant settings, or at least what I
think are the relevant settings:

-- httpd.conf --
Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
MaxClients 150
MaxRequestsPerChild 1000
</IfModule>
<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
-- end httpd.conf --

-- one2team.conf, included file --
LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf.d/workers.properties
JkLogFile logs/jk.log
JkLogLevel info
JkMount /*.jsp ajp13
JkMount /servlet/* ajp13
JkMount /webdav ajp13
JkMount /webdav/* ajp13
[...]
<VirtualHost the.server.name:443>
[...]
RedirectMatch permanent ^/$
https://edf.one2team.com/jsp/login/login.jsp
RedirectMatch permanent .*/favicon\.ico$
https://edf.one2team.com/images/one2team.ico
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite
ALL:!ADH:RC4+RSA:+HIGH:-MEDIUM:-SSLv2:-LOW:-eNULL
SSLOptions +StdEnvVars +ExportCertData
SSLCertificateKeyFile conf/ssl.key/edf.key
SSLCertificateFile conf/ssl.crt/edf.crt
ExpiresActive On
ExpiresDefault "access plus 2 month"
Header set Author "One2team"
AddDefaultCharset UTF-8
SetOutputFilter DEFLATE
SetInputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
dont-vary
SetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:exe|com)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI
\.(?:t?gz|zip|bz2|sit|rar|lha|Z|arc|jar|war)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI
\.(?:pdf|stc|std|sti|stw|sxc|sxd|sxg|sxi|sxm|sxw)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.js$ no-gzip dont-vary
DeflateMemLevel 9
DeflateCompressionLevel 3
DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
<Location "/jsp">
Header append Pragma "no-cache"
ExpiresDefault "now"
Header set Cache-Control "no-store, private, no-cache,
must-revalidate, proxy-revalidate"
Header unset Vary
</Location>
<LocationMatch "/(webdav/|servlet/)">
Header append Pragma "no-cache"
ExpiresDefault "now"
Header set Cache-Control "no-store, private, no-cache,
must-revalidate, proxy-revalidate"
Header unset Vary
</LocationMatch>
<LocationMatch
"/servlet/(ShowDocumentContent|ShowBinary|DownloadServlet|Ch artServlet)">
Header set Cache-Control "private, must-revalidate,
proxy-revalidate, max-age=0"
Header unset Pragma
Header unset Vary
</LocationMatch>
<Location "/html/doc/">
Options Indexes FollowSymlinks
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_ACCEPT} application/xhtml\+xml
RewriteCond %{HTTP_ACCEPT}
!application/xhtml\+xml\s*;\s*q=0
RewriteCond %{REQUEST_URI} \.html$
RewriteCond %{THE_REQUEST} HTTP/1\.1
RewriteRule .* - [T=application/xhtml+xml]
</Location>
<Location "/html/doc/fr">
DefaultLanguage fr
</Location>
<Location "/WEB-INF">
AllowOverride None
Order allow,deny
Deny from all
</Location>
<Directory "/var/lib/tomcat5/webapps/one2team/WEB-INF">
AllowOverride None
Order allow,deny
Deny from all
</Directory>
<Directory "/var/lib/tomcat5/webapps/one2team/">
Options FollowSymLinks
</Directory>
</VirtualHost>

About the httpd.conf: I don't see the prefork or worker module inserted
anywhere via a LoadModule, does that mean that these settings are
totally ignored, or is at least the prefork module built-in? I do see
several Apache processes forking on startup... I've read numerous
documents about performance settings, but I still don't know what
setting to modify. I'd guess some candidates would be StartServers,
MinSpareServers and MaxSpareServers, but then they're in an <ifmodule>
which I don't even know whether it's used at all :(

TIA, and have a nice day!

"Francis Galiegue" <fgaliegue@gmail.com> schreef in bericht
news:1147086002.170834.72780@g10g2000cwb.googlegro ups.com...
> I have a problem with Apache 2.0.46, packaged by RedHat for its EL3,
> and the number of requests it can handle, or at least I think the
> problem lies there.
Share with us the error_log lines that stems for this idea.

> I have made a simple test: my firewall, based on iptables, can tell me
> how many connections there were to port 443 during a period of time, so
> I reset the counter, counted the lines in the access_log, slept for 15
> minutes and then displayed the iptables counter and the number of lines
> in the access_log again: each time, there are more connections to port
> 443 than there are new lines in the access_log (approx. 30 to 50).
> Which is why I believe Apache is misconfigured. The trouble is, nothing
> at all shows in the error logs...
In theory one can make a connection without issuing a request.
Without making a request, there is nothing apache will ever log.

On the other hand one can issue a series of requests on a single connection
....

Did you count _unique_ IP adresses?

[snipped]
> About the httpd.conf: I don't see the prefork or worker module inserted
> anywhere via a LoadModule, does that mean that these settings are
> totally ignored, or is at least the prefork module built-in?
http://httpd.apache.org/docs/2.0/mpm.html
MPMs must be chosen during configuration, and compiled into the server.
Compilers are capable of optimizing a lot of functions if threads are used,
but only if they know that threads are being used.
To actually choose the desired MPM, use the argument --with-mpm=NAME with
the configure script. NAME is the name of the desired MPM.

Once the server has been compiled, it is possible to determine which MPM was
chosen by using ./httpd -l. This command will list every module that is
compiled into the server, including the MPM.

> I do see
> several Apache processes forking on startup... I've read numerous
> documents about performance settings, but I still don't know what
> setting to modify. I'd guess some candidates would be StartServers,
> MinSpareServers and MaxSpareServers, but then they're in an <ifmodule>
> which I don't even know whether it's used at all :(

A better one to fiddle might be 'maxclients'; your error_log may even refer
to it.

http://httpd.apache.org/docs/2.0/mod...tml#maxclients


HansH

HansH wrote:
> "Francis Galiegue" <fgaliegue@gmail.com> schreef in bericht
> news:1147086002.170834.72780@g10g2000cwb.googlegro ups.com...
> > I have a problem with Apache 2.0.46, packaged by RedHat for its EL3,
> > and the number of requests it can handle, or at least I think the
> > problem lies there.
> Share with us the error_log lines that stems for this idea.
>

That's the trouble: there isn't any. What I do see in the ip-label
report though is that some static objects are not delivered on time. I
couldn't establish a relationship yet between the missing objects and
the access_log, that's the point I'm looking at right now.

> > I have made a simple test: my firewall, based on iptables, can tell me
> > how many connections there were to port 443 during a period of time, so
> > I reset the counter, counted the lines in the access_log, slept for 15
> > minutes and then displayed the iptables counter and the number of lines
> > in the access_log again: each time, there are more connections to port
> > 443 than there are new lines in the access_log (approx. 30 to 50).
> > Which is why I believe Apache is misconfigured. The trouble is, nothing
> > at all shows in the error logs...
> In theory one can make a connection without issuing a request.
> Without making a request, there is nothing apache will ever log.
>
> On the other hand one can issue a series of requests on a single connection
> ...
>

I suppose that's what the MaxRequestPerChild means?

> Did you count _unique_ IP adresses?
>

No, that's right. I'll do it at the firewall level as well and try and
compare this with error_log. Good idea!

> [snipped]
> > About the httpd.conf: I don't see the prefork or worker module inserted
> > anywhere via a LoadModule, does that mean that these settings are
> > totally ignored, or is at least the prefork module built-in?
> http://httpd.apache.org/docs/2.0/mpm.html
> MPMs must be chosen during configuration, and compiled into the server.
> Compilers are capable of optimizing a lot of functions if threads are used,
> but only if they know that threads are being used.
> To actually choose the desired MPM, use the argument --with-mpm=NAME with
> the configure script. NAME is the name of the desired MPM.
>
> Once the server has been compiled, it is possible to determine which MPM was
> chosen by using ./httpd -l. This command will list every module that is
> compiled into the server, including the MPM.
>

Thanks for the tip! httpd -l does show prefork module built in.

> > I do see
> > several Apache processes forking on startup... I've read numerous
> > documents about performance settings, but I still don't know what
> > setting to modify. I'd guess some candidates would be StartServers,
> > MinSpareServers and MaxSpareServers, but then they're in an <ifmodule>
> > which I don't even know whether it's used at all :(
>
> A better one to fiddle might be 'maxclients'; your error_log may even refer
> to it.
>

As stated above, nothing in the logs, that's the real trouble :(

"Francis Galiegue" <fgaliegue@gmail.com> schreef in bericht
news:1147160527.412414.85090@j33g2000cwa.googlegro ups.com...
> > In theory one can make a connection without issuing a request.
> > Without making a request, there is nothing apache will ever log.
> >
> > On the other hand one can issue a series of requests on a single
connection
> I suppose that's what the MaxRequestPerChild means?
Negative,
http://httpd.apache.org/docs/2.0/mod...paliverequests
Btw MaxRequestPerChild does not apply to the prefork mpm you use.

> As stated above, nothing in the logs, that's the real trouble :(
There is one log not related to the virtual hosts, did you check that one
too?

http://httpd.apache.org/docs/2.0/mod/core.html#timeout


If IP-label is running at the far end of the internet -having your webserver
at the near end-, any networking component may distort the report.


HansH

Well OK, there was a coupe of things:

1. I did look in the VHost specific log, what I didn't understand at
first is that all requests were logged twice except for the Nagios
probes. So, I cured it by reformatting completely the logs and not
append the CustomLog to the access file.

2. KeepAlive was off. I put it on. Since then the monitor doesn't show
any errors anymore... I also put MaxKeepAliveRequests to 0 and let the
KeepAliveTimeout to its default value of 15 (seconds).

All in all, the problem is now solved. There are still quirks with
Tomcat 5 though, but that's offtopic for this forum I guess...