(Problem) Apache Connection Rate Measurement on Intel Servers

Problem in brief: I am unable to increase HTTP/SSL connection rate,
while my Apache server platform shows a huge (98%) CPU slack.

----------------



I benchmark the performance of several Linux/Open Source applications
on Intel Architecture (IA32/x86) servers. In this context, I am trying
to measure the peak HTTP connection rate sustained when SSL is enabled
in Apache. A 'connection' in this benchmark consists of:



TCP Connect

HTTP Connect

GET_SSL(a small file with a few bytes)

HTTP Disconnect

TCP Disconnect



My test server side test setup consists of:



(1) Intel 5300 Xeon (quad core) 2.67GHz, dual socket platform (with
Intel 5000P chipset) - i.e., an 8 way SMP platform

(2) Linux 2.6.13 (CentOS 4.1)

(3) Apache 2.0.52

(4) OpenSSL 0.9.7e

(5) Intel 82571 PCI Express Gigabit Ethernet NIC



My client side test set up consists of:

IxIA L4-L7 network traffic generator that is capable of generating upto
***-220 connections/sec-*** and emulate several thousand simultaneous
HTTP/SSL clients. (Aptixia XM12 test chassis with IxIA Application
Logic Module test card).



The IxIA test module, when tested against Windows 2000/IIS has measured
up to ***-220 HTTPS connections per second-*** on a 2.8GHz single
CPU/single core Xeon platform.



PROBLEM: When I test my Linux/Apache server for the same HTTPS
connection rate, my IxIA tester measures ***-at most 10
connections/sec-***, while my server shows at least a 98% CPU slack (as
measured using Linux TOP utility). I have played with various TCP
connection/persistence/buffer options, but with no improvement what so
ever.



Also, when I run the same test without SSL, I get a huge connection
rate of ***-40000 HTTP connections/sec-***. This leads me to believe
that the problem lies somewhere with the modSSL module & not Apache
itself, ...but I am totally clueless after probing OpenSSL stack for
several weeks. Some basic Ethereal analysis shows that the server is
issuing a TCP RESET for nearly almost all connection requests from the
IxIA (virtual) clients.



NOTE: The client side test configuration & load is identical for both
Windows 2000/IIS and Linux/Apache.



I would greatly appreciate any known insights into this issue.



- Hari



<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Hari Tadepalli

Sr Staff Performance Engineer

Intel Corporation

Infrastructure Processor Division

Chandler, AZ 85226

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>