inconsistent HTTP download speeds on a LAN

Hello,

We have an Apache 2.0.52 server on RHEL ES4u4 (2.6.9-42.ELsmp) that is running
a CGI application. As part of the application, attachments can be downloaded by
users. The server is connect via GigE to a central switch, and various clients
are connected via LAN to various switches, which then connect to the central
switch.

For some clients the attachments are downloaded at 50-100 KB/s, but for others
they get 2+ MB/s. The slow clients are of different OSes (Linux, OS X, and
Windows) and are all on different VLANs. The fast clients are also various
OSes and VLANs (i.e., switches / routers don't seem to be a common element).

I'm trying to figure what the fast clients have in common, and how they are
different from the slow ones.

I've done a few tcpdumps, and the only difference between fast and slow systems
(using the same OS) is that TCP window size gets to 60 KB in ~0.02s for the
fast clients, but takes up to 0.80-1.14 seconds for the slow ones. I'm thinking
this has something to do with TCP ramp up algorithms, but can't seem to figure
out the differences between the two.

One of the slow machines is Fedora 8:

Linux lnx-slowA 2.6.24.3-50.fc8 #1 SMP Thu Mar 20 13:39:08 EDT 2008
x86_64 x86_64 x86_64 GNU/Linux

Another is Fedora 6:

Linux lnx-slowB 2.6.18-1.2798.fc6 #1 SMP Mon Oct 16 14:54:20 EDT 2006
i686 i686 i386 GNU/Linux

One of the fast machines is Fedora 7:

Linux lnx-fast 2.6.23.15-80.fc7 #1 SMP Sun Feb 10 17:29:10 EST 2008
i686 i686 i386 GNU/Linux

Any ideas?

Thanks for any info.

--
David Magda <dmagda at ee.ryerson.ca>
Because the innovator has for enemies all those who have done well under
the old conditions, and lukewarm defenders in those who may do well
under the new. -- Niccolo Machiavelli, _The Prince_, Chapter VI

Seems like the easiest thing to eliminate network hardware would be to
pull a slow machine and move it the the location of a fast one, using
the fast ones connection - right down to the cable. That would
positively eliminate everything but the machine itself. From there you'd
have to look at similarities between the slow computers.

Kurt

David Magda wrote:
> Hello,
>
> We have an Apache 2.0.52 server on RHEL ES4u4 (2.6.9-42.ELsmp) that is running
> a CGI application. As part of the application, attachments can be downloaded by
> users. The server is connect via GigE to a central switch, and various clients
> are connected via LAN to various switches, which then connect to the central
> switch.
>
> For some clients the attachments are downloaded at 50-100 KB/s, but for others
> they get 2+ MB/s. The slow clients are of different OSes (Linux, OS X, and
> Windows) and are all on different VLANs. The fast clients are also various
> OSes and VLANs (i.e., switches / routers don't seem to be a common element).
>
> I'm trying to figure what the fast clients have in common, and how they are
> different from the slow ones.
>
> I've done a few tcpdumps, and the only difference between fast and slow systems
> (using the same OS) is that TCP window size gets to 60 KB in ~0.02s for the
> fast clients, but takes up to 0.80-1.14 seconds for the slow ones. I'm thinking
> this has something to do with TCP ramp up algorithms, but can't seem to figure
> out the differences between the two.
>
> One of the slow machines is Fedora 8:
>
> Linux lnx-slowA 2.6.24.3-50.fc8 #1 SMP Thu Mar 20 13:39:08 EDT 2008
> x86_64 x86_64 x86_64 GNU/Linux
>
> Another is Fedora 6:
>
> Linux lnx-slowB 2.6.18-1.2798.fc6 #1 SMP Mon Oct 16 14:54:20 EDT 2006
> i686 i686 i386 GNU/Linux
>
> One of the fast machines is Fedora 7:
>
> Linux lnx-fast 2.6.23.15-80.fc7 #1 SMP Sun Feb 10 17:29:10 EST 2008
> i686 i686 i386 GNU/Linux
>
> Any ideas?
>
> Thanks for any info.
>

David Magda wrote:
> Hello,
>
> We have an Apache 2.0.52 server on RHEL ES4u4 (2.6.9-42.ELsmp) that is running
> a CGI application. As part of the application, attachments can be downloaded by
> users. The server is connect via GigE to a central switch, and various clients
> are connected via LAN to various switches, which then connect to the central
> switch.
>
> For some clients the attachments are downloaded at 50-100 KB/s, but for others
> they get 2+ MB/s. The slow clients are of different OSes (Linux, OS X, and
> Windows) and are all on different VLANs. The fast clients are also various
> OSes and VLANs (i.e., switches / routers don't seem to be a common element).
>

Stupid trivial question, but are all the clients using correct duplex?

On 2008-04-24, Jurgen Haan <jurgen@fake.tld> wrote:
> Stupid trivial question, but are all the clients using correct duplex?

Not the case here.

Something that I forgot to mention in my original post: the "slow" clients
are not acutally slow everywhere, just with that particular web server. They
can download at relatively high speeds (2+ MB/s) from a different web server,
that's also running Apache and Red Hat (though different versions).

The TCP settingss (as shown via 'sysctl -a | grep tcp') are the same between
the two servers.

It's just with this particular web server that some workstations are fast and
some are slow.

On 2008-04-23, Kurt <kurt@nospam.olypen.com> wrote:
> Seems like the easiest thing to eliminate network hardware would be to
> pull a slow machine and move it the the location of a fast one, using
> the fast ones connection - right down to the cable. That would
> positively eliminate everything but the machine itself. From there you'd
> have to look at similarities between the slow computers.

Still slow.

One thing that I forgot to mention: the "slow" clients are only slow for
this particular web server. If I try to download a file from another internal
web server the speed is fine.

It's just with this particular one some clients are fast and some are slow.
The TCP settings ('sysctl -a | grep tcp') are the same between the two
servers, though there are slight differences between the two web servers: one
has httpd-2.0.52-22.ent, while the other has httpd-2.0.52-25.ent. One is
RHEL ES 4u4 (2.6.9-42.ELsmp), while the other is RHEL AS 4u2 (2.6.9-22.ELsmp).

On 2008-04-24, David Magda <dmagda+trace080423@ee.ryerson.ca> wrote:
> Something that I forgot to mention in my original post: the "slow" clients
> are not acutally slow everywhere, just with that particular web server. They
> can download at relatively high speeds (2+ MB/s) from a different web server,
> that's also running Apache and Red Hat (though different versions).
>
> The TCP settingss (as shown via 'sysctl -a | grep tcp') are the same between
> the two servers.
>
> It's just with this particular web server that some workstations are fast and
> some are slow.

Just a quick update on the two web servers: going through 'sysctl -a' output
for both machines and doing a diff(1), there are no notable differences
between the two besides things like hostname and kernel versions. The only
differences under net.* is that one machine has an extra interface plumbed.