Gigabit poor performance

Hi all,
I've just installed a Gigabit network on my cluster and I'm trying to
do some benchmarking. I'm measuring the bandwidth between two machines
each equipped with 2000+ AMD processor, 1 GBytes RAM and an Atlantis
Land A02-SG32 Gigabit Ethernet NIC (copper). They are connected
through an Atlantis Land A02-G8 Gigabit Switch and I even tried Cat-5,
Cat-5E and Cat-6 cables. I used both the tg3 driver (in kernel 2.4.20)
and the ac100 driver taken from the manufacturer site.The measured
have been made with the Netperf Benchmark. In any case I can't get
more then 195 Mbps. I also made the same measures with point-to-point
connection (It doesn't need crossed cable!!!) but nothing different.
What's wrong with my system? Is there anything particular to do to
make it work properly? Is there any of you that has the same NIC as me
and obtains better results?
Help me please!!!
Thanks

Alfredo

Hi Alfredo,

you wrote:
> What's wrong with my system? Is there anything particular to do to
> make it work properly? Is there any of you that has the same NIC as me
> and obtains better results?

No experience with GBit-Ethernet yet, but did you try to enable jumbo
frames?

For the tg3 driver in Linux 2.4.22 simple add "mtu=9000" to the ifconfig
parameters. You can change the MTU any time after the system startup
too. But:
- you have to enable all systems to use it,
- you have to check wether your GBit Hub does support jumbo frames,
- earlier releases of the driver might require a parameter when loading
the module, "mtu=9000" too IIRC.


Michael

--
Linux@TekXpress
http://www-users.rwth-aachen.de/Mich...kxp/tekxp.html

Alfredo Buttari <buttari@ing.uniroma2.it> wrote:
> have been made with the Netperf Benchmark. In any case I can't get
> more then 195 Mbps. I also made the same measures with point-to-point

What socket buffer sizes are you using? Cut and paste the netperf
command line and output here for us to see. Compile netperf
-DUSE_PROC_STAT and then also ask for CPU utilization figures with -c
and -C in the command line.

I believe in Linux there are tunables in the stack that will place
upper-bounds on socket buffer sizes, you might want to make sure those
are "large enough"

Gigabit Ethernet "requires" rather larger socket buffer sizes to get
link-rate (when you have enough CPU and such).

rick jones
--
portable adj, code that compiles under more than one compiler
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

On 10 Sep 2003 07:31:31 -0700, Alfredo Buttari <buttari@ing.uniroma2.it> wrote:
> Hi all,
> I've just installed a Gigabit network on my cluster and I'm trying to
> do some benchmarking. I'm measuring the bandwidth between two machines
> each equipped with 2000+ AMD processor, 1 GBytes RAM and an Atlantis
> Land A02-SG32 Gigabit Ethernet NIC (copper). They are connected
> through an Atlantis Land A02-G8 Gigabit Switch and I even tried Cat-5,
> Cat-5E and Cat-6 cables. I used both the tg3 driver (in kernel 2.4.20)
> and the ac100 driver taken from the manufacturer site.The measured
> have been made with the Netperf Benchmark. In any case I can't get
> more then 195 Mbps. I also made the same measures with point-to-point

that's about 20MB/s; what's the performance on your hard drive(s)?
what performance do you get going through the loopback device (having the
computer talk to itself)?

Rick Jones wrote:
>
> Alfredo Buttari <buttari@ing.uniroma2.it> wrote:
> > have been made with the Netperf Benchmark. In any case I can't get
> > more then 195 Mbps. I also made the same measures with point-to-point
>
> What socket buffer sizes are you using? Cut and paste the netperf
> command line and output here for us to see. Compile netperf
> -DUSE_PROC_STAT and then also ask for CPU utilization figures with -c
> and -C in the command line.
>
> I believe in Linux there are tunables in the stack that will place
> upper-bounds on socket buffer sizes, you might want to make sure those
> are "large enough"
>
> Gigabit Ethernet "requires" rather larger socket buffer sizes to get
> link-rate (when you have enough CPU and such).

I did some tests a while ago using SuSE Enterprise Server 7 (2.4.7) and
Intel EtherExpress Pro 1000 NICs (built onto the motherboard of IBM X
series 345 boxes) and a Cisco 6000 series switch. I was able to get a
true gigabit (112Mbyte/s from an FTP) under optimal conditions
(filesystem cache primed, writing to /dev/null) without tuning anything
in Linux.

Of course, SuSE may have pre-tuned some things, as SLES is intended for
server use. They also patch the kernel. TBH I havn't looked into this
much, as it works so well out of the box.

These boxes are plenty powerful, I think the ones I was using were dual
2GHz Xeons with 2.5GB. Top didn't show high CPU utilisation during the
tests though. It did show most of the memory in the filesystem cache but
that is to be expected.

Regards, Ian

TCS <The.Central.Scrutinizer@p.o.b.o.x.com> wrote:
> On 10 Sep 2003 07:31:31 -0700, Alfredo Buttari <buttari@ing.uniroma2.it> wrote:
>> The measured have been made with the Netperf Benchmark. In any case
>> I can't get more then 195 Mbps. I also made the same measures with
>> point-to-point

> that's about 20MB/s; what's the performance on your hard drive(s)?

Netperf does not do disc I/O during the run.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

In article <84053e81.0309100631.67052b16@posting.google.com >, Alfredo Buttari wrote:
> Hi all,
> I've just installed a Gigabit network on my cluster and I'm trying to
> do some benchmarking. I'm measuring the bandwidth between two machines
> each equipped with 2000+ AMD processor, 1 GBytes RAM and an Atlantis
> Land A02-SG32 Gigabit Ethernet NIC (copper). They are connected
> through an Atlantis Land A02-G8 Gigabit Switch and I even tried Cat-5,
> Cat-5E and Cat-6 cables. I used both the tg3 driver (in kernel 2.4.20)
> and the ac100 driver taken from the manufacturer site.The measured
> have been made with the Netperf Benchmark. In any case I can't get
> more then 195 Mbps. I also made the same measures with point-to-point
> connection (It doesn't need crossed cable!!!) but nothing different.
> What's wrong with my system? Is there anything particular to do to
> make it work properly? Is there any of you that has the same NIC as me
> and obtains better results?
>
Well according to folk who have put my laptop on gigabit ethernet (a IBM T40p
with a Intel PRO/1000) it easily gets 400Mbps on a 1.6Ghz Pentium M (P4
mobile). CPU speed is not the issue.

What you probably want a look at is as you are using gigabit ethernet cards
on an architecture from the 80's hardware interrupts are probably toasting the
CPU's throughput. Have a look at 'top' and how much is dedicated to
system/idle time. Look at /proc/interrupts with a

watch -n1 cat /proc/interrupts

as you do your test and see how hard your machine is being hit. There is a
option for my Gigabit card which turns off the interrupt to the CPU and get
the CPU to poll the card instead. NAPI is what you are looking for.

Regards

Alex

buttari@ing.uniroma2.it (Alfredo Buttari) wrote in message news:<84053e81.0309100631.67052b16@posting.google. com>...
> Hi all,
> I've just installed a Gigabit network on my cluster and I'm trying to
> do some benchmarking. I'm measuring the bandwidth between two machines
> each equipped with 2000+ AMD processor, 1 GBytes RAM and an Atlantis
> Land A02-SG32 Gigabit Ethernet NIC (copper). They are connected
> through an Atlantis Land A02-G8 Gigabit Switch and I even tried Cat-5,
> Cat-5E and Cat-6 cables. I used both the tg3 driver (in kernel 2.4.20)
> and the ac100 driver taken from the manufacturer site.The measured
> have been made with the Netperf Benchmark. In any case I can't get
> more then 195 Mbps. I also made the same measures with point-to-point
> connection (It doesn't need crossed cable!!!) but nothing different.
> What's wrong with my system? Is there anything particular to do to
> make it work properly? Is there any of you that has the same NIC as me
> and obtains better results?
> Help me please!!!
> Thanks
>
> Alfredo

Thanks all,
anyway I forgot to tell that my NIC is a 32 bit one for a 33 Mhz PCI
bus. I think that's the great problem for I found other people with
similar NICS saying that the bottleneck is in the PCI bus. Maybe I'm
going to change my NICS even because they don't support jumbo frames.
Thanks all of you for your precious suggests.
Alfredo

Alfredo Buttari <buttari@ing.uniroma2.it> wrote:
> Thanks all, anyway I forgot to tell that my NIC is a 32 bit one for
> a 33 Mhz PCI bus. I think that's the great problem for I found other
> people with similar NICS saying that the bottleneck is in the PCI
> bus.

A 32x33 PCI bus has a "marketing" rating of 100+MB/s. Now, one cannot
expect to take a PCI bus to 100% utilization, so one cannot expect
link-rate GbE from a PCI-1X bus/NIC combination, but unless your
system has really bad DMA latency/performance, you aught to be able to
do better than 195 Mbit/s. (Assuming of course you have the CPU
horsepower, which you can tell from either top while netperf is
running, or by compiling netperf -DUSE_PROC_STAT and asking it to
report CPU util.

> Maybe I'm going to change my NICS even because they don't
> support jumbo frames.

Neither do many switches. And with JumboFrames, it is an "entire
subnet or nothing" sort of thing.

rick jones
--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...