linux nfs server tuning

The following shell clause shows the send and receive buffers for the
NFS server:

#!/bin/sh

while true; do
netstat -nua | egrep :2049
sleep 1
done

The output is this (on a loaded machine):

(Proto Recv-Q Send-Q Local Address Foreign Address)
udp 81340 5456 0.0.0.0:2049 0.0.0.0:*
udp 167328 0 0.0.0.0:2049 0.0.0.0:*
udp 202188 0 0.0.0.0:2049 0.0.0.0:*
udp 160356 0 0.0.0.0:2049 0.0.0.0:*
udp 190568 0 0.0.0.0:2049 0.0.0.0:*
udp 202188 0 0.0.0.0:2049 0.0.0.0:*
udp 167328 0 0.0.0.0:2049 0.0.0.0:*
udp 202188 0 0.0.0.0:2049 0.0.0.0:*
udp 125496 0 0.0.0.0:2049 0.0.0.0:*
udp 109228 0 0.0.0.0:2049 0.0.0.0:*
udp 169652 0 0.0.0.0:2049 0.0.0.0:*
udp 202188 0 0.0.0.0:2049 0.0.0.0:*
udp 202188 0 0.0.0.0:2049 0.0.0.0:*

You'll notice that size of the recv buffer never seems to exceed
202188.
I thought that this would be the size of the recv buffer, but it
doesn't
appear to be (cat /proc/sys/net/core/rmem_max shows 131071). I couldn't
find any indication that the recv buffer size was be set before the nfs
server starts and reset after.

Does anyone know why the recv buffer seems to be topping out at 202188
bytes? Can this ceiling be raised?

This is 2.4.24 on Redhat 7.1. The system is a dual 2.8GHz Xeon with 4GB
of RAM and 100Mbit ethernet.

"Chris Walker" <cwalkatron@gmail.com> wrote in message news:<cadpj8$54b@odah37.prod.google.com>...
> The following shell clause shows the send and receive buffers for the
> NFS server:
>
> #!/bin/sh
>
> while true; do
> netstat -nua | egrep :2049
> sleep 1
> done
>
> The output is this (on a loaded machine):
>
> (Proto Recv-Q Send-Q Local Address Foreign Address)
> udp 81340 5456 0.0.0.0:2049 0.0.0.0:*
> udp 167328 0 0.0.0.0:2049 0.0.0.0:*
> udp 202188 0 0.0.0.0:2049 0.0.0.0:*
> udp 160356 0 0.0.0.0:2049 0.0.0.0:*
> udp 190568 0 0.0.0.0:2049 0.0.0.0:*
> udp 202188 0 0.0.0.0:2049 0.0.0.0:*
> udp 167328 0 0.0.0.0:2049 0.0.0.0:*
> udp 202188 0 0.0.0.0:2049 0.0.0.0:*
> udp 125496 0 0.0.0.0:2049 0.0.0.0:*
> udp 109228 0 0.0.0.0:2049 0.0.0.0:*
> udp 169652 0 0.0.0.0:2049 0.0.0.0:*
> udp 202188 0 0.0.0.0:2049 0.0.0.0:*
> udp 202188 0 0.0.0.0:2049 0.0.0.0:*
>
> You'll notice that size of the recv buffer never seems to exceed
> 202188.
> I thought that this would be the size of the recv buffer, but it
> doesn't

See below in general ... but there can be a number of
settings/variables involved with any _particular_ observation.

> appear to be (cat /proc/sys/net/core/rmem_max shows 131071). I couldn't

65535 * 2 = 13170 plus there are several "conflicting/confusing"
settings -- I have trouble keeping them sorted out ;-(

> find any indication that the recv buffer size was be set before the nfs
> server starts and reset after.
>
> Does anyone know why the recv buffer seems to be topping out at 202188
> bytes? Can this ceiling be raised?

These figures are reflective of the interaction of the IP stack _and_
the specific app.

> This is 2.4.24 on Redhat 7.1. The system is a dual 2.8GHz Xeon with 4GB
> of RAM and 100Mbit ethernet.

$man netstat
....
OUTPUT
....
Recv-Q
The count of bytes not copied by the user program connected to
this socket.

Ie., it is not _determined_ by the size of rmem buffers alone.
netstat is OK as far as it goes, but I usually prefer to break out the
sniffer early on and see just what's happening on a packet level.
Disk I/O will ultimately limit the tranfer rates of larger files --
especially packets read from wire, then _written_ to disk.

So far as "tuning" some of the net variables check these out:
file:/usr/src/linux-2.4.?/Documentation/networking/ip-sysctl.txt
http://ipsysctl-tutorial.frozentux.n...-tutorial.html
http://lartc.org/howto/lartc.kernel.obscure.html
I think these pretty much require a "tune/change by observation"
approach except for the more "obvious" settings ;-)

I assume you're looking to improve throughput with larger transfers
;-)

The NFS HOWTO has some pretty good -- general -- tuning pointers:
http://www.tldp.org/HOWTO/NFS-HOWTO/performance.html

Also will depend on the specific NFS server you're running.

BTW, you may also have some peculiaritues relating to the smp setup
you have. You may want to double check just how smp friendly your NFS
version is -- upgrade? update? cpu affinity? I have no specific
knowledge of running NFS with smp/RH7.1 -- this is just a general
suggestion.

hth,
prg
email above disabled