[Samba] Samba ignoring socket options?

Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in
11.0) on a storage server connected to multiple windows based clients
over a gigabit ethernet link. The server is a quad core Intel CPU and
is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients.

I am seeing performance issues on transfers over the gigabit ethernet network,
and was trying to play with the socket options settings in the smb.conf
file to improve transfers rates, but no matter what I set the SO_RCVBUF
and SO_SNDBUF values too, the transfer rates are unchanged, even if I
set the buffer sizes down to 512, which should have the effect of at
least slowing things down dramatically, leading me to question if Samba
is actually using these settings at all. TCP_NODELAY is set, but it
doesn't seem to matter much if I include it or not on the socket
options line, and the line is definitely not commented out, as if I
misspell something on that line, samba terminates with an error when I
try and restart the daemon.

Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client,
and the disks on both windows machines are RAID0 (4 and 2 disk RAID0
sets respectively), so I don't think I am running into filesystem
performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates.

The
raid array on the samba server consist of 2 6 disk raid5 sets with fast
disks on them, running lvm and XFS for a filesysteem. I can do a dd of
a multigigabyte file to /dev/null and get roughly 500-600 MB/'s
transfer rates through the filesystem, so I don't think the raid array
and file system is a bottleneck.

I have run netperf tests
between the server and the clients to see if I had some network
plumbing problems. With default socket settings for netperf (8182
buffer size), I get about 300 mbps transfer rates between the clients
and the server (which matches approximately the 30 MB/s transfer
rates). With 65536 byte buffers, that number goes to 970 or so Mbps,
so I think the interface cards, TCP stack, switches are all ok.

If I do an FTP from the server to a client, I get 45-50 MB/s transfer rates, so I think the problem is somewhere in Samba.

Again,
if I change the SO_SNDBUF and SO_RCVBUF values, either up or down, or
keep them unset, I get almost no variance in transfer rates. Vista is
set to use autotuning in it's TCP configuration, and window scaling and
RFC 1323 options are enabled, but I see the same exact performance on
XP as well, so I don't think it's a client issue. I get somewhat
faster copying from a windows 2003 server on the same LAN to the same
clients, even though it is not equipped with a raid array and is just
reading from one disk with an unoptimized NTFS filesystem.

Is there a bug here or am I missing something in the configuration?

thanks,
Mike




--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, Aug 28, 2008 at 12:53:00PM -0700, Mike Myers wrote:
> Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in
> 11.0) on a storage server connected to multiple windows based clients
> over a gigabit ethernet link. The server is a quad core Intel CPU and
> is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients.
>
> I am seeing performance issues on transfers over the gigabit ethernet network,
> and was trying to play with the socket options settings in the smb.conf
> file to improve transfers rates, but no matter what I set the SO_RCVBUF
> and SO_SNDBUF values too, the transfer rates are unchanged, even if I
> set the buffer sizes down to 512, which should have the effect of at
> least slowing things down dramatically, leading me to question if Samba
> is actually using these settings at all. TCP_NODELAY is set, but it
> doesn't seem to matter much if I include it or not on the socket
> options line, and the line is definitely not commented out, as if I
> misspell something on that line, samba terminates with an error when I
> try and restart the daemon.

Samba is definately setting these options.

> Samba is getting roughly 30 MB/s tranfer rates from the linux server to a windows vista and a windows XP client,
> and the disks on both windows machines are RAID0 (4 and 2 disk RAID0
> sets respectively), so I don't think I am running into filesystem
> performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates.
>
> The
> raid array on the samba server consist of 2 6 disk raid5 sets with fast
> disks on them, running lvm and XFS for a filesysteem. I can do a dd of
> a multigigabyte file to /dev/null and get roughly 500-600 MB/'s
> transfer rates through the filesystem, so I don't think the raid array
> and file system is a bottleneck.
>
> I have run netperf tests
> between the server and the clients to see if I had some network
> plumbing problems. With default socket settings for netperf (8182
> buffer size), I get about 300 mbps transfer rates between the clients
> and the server (which matches approximately the 30 MB/s transfer
> rates). With 65536 byte buffers, that number goes to 970 or so Mbps,
> so I think the interface cards, TCP stack, switches are all ok.

Can you try using smbclient to do a large file transfer from
another client Linux box and time that please ? That eliminates
the Windows clients from the equation, and allows us to test
only with things we can examine directly.

Jeremy.
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, 28 Aug 2008, Jeremy Allison wrote:

> Can you try using smbclient to do a large file transfer from
> another client Linux box and time that please ?

Minor rant. One thing that slightly bugs me about smbclient is that it
reports the transfer rate as "kb/s", which means nothing to me. Is this
"KB/s" or "Kb/s"? Well, it's the former: kilobytes per second. So
shouldn't it say "KB/s"?

BTW, I get 43 MB/s with a single 12 MB file on GbE without any socket
options; linux -> linux.

Steve
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, Aug 28, 2008 at 05:31:50PM -0400, Steve Thompson wrote:
> On Thu, 28 Aug 2008, Jeremy Allison wrote:
>
> >Can you try using smbclient to do a large file transfer from
> >another client Linux box and time that please ?
>
> Minor rant. One thing that slightly bugs me about smbclient is that it
> reports the transfer rate as "kb/s", which means nothing to me. Is this
> "KB/s" or "Kb/s"? Well, it's the former: kilobytes per second. So
> shouldn't it say "KB/s"?

Easily fixed. I'll probably do that.

> BTW, I get 43 MB/s with a single 12 MB file on GbE without any socket
> options; linux -> linux.

Ok, that's the same as the Windows systems right ? Should be
higher than that. Ok, at least we've removed the black box
from the system - everything can be examined in open source
code now. You should be able to get 100MB/sec (or close to
it) I think.

You might need to try looking into tbench/dbench to examine
where the bottleneck is :

http://samba.org/ftp/tridge/dbench/README

Jeremy.
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, 2008-08-28 at 14:53 -0700, Jeremy Allison wrote:
> On Thu, Aug 28, 2008 at 05:31:50PM -0400, Steve Thompson wrote:
> > On Thu, 28 Aug 2008, Jeremy Allison wrote:
> >
> > >Can you try using smbclient to do a large file transfer from
> > >another client Linux box and time that please ?
> >
> > Minor rant. One thing that slightly bugs me about smbclient is that it
> > reports the transfer rate as "kb/s", which means nothing to me. Is this
> > "KB/s" or "Kb/s"? Well, it's the former: kilobytes per second. So
> > shouldn't it say "KB/s"?
>
> Easily fixed. I'll probably do that.

If we want to be standards compliant then we should write KiB/s[1] not
Kb/s and MiB/s[2] and GiB/s[3]

:-D

Simo.


[1] http://en.wikipedia.org/wiki/Kibibyte
[2] http://en.wikipedia.org/wiki/Mebibyte
[3] http://en.wikipedia.org/wiki/Gibibyte

--
Simo Sorce
Samba Team GPL Compliance Officer <simo@samba.org>
Senior Software Engineer at Red Hat Inc. <simo@redhat.com>

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, Aug 28, 2008 at 10:02:51PM +0000, simo wrote:
> On Thu, 2008-08-28 at 14:53 -0700, Jeremy Allison wrote:
> > On Thu, Aug 28, 2008 at 05:31:50PM -0400, Steve Thompson wrote:
> > > On Thu, 28 Aug 2008, Jeremy Allison wrote:
> > >
> > > >Can you try using smbclient to do a large file transfer from
> > > >another client Linux box and time that please ?
> > >
> > > Minor rant. One thing that slightly bugs me about smbclient is that it
> > > reports the transfer rate as "kb/s", which means nothing to me. Is this
> > > "KB/s" or "Kb/s"? Well, it's the former: kilobytes per second. So
> > > shouldn't it say "KB/s"?
> >
> > Easily fixed. I'll probably do that.
>
> If we want to be standards compliant then we should write KiB/s[1] not
> Kb/s and MiB/s[2] and GiB/s[3]
>
> :-D

Yuk. I was thinking more this...

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, 2008-08-28 at 15:04 -0700, Jeremy Allison wrote:
> Yuk. I was thinking more this...

> - DEBUG(1,("(%3.1f kb/s) (average %3.1f kb/s)\n",
> + DEBUG(1,("(%3.1f KiloBytes/sec) (average %3.1f
> KiloBytes/sec)\n",

Fine by me, I was just pointing out the standard to be pedant :-)

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <simo@samba.org>
Senior Software Engineer at Red Hat Inc. <simo@redhat.com>

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, 28 Aug 2008, Jeremy Allison wrote:

> On Thu, Aug 28, 2008 at 05:31:50PM -0400, Steve Thompson wrote:
>> BTW, I get 43 MB/s with a single 12 MB file on GbE without any socket
>> options; linux -> linux.
>
> Ok, that's the same as the Windows systems right ? Should be
> higher than that. Ok, at least we've removed the black box
> from the system - everything can be examined in open source
> code now. You should be able to get 100MB/sec (or close to
> it) I think.

*sound of stick hitting head*

My destination filesystem was an NFS-mounted volume. Copying instead to a
local software RAID-1 volume, I get 63 MB/sec, which I think is quite
reasonable. The systems were quite busy at the time. Samba 3.0.24 on
CentOS 4.6/x86_64.

What is also interesting is that changing 'socket options' to a variety of
different values has no effect whatsoever on the performance (and I did
restart smbd each time).

Steve
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, Aug 28, 2008 at 07:20:22PM -0400, Steve Thompson wrote:
> On Thu, 28 Aug 2008, Jeremy Allison wrote:
>
> >On Thu, Aug 28, 2008 at 05:31:50PM -0400, Steve Thompson wrote:
> >>BTW, I get 43 MB/s with a single 12 MB file on GbE without any socket
> >>options; linux -> linux.
> >
> >Ok, that's the same as the Windows systems right ? Should be
> >higher than that. Ok, at least we've removed the black box
> >from the system - everything can be examined in open source
> >code now. You should be able to get 100MB/sec (or close to
> >it) I think.
>
> *sound of stick hitting head*
>
> My destination filesystem was an NFS-mounted volume. Copying instead to a
> local software RAID-1 volume, I get 63 MB/sec, which I think is quite
> reasonable. The systems were quite busy at the time. Samba 3.0.24 on
> CentOS 4.6/x86_64.

Ah, twice over the network. Always good for performance :-).

Glad to be able to help.

Jeremy.
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba

On Thu, Aug 28, 2008 at 12:53:00PM -0700, Mike Myers wrote:
> Hi everyone. I am running Samba 3.2.0-22.1 (as packaged by OpenSUSE in
> 11.0) on a storage server connected to multiple windows based clients
> over a gigabit ethernet link. The server is a quad core Intel CPU and
> is equipped with an Intel e1000 based gigabit ethernet controller and plugged into a common gigabit ethernet switch with the windows clients.

Is this still the case?

In addition to Jeremy's suggestion I'd like to see you testing this with
a straight connection between the two systems. But after reading the
rest of your posting it's not very liekly to help.

For openSUSE 11.0 we'll also soon see an official update to 3.2.3.

Meanwhile you might use the packages provided by the openSUSE Build
Service. See http://en.opensuse.org/Samba#openSUSE_Build_Service for
how to access them.

> Samba is getting roughly 30 MB/s tranfer rates from the linux server to awindows vista and a windows XP client,
> and the disks on both windows machines are RAID0 (4 and 2 disk RAID0
> sets respectively), so I don't think I am running into filesystem
> performance issues on the target. Moving from the windows systems to Samba, I see about 45 MB/sec transfers rates.

The write case is faster than read. Strange to me.

> The
> raid array on the samba server consist of 2 6 disk raid5 sets with fast
> disks on them, running lvm and XFS for a filesysteem. I can do a dd of
> a multigigabyte file to /dev/null and get roughly 500-600 MB/'s
> transfer rates through the filesystem, so I don't think the raid array
> and file system is a bottleneck.

Do you have some space on a local disk with ext2, no raid, no lvm? Only
to ensure your issue isn't caused by one of these components.

> I have run netperf tests
> between the server and the clients to see if I had some network
> plumbing problems. With default socket settings for netperf (8182
> buffer size), I get about 300 mbps transfer rates between the clients
> and the server (which matches approximately the 30 MB/s transfer
> rates). With 65536 byte buffers, that number goes to 970 or so Mbps,
> so I think the interface cards, TCP stack, switches are all ok.

Therefore this might be caused by the file system in use. Please try to
test the way as suggested above. If it isn't solved already.

Lars
--
Lars Müller [ˈla�(r)z ˈm�l�]
Samba Team
SUSE Linux, Maxfeldstraße 5, 90409 Nürnberg, Germany

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/listinfo/samba
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iD8DBQFIt98+4NBiK0FKV8MRAug1AJ0dFiU/h6nrp0oqTikKrseN5m7fMwCfRNq5
oSc381mSEMIZoDEJkfFIC3g=
=Yoec
-----END PGP SIGNATURE-----