Setting up a linux fileserver

Hello. I'm trying to find a solution that would work for me. I need a
file sever that would work for about 40 different nodes. Those nodes
might need large amounts of data at the same time (like 30 computers
trying go retrieve 500 MB of data each, that data is scattered in coule
hundreds of files), those peaks don't happen very often but there is a
general tendency of having new connections happening simultaneously.
Currently I have linux box serving samba to windows nodes but I can't
find any solution go give any load balancing for my connections. I use
gigabit ethernet and at some times it chokes my file server (not it's
disks, not it's cpu but it's network). I was wondering on some solutions
that would use multiple network cards from one fileserver. It would be
possible for me to change other nodes from windows to linux (but no
solaris please) so linux to linux solutions are welcome (linux to
windows would be perfect). I found some infomations about NFS v4.1 but
it might be an overkill for my situation (having multiple servers
working as one).

I hope that you can give me some advice in my situation.

Kuba

On Mon, 27 Apr 2009 12:40:14 +0200, groblus wrote:

> I was wondering on some solutions
> that would use multiple network cards from one fileserver.

Why not just break the network up into subnetworks on separate LANs?


If it is the NIC in the server that is saturating, depending on your
router, you could install multiple NICs and use your (managed) router to
isolate different boxen into different subnets

Alternatively, you could just have different routers connected to
different subnets and different groups of boxen.

About 15 years ago, I did install dual NICs into
pC boxen, one went to WAN gateway and Novell server) and tother went to
*nix boxen for data, printing, backups, xterm, etc.

terryc pisze:
> On Mon, 27 Apr 2009 12:40:14 +0200, groblus wrote:
>
>> I was wondering on some solutions
>> that would use multiple network cards from one fileserver.
>
> Why not just break the network up into subnetworks on separate LANs?
>
>
> If it is the NIC in the server that is saturating, depending on your
> router, you could install multiple NICs and use your (managed) router to
> isolate different boxen into different subnets
>
> Alternatively, you could just have different routers connected to
> different subnets and different groups of boxen.
>
> About 15 years ago, I did install dual NICs into
> pC boxen, one went to WAN gateway and Novell server) and tother went to
> *nix boxen for data, printing, backups, xterm, etc.
>
this might be a nice solution especially for now, but is there some nice
and elegant solution having one network (without subnetworks) and load
balance samba shares ? or if not by using samba perhaps use of nfs ?
or I could use some form of hardware link aggregation ? Perhaps thera
are some double port network card that allow "adapter teaming" under
linux ? There are number of possibilities and I don't know which to
explore. Whis will be easiest to implement ?

On Mon, 27 Apr 2009 16:21:54 +0200, groblus wrote:

> terryc pisze:
>> On Mon, 27 Apr 2009 12:40:14 +0200, groblus wrote:
>>
>>> I was wondering on some solutions
>>> that would use multiple network cards from one fileserver.
>>
>> Why not just break the network up into subnetworks on separate LANs?
>>
>>
>> If it is the NIC in the server that is saturating, depending on your
>> router, you could install multiple NICs and use your (managed) router
>> to isolate different boxen into different subnets
>>
>> Alternatively, you could just have different routers connected to
>> different subnets and different groups of boxen.

This is the way it is usually done.


> this might be a nice solution especially for now, but is there some nice
> and elegant solution having one network (without subnetworks) and load
> balance samba shares ?

AFAIK, not for samba, and not for any other service. The load balancers I
know off are to balance/share loads between multiple servers. then you
have the problem of separate data stores unless you have a modern version
of dual ported scsi arrays (aka DEC Storage works).

Can you go FULL DUPLEX on the server NIC.
Have you also checked for network errors?
In the really old days, sometimes puting the server onto a token ring
solved NIC bottlenecks as token ring gave 16MB/sec compared to practical
8MB/sec for ethenet. Sadly, it never made a similar step up.

Bottom line is something will continue to cause "the bottleneck"

or if not by using samba perhaps use of nfs ? or
> I could use some form of hardware link aggregation ? Perhaps thera are
> some double port network card that allow "adapter teaming" under linux ?
> There are number of possibilities and I don't know which to explore.
> Whis will be easiest to implement ?

Naah. I stand to be corrected but unless you are talking point to point,
hardware link aggregtion doesn't work AFAIK.

Put a second NIC in the server, get a second LAN router/switch and move
half the load to a new IP range and the new router/switch network.

terryc <newssevenspam-spam@woa.com.au> wrote:
> Why not just break the network up into subnetworks on separate LANs?

Also, if you are using NFS, it produces less network traffic than samba, and
the kernel mode NFS server is relatively efficient.

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/

"groblus" <groblus@wp.pl> wrote in message
news:gt3pef$o7m$1@node1.news.atman.pl...
> Hello. I'm trying to find a solution that would work for me. I need a file
> sever that would work for about 40 different nodes. Those nodes might need
> large amounts of data at the same time (like 30 computers trying go retrieve
> 500 MB of data each, that data is scattered in coule hundreds of files), those
> peaks don't happen very often but there is a general tendency of having new
> connections happening simultaneously. Currently I have linux box serving samba
> to windows nodes but I can't find any solution go give any load balancing for
> my connections. I use gigabit ethernet and at some times it chokes my file
> server (not it's disks, not it's cpu but it's network). I was wondering on
> some solutions that would use multiple network cards from one fileserver. It
> would be possible for me to change other nodes from windows to linux (but no
> solaris please) so linux to linux solutions are welcome (linux to windows
> would be perfect). I found some infomations about NFS v4.1 but it might be an
> overkill for my situation (having multiple servers working as one).
>
> I hope that you can give me some advice in my situation.
>
> Kuba
>

google iSCSI, maybe that's what you are looking for.

terryc pisze:
> On Mon, 27 Apr 2009 16:21:54 +0200, groblus wrote:
>(...)
>
> Naah. I stand to be corrected but unless you are talking point to point,
> hardware link aggregtion doesn't work AFAIK.
>
> Put a second NIC in the server, get a second LAN router/switch and move
> half the load to a new IP range and the new router/switch network.
>

how about ?
http://www.howtoforge.com/nic_bonding

will samba benefit from that solution ?

On Mon, 27 Apr 2009, in the Usenet newsgroup comp.os.linux.networking, in
article <gt3pef$o7m$1@node1.news.atman.pl>, groblus wrote:

>I need a file sever that would work for about 40 different nodes.
>Those nodes might need large amounts of data at the same time (like
>30 computers trying go retrieve 500 MB of data each, that data is
>scattered in coule hundreds of files), those peaks don't happen very
>often but there is a general tendency of having new connections
>happening simultaneously.

30 computers - 500 Megs each = 15000 total. With those numbers,
there isn't that much that you can do, as that is beyond the
capability of the network card, and probably the disks and CPU.

>Currently I have linux box serving samba to windows nodes but I can't
>find any solution go give any load balancing for my connections. I
>use gigabit ethernet and at some times it chokes my file server (not
>it's disks, not it's cpu but it's network). I was wondering on some
>solutions that would use multiple network cards from one fileserver.

Divide the network in two, using two NICs. That will relieve some of
the pressure there, and cause the bottleneck to move to the bus or
disks. Note that putting two NICs on the same wire isn't going
to do anything - the wire is busy. Switching to 10GigE is essentially
the same as adding a divided network, in that you will merely move the
bottleneck elsewhere.

>I found some infomations about NFS v4.1 but it might be an overkill
>for my situation (having multiple servers working as one).

The only way that would improve things is to isolate the individual
servers to separate networks. If you really have to have that 15
Gigabyte capability, that's probably the only solution - expensive,
but that's your need.

Old guy

groblus wrote:
> Hello. I'm trying to find a solution that would work for me. I need a
> file sever that would work for about 40 different nodes. Those nodes
> might need large amounts of data at the same time (like 30 computers
> trying go retrieve 500 MB of data each, that data is scattered in coule
> hundreds of files), those peaks don't happen very often but there is a
> general tendency of having new connections happening simultaneously.
> Currently I have linux box serving samba to windows nodes but I can't
> find any solution go give any load balancing for my connections. I use
> gigabit ethernet and at some times it chokes my file server (not it's
> disks, not it's cpu but it's network). I was wondering on some solutions
> that would use multiple network cards from one fileserver. It would be
> possible for me to change other nodes from windows to linux (but no
> solaris please) so linux to linux solutions are welcome (linux to
> windows would be perfect). I found some infomations about NFS v4.1 but
> it might be an overkill for my situation (having multiple servers
> working as one).
>
> I hope that you can give me some advice in my situation.
>
> Kuba

I have never done anything like this in practice - I'm relying on some
of the more experienced posters to tell me if these are daft ideas!

One idea might be to have multiple NICs on the server connected to a
serious-sized network switch. Let each NIC have a different IP address,
and set up a dns server to round-robin these IP addresses for the
server's name, using very short timeouts. That way different clients
will get randomly distributed IP addresses for the server, and therefore
target different NICs. If the switch can handle the simultaneous
traffic, this should increase your average throughput.

(I presume the server is pretty well equipped, with lots of RAM for
caching.)

Have you looked at any sort of distributed file system, so that you
could have several servers each able to supply the same data? Some
vague ideas might be:

http://en.wikipedia.org/wiki/Lustre_(file_system)
http://en.wikipedia.org/wiki/OCFS
http://en.wikipedia.org/wiki/GlusterFS
http://www.redhat.com/gfs

Moe Trin <ibuprofin@painkiller.example.tld> wrote:
> 30 computers - 500 Megs each = 15000 total. With those numbers,
> there isn't that much that you can do, as that is beyond the
> capability of the network card, and probably the disks and CPU.

It would be better to cluster the computers together and loadshare with this
amount of data. What sort of data is this? Maybe it would be worth a look at
how this data is generated and used. There may be a more efficient way of
handling this.

Mark.

--
Mark Hobley
Linux User: #370818 http://markhobley.yi.org/