Using multiple NICs

I have this application that transfers huge amounts of data. The data source
is an NFS share. What I would like to do is mount the NFS share so that the
reads are from one NIC and move the data out through a different NIC.

Has anyone tried that before? What do you configure something like that?

Thanks in advance,
TH

"TH" <noaddress@nodomain.com>,
In a message on Tue, 2 May 2006 13:01:23 -0500, wrote :

"> I have this application that transfers huge amounts of data. The data source
"> is an NFS share. What I would like to do is mount the NFS share so that the
"> reads are from one NIC and move the data out through a different NIC.

It is doubtful that you can do *specificly* this.

">
"> Has anyone tried that before? What do you configure something like that?

What you really want to do is called 'bonding', where you bond two (or
more) NICs into a single logical NIC. Linux does support this. You
need to be sure your EtherNet switch supports it as well. Oh, it really
only makes sense if both machines (server AND client) are doing this.

You will get the same bandwidth effect as you want, since if you really
pull/push data at a rate high enough, the kernel will load share across
the two (or more) NICs, although you won't have any one NIC *dedicated*
to reads or writes (just like on a SMP system you won't have any one
processor dedicated to any specific processing).

">
"> Thanks in advance,
"> TH
">
">
">

Robert Heller -- 978-544-6933
Deepwoods Software -- Linux Installation and Administration
http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk

Not the answer I was looking for but thanks! :) Had to give it a try.

TH

"Robert Heller" <heller@deepsoft.com> wrote in message
news:9ebe3$4457c4f6$cb248f0$15561@news.news-service.com...
> "TH" <noaddress@nodomain.com>,
> In a message on Tue, 2 May 2006 13:01:23 -0500, wrote :
>
> "> I have this application that transfers huge amounts of data. The data
> source
> "> is an NFS share. What I would like to do is mount the NFS share so that
> the
> "> reads are from one NIC and move the data out through a different NIC.
>
> It is doubtful that you can do *specificly* this.
>
> ">
> "> Has anyone tried that before? What do you configure something like
> that?
>
> What you really want to do is called 'bonding', where you bond two (or
> more) NICs into a single logical NIC. Linux does support this. You
> need to be sure your EtherNet switch supports it as well. Oh, it really
> only makes sense if both machines (server AND client) are doing this.
>
> You will get the same bandwidth effect as you want, since if you really
> pull/push data at a rate high enough, the kernel will load share across
> the two (or more) NICs, although you won't have any one NIC *dedicated*
> to reads or writes (just like on a SMP system you won't have any one
> processor dedicated to any specific processing).
>
> ">
> "> Thanks in advance,
> "> TH
> ">
> ">
> ">
>
> Robert Heller -- 978-544-6933
> Deepwoods Software -- Linux Installation and Administration
> http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
> heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk
>
>
>
>
>
>
>

TH wrote:
> I have this application that transfers huge amounts of data. The data
> source is an NFS share. What I would like to do is mount the NFS share so
> that the reads are from one NIC and move the data out through a different
> NIC.
>
Are you talking about more than two machines?

If the NFS share is on machine A and you are on machine B and you move data
from machine A to machine C, and if machine B has two NICs, one to A and one
to B, you can just do what you want without doing anything special.


--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 17:25:01 up 22 days, 6:52, 3 users, load average: 4.10, 4.18, 4.15

"TH" <noaddress@nodomain.com>,
In a message on Tue, 2 May 2006 16:04:10 -0500, wrote :

"> Not the answer I was looking for but thanks! :) Had to give it a try.
">
"> TH
">
"> "Robert Heller" <heller@deepsoft.com> wrote in message
"> news:9ebe3$4457c4f6$cb248f0$15561@news.news-service.com...
"> > "TH" <noaddress@nodomain.com>,
"> > In a message on Tue, 2 May 2006 13:01:23 -0500, wrote :
"> >
"> > "> I have this application that transfers huge amounts of data. The data
"> > source
"> > "> is an NFS share. What I would like to do is mount the NFS share so that
"> > the
"> > "> reads are from one NIC and move the data out through a different NIC.
"> >
"> > It is doubtful that you can do *specificly* this.

On re-reading it *sounds* like you might be thinking of having two
networks, one with the NFS server and one without the NFS server and a
machine with two NICs, one on each network. (I thought at first you
wanted to separate NFS reads from NFS writes to/from the same server.)
This is trivial to do:

Lets say your NFS server is 192.168.1.100 and is thus on network
192.168.1.0. Your data is meant to land on some machine with an IP of
192.168.2.199 and your compute box has two NICs, eth0 and eth1. eth0
has an IP of 192.168.1.99 and eth1 an IP of 192.168.2.27. You also
have two ethernet switches, A and B. You plug the ethernet cable from
the NFS server into switch A along with a cable from the compute box's eth0
card. You connect the machine with an IP of 192.168.2.199 to switch B
along with a cable from the compute box's eth1 card. You export your
NFS share to 192.168.1.99 and mount 192.168.1.100:/whatever on your
compute box. The NFS traffic will go through eth0 and on switch A. If
your output is headed to 192.168.2.199, it will go through eth1 and
through switch B. (It is possible to use a single switch, but I am not
sure if you will get burned by network contention -- it would depend on
how smart the switch is.)

If you only have one network (eg one network switch) and a machine with
two NICs, then what you want to do is bonding. You can have a machine
with two IP numbers (on the same network), one per NIC, but things are
strange: what would the default route be? With bonding, the kernel
merges the traffic on the two NICs into one logical data path, so you
have one logical network interface with single IP address and a single
route. It just has twice the traffic capacity.

Imagine the difference between a four lane highway, with two lanes in
each direction vs two parallel two lane roads (assumes full duplex
NICs). The two *separate* roads get you the same place, but you can't
shift lanes from one road to the other, should one become congested.
Bonding has the advantage of sharing the load on a demand basis. You
can't be sure that the NFS read traffic will *exactly* be only 1/2 of
the total traffic. It might work out to only be 1/3 of the traffic (or
2/3). With bonding, the difference is made up so that you get to use
all of the available bandwidth (assuming your machines can really push
that much data). If you do the two network 'trick' above, one NIC
could end up being underutilized and the other will be maxed out and
total thoughput will be less than the max possible. With bonding you
get to use ALL of the available bandwidth, even if the traffic is not
even.

"> >
"> > ">
"> > "> Has anyone tried that before? What do you configure something like
"> > that?
"> >
"> > What you really want to do is called 'bonding', where you bond two (or
"> > more) NICs into a single logical NIC. Linux does support this. You
"> > need to be sure your EtherNet switch supports it as well. Oh, it really
"> > only makes sense if both machines (server AND client) are doing this.
"> >
"> > You will get the same bandwidth effect as you want, since if you really
"> > pull/push data at a rate high enough, the kernel will load share across
"> > the two (or more) NICs, although you won't have any one NIC *dedicated*
"> > to reads or writes (just like on a SMP system you won't have any one
"> > processor dedicated to any specific processing).
"> >
"> > ">
"> > "> Thanks in advance,
"> > "> TH
"> > ">
"> > ">
"> > ">
"> >
"> > Robert Heller -- 978-544-6933
"> > Deepwoods Software -- Linux Installation and Administration
"> > http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
"> > heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk
"> >
"> >
"> >
"> >
"> >
"> >
"> >
">
">
">

Robert Heller -- 978-544-6933
Deepwoods Software -- Linux Installation and Administration
http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk

Robert Heller wrote:
> "TH" <noaddress@nodomain.com>,
> In a message on Tue, 2 May 2006 13:01:23 -0500, wrote :
>
> "> I have this application that transfers huge amounts of data. The data source
> "> is an NFS share. What I would like to do is mount the NFS share so that the
> "> reads are from one NIC and move the data out through a different NIC.
>
> It is doubtful that you can do *specificly* this.
>
> ">
> "> Has anyone tried that before? What do you configure something like that?
>
> What you really want to do is called 'bonding', where you bond two (or
> more) NICs into a single logical NIC. Linux does support this. You
> need to be sure your EtherNet switch supports it as well. Oh, it really
> only makes sense if both machines (server AND client) are doing this.
>
> You will get the same bandwidth effect as you want, since if you really
> pull/push data at a rate high enough, the kernel will load share across
> the two (or more) NICs, although you won't have any one NIC *dedicated*
> to reads or writes (just like on a SMP system you won't have any one
> processor dedicated to any specific processing).

Actually, given that modern Ethernet NICs and switches are generally
full duplex devices, I would wager that bonding two NICs will offer
better bandwidth than the OP's original proposal. After all, a 100 Mbit
NIC (as an example, it applies equally to Gigabit NICs) can
simultaneously read and write at 100 Mbit/sec (theoretically). With his
proposal he would still have a 100 Mbit/sec read channel and a 100 Mbit
write channel. With your suggestion for bonding two NICs he'd have 2 x
100 Mbit for reading and 2 x 100 Mbit for writing. I don't see that the
dedicated-read-NIC and dedicated-write-NIC would offer any benefit
whatsoever (assuming he's connecting both NICs to the same network).

A side note for the OP: if you're using Gigabit Ethernet, beware of the
fact that a single GigE channel can theoretically consume almost all of
the bandwidth on an ordinary 32-bit, 33MHz PCI bus. You'll need PCI-X
or PCI Express (and/or multiple PCI busses) to get the benefit of
multiple GigE NICs. You don't say what you're currently using so this
may or may not be a consideration for you.

Thanks! The NICs would go to a switch so that may not be possible.

"Jean-David Beyer" <jeandavid8@verizon.net> wrote in message
news:pcQ5g.11499$Un3.9385@trnddc05...
> TH wrote:
>> I have this application that transfers huge amounts of data. The data
>> source is an NFS share. What I would like to do is mount the NFS share so
>> that the reads are from one NIC and move the data out through a different
>> NIC.
>>
> Are you talking about more than two machines?
>
> If the NFS share is on machine A and you are on machine B and you move
> data
> from machine A to machine C, and if machine B has two NICs, one to A and
> one
> to B, you can just do what you want without doing anything special.
>
>
> --
> .~. Jean-David Beyer Registered Linux User 85642.
> /V\ PGP-Key: 9A2FC99A Registered Machine 241939.
> /( )\ Shrewsbury, New Jersey http://counter.li.org
> ^^-^^ 17:25:01 up 22 days, 6:52, 3 users, load average: 4.10, 4.18, 4.15

I thinking bonding may be a better option. Better yet get a machine that
drive a Gig NIC. It was just a thought to segregate the read/write traffic
to improve performance but it seems that I may not gain much.


"Robert Heller" <heller@deepsoft.com> wrote in message
news:b172e$4457d8f1$cb248f0$13073@news.news-service.com...
> "TH" <noaddress@nodomain.com>,
> In a message on Tue, 2 May 2006 16:04:10 -0500, wrote :
>
> "> Not the answer I was looking for but thanks! :) Had to give it a try.
> ">
> "> TH
> ">
> "> "Robert Heller" <heller@deepsoft.com> wrote in message
> "> news:9ebe3$4457c4f6$cb248f0$15561@news.news-service.com...
> "> > "TH" <noaddress@nodomain.com>,
> "> > In a message on Tue, 2 May 2006 13:01:23 -0500, wrote :
> "> >
> "> > "> I have this application that transfers huge amounts of data. The
> data
> "> > source
> "> > "> is an NFS share. What I would like to do is mount the NFS share so
> that
> "> > the
> "> > "> reads are from one NIC and move the data out through a different
> NIC.
> "> >
> "> > It is doubtful that you can do *specificly* this.
>
> On re-reading it *sounds* like you might be thinking of having two
> networks, one with the NFS server and one without the NFS server and a
> machine with two NICs, one on each network. (I thought at first you
> wanted to separate NFS reads from NFS writes to/from the same server.)
> This is trivial to do:
>
> Lets say your NFS server is 192.168.1.100 and is thus on network
> 192.168.1.0. Your data is meant to land on some machine with an IP of
> 192.168.2.199 and your compute box has two NICs, eth0 and eth1. eth0
> has an IP of 192.168.1.99 and eth1 an IP of 192.168.2.27. You also
> have two ethernet switches, A and B. You plug the ethernet cable from
> the NFS server into switch A along with a cable from the compute box's
> eth0
> card. You connect the machine with an IP of 192.168.2.199 to switch B
> along with a cable from the compute box's eth1 card. You export your
> NFS share to 192.168.1.99 and mount 192.168.1.100:/whatever on your
> compute box. The NFS traffic will go through eth0 and on switch A. If
> your output is headed to 192.168.2.199, it will go through eth1 and
> through switch B. (It is possible to use a single switch, but I am not
> sure if you will get burned by network contention -- it would depend on
> how smart the switch is.)
>
> If you only have one network (eg one network switch) and a machine with
> two NICs, then what you want to do is bonding. You can have a machine
> with two IP numbers (on the same network), one per NIC, but things are
> strange: what would the default route be? With bonding, the kernel
> merges the traffic on the two NICs into one logical data path, so you
> have one logical network interface with single IP address and a single
> route. It just has twice the traffic capacity.
>
> Imagine the difference between a four lane highway, with two lanes in
> each direction vs two parallel two lane roads (assumes full duplex
> NICs). The two *separate* roads get you the same place, but you can't
> shift lanes from one road to the other, should one become congested.
> Bonding has the advantage of sharing the load on a demand basis. You
> can't be sure that the NFS read traffic will *exactly* be only 1/2 of
> the total traffic. It might work out to only be 1/3 of the traffic (or
> 2/3). With bonding, the difference is made up so that you get to use
> all of the available bandwidth (assuming your machines can really push
> that much data). If you do the two network 'trick' above, one NIC
> could end up being underutilized and the other will be maxed out and
> total thoughput will be less than the max possible. With bonding you
> get to use ALL of the available bandwidth, even if the traffic is not
> even.
>
> "> >
> "> > ">
> "> > "> Has anyone tried that before? What do you configure something like
> "> > that?
> "> >
> "> > What you really want to do is called 'bonding', where you bond two
> (or
> "> > more) NICs into a single logical NIC. Linux does support this. You
> "> > need to be sure your EtherNet switch supports it as well. Oh, it
> really
> "> > only makes sense if both machines (server AND client) are doing this.
> "> >
> "> > You will get the same bandwidth effect as you want, since if you
> really
> "> > pull/push data at a rate high enough, the kernel will load share
> across
> "> > the two (or more) NICs, although you won't have any one NIC
> *dedicated*
> "> > to reads or writes (just like on a SMP system you won't have any one
> "> > processor dedicated to any specific processing).
> "> >
> "> > ">
> "> > "> Thanks in advance,
> "> > "> TH
> "> > ">
> "> > ">
> "> > ">
> "> >
> "> > Robert Heller -- 978-544-6933
> "> > Deepwoods Software -- Linux Installation and Administration
> "> > http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
> "> > heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk
> "> >
> "> >
> "> >
> "> >
> "> >
> "> >
> "> >
> ">
> ">
> ">
>
> Robert Heller -- 978-544-6933
> Deepwoods Software -- Linux Installation and Administration
> http://www.deepsoft.com/ -- Web Hosting, with CGI and Database
> heller@deepsoft.com -- Contract Programming: C/C++, Tcl/Tk
>
>
>
>
>
>
>

Let me see if I understand you. Are you sayng that a 100 Mbps NIC can
write/read (each) at 100 Mbps or it can write/read at 100 Mpbs max? If the
later, then it is possible to overrun (or try to :)) the 100 Mbps card.
Depends on the bus speed.

You are right that a dedicated NIC for read and dedicated NIC for write
would provide any benefit. It was just a thought ....

Thanks!

"John-Paul Stewart" <jpstewart@binaryfoundry.ca> wrote in message
news:v63li3-bv8.ln1@mail.binaryfoundry.ca...
> Robert Heller wrote:
>> "TH" <noaddress@nodomain.com>,
>> In a message on Tue, 2 May 2006 13:01:23 -0500, wrote :
>>
>> "> I have this application that transfers huge amounts of data. The data
>> source "> is an NFS share. What I would like to do is mount the NFS share
>> so that the "> reads are from one NIC and move the data out through a
>> different NIC.
>>
>> It is doubtful that you can do *specificly* this.
>>
>> "> "> Has anyone tried that before? What do you configure something like
>> that?
>>
>> What you really want to do is called 'bonding', where you bond two (or
>> more) NICs into a single logical NIC. Linux does support this. You
>> need to be sure your EtherNet switch supports it as well. Oh, it really
>> only makes sense if both machines (server AND client) are doing this.
>>
>> You will get the same bandwidth effect as you want, since if you really
>> pull/push data at a rate high enough, the kernel will load share across
>> the two (or more) NICs, although you won't have any one NIC *dedicated*
>> to reads or writes (just like on a SMP system you won't have any one
>> processor dedicated to any specific processing).
>
> Actually, given that modern Ethernet NICs and switches are generally full
> duplex devices, I would wager that bonding two NICs will offer better
> bandwidth than the OP's original proposal. After all, a 100 Mbit NIC (as
> an example, it applies equally to Gigabit NICs) can simultaneously read
> and write at 100 Mbit/sec (theoretically). With his proposal he would
> still have a 100 Mbit/sec read channel and a 100 Mbit write channel. With
> your suggestion for bonding two NICs he'd have 2 x 100 Mbit for reading
> and 2 x 100 Mbit for writing. I don't see that the dedicated-read-NIC and
> dedicated-write-NIC would offer any benefit whatsoever (assuming he's
> connecting both NICs to the same network).
>
> A side note for the OP: if you're using Gigabit Ethernet, beware of the
> fact that a single GigE channel can theoretically consume almost all of
> the bandwidth on an ordinary 32-bit, 33MHz PCI bus. You'll need PCI-X or
> PCI Express (and/or multiple PCI busses) to get the benefit of multiple
> GigE NICs. You don't say what you're currently using so this may or may
> not be a consideration for you.

On Thu, 4 May 2006 17:06:17 -0500, "TH" <noaddress@nodomain.com> wrote:

>Let me see if I understand you. Are you sayng that a 100 Mbps NIC can
>write/read (each) at 100 Mbps or it can write/read at 100 Mpbs max? If the
>later, then it is possible to overrun (or try to :)) the 100 Mbps card.
>Depends on the bus speed.

Don't confuse bits and bytes. A 100Mbps is full speed each direction
with a full duplex link, there is a separate twisted pair in the cable
for each data direction.

What brand NICs are you using? Switching to 'name' brands (not Realtek)
may improve performance enormously. I chucked out all my 8139 based
NICs in favour of s/h Intel pro/100 and can achieve saturated link data
transfers between a couple faster boxen (P4-HT 2.4GHz + Sempron 2600+).
>
>You are right that a dedicated NIC for read and dedicated NIC for write
>would provide any benefit. It was just a thought ....

Separating read / write gives no benefit, 'bonding'[1] a pair of quality
NICs between the boxen will double speed. S/h pro/100 NICs sell for a
few bucks, better than buying new commodity NICs with rtl8139 chip.

[1] Intel call this 'teaming'. Can be used linux <-> windoze too.

Grant.
--
Memory fault -- brain fried