localhost optimization

I seem to have read in numerous places, that using a form of IP based
network connectivity to localhost on Linux is something especially
optimized. So much so, that there would hardly have to be any
difference between, for example, setting up a TCP/IP connection to
localhost and setting up a connection through a UNIX socket. I worked
with that assumption for a good while until I got to test it and I got
the shock of my life. I know it's just a single test over a self-
invented protocol using a self-written implementation, but a
difference of a factor thousand in speed ? That seems like a lot. Am
I doing something wrong ? Are there flags one has to set in ioctl or
something ? I won't be posting any code just yet, since I'd really
like a simple answer: has it or has it not been optimized, and if so,
what is a reasonable difference to expect ? Thank you for your time.

On May 1, 2:46 pm, k...@pink-frog.com wrote:

> I seem to have read in numerous places, that using a form of IP based
> network connectivity to localhost on Linux is something especially
> optimized. So much so, that there would hardly have to be any
> difference between, for example, setting up a TCP/IP connection to
> localhost and setting up a connection through a UNIX socket. I worked
> with that assumption for a good while until I got to test it and I got
> the shock of my life. I know it's just a single test over a self-
> invented protocol using a self-written implementation, but a
> difference of a factor thousand in speed ? That seems like a lot. Am
> I doing something wrong ? Are there flags one has to set in ioctl or
> something ? I won't be posting any code just yet, since I'd really
> like a simple answer: has it or has it not been optimized, and if so,
> what is a reasonable difference to expect ? Thank you for your time.

My bet is that your code triggered Nagle in a bad way.

DS

On May 2, 11:33 am, David Schwartz <dav...@webmaster.com> wrote:
....
>
> My bet is that your code triggered Nagle in a bad way.
>
> DS

I've thought about that; my packets are indeed (or would be, if we did
actually send those packets) typically very short - say, 20, 30
bytes. But I was under the impression that optimization took care of
the fact that those packets to localhost were never actually sent.
And that there would be no delay in sending them either. I guess I
was wrong. I'll try setting TCP_NODELAY.

On May 2, 11:33 am, David Schwartz <dav...@webmaster.com> wrote:

>
> My bet is that your code triggered Nagle in a bad way.
>
> DS

I set TCP_NODELAY. That helped a lot. We're down to:

0m0.051s

versus:

0m4.078s

Which is, roughly, a factor of eighty. Which is good, but still a
huge factor. Do you have any more ideas ?

On May 2, 12:41 pm, k...@pink-frog.com wrote:

> I set TCP_NODELAY. That helped a lot. We're down to:
>
> 0m0.051s
>
> versus:
>
> 0m4.078s
>
> Which is, roughly, a factor of eighty. Which is good, but still a
> huge factor. Do you have any more ideas ?

You're still making a complex and expensive system call for every
20-30 bytes you need to send. TCP_NODELAY minimized the symptoms of
the problem but didn't fix the problem. You still have a fundamental
difference -- TCP is a bytestream protocol and UDP is a datagram
protocol. Your code does not sensibly use a bytestream protocol.

Accumulate your data into larger chunks. Do not call 'send' or 'write'
unless one of two things are the case:

1) You have at least 1Kb to send.

2) You do not expect to need to send any more data in the immediate
future, until either some event occurs or you receive data from the
other side.

You may also be interacting badly with the scheduler, switching
between sending and receiving more often than is needed. It's hard to
know without seeing your test code, but this can be a source of poor
performance.

Imagine if your UDP code was doing the following:

1) Send 20 datagrams, each 20-30 bytes
2) Receive the 20 datagrams, each 20-30 bytes
3) Repeat

and your TCP code was doing the following:

1) Send one small chunk of 10-20 bytes.
2) Receive one small chunk of 10-20 bytes.
3) Repeat

The TCP code would require 20 times as many loops as the first. If the
loop overhead is a significant fraction of the time, then the first
bit of code will be much faster than the second. Sending larger chunks
is one way to remove this cost.

Try to take advantage of TCP's transmit pacing by calling 'send' in a
tight loop until it blocks.

DS