Proxy that connects 2 different networks

Hi,

I've encountered a problem that i can't easily resolve, so i'd
appreciate any help.
I'm writing a HTTP proxy - the client (wget/browser) connects to a
listening-socket, and the proxy passes the HTTP request to a
webserver, then passing the reply from the webserver to the client.

My proxy do NOT accumulate the data returned from the WebServer : i'm
using epoll() , and each time my read() request is filled with data, i
send the data to the other side and free the buffer .

Everything works great when both the client and the webserver running
in the same network or using the same speed (100MB , 1GB) . however
when the client is living in a low-speed network (10MB) and the
webserver lives in a high-speed network (1GB) , i get performance
problems due to swapping .

My questions are practical and theoretical as well :
1. since i'm dealing with two different sockets (client socket+server
socket) , how can i tell the server-socket to use the speed of the
client-socket ? is it the Window-Size ?
2. since my process do not accumulate the memory , why do i see the
memory gets bigger ? (top command), i would expect the kernel/machine
to keep the memory, but not the process...
3. finally, how proxy usually deals with this problem ? i guess that
i'm not the first one trying to connect two different networks with
two statefull connections ...

thank you very much
zferentz

zferentz@gmail.com wrote:
> Hi,
>
> I've encountered a problem that i can't easily resolve, so i'd
> appreciate any help.
> I'm writing a HTTP proxy - the client (wget/browser) connects to a
> listening-socket, and the proxy passes the HTTP request to a
> webserver, then passing the reply from the webserver to the client.
>
> My proxy do NOT accumulate the data returned from the WebServer : i'm
> using epoll() , and each time my read() request is filled with data, i
> send the data to the other side and free the buffer .

What do you do if the send() blocks or otherwise doesn't succeed?

Robert

>
> Everything works great when both the client and the webserver running
> in the same network or using the same speed (100MB , 1GB) . however
> when the client is living in a low-speed network (10MB) and the
> webserver lives in a high-speed network (1GB) , i get performance
> problems due to swapping .
>
> My questions are practical and theoretical as well :
> 1. since i'm dealing with two different sockets (client socket+server
> socket) , how can i tell the server-socket to use the speed of the
> client-socket ? is it the Window-Size ?
> 2. since my process do not accumulate the memory , why do i see the
> memory gets bigger ? (top command), i would expect the kernel/machine
> to keep the memory, but not the process...
> 3. finally, how proxy usually deals with this problem ? i guess that
> i'm not the first one trying to connect two different networks with
> two statefull connections ...
>
> thank you very much
> zferentz
>

On Sep 17, 1:39 pm, Robert Harris <robert.f.har...@blueyonder.co.uk>
wrote:
> zfere...@gmail.com wrote:
> > Hi,
>
> > I've encountered a problem that i can't easily resolve, so i'd
> > appreciate any help.
> > I'm writing a HTTP proxy - the client (wget/browser) connects to a
> > listening-socket, and the proxy passes the HTTP request to a
> > webserver, then passing the reply from the webserver to the client.
>
> > My proxy do NOT accumulate the data returned from the WebServer : i'm
> > using epoll() , and each time my read() request is filled with data, i
> > send the data to the other side and free the buffer .
>
> What do you do if the send() blocks or otherwise doesn't succeed?
>
> Robert
>
>
>
>
>
> > Everything works great when both the client and the webserver running
> > in the same network or using the same speed (100MB , 1GB) . however
> > when the client is living in a low-speed network (10MB) and the
> > webserver lives in a high-speed network (1GB) , i get performance
> > problems due to swapping .
>
> > My questions are practical and theoretical as well :
> > 1. since i'm dealing with two different sockets (client socket+server
> > socket) , how can i tell the server-socket to use the speed of the
> > client-socket ? is it the Window-Size ?
> > 2. since my process do not accumulate the memory , why do i see the
> > memory gets bigger ? (top command), i would expect the kernel/machine
> > to keep the memory, but not the process...
> > 3. finally, how proxy usually deals with this problem ? i guess that
> > i'm not the first one trying to connect two different networks with
> > two statefull connections ...
>
> > thank you very much
> > zferentz- Hide quoted text -
>
> - Show quoted text -

Hi Robert,

if the send() fails i disconnect from both sides (close client+server
sockets),
However, if the send() partially succeeded (only part of the data was
sent) , i'm trying to resend the rest .

In Windows, if i'm trying to send too much data , beyond some kernel-
buffers , the send() will fail. however , i'm not sure whether this is
the case with Linux... maybe the kernel accumulate the memory and i
cannot do anything about it or cannot even tell that i'm sending too
much memory...

Any ideas ? how usually proxies deals with such problems ? Does it
have anything with the "TCP congestion" or TCP-window-size ?

tnx again,
zferentz

On Sep 18, 12:07 pm, zfere...@gmail.com wrote:
> On Sep 17, 1:39 pm, Robert Harris <robert.f.har...@blueyonder.co.uk>
> wrote:
>
>
>
>
>
> > zfere...@gmail.com wrote:
> > > Hi,
>
> > > I've encountered a problem that i can't easily resolve, so i'd
> > > appreciate any help.
> > > I'm writing a HTTP proxy - the client (wget/browser) connects to a
> > > listening-socket, and the proxy passes the HTTP request to a
> > > webserver, then passing the reply from the webserver to the client.
>
> > > My proxy do NOT accumulate the data returned from the WebServer : i'm
> > > using epoll() , and each time my read() request is filled with data, i
> > > send the data to the other side and free the buffer .
>
> > What do you do if the send() blocks or otherwise doesn't succeed?
>
> > Robert
>
> > > Everything works great when both the client and the webserver running
> > > in the same network or using the same speed (100MB , 1GB) . however
> > > when the client is living in a low-speed network (10MB) and the
> > > webserver lives in a high-speed network (1GB) , i get performance
> > > problems due to swapping .
>
> > > My questions are practical and theoretical as well :
> > > 1. since i'm dealing with twodifferentsockets (client socket+server
> > > socket) , how can i tell the server-socket to use the speed of the
> > > client-socket ? is it the Window-Size ?
> > > 2. since my process do not accumulate the memory , why do i see the
> > > memory gets bigger ? (top command), i would expect the kernel/machine
> > > to keep the memory, but not the process...
> > > 3. finally, how proxy usually deals with this problem ? i guess that
> > > i'm not the first one trying to connect twodifferentnetworkswith
> > > two statefull connections ...
>
> > > thank you very much
> > >zferentz- Hide quoted text -
>
> > - Show quoted text -
>
> Hi Robert,
>
> if the send() fails i disconnect from both sides (close client+server
> sockets),
> However, if the send() partially succeeded (only part of the data was
> sent) , i'm trying to resend the rest .
>
> In Windows, if i'm trying to send too much data , beyond some kernel-
> buffers , the send() will fail. however , i'm not sure whether this is
> the case with Linux... maybe the kernel accumulate the memory and i
> cannot do anything about it or cannot even tell that i'm sending too
> much memory...
>
> Any ideas ? how usually proxies deals with such problems ? Does it
> have anything with the "TCP congestion" or TCP-window-size ?
>
> tnx again,zferentz- Hide quoted text -
>
> - Show quoted text -

Hi Robert and thank you for your help.
I've found that problem , it has absolutely nothing with the kernel or
I/O, . since i'm working in nonblocking mode, i'm using queues to pass
data between one side and the other , the problem was that the queues
got huge due to the different push/pop differences (due to the network
differences) .

I completely forgot that i'm sending the data when the socket "is
ready" , so all the window-size stuff is completely irrelevant.

thnx