large file transfer speed

I am running a couple of intel servers on RH ES 3.0. Dual cpu, 2 gig
ram, hardware raid.

I need to transfer a huge file (2.8gig) from one machine to the other
every night at midnight. I need it to complete by 5AM. I have a T1
(1.5Mx1.5M) at corporate, and SDSL (384x384) at the remote site.

When I use scp to start the file transfer (at night I just run it from a
script) manually, the transfer speed starts at 132k/s, and then drops
down right away to about 38k/sec.

Is there a better/faster way to transfer a file of that size?

Thanks -D

DM wrote:
> When I use scp to start the file transfer (at night I just run it from a
> script) manually, the transfer speed starts at 132k/s, and then drops
> down right away to about 38k/sec.

Let's see. The bottleneck is the SDSL line, which is limited to
384Kb (38KB)/sec. That's consistent with the value reported by ssh,
which, I believe, reports in bytes, not bits. So you're pushing
the line as fast as you can. The high initial speed is probably
an artifact of filling buffers, not the actual initial transfer
rate. Is the file compressed? Is it compressible? That's the only
way I can see to possibly get faster transfers, short of paying
for an upgrade to the SDSL line...

On Mon, 17 Apr 2006 13:01:43 -0400, DM wrote:

> I am running a couple of intel servers on RH ES 3.0. Dual cpu, 2 gig
> ram, hardware raid.
>
> I need to transfer a huge file (2.8gig) from one machine to the other
> every night at midnight. I need it to complete by 5AM. I have a T1
> (1.5Mx1.5M) at corporate, and SDSL (384x384) at the remote site.
>
> When I use scp to start the file transfer (at night I just run it from a
> script) manually, the transfer speed starts at 132k/s, and then drops
> down right away to about 38k/sec.
>
> Is there a better/faster way to transfer a file of that size?
>
> Thanks -D
>
It is implied in your post the the 2.8G file is being transferred
between the corporate and remote site. Correct?

As you explained, you have a 384kbs bottleneck through the SDSL at the
remote site. Do you think that your T1 at corporate has magic properties
to overcome that bottleneck? It doesn't. The 38kB throug scp is a rate
that is within the expected transfer speed (i.e. it less than the
theoretical maximum of 48kB.) At your rate, it will take 21.5 hours. The
fastest rate reported is probably incorrect (burst or bad measurement).
But even using that rate (132kB/s), the transfer would still require about
6 hours.

It is not easily solved if there are political reasons why the transfer
must be completed by 5AM. For example, I am assuming that the transfer
must be completed to avoid consuming all bandwidth which should normally
be assigned to office workers during their normal business hours. One
approach is to buy more bandwidth, which may be as easy as buying
a second DSL line. The transfer which will take 21.5 hours could be okay
because at least one complete backup is accomplished before the next
backup job starts (but there's less than 3 hours to spare.)

There is also a higher rate DSL line which is assymetric (8 down, 1.5 up),
but it is not universally available, and from what I've heard, it is
difficult to achieve the quoted data rate.

One other idea: is your 2.8g data compressible? If a fast computer could
spend some time working compressing the data before it is sent over the
wire, then you might achieve your original goal. gzip or bzip2 may be
able to get 9:1 compression or better, depending on the nature of your dta.

--
Douglas Mayne

DM wrote:
> Is there a better/faster way to transfer a file of that size?

Another thought... Is this a repeated transfer of similar information
every night, or is the data significantly different each time? If it's
similar, you might pay the price of a too-long initial transfer and then
simply pass the *differences* each night. That *might* help you get
done in 5 hours after the first time. Look into 'rsync' and similar
programs if this sounds feasible.

DM wrote:
> I am running a couple of intel servers on RH ES 3.0. Dual cpu, 2 gig
> ram, hardware raid.
>
> I need to transfer a huge file (2.8gig) from one machine to the other
> every night at midnight. I need it to complete by 5AM. I have a T1
> (1.5Mx1.5M) at corporate, and SDSL (384x384) at the remote site.
>
> When I use scp to start the file transfer (at night I just run it from a
> script) manually, the transfer speed starts at 132k/s, and then drops
> down right away to about 38k/sec.
>
> Is there a better/faster way to transfer a file of that size?
>
> Thanks -D

Depending on the distance between the corporate and remote sites, burn
a dvd with the file and drive it over. Otherwise, pay to have your
SDSL upgraded.

Doug

On Mon, 17 Apr 2006 13:57:34 -0400, Douglas O'Neal wrote:

> DM wrote:
>
<snip>
> Depending on the distance between the corporate and remote sites, burn
> a dvd with the file and drive it over. Otherwise, pay to have your
> SDSL upgraded.
>
> Doug
>
"Never underestimate the bandwidth of a station wagon full of tapes
hurtling down the highway."

--
http://en.wikipedia.org/wiki/Andrew_Tanenbaum
http://en.wikipedia.org/wiki/Sneakernet

DM <dont_spam_me@reply_to_group.instead> writes:

>I am running a couple of intel servers on RH ES 3.0. Dual cpu, 2 gig
>ram, hardware raid.

>I need to transfer a huge file (2.8gig) from one machine to the other
>every night at midnight. I need it to complete by 5AM. I have a T1
>(1.5Mx1.5M) at corporate, and SDSL (384x384) at the remote site.

Assuming that the file does not change much from night to night, it would
be far far more efficient to use rsync. It will transfer only the changes.



>When I use scp to start the file transfer (at night I just run it from a
>script) manually, the transfer speed starts at 132k/s, and then drops
>down right away to about 38k/sec.

Sounds right 384k is presumably the speed in bits per second, which is
about 40Kbytes /sec, which is what you are seeing.



>Is there a better/faster way to transfer a file of that size?

See DHL replaces DSL at
http://www.embedded.com/showArticle....leID=184417070 .

Harold

HAHA - indeed. I've shared this logistical genius with my geek
associates - thanks.

-D

harold@hallikainen.com wrote:
> See DHL replaces DSL at
> http://www.embedded.com/showArticle....leID=184417070 .
>
> Harold
>

Unruh wrote:
> DM <dont_spam_me@reply_to_group.instead> writes:
>
>
>>I am running a couple of intel servers on RH ES 3.0. Dual cpu, 2 gig
>>ram, hardware raid.
>
>
>>I need to transfer a huge file (2.8gig) from one machine to the other
>>every night at midnight. I need it to complete by 5AM. I have a T1
>>(1.5Mx1.5M) at corporate, and SDSL (384x384) at the remote site.
>
>
> Assuming that the file does not change much from night to night, it would
> be far far more efficient to use rsync. It will transfer only the changes.

Will rsync make changes within an individual file, or just replace the
file itself?

>
>
>
>
>>When I use scp to start the file transfer (at night I just run it from a
>>script) manually, the transfer speed starts at 132k/s, and then drops
>>down right away to about 38k/sec.
>
>
> Sounds right 384k is presumably the speed in bits per second, which is
> about 40Kbytes /sec, which is what you are seeing.
>
>
>
>
>>Is there a better/faster way to transfer a file of that size?
>
>