secure transfer of huge files between two hosts?

How do you transfer huge files (2-18Gigs) between two hosts?
It needs to be secure and it must offer a REGET-possibility (continue
download after the transfer was interrupted)

My first attempt was simply using https, which turned out to be
difficult, cause apache does not support big files. One can recompile it
to include big-file support, but there seem to be several problems and I
really dont want to go out to 10machines and recompile apache just for
this purpose. (and one would need to recompile all modules as well,
which always is a pain in the back)

Second attempt was scp, which does not support REGET.

Third attempt was ftp via a ssh-tunnel. I didnt get it to work.
(ssh -L 4021:localhost.21 user@remote.host -> passive -> dir ->
connection refused) and I remember always having troubles ftp via ssh.

Fourth attempt was psftp (there is a unix-port of putty), which supports
REGET (great tool : psftp) but does not support big files either. 2GB
ist the limit here as well.


Fifth attempt was rsync via ssh, which takes a *very* long time to
determine the needed delta-pieces and then fails anyway (there was
enough space on the targetdrive ...)
write failed on "/extra/monthly.imap.00-06-15.tgz": No space left on device
rsync error: error in file IO (code 11) at receiver.c(305)
rsync: connection unexpectedly closed (77 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)

I tried with different checksum-blocksizes, but always get the same result.
Rsync is an overkill anyway, cause it assumes possible changes all over
the file and not only at the end.



My last attempt will be to invoke dd or split to split the file to
chunks<2GB and transfer them seperately. On the other hand: we almost
have 2005 and I'm sure there is a solution to do it in once and I just
dont think about it by now.

thnx,
peter




--
http://www2.goldfisch.at/know_list
http://leblogsportif.sportnation.at

peter pilsl wrote:

> Third attempt was ftp via a ssh-tunnel. I didnt get it to work.
> (ssh -L 4021:localhost.21 user@remote.host -> passive -> dir ->
> connection refused) and I remember always having troubles ftp via ssh.
>

I don't know what your problem is with ssh, but there is that and scp.
Also, you might consider a VPN.

--

(This space intentionally left blank)

>
> I don't know what your problem is with ssh, but there is that and scp.
> Also, you might consider a VPN.
>

as I stated in my posting scp does not support REGET. To be more
detailed: the scp that comes with open-ssh does not. psftp that comes
with putty (there is a unix-port for it) does support it, but does not
support files>2GB.

VPN would be a possibility, but also a very heavy weapon for simply
transfering a simple file. Installing pptpd or whatever on many server
and maintaining the user, doing the routing whatever ... its just a bit
too much of work, when - what I want really do is:

downloadsecure -reget user@host:/file ./

peter


--
http://www2.goldfisch.at/know_list
http://leblogsportif.sportnation.at

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
NotDashEscaped: You need GnuPG to verify this message

In comp.os.linux.networking peter pilsl <pilsl@goldfisch.at>:
[..]

> as I stated in my posting scp does not support REGET. To be more
> detailed: the scp that comes with open-ssh does not. psftp that comes
> with putty (there is a unix-port for it) does support it, but does not
> support files>2GB.

> VPN would be a possibility, but also a very heavy weapon for simply
> transfering a simple file. Installing pptpd or whatever on many server
> and maintaining the user, doing the routing whatever ... its just a bit
> too much of work, when - what I want really do is:

> downloadsecure -reget user@host:/file ./

I'd try out if "unison - File Synchronizer" works better.

http://www.cis.upenn.edu/~bcpierce/unison/

--
Michael Heiming (GPG-Key ID: 0xEDD27B94)
mail: echo zvpunry@urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBaHmsAkPEju3Se5QRAmLZAKCK1p+9/nYCyo2MecPXQ2NEp5SfSwCffkqY
MK6enn7YiTeMOyW81LDlcJc=
=Sogt
-----END PGP SIGNATURE-----

On Sat, 09 Oct 2004 23:00:38 +0200, peter pilsl wrote:

> How do you transfer huge files (2-18Gigs) between two hosts?

FTP/HTTP/RSYNC in a SSH tunnel?

HTTPS?

--
BOFH Excuse #18:

excess surge protection

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2004-10-09, peter pilsl <pilsl@goldfisch.at> wrote:

> Fifth attempt was rsync via ssh, which takes a *very* long time to
> determine the needed delta-pieces and then fails anyway (there was
> enough space on the targetdrive ...)

[...]

> My last attempt will be to invoke dd or split to split the file to
> chunks<2GB and transfer them seperately.


I use a similar strategy for a slightly different problem: I need to
transfer largish files over a poor, slow, and unreliable link. I use
bzip2 compression, split the tarball, then rsync. This way, if the link
dies midway, I've gotten at least something.

I usually split into sizes which will go over in 5-10 minutes. I don't
mind having to redo that much copying, but much longer than that is
too much wasted time. (Too much shorter generates too many split pieces,
which is just an aesthetic issue; you may not care about that.)

- --keith

- --
kkeller-usenet@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://wombat.san-francisco.ca.us/cgi-bin/fom

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBaLiJhVcNCxZ5ID8RAgkjAJ9HW+D+0xnqboYhuysM6m 2WiFB0KQCeKIET
sHyqySRgCTg7RLX54A8zM8U=
=s3Lh
-----END PGP SIGNATURE-----

On Sat, 09 Oct 2004 23:00:38 +0200, peter pilsl <pilsl@goldfisch.at>
wrote:

>
>How do you transfer huge files (2-18Gigs) between two hosts?
>It needs to be secure and it must offer a REGET-possibility (continue
>download after the transfer was interrupted)
>
>My first attempt was simply using https, which turned out to be
>difficult, cause apache does not support big files. One can recompile it
>to include big-file support, but there seem to be several problems and I
>really dont want to go out to 10machines and recompile apache just for
>this purpose. (and one would need to recompile all modules as well,
>which always is a pain in the back)
>
>Second attempt was scp, which does not support REGET.
>
>Third attempt was ftp via a ssh-tunnel. I didnt get it to work.
>(ssh -L 4021:localhost.21 user@remote.host -> passive -> dir ->
>connection refused) and I remember always having troubles ftp via ssh.
>
>Fourth attempt was psftp (there is a unix-port of putty), which supports
>REGET (great tool : psftp) but does not support big files either. 2GB
>ist the limit here as well.
>
>
>Fifth attempt was rsync via ssh, which takes a *very* long time to
>determine the needed delta-pieces and then fails anyway (there was
>enough space on the targetdrive ...)
>write failed on "/extra/monthly.imap.00-06-15.tgz": No space left on device
>rsync error: error in file IO (code 11) at receiver.c(305)
>rsync: connection unexpectedly closed (77 bytes read so far)
>rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>
>I tried with different checksum-blocksizes, but always get the same result.
>Rsync is an overkill anyway, cause it assumes possible changes all over
>the file and not only at the end.
>
>
>
>My last attempt will be to invoke dd or split to split the file to
>chunks<2GB and transfer them seperately. On the other hand: we almost
>have 2005 and I'm sure there is a solution to do it in once and I just
>dont think about it by now.
>
>thnx,
>peter

Trying to accomplish the same task I find that large files over VPN or
tunnel seem to go quite slowly most likely due to the
encryption/decryption in real time.

What we do for very large files is to pgp them (which encrypts and
compresses them) after which we use a straight ftp transfer where the
remote server is limiting access to specify IP's and username/password
combos.

Of course, on the remote server they must be decrypted and expanded
appropriately. You may have an issue with large files but you'll have
to be sure that the OS and the interim programs can handle them.

Good luck,

Bob

peter pilsl wrote:
>
> How do you transfer huge files (2-18Gigs) between two hosts?
> It needs to be secure and it must offer a REGET-possibility (continue
> download after the transfer was interrupted)
>

thnx a lot for all answers.

I finally came to two solutions:

A) if the file does not exist already then I use split as filter when
creating to get it into appropriate junks<2GB and then use https or
psftp to transfer it. Nevertheless I dont like split too much. I always
fear that one byte is lost when rejoining them with cat :)

B) if the file already exists I do the following: just download using
psftp also - which will stop after 2GB-limit is reached.
Additionally I use dd on the remote side (using the skip-option) to
create a file that contains the rest which is bigger than 2GB. (I use
bs=100000 here). I can download this file then and use dd on the local
side (using the seek-option) to add this rest to the already downloaded
first part.
This way is very handy cause:
i) I can start download immediately without needing to split first
ii) Splitting an existing 3GB-File into two parts needs 3GB emtpy space
and is timeconsuming after all (dealing with IDE here). Using seek and
split I need only one third of empty space and systemresources. Same on
local side.

conclusion:
*rsync is totally lost with such big files.
*support of big files is (and I fully understand this) not a
top-priority in many tools and ports.
* md5checksum is quite fine with big files for checking the transfer. On
my celeron700 with slow IDE it tooks about 7minutes to calculate the
checksum which is ok to me.
* unison looks great but its not a standardtool so I will not rely on
it. thnx.


peter


--
http://www2.goldfisch.at/know_list
http://leblogsportif.sportnation.at

peter pilsl wrote:
>
> How do you transfer huge files (2-18Gigs) between two hosts?
> It needs to be secure and it must offer a REGET-possibility (continue
> download after the transfer was interrupted)
>
> My first attempt was simply using https, which turned out to be
> difficult, cause apache does not support big files. One can recompile it
> to include big-file support, but there seem to be several problems and I
> really dont want to go out to 10machines and recompile apache just for
> this purpose. (and one would need to recompile all modules as well,
> which always is a pain in the back)
>
> Second attempt was scp, which does not support REGET.
>
> Third attempt was ftp via a ssh-tunnel. I didnt get it to work.
> (ssh -L 4021:localhost.21 user@remote.host -> passive -> dir ->
> connection refused) and I remember always having troubles ftp via ssh.
>
> Fourth attempt was psftp (there is a unix-port of putty), which supports
> REGET (great tool : psftp) but does not support big files either. 2GB
> ist the limit here as well.
>
>
> Fifth attempt was rsync via ssh, which takes a *very* long time to
> determine the needed delta-pieces and then fails anyway (there was
> enough space on the targetdrive ...)
> write failed on "/extra/monthly.imap.00-06-15.tgz": No space left on device
> rsync error: error in file IO (code 11) at receiver.c(305)
> rsync: connection unexpectedly closed (77 bytes read so far)
> rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>
> I tried with different checksum-blocksizes, but always get the same result.
> Rsync is an overkill anyway, cause it assumes possible changes all over
> the file and not only at the end.
>
>
>
> My last attempt will be to invoke dd or split to split the file to
> chunks<2GB and transfer them seperately. On the other hand: we almost
> have 2005 and I'm sure there is a solution to do it in once and I just
> dont think about it by now.
>
> thnx,
> peter
>
>
>
>
How about using something like gtk-gnutella, or the torrent over ssl?
Both of these support very large files (eg. DVD's, distro's) and give
the capability to serve the files from multiple machines as well as
restartability and automatic reassembly.

Rob Mitchell wrote:
>
> How about using something like gtk-gnutella, or the torrent over ssl?
> Both of these support very large files (eg. DVD's, distro's) and give
> the capability to serve the files from multiple machines as well as
> restartability and automatic reassembly.
>

actually thats an interesting point. What I need to transfer this big
files for is distributed backup. Some kind of p2p or torrent might be
very handy here, cause it automatically loadbalances and looks for the
fastest connections. Maybe I'll give it a try.

thnx,
peter



--
http://www2.goldfisch.at/know_list
http://leblogsportif.sportnation.at