On Sat, 09 Oct 2004 23:00:38 +0200, peter pilsl <pilsl@goldfisch.at>
wrote:

>
>How do you transfer huge files (2-18Gigs) between two hosts?
>It needs to be secure and it must offer a REGET-possibility (continue
>download after the transfer was interrupted)
>
>My first attempt was simply using https, which turned out to be
>difficult, cause apache does not support big files. One can recompile it
>to include big-file support, but there seem to be several problems and I
>really dont want to go out to 10machines and recompile apache just for
>this purpose. (and one would need to recompile all modules as well,
>which always is a pain in the back)
>
>Second attempt was scp, which does not support REGET.
>
>Third attempt was ftp via a ssh-tunnel. I didnt get it to work.
>(ssh -L 4021:localhost.21 user@remote.host -> passive -> dir ->
>connection refused) and I remember always having troubles ftp via ssh.
>
>Fourth attempt was psftp (there is a unix-port of putty), which supports
>REGET (great tool : psftp) but does not support big files either. 2GB
>ist the limit here as well.
>
>
>Fifth attempt was rsync via ssh, which takes a *very* long time to
>determine the needed delta-pieces and then fails anyway (there was
>enough space on the targetdrive ...)
>write failed on "/extra/monthly.imap.00-06-15.tgz": No space left on device
>rsync error: error in file IO (code 11) at receiver.c(305)
>rsync: connection unexpectedly closed (77 bytes read so far)
>rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>
>I tried with different checksum-blocksizes, but always get the same result.
>Rsync is an overkill anyway, cause it assumes possible changes all over
>the file and not only at the end.
>
>
>
>My last attempt will be to invoke dd or split to split the file to
>chunks<2GB and transfer them seperately. On the other hand: we almost
>have 2005 and I'm sure there is a solution to do it in once and I just
>dont think about it by now.
>
>thnx,
>peter

Trying to accomplish the same task I find that large files over VPN or
tunnel seem to go quite slowly most likely due to the
encryption/decryption in real time.

What we do for very large files is to pgp them (which encrypts and
compresses them) after which we use a straight ftp transfer where the
remote server is limiting access to specify IP's and username/password
combos.

Of course, on the remote server they must be decrypted and expanded
appropriately. You may have an issue with large files but you'll have
to be sure that the OS and the interim programs can handle them.

Good luck,

Bob