[Bug 3117] New: rsyncing a file to a partially downloaded copy is

https://bugzilla.samba.org/show_bug.cgi?id=3117

Summary: rsyncing a file to a partially downloaded copy is
extremely slow
Product: rsync
Version: 2.6.6
Platform: All
OS/Version: Linux
Status: NEW
Severity: enhancement
Priority: P3
Component: core
AssignedTo: wayned@samba.org
ReportedBy: prumpf@gmail.com
QAContact: rsync-qa@samba.org


I would imagine that it is common to use rsync to rsync a file to a partial
download of itself (or a prefix of the file that might arise from something
being appended to the source file). However, for large files, this seems to be
extremely slow since many small chunks of constant size are compared.

While the --block-size option can help with this, the block size to use has to
be calculated for each rsync invocation to avoid retransmitting an average of
block-size/2 bytes.

I would suggest that modifying the rsync algorithm to initially compare chunks
of exponentially increasing size until a mismatch is found would probably be
worth it in terms of the total bandwidth saved. Even if you disagree with that,
a quick-and-dirty fix would be an option that would cause rsync to check for the
case that the larger file results from the smaller file by appending data before
going into the full rsync algorithm. I believe this wouldn't take more than a
couple of minutes for someone familiar with rsync internals.

It certainly seems odd that rsync is essentially unusable for something that
wget --continue deals with.

To help searching for this bug: log files append appending live streams partial
download aborted download interrupted download restarting rsync restart

--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html