How to Resume Partial File Transfers
I work primarily with UNIX and Linux machines and scp is my main choice to transfer files with. It is both convenient, short and secure.
Example:
scp localfile user@remotecomputer:/path/to/target/dir
Recently I was transferring an 8GB file and due to a network issue, the transfer was interrupted at nearly 40%.
I have learned over the years that there is often little which can be done to prevent such interruptions. I know that many people go to great lengths to find Broadband which is reliable, but sometimes disruptions simply cannot be avoided. Of course, this can be both frustrating and time-consuming, but there is a quick fix. Although such disruptions can’t be prevented, there is a fast and easy way to resume them. Resuming has relatively few system requirements, and can save a lot of time and hassle when transferring large files.
I found a solution at joen.dk ,which uses rsync to resume the transfer:
rsync --partial --progress --rsh=ssh host:remote_file local_file
Now we can improve this slightly by shortening the above command. We can substitute –rsh=ssh with -e ssh, and use -P instead of –partial –progress. Also, you can add user@host if you need to specify a different remote shell user:
rsync -P -e ssh user@host:remote_file local_file
This above example will work with any file that was partially transfered. How the transfer was started does not really matter. It could be through scp, nc or even ftp. After you execute the above command it will take rsync a little time to verify the previously downloaded part before it continues with the rest. Be patient, depending on your network speed rsync could take some time to go through what you have already transfered. Of course this is much faster than if you were to start the download all over again and it shows you the progress in percentages.
Keep in mind that there have to be a couple of requirements in place in order to resume the file transfer with rsync:
1. You should have remote shell access.
2. The remote machine should have rsync installed. Since rsync is by default on most Linux distributions that generally should not be an issue.
6 Comments to How to Resume Partial File Transfers
Leave a comment
Search
Archive
| M | T | W | T | F | S | S |
|---|---|---|---|---|---|---|
| « Mar | ||||||
| 1 | 2 | 3 | 4 | 5 | 6 | |
| 7 | 8 | 9 | 10 | 11 | 12 | 13 |
| 14 | 15 | 16 | 17 | 18 | 19 | 20 |
| 21 | 22 | 23 | 24 | 25 | 26 | 27 |
| 28 | 29 | 30 | 31 | |||
Recent Comments
- Aron on Clone Disk Drives with Ubuntu. Make an Exact Copy of Your Hard Drive.
- letroll on Android – Displaying Dialogs From Background Threads
- When Wireless Goes Rogue « Ham Radio Weblog PD0AC on KARMA on the Fon and Sniffing Wireless Network Traffic with Ubuntu – Step by Step
- jornando junior on Android – Displaying Dialogs From Background Threads
- Geo on Extract Audio (.mp3) from Video Files Like .flv, .mov, .avi and Others with Ubuntu
Categories
Blogroll
Online Tools
Other
BLOG ARCHIVE
- March 2012 (1)
- November 2011 (1)
- August 2011 (1)
- April 2011 (1)
- January 2011 (2)
- September 2010 (1)
- August 2010 (2)
- July 2010 (2)
- June 2010 (2)
- May 2010 (1)
- January 2010 (2)
- December 2009 (2)
- November 2009 (3)
- October 2009 (1)
- September 2009 (3)
- July 2009 (1)
- May 2009 (1)
- March 2009 (1)
- February 2009 (2)
- January 2009 (2)
- December 2008 (1)
- November 2008 (4)
- October 2008 (5)
Great tip, though I think you’ve reversed the source/destination of the rsync example.
Its not reversed, the example synchronizing from a remote computer to a local computer.
yes but shouldnt the order be
‘ssh source dest’ and ‘rsync source dest’.
your ssh example is inconsistent with your rsync example
Great tip, thanks!
You could add that if the person is using a non-standard ssh port he should add it like this:
rsync -P -e “ssh -p $portNumber” user@host:remote_file local_file
Including the quotation marks.
I thought of coming back here and sharing this info:
If you wish resume downloading of a whole folder with several files, instead of a single file, do this:
rsync -P -avz -e ssh user@host:remote_folder/ local_folder/
or if you’re using a port different from 22:
rsync -P -avz -e “ssh -p $portNumber” user@host:remote_folder/ local_folder/
‘-a’ there is the crucial parameter to be added, ‘-v’ and ‘-z’ are optional. Check man rsync for further info.
Cheers.
either the scp example should reverse the source and dest or you should specify which machine you are logged into for each command. clearly in the scp example you are copying a local file to a remote machine, and in the rsync example you appear to be doing the reverse. one would assume you are logged into different machines in each case if one knows the syntax of the commands. but it might not be obvious.