Backup and recovery describe the process of creating and storing copies of data that can be used to protect organizations against data loss. Recovery from a backup usually entails restoring the data to its original/ different location where it may be utilized in lieu of the lost or damaged data.
Primary data failures can be the result of hardware or software failure, data corruption, or a human-caused event, such as a malicious attack (virus or malware), or accidental deletion of data.
Manually syncing folders or copying files from one location to another can be incredibly time-consuming to do. That’s where
rsync comes in super, hyper handy.
Rsync is a fast and extraordinarily versatile files and folders synchronizing tool. It can copy locally, to/from another host over any remote shell, or to/from a remote Rsync daemon. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. After that first sync,
rsync will only move the changes, keeping the mirror identical to the primary server and minimizing network traffic. Scared of losing connection during transfer – this tool will start exactly where it left off, once the connection is re-established.
Most Linux distributions and macOS come with the
rsync tool pre-installed. If you don’t already have
rsync on your machine, you can easily install it using your distribution’s package manager.
$ sudo apt install rsync
Rsync Command Syntax and Common Options
rsync utility expressions are as follows:
Local to Local: rsync [OPTION]... [SRC]... DEST Local to Remote: rsync [OPTION]... [SRC]... [USER@]HOST:DEST Remote to Local: rsync [OPTION]... [USER@]HOST:SRC... [DEST]
OPTION– The rsync options.
SRC– Source directory.
DEST– Destination directory.
USER– Remote username.
HOST– Remote hostname or IP Address.
rsync has a ton of options that control how the command behaves. The following are the most commonly used options:
--archive, archive mode, equivalent to combining
-rlptgoD. This option tells
rsyncto syncs directories recursively, transfer special and block devices, preserve symbolic links, modification times, groups, ownership, and permissions.
--recursiverecurses into directories
--relativepreserves the full path, it creates subdirectories in the target location.
rsyncto compress the data as it is sent to the destination machine. Use this option only if the connection to the remote machine is slow.
-P, is equivalent to
--progress. When this option is used,
rsyncshows a progress bar during the transfer and keeps the partially transferred files. It is useful when transferring large files over slow or unstable network connections.
--delete, option is used,
rsyncdeletes extraneous files from the destination location. It is useful for mirroring.
--quietuse this option if you want to suppress non-error messages.
-e, option allows you to choose a different remote shell. By default,
rsyncis configured to use
--human-readableoutputs numbers in a human-readable format.
Copy/Sync Files and Directory Locally
The following command will sync files and directories on a local machine from one location to another location. The user running the command must have read permissions on the source location and write permissions on the destination.
$ sudo rsync -avh /var/log/ ~/Desktop/log-backups/ password for clinerds: sending incremental file list created directory /home/clinerds/Desktop/log-backups log/ log/auth.log log/btmp log/btmp.1 log/kern.log log/syslog ... ... sent 17.18M bytes received 2.80K bytes 3.82M bytes/sec total size is 17.17M speedup is 1.00
Omitting the filename from the destination location copies the file with the later current name. If you want to save the file under a different name, you need to specify the new name on the destination part. Here we specifically chose
log-backups. To cut it short, if the destination directory doesn’t exist,
rsync will create it.
Trailing slash (/)
It’s worth mentioning that
rsync treats source folders with a trailing slash (/) differently. If the source directory contains a trailing slash, the command will only transfer the contents of that directory to the destination directory; if not,
rsync transfers the source directory to the destination directory.
Here are some examples:
with a trailing slash (/)
$ rsync -avgrtHP ~/Desktop/Out/ ~/Desktop/public OUPUT $ ls Out
without a trailing slash (/)
$ rsync -avgrtHP ~/Desktop/Out ~/Desktop/private OUPUT $ ls TopG samael_caillard_hipster_in_stone-19.jpg samael_caillard_hipster_in_stone-3.jpg Popsicle-gif.gif Prints 'Screenshot-2017-12-2 Kiss Mix.png' 'Screenshot-2017-12-2 Mojito - Kiss Mix.png'
Copy\Sync Files and Directory from a Local to a Remote Server
In other to use
rsync to synchronize data remotely, it must be installed on both the source and the destination machine.
rsync is configured to use
SSH as default remote shell.
In the following example, we are transferring a directory from a local to a remote server:
$ rsync -avgtrHP /Users/probe/www/clinerds-Blog/ firstname.lastname@example.org:/Users/luna/Backups/Blog-backupX/ Password: building file list ... 172 files to consider created directory /Users/luna/Backups/Blog-backupX
Copy\Sync Files and Directory from a Remote to a Local Server
To transfer data from a remote to a local machine, use the remote location as a source:
$ rsync -avgtrHP email@example.com:/Users/luna/Backups/Blog-backupX/ ~/desktop/desktop-backups/ Password: receiving file list ... 642 files to consider created directory /Users/probe/desktop/desktop-backups
-avgtrHP options, they will tell you a lot.
Rsync over SSH
With all that said, Rsync communication is plain clear text by default.
This makes Rsync is vulnerable to the Man-in-the-middle attacks (MITM). It doesn’t natively provide encryption of any of the data transferred over.
The usual and smart workaround is to use the Secure Shell (SSH) protocol to tunnel
-e option specify the remote shell to use; here
$ rsync -avgtrHP -e ssh ~/desktop/expenses/ firstname.lastname@example.org:~/Desktop/ email@example.com's password: building file list ... 202 files to consider
Rsync with Non-standard SSH Port
Rsync uses the default port 22 over
SSH to communicate with the remote host. Most Linux system users set the default
SSH port to a non-standard port for security reasons. If such a case, you must specify a
SSH port to connect
rsync command on a non-standard port.
SSH on the remote host is listening on a port other than the default 22, specify the port using the
-e option to.
Local to a Remote Server with Non-standard SSH Port
$ rsync -avgtrHP -e "ssh -p 6656" ~/desktop/desktop-backups/ firstname.lastname@example.org:~/Desktop/ email@example.com's password: building file list ... 619 files to consider
Remote to a Local Server with Non-standard SSH Port
$ rsync -avgtrHP -e "ssh -p 6656" firstname.lastname@example.org:~/desktop/ ~/desktop/desktop-backups/ email@example.com's password: building file list ... 619 files to consider
These options allow us to define which files or directories we want to include in our synchronization and which we do not.
In the following example, we will include files and directories only if they begin with the capitalized letter ‘P‘ and exclude everything else.
ls in our current directory on our local end:
$ ls Popsicle-gif.gif Prints Screenshot-2017-12-2 Kiss Mix.png Screenshot-2017-12-2 Mojito - Kiss Mix.png Screenshot-2017-12-2 Our Story- Kiss Mix(1).png Screenshot-2017-12-2 Our Story- Kiss Mix(2).png Screenshot-2017-12-2 Our Story- Kiss Mix.png instG samael_caillard_hipster_in_stone-19.jpg samael_caillard_hipster_in_stone-3.jpg TopG
rsync -avgrtHP -e "ssh -p 6656" --include 'P*' --exclude '*' ~/desktop/Out/ firstname.lastname@example.org:~/desktop/include email@example.com's password:
Now, let’s check on the remote side:
$ cd ~/desktop/include $ls Popsicle-gif.gif Prints
We can also specify the Max file size to be synced.
Here, we will synchronize only those files which are equal to or smaller than 300k.
rsync -avgrtHP -e "ssh -p 6656" --max-size '300k' ~/desktop/Out/ firstname.lastname@example.org:~/desktop/max300k email@example.com's password:
Remove source files after transfer
--remove-source-files option lets you remove the files from the source after the sync. is done.
rsync -avgrtHP -e "ssh -p 6656" --remove-source-files ~/firstname.lastname@example.org:~/desktop/music email@example.com's password:
After executing the above command, everything inside the mp3 directory will be removed after successful transfer.
As we’ve seen, rsync could be is a great replacement for
sftp , and
cp commands. For best results, backup copies are made on a consistent, regular basis to minimize the amount of data lost between backups. The more time passes between backup copies, the more potential for data loss when recovering from a backup.
That’s all with rsync now, stay connected with Clinerds for more exciting and interesting future posts.