Making backups with rsync

rsync is a tool used for synchronize files or directories minimizing the data transfer, due to its algorithm based in the difference of the files and the data compression. rsync allow us work in a local machine and with remote servers using ssh server or using rsync as a server listening the port number 873/tcp.
This tutorial is a brief introduction to the use of rsync and the main options and uses.

Basic use of rsync

The basic way of use rsync is with the -av options:

-a (archive mode) : same options as -rlptgoD (-r recursively, -l copy the symbolic links as symbolic links, -p keep the permissions, -t keep the modify time, -g keep the group, -o keep the owner, -D keep the devices files)

root@opentodo-server:~# ls
dir1 dir2
root@opentodo-server:~# rsync -av dir1/ dir2/
sending incremental file list
./
bar
fic1
foo
dira/
dirb/

sent 239 bytes received 80 bytes 638.00 bytes/sec
total size is 0 speedup is 0.00

If one file is changed rsync by default use the option -W for local copy. This causes the entire file is copied:

root@opentodo-server:~# echo "one small step for a man, one giant leap for mankind" >> dir1/foo
root@opentodo-server:~# rsync -av dir1/ dir2/
sending incremental file list
./
foo

<strong>sent 4871 bytes</strong> received 36 bytes 9814.00 bytes/sec
total size is 4712 speedup is 0.96

To avoid this behavior we can use the option –no-whole-file:

root@opentodo-server:~# rsync -av --no-whole-file dir1/ dir2/
sending incremental file list
./
foo

<strong>sent 695 bytes</strong> received 78 bytes 1546.00 bytes/sec
total size is 4712 speedup is 6.10

Incremental backups

With rsync we can make incremental backups saving the changed files in a different directory with the -b option:

root@opentodo-server:~# rsync -avvb --backup-dir=~/BackupDir_$(date +%H%M%m%d%y) dir1/ dir2/
backup_dir is ~/BackupDir_2015082612/
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
bar
foo is uptodate
<strong>backed up bar to ~/BackupDir_2015082612/bar</strong>
total: matches=0 hash_hits=0 false_alarms=0 data=21

sent 166 bytes received 45 bytes 422.00 bytes/sec
total size is 4733 speedup is 22.43

Including and excluding files for the backup

We can include and exclude by the file or directory name with the options –include or –exclude:

root@opentodo-server:~# rsync -av --exclude='bar' dir1/ dir2/
sending incremental file list
./

sent 81 bytes received 17 bytes 196.00 bytes/sec
total size is 4712 speedup is 48.08

root@opentodo-server:~# rsync -av --include='bar' dir1/ dir2/
sending incremental file list
bar
sent 154 bytes received 33 bytes 374.00 bytes/sec
total size is 4733 speedup is 25.31

Also we can create filters in a text file for rsync for include or exclude directories:

root@opentodo-server:~# cat dir.txt
+ */
+ /var/log/**
+ /usr/share/doc/procmail/**
- *

• + : match.
• – : discard.
• */ : is necessary for match the own directories (/var, /var/log and /usr/, /usr/share/ …) .
• – * :determine all that doesn’t match with the pattern will exclude for the backup.
• ** : all files and directories will match with the pattern.

root@opentodo-server:~# rsync -av --prune-empty-dirs --include-from=dir.txt / backup/
building file list ... done
(...)
var/log/news/
var/log/news/news.crit
var/log/news/news.err
var/log/news/news.notice

sent 1702534 bytes received 1148 bytes 3407364.00 bytes/sec
total size is 1612678 speedup is 0.95

The –prune-empty-dirs option is used for don’t copy the empty directories.

Other way without –prune-empty-dirs option specifying all the directories:

root@opentodo-server:~# cat dir.txt
+ /var/
+ /var/log/
+ /usr/
+ /usr/share/
+ /usr/share/doc/
+ /usr/share/doc/procmail/
+ /var/log/**
+ /usr/share/doc/procmail/**
- *

This are two valid patterns, the first is more simple than the second, but It will be more slow for search the directories recursively.

Remote rsync

We can use rsync for remote backups using ssh. For do this more easy, I’ll copy my key to the other server for connect with the server without using a password. First we have to generate a rsa key pair in our servers:

# ssh-keygen -t rsa

Next copy our public key to the destination backup server as authorized key:

# scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/authorized_keys

Now we can make a backup script to syncronize two directories:

root@opentodo-server:~# cat syncbackup.sh

#!/bin/bash

export source=/root/backup/
export destination=192.168.0.110:/backup
export logfile=/var/log/syncbackup.log

echo -e "n** Begin at $(date +%H:%M:%S_%m/%d/%y) **" >> $logfile
nice -n19 ionice -n2 rsync -avz --delete --no-whole-file $source root@$destination 2>&1 >> $logfile
echo -e "n** Finish at $(date +%H:%M:%S_%m/%d/%y) **" >> $logfile

nice and ionice command are used for the best performance for the server and the backup don’t interrupt the work of the server. nice specify the priority for the process between the range -20 (High priority) and 19 (low priority). ionice is used to determine the priority that one process access to the disk.[/sourcecode]

And execute every hour with cron:

# crontab -e

# m h dom mon dow command
0 * * * * /root/syncbackup.sh

Official website: http://rsync.samba.org/
rsync algorithm website: http://rsync.samba.org/tech_report/