Backups with rsync

I am currently running an unRAID homeserver to provide data and a few services on my local network. The Synology Diskstation 416 that it replaced, now serves as the backup target.
After testing several backup solutions, I think a simple rsync script is enough for what I need. It cannot offer fancy stuff but is a reliable and well tested tool. A few notes on how I came to this conclusion: The backup target (DS 416) is local and set up with disk encryption, so I don't need a tool with built-in encryption (neither storage, nor transmission). Deduplication is nice to have but in my case reduced the backup size only by a few hundred GB - almost nothing compared to several TB of data. Incremental backups will keep the required disk space low anyway.

Hardware

function name description
backup source homeserver unRAID server with a couple of SMB shares that should be backed up.
backup target BackupNAS Synology Diskstation 416, formerly used as main NAS/server, but to slow now

Set up ssh to allow passwordless access with keyfiles

The aim is to have this backup run automatically once the Diskstation starts. Rsync must thus be able to access the files to be backed up without manual interaction. This can be done with keyfile access in ssh and then let rsync use ssh to connect to the backup source.

Following e.g. this tutorial, it is trivial to set up keyfile authentification. The commands below need to be run on BackupNAS, e.g. via ssh.

Create a key pair on BackupNAS:

user@backupnas:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Created directory '/home/user/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/a/.ssh/id_rsa.
Your public key has been saved in /home/a/.ssh/id_rsa.pub.
The key fingerprint is:
12:34:56:78:90:ab:cd:ef:fe:dc:ba:09:87:65:43:21 user@backupnas

Create the .ssh directory in case it does not yet exist:

ssh user@homeserver mkdir -p .ssh

Allow user on backupnas to authentificate with his key when accessing homeser:

user@backupnas:~> cat .ssh/id_rsa.pub | ssh user@homeserver 'cat >> .ssh/authorized_keys'

Try out, now without having to enter the password:

user@backupnas:~> ssh user@homeserver

Enable sendmail on DSM

For an automated system, it's nice to get notifications after the backup has finished (successfully or unsuccessfully). This can be done on DSM, the Synology operating system by installing the "Mail Server" package from the Package Center.

In Mail Server, enable SMTP and SMTP relay.

SMTP settings relay settings
SMTP and SMTP relay settings in "Mail Server".

Now, the sendmail command is available on the command line:

user@backupnas:~> sendmail           \
-F "BackupNAS"                       \
-f "backupnas@provider.com"          \
-t "an-email@nprovider.com" << EOF
Subject: A test email
This is the body of the email.
EOF

backup with rsync

Rsync offers a huge amount of options. For my backup, the following are relevant.

-aP --archive, --partial, --progress. Archive the data, resume partial transfers after a failure, show sync progress.
--itemize-changes List updated files and what's happening to them. See the manual for what the update codes mean.
--prune-empty-dirs Delete empty directory trees without any files.
--delete Delete files in the backup directory that have been deleted from the source directory.
--link-dest="../old_backup" Do not copy over all files but first check if they are already present in this directory. It they are present set a hard link. Allows for incremental backups. Careful: the path to this previous backup to link to is relative to the current backup!
--exclude="cache" Exclude a directory or pattern from the sync, in this example the cache directory on the top level of the backup source
--log-file="some_filename.log" Write a log file for later inspection of the sync process
-e ssh Use ssh to access the subsequent directory

I usually run the first backup manually to check for errors, monitor the performance, and so on. The --link-dest option must not be given since there is no old backup to link to. Some options (--delete and some implicitely given by -a) have no effect in this case.

rsync -aP                                             \
      --itemize-changes                               \
      --prune-empty-dirs                              \
      --delete                                        \
      --exclude="cache"                               \
      --log-file="volume1/backup/my_first_backup.log" \
      -e ssh user@homeserver:/mnt/user/               \
      /volume1/backup/my_first_backup/

This will backup all shares of homeserver (unRAID lists the shares in /mnt/user/) except for the cache share to the folder backup on DSM volume1.

For a consistent organization of backups, it is useful to store them by date instead of some name ("my_first_backup"). The date function provides the current date and time in many formats. I plan to run this backup once a week, so labeling by date (without time) is enough.

today=`date +"%Y-%m-%d"`

The format I store my backups is simple with a directory per backup named as YYYY-MM-DD. This format can easily be parsed to automatically determine the previous backup to link to.

get_previous () {
    local previous=`ls /volume1/backup/*.log | tail -1`
    local previous=${previous#"/volume1/backup/"}
    local previous=${previous#*_}
    local previous=${previous%".log"}
    latest=$previous
    echo "Latest backup for $backupdir was on: $latest".
    echo "Will do a new one today: $today"
}

The final script is hard to read and redundant because the email notifications can be lengthy depending on what you want to send. These functions provide notify_start and notify_end as a wrapper to Synology's sendmail command.

send_status () {
    sendmail -F "BackupNAS" -f "email@provider.com" -t "email@provider.com" << EOF
    Subject: $1
    $2
    EOF
}

notify_start () {
    send_status "Starting backup" "Starting backup of homeserver now (`date +"%Y-%m-%d %T"`).

Backing up to $backupdir
Previous backup was on $latest and will be updated to $today."
}

notify_end () {
    send_status "Finished backup" "Finished backup of $backupdir now (`date +"%Y-%m-%d %T"`).

rsync return code: $?
Check tmux or sync log session for potential problems."
}

Full backup script

The final backup script then looks like this below. It has worked very well for me so far but I plan to expand it with some more options in the future. The BackupNAS can run this script automatically, e.g. upon startup or weekly with the Task Scheduler (see User-defined script).

To be able to monitor what is going on in real time, I let the script run in a tmux session. By ssh-ing into BackupNAS and attaching the session, it is possible to view and interact with the backup script. It can be particularly useful for debugging. Once the script is finished, the tmux session is also closed but all relevant information is stored in the log file created by rsync. The Synology Task Scheduler executes this code

tmux new-session -d -s 'backup' '/var/services/homes/myuser/rsync_homeserver.sh'

which in turn calls the backup script located in the homedirectory of "myuser".

The full script:


#####################
# RSYNC AUTO BACKUP #
#####################

# A simple bash script to backup homeserver in two stages (data + media) to BackupNAS.
# Email notifications are send out when the backup starts and when it finishes.


####################################################################################################
# INFO #
####################################################################################################

# rsync -aP                                                       \   # --archive, --partial, --progress
#     --itemize-changes                                           \   # list updated files and what's happening to them
#     --prune-empty-dirs                                          \   # delete empty directory trees
#     --delete                                                    \   # delete removed files
#     --link-dest="../${previous}"                                \
#     --exclude="movies"                                          \
#     --log-file="/volume2/homeserver/homeserver_${today}.log"    \
#     -e ssh root@homeserver:/mnt/user/                           \
#     /target/


####################################################################################################
# SETUP
####################################################################################################

get_previous () {
    local previous=`ls $backupdir/*.log | tail -1`
    local previous=${previous#"$backupdir/"}
    local previous=${previous#*_}
    local previous=${previous%".log"}
    latest=$previous
    echo "Latest backup for $backupdir was on: $latest".
    echo "Will do a new one today: $today"
}

send_status () {
    sendmail -F "BackupNAS" -f "email@provider.com" -t "email@provider.com" << EOF
Subject: $1
$2
EOF
}

notify_start () {
    send_status "Starting backup" "Starting backup of homeserver now (`date +"%Y-%m-%d %T"`).

Backing up to $backupdir
Previous backup was on $latest and will be updated to $today."
}

notify_end () {
    send_status "Finished backup" "Finished backup of $backupdir now (`date +"%Y-%m-%d %T"`).

rsync return code: $?
Check tmux or sync log session for potential problems."
}


####################################################################################################
# RUN
####################################################################################################

# homeserver data
####################################################################################################

today=`date +"%Y-%m-%d"`
backupdir="/volume1/data"
get_previous
notify_start

rsync -aP                                        \
    --itemize-changes                            \
    --prune-empty-dirs                           \
    --delete                                     \
    --link-dest="../${latest}"                   \
    --exclude="media"                            \
    --log-file="$backupdir/data_${today}.log"    \
    -e ssh user@homeserver:/mnt/user/            \
    $backupdir/${today}/

notify_end


# media
####################################################################################################

today=`date +"%Y-%m-%d"`
backupdir="/volume3/media"
get_previous
notify_start

rsync -aP                                        \
    --itemize-changes                            \
    --prune-empty-dirs                           \
    --delete                                     \
    --link-dest="../${latest}"                   \
    --include "media/***"                        \
    --exclude "*"                                \
    --log-file="$backupdir/media_${today}.log"   \
    -e ssh user@homeserver:/mnt/user/            \
    $backupdir/${today}/

notify_end


####################################################################################################
#
####################################################################################################