Simple and Secure External Backup

March 6th, 2018 by Philip Iezzi 14 min read
cover image

What we are going to set up here is a simple and secure offsite and offline backup server. Let's assume you already have an existing backup server that is connected to the internet 24/7 and does daily/weekly/monthly backups. We would now like to set up a second offsite backup server that just cares about storing data to encrypted external drive and after each backup run, you are going to physically detach that drive.

So, we are talking about offline backups in addition to the fact having this server offsite - at a different location than your main backup server.

Preferably, your main backup server would also be offsite. But as it needs to pull data frequently, its storage is always available and not getting detached.

Let's call your main backup server backup and the one we are going to set up here extbackup.

Requirements

We have the following – from a security point of view – rather strict requirements:

  • Simple hardware setup: cheap but reliable fanless system that could be mounted to any office desk.
  • System: Standard Debian Linux
  • No backup data should ever be stored (not even temporarily!) on our root partition, only on external drives.
  • System is installed on SSD, external drives attached over USB 3.
  • External drives fully encrypted using standard encryption. My choice: LUKS
  • LUKS encryption key not stored anywhere on SSD, only generated at runtime.
  • Backup data is pulled from backup over SSH.
  • SSH private key to connect to backup is not stored anywhere on extbackup in plaintext, only getting generated at runtime.
  • Both LUKS encryption key and SSH private key should not be required to be stored in any password manager or anywhere else.
  • Unlock password for those keys should only be used as a 2nd factor, so encryption security does not depend on its length.
  • External disks should only get decrypted during a backup run and getting unmounted again right afterwards.
  • Backup data is directly streamed to external disks without temporary storage on extbackup server.

As this tutorial is about a simple yet secure external backup server, I am not going to talk about any snapshot based backup solutions (personally, I love zfs send|receive ...), but using rsync instead.

Hardware Setup

I do recommend the following hardware for a small robust fanless system that is built for 24/7 operation:

componentspec
SystemShuttle Barebone XPC slim DS77 Series (DS77U3)
RAMSO-DDR4-RAM 2133 MHz 1x 8GB
SSDSamsung SSD 960 EVO NVMe M.2 2280 250GB

This hardware currently (March 2018) is available for $600 in total. You can go much cheaper with some Intel NUC system that comes delivered with a basic RAM / SSD setup. But we do recommend that Shuttle XPC line as it is an industry-grade platform and we never had any issues with its predecessor DS57U.

As external 4TB USB 3.0 drive, I do recommend the following: Samsung P3 Portable 4TB (HX-MTD40EF/G2) – Choose any kind of external USB 3.0 drive but make sure it is USB-powered and does not need any extra power cable, so better go for 2.5".

System Setup

Install a standard Debian from netinst, e.g. via USB stick:

$ wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.3.0-amd64-netinst.iso
$ cat debian-9.3.0-amd64-netinst.iso > /dev/disk2

On macOS, use diskutil list to identify the correct device name of your USB stick and run diskutil unmountDisk /dev/disk2 prior to writing to it.

I am not going to explain how to set up Debian Linux here as I'm sure you are going to manage that by yourself.

Build encryption key

As promised, the LUKS encryption key is only going to be stored at runtime in ramfs (volatile memory aka. RAM). It is built upon first login as root. So let's hook below script into /root/.profile:

echo "/usr/local/sbin/build-encryption-key.sh" >> /root/.profile

Deploy the following script to /usr/local/sbin/build-encryption-key.sh:

build-encryption-key.sh
#!/bin/bash
#
# This script usually is called on the first login and asks for a password
# to build the LUKS encryption key which then is stored only in volatile
# memory: /mnt/ramfs/luks_pw (ramfs).
# This script is added to your /root/.profile in order you won't forget to
# build the encryption key each time you reboot the server.
#
# We are using ramfs instead of tmpfs as there is no swapping support in 
# ramfs which is good in a security perspective.
# see: http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux
#
# If ever you need to change the password of an LUKS encrypted volume:
#
#  $ cryptsetup luksChangeKey /dev/sdb1 /mnt/ramfs/luks_pw
#  Enter passphrase to be changed: (old password)
#


################### CONFIGURATION #######################
RAMFS_PATH=/mnt/ramfs
RAMFS_SIZE=20M
KEYFILE=$RAMFS_PATH/luks_pw
SALT='bo1AZ+pX1H%qgIiPDFy74.GB7GvJ6d/dqlV5%bjf861PV-evbC'
SHA1_CHECK='fb9e740efe20f541349d37eff7aa34efd4ac823d'
#########################################################

printinfo() {
    echo "[INFO] $1"
}

printwarn() {
    echo "[WARNING] $1" | grep --color "WARNING"
}

if [ -f "$KEYFILE" ]; then
    # exit silently, as the key file already exists
    exit
else
    printwarn "The LUKS key file ($KEYFILE) does not yet exist!"
fi

# set up RAM disk if it is not yet mounted
if ! mountpoint -q "$RAMFS_PATH"; then
    echo "Setting up ramfs on $RAMFS_PATH (size=$RAMFS_SIZE) ..."
    mkdir -p $RAMFS_PATH
    mount -t ramfs -o size=$RAMFS_SIZE ramfs $RAMFS_PATH
    if ! grep -q "$RAMFS_PATH" /etc/fstab; then
        echo "Adding line to /etc/fstab to persist mounting of $RAMFS_PATH ..."
        echo "ramfs   $RAMFS_PATH              ramfs   defaults,size=$RAMFS_SIZE        0 0" >> /etc/fstab
    fi
fi

# make this script start on each login
SCRIPT=$(readlink -f $0)
if ! grep -q "$SCRIPT" /root/.profile; then
    echo "$SCRIPT" >> /root/.profile
fi

# get password from interactive user input
while read -s -p 'Unlock LUKS encryption key: ' PASS && [[ $(echo -n "$PASS" | wc --chars) -lt 8 ]] ; do
    echo
    echo "Your password must be at least 8 characters long!"
done
echo

# calculate encryption key (SHA-512 hash of salt.password concatenation)
KEY=`echo -n "$SALT.$PASS" | sha512sum | cut -d' ' -f1`

# store LUKS key file to ramfs
touch $KEYFILE && chmod 600 $KEYFILE
echo -n "$KEY" > $KEYFILE

# SHA-1 check of the key - assure you have correctly built it by entering the correct password
KEY_SHA1=`cat $KEYFILE | sha1sum | cut -d' ' -f1`
if [ "$KEY_SHA1" != "$SHA1_CHECK" ]; then
    printwarn "Your key does not seem to be correct. You might have entered the wrong password. Please run $SCRIPT again!"
    printinfo "If you are sure you have entered the right password, try to set SHA1_CHECK='$KEY_SHA1' in `basename $0`."
    rm -f $KEYFILE
    exit 1
fi

printinfo "The LUKS key file was successfully stored in $KEYFILE."


# SSH private key encryption
#
# SETUP:
# Initially, we need to create an encrypted version of id_rsa and destroy the plaintext private key:
#   $ cat ~/.ssh/id_rsa | openssl enc -e -aes-256-cbc -a -k "$(cat /mnt/ramfs/luks_pw)" > ~/.ssh/id_rsa.encrypted
# or better just copy the password from /mnt/ramfs/luks_pw and enter it interactively:
#   $ cat ~/.ssh/id_rsa | openssl enc -e -aes-256-cbc -a > ~/.ssh/id_rsa.encrypted
#   $ enter aes-256-cbc encryption password: (...)
#   $ chmod 600 ~/.ssh/id_rsa.encrypted
#   $ shred -u ~/.ssh/id_rsa
#
if [ -f ~/.ssh/id_rsa.encrypted ]; then
    touch $RAMFS_PATH/id_rsa.decrypted && chmod 600 $RAMFS_PATH/id_rsa.decrypted
    cat ~/.ssh/id_rsa.encrypted | openssl base64 -d | openssl enc -d -aes-256-cbc -k "$(cat $KEYFILE)" > $RAMFS_PATH/id_rsa.decrypted
    ln -sf $RAMFS_PATH/id_rsa.decrypted ~/.ssh/id_rsa
    printinfo "SSH private key was successfully decrypted to $RAMFS_PATH/id_rsa.decrypted."
else
    printwarn "Please encrypt your SSH private key to ~/.ssh/id_rsa.encrypted so I can decrypt it to ramfs."
fi

But before using this script in production, ensure you have set your own SALT in the CONFIGURATION section. Don't yet care about SHA1_CHECK - this is only used to verify that the key got correctly built. Upon first run, the script will provide you with the right information (the unlock password can be freely chosen, just please remember it!):

$ build-encryption-key.sh
Unlock LUKS encryption key: 
[WARNING] Your key does not seem to be correct. You might have entered the wrong password. Please run build-encryption-key.sh again!
[INFO] If you are sure you have entered the right password, try to set SHA1_CHECK='fb9e740efe20f541349d37eff7aa34efd4ac823d' in build-encryption-key.sh.

The unlock password is only needed to unlock/generate your LUKS encryption key and is only used as a second factor. For LUKS encryption, we are then only going to use the generated /mnt/ramfs/luks_pw which you should not write down anywhere!

Set correct SHA1_CHECK in /usr/local/sbin/build-encryption-key.sh and test if the script gets correctly invoked upon first login as root:

extbackup$ sudo su -
[WARNING] The LUKS key file (/mnt/ramfs/luks_pw) does not yet exist!
Unlock LUKS encryption key: 
[INFO] The LUKS key file was successfully stored in /mnt/ramfs/luks_pw.
[INFO] SSH private key was successfully decrypted to /mnt/ramfs/id_rsa.decrypted.

LUKS encrypt USB drive

Repeat the following steps as initial setup for every USB disk you wish to use for your external backups.

Install cryptsetup if not already installed. Connect the external drive, leave it unmounted, and make note of the device label (e.g. /dev/sdb):

$ apt-get install cryptsetup
$ lsblk

Create a single partition using a favorite partitioning utility (fdisk, gparted,...) that fills the entire drive:

$ parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary ext3 0% 100%
(parted) quit

Encrypt the partition (using Cryptsetup encryption option defaults) and use our keyfile instead of interactively providing the password:

$ cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha256 --iter-time 2000 --key-file=/mnt/ramfs/luks_pw luksFormat /dev/sdb1
$ cryptsetup luksOpen --key-file=/mnt/ramfs/luks_pw /dev/sdb1 sdb1_crypt

Install ext4 filesystem and mount the partition to gain access to the storage:

$ mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/mapper/sdb1_crypt
$ mount -t ext4 /dev/mapper/sdb1_crypt /mnt/plaintext
$ echo -n "BACKUP_1" > /mnt/plaintext/LABEL

Before physically disconnecting the drive the partition must be unmounted and the encrypted device must be closed:

$ umount /mnt/plaintext
$ cryptsetup luksClose /dev/mapper/sdb1_crypt

LUKS mounting

Let's create /etc/profile.d/luks-encryption.sh to define the following aliases:

luks-encryption.sh
luksMount() {
    DEVNAME=${1:-sdb}
    mkdir -p /mnt/plaintext
    parted /dev/$DEVNAME print > /dev/null && sleep 5 && cryptsetup luksOpen --key-file=/mnt/ramfs/luks_pw /dev/${DEVNAME}1 ${DEVNAME}1_crypt && mount -t ext4 /dev/mapper/${DEVNAME}1_crypt /mnt/plaintext
}
alias mount-extbackup=luksMount

luksUmount() {
    DEVNAME=${1:-sdb}
    umount -l /mnt/plaintext && cryptsetup luksClose /dev/mapper/${DEVNAME}1_crypt
}
alias umount-extbackup=luksUmount

The parted command in luksMount() is only used to ensure our USB drive spins up in case it went to sleep.

We can now use those aliases to mount/unmount our external drive to /mnt/plaintext:

$ mount-extbackup
$ cat /mnt/plaintext/LABEL
$ umount-extbackup

Secure rsyncing over SSH

Encrypt SSH private key

You may have noticed that our build-encryption-key.sh script also tries to decrypt your SSH private key to ramfs upon first login as root. We are going to use our default identity file ~/.ssh/id_rsa for rsyncing over ssh. But we don't want to have it stored plaintext anywhere on persistent storage.

If you don't have any SSH public/private keypair yet, create it (ssh-keygen defaults to RSA 2048 bit, but we prefer 4096 bit), with an empty passphrase:

$ ssh-keygen -b 4096 -P ''

We are now going to encrypt our private key with OpenSSL, using the same encryption key as we use to encrypt our LUKS devices:

$ cd ~/.ssh/
$ cat id_rsa | openssl enc -e -aes-256-cbc -a -k "$(cat /mnt/ramfs/luks_pw)" > id_rsa.encrypted
$ chmod 600 id_rsa.encrypted

Let's now secure erase our plaintext private key:

$ shred -u ~/.ssh/id_rsa

To test decryption, remove LUKS key in ramfs and login again as root (which invokes build-encryption-key.sh):

$ rm -f /mnt/ramfs/*
$ exit
$ sudo su -
[WARNING] The LUKS key file (/mnt/ramfs/luks_pw) does not yet exist!
Unlock LUKS encryption key: 
[INFO] The LUKS key file was successfully stored in /mnt/ramfs/luks_pw.
[INFO] SSH private key was successfully decrypted to /mnt/ramfs/id_rsa.decrypted.

You should then also find /mnt/ramfs/id_rsa.decrypted correctly symlinked to ~/.ssh/id_rsa.

Limit rsync to a directory (rrsync)

extbackup needs full root permissions to pull data from backup server, but its access should be limited to rsync from a specific directory and not executing any other commands on backup.

There is a simple rsync wrapper for this which comes shipped with Debian's rsync package but first needs to be unpacked/installed:

backup$ zcat /usr/share/doc/rsync/scripts/rrsync.gz > /usr/local/bin/rrsync
backup$ chmod 0755 /usr/local/bin/rrsync
backup$ ln -s /usr/local/bin/rrsync /usr/bin/rrsync

We then add this to /etc/sudoers.d/extbackup on backup:

%backuppers ALL= NOPASSWD:SETENV: /usr/bin/rrsync

Let's now create a specific user extbackup and add him to group backuppers:

backup$ adduser extbackup
backup$ adduser extbackup backuppers

Add the public key of extbackup (found in extbackup:~/.ssh/id_rsa.pub) to backup:/home/extbackup/.ssh/authorized_keys, prefixed by the following options:

command="sudo -E /usr/bin/rrsync -ro /backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAAABxyz...

This limits extbackup to only pull data from backup:/backups/ directory over rsync. Test-run:

extbackup$ rsync -aH --numeric-ids --delete --delete-before extbackup@backup: /mnt/plaintext/backups

Explanation: extbackup's root user rsyncs data from backup:/backups/ to /mnt/plaintext (decrypted external USB drive) but only needs to provide the relative path to backup as rsyncing is enforced over rrsync to /backups/ directory on the remote side. If we want to pull all data from backup:/backups/, simply provide an empty relative path after the colon.

Isn't that cool? We can pull data with root rights from our backup server but are limited to one specific directory and are not able to execute any other commands except rsync.

Final backup script

The following script basically decrypts and mounts the external USB device, then rsyncs the whole data from backup:/backups/ to it and unmounts / closes the encrypted device right afterwards.

extbackup.sh
#!/bin/bash

######### CONFIGURATION ############
BKUP_USER=extbackup
BKUP_SERVER=backup.example.com
BKUP_SRCDIR=/backups
TSFORMAT="%Y-%m-%d %H:%M:%S"
RSYNC_CMD="/usr/bin/nice -n19 /usr/bin/ionice -c3 /usr/bin/rsync"
KEYFILE=/mnt/ramfs/luks_pw
USB_DEVICE=/dev/sdb
USB_PARTITION=/dev/sdb1
MAPPER=sdb1_crypt
MNT_PLAINTEXT=/mnt/plaintext
####################################

printinfo() {
    echo "[`date +"$TSFORMAT"`] $1"
}

printwarn() {
    echo "[`date +"$TSFORMAT"`] WARNING: $1" | grep --color "WARNING"
}

printerr() {
    >&2 echo "[`date +"$TSFORMAT"`] ERROR: $1"
}

errquit() {
    if [ ! -z "$1" ]; then
        >&2 echo "[`date +"$TSFORMAT"`] ERROR: $1"
    fi
    exit 1
}

# check if key file exists
if [ ! -f $KEYFILE ]; then
    errquit "The LUKS key file $KEYFILE does not exist yet. Please run build-encryption-key.sh first!"
fi

# create mountpoints if they don't exist yet
mkdir -p $MNT_PLAINTEXT

# First, make sure $USB_DEVICE will wake up prior to mounting $USB_PARTITION
# (using sfdisk as a simple workaround)
#sfdisk -d $USB_DEVICE > /dev/null
parted $USB_DEVICE print > /dev/null
sleep 5

# open luks device
cryptsetup luksOpen --key-file=$KEYFILE $USB_PARTITION $MAPPER

# abort on errors
if [ "$?" -ne "0" ]; then
  errquit "Could not open encrypted device $USB_PARTITION. Backup script aborted!!!"
fi

# mount mapped device
mount -t ext4 /dev/mapper/$MAPPER $MNT_PLAINTEXT

# abort, if $MNT_PLAINTEXT is already mounted
if [ "$?" -ne "0" ]; then
  errquit "Could not mount /dev/mapper/$MAPPER to $MNT_PLAINTEXT. Backup script aborted!!!"
fi

# abort on missing label
if [ ! -f $MNT_PLAINTEXT/LABEL ]; then
  errquit "Disk label ($MNT_PLAINTEXT/LABEL) does not exist. Backup script aborted!!!"
fi

# TEMP
cd $MNT_PLAINTEXT/
if [ -d backup ] && [ ! -h backup ]; then
    printwarn "removing legacy $MNT_PLAINTEXT/backup first ..."
    rm -rf backup
fi
if [ -d remotebackup ] && [ ! -e backup ]; then
    printinfo "Creating symlink $MNT_PLAINTEXT/backup -> $MNT_PLAINTEXT/remotebackup ..."
    ln -sf remotebackup backup
fi

printinfo "start syncing backups from $BKUP_SERVER:$BKUP_SRCDIR ..."
RSYNC_PARAMS=""
$RSYNC_CMD -aH --numeric-ids -e ssh --delete --delete-before $BKUP_USER@$BKUP_SERVER: $MNT_PLAINTEXT/backups

printinfo "DONE."
echo
echo

# print disk usage
DISK_LABEL=`cat $MNT_PLAINTEXT/LABEL`
echo "------------------------------"
echo "DISK LABEL: $DISK_LABEL"
echo "------------------------------"
df -h | grep /mnt/

# set modification timestamp for LABEL (as reference for last successful backup run)
touch $MNT_PLAINTEXT/LABEL

# make sure file system is idle
sync
sleep 5

# unmount mapped device
# Lazy  unmount. Detach the filesystem from the filesystem hierarchy now,
# and cleanup all references to the filesystem as soon as it is
# not busy anymore
umount -l $MNT_PLAINTEXT

# abort on errors
if [ "$?" -ne "0" ]; then
  echo "Could not unmount $MNT_PLAINTEXT. Backup script aborted!!!"
  echo "lsof output:"
  echo ""
  lsof
  exit 1
fi

# make sure file system is idle
sync
sleep 60

# close encrypted device
cryptsetup luksClose /dev/mapper/$MAPPER

# abort on errors
if [ "$?" -ne "0" ]; then
  echo "Could not close encrypted LUKS device /dev/mapper/$MAPPER!!!"
  echo "lsof output:"
  echo ""
  # http://askubuntu.com/questions/429612/device-mapper-remove-ioctl-on-luks-xxxx-failed-device-or-resource-busy
  # the following sed replacement won't work...
  # dmsetup ls | grep sdb1_crypt | sed 's/.*\((\d+):(\d+)\)/\1,\2/'
  lsof | grep 254,2
  exit 1
fi

Run this as a weekly cronjob, e.g. via /etc/crontab, starting Sat morning:

00 07   * * 6   root    extbackup.sh