Yunohost backup with restic

February 15, 2020

It was suggested that I create a package on the forum after I published this. It’s done!

What follows stays valid but it would be a lot simpler to use the package instead, which furthermore handles repositories check.

The app is still marked as experimental so to install it run: yunohost app install https://github.com/YunoHost-Apps/restic_ynh.git

To install it: yunohost app install restic or simply choose it from the list on the web interface.

Description

This procedure is adapted from this one. I decided to use restic instead of Borg because:

I tried the yunohost packaged borg but failed to make it work
I did not want to have to install a dependency on the target server, restic only needs a sftp server as target which qualifies to any gnu/linux server with openssh-server installed on.
restic has the same features as borg (at least all the ones that matter to me)
restic being written in go has no dependency, you copy the binary and that’s it, no packaging problem
I was used to restic
Why not? It makes an alternative to the existing methods. Everybody is free to choose their favorite one :)

The goal here is to backup my Yunohost instance daily and leverage restic’s deduplication and snapshots to keep multiple archives of my data and configuration.

Here we consider the following use case:

Backup method: myrestic
Server hosting the Yunohost instance to be backed up
- name mysourceserver
- address mysourceserver.mysourcedomain.tld
- private key /root/.ssh/id_rsa_restic
- public key /root/.ssh/id_rsa_restic.pub
Server to backup Yunohost to:
- Name mytargetserver
- address mytargetserver.mytargetdomain.tld
- sftp login user mytargetuser
- sftp port 2222
- directory hosting the backup repositories: /home/backup/mysourceserver.mysourcedomain.tld

Adapt those to your needs in the following steps.

You could also use a local directory on an external drive for example and benefit restic’s deduplication and snapshots features.

In that case you would not need any target server configuration other than creating the directory where the backups should be stored on your yunohost server.

You would not need source server ssh configuration either.

Target server configuration

On the server where you want to send your backups

Create a user mytargetuser and set its password. Refer to your distribution documentation for this but it should look like something like this:
```
# as root or using sudo
useradd -m mytargetuser
passwd mytargetuser
```

Create the directory where backups will be stored and make mytargetuser the owner

mkdir /home/backup/mysourceserver.mysourcedomain.tld
chown mytargetuser: /home/backup/mysourceserver.mysourcedomain.tld

Security could be hardened:

Disable ssh and shell login for mytargetuser
restrain mytargetuser to a sftp jail

Source server configuration

On the Yunohost server to be backed up (all this should be done with the root user):

Install restic
```
cd /tmp
wget https://github.com/restic/restic/releases/download/v0.9.6/restic_0.9.6_linux_amd64.bz2  (1)
bunzip2 restic_*.bz2 -c > /usr/local/bin/restic
chmod +x /usr/local/bin/restic
```
1 Update this with the latest release that can be found on the release page and select the one matching your architecture (e.g: linux_arm64 for a raspberry).

Any other installation method would do, check here

Configure SSH

Generate an SSH keypair

ssh-keygen -b 4096 -t rsa -f /root/.ssh/id_rsa_restic -q -N ""

Set ssh config file

cat <<EOCONF >> ~/.ssh/config
Host mytargetserver
  Hostname mytargetserver.mytargetdomain.tld
  Port 2222
  User mytargetuser
  IdentityFile /root/.ssh/id_rsa_restic
EOCONF

Copy ssh key to target server

ssh-copy-id -i /root/.ssh/id_rsa_restic mytargetserver

Check that everything works up to that point

Now that we have configured both our source and target server, we want to make sure we can actually make a restic backup from source to target.

On mysourceserver as root:

Try to initialize a repository on mytargetserver from mysourceserver

restic -r sftp:mytargetserver:/home/backup/mysourceserver.mysourcedomain.tld/test-repository init

This should ask for a password (use any one you like) and return a success.

Backup script

It’s time to create the script that will backup our system and applications using restic.

Create a file /etc/yunohost/hooks.d/backup_method/05-myrestic with this content:

#!/bin/bash

set -e

RESTIC_PASSWORD="mysupersecretpassword"  # change it but DO NOT loose this password, you can't recover it
RESTIC_REPOSITORY_BASE=sftp:mytargetserver:/home/backup/mysourceserver.mysourcedomain.tld
RESTIC_COMMAND=/usr/local/bin/restic

do_need_mount() {
    work_dir="$1"
    name="$2"
    repo="$3"
    size="$4"
    description="$5"
    export RESTIC_PASSWORD
    export RESTIC_REPOSITORY=${RESTIC_REPOSITORY_BASE}/$name

    # Try to list snapshot from repository, otherwise initialize repository
    $RESTIC_COMMAND list snapshots || $RESTIC_COMMAND init
}

do_backup() {

    work_dir="$1"
    name="$2"
    repo="$3"
    size="$4"
    description="$5"
    export RESTIC_PASSWORD
    export RESTIC_REPOSITORY=${RESTIC_REPOSITORY_BASE}/$name
    LOGFILE=/var/log/backup_restic.log
    ERRFILE=/var/log/backup_restic.err
    current_date=$(date +"%d_%m_%y_%H:%M")
    pushd $work_dir
    $RESTIC_COMMAND backup ./ >> $LOGFILE 2>> $ERRFILE
    return_code="$?"
    popd

    # cleanup old archives only if the backup succeeded
    if [ "$return_code" -eq "0" ];then
            $RESTIC_COMMAND forget --keep-daily 7 --keep-weekly 8 --keep-monthly 12 >> $LOGFILE 2>> $ERRFILE
    fi
}

work_dir=$2
name=$3

size=$5
description=$6

case "$1" in
  need_mount)
    do_need_mount $work_dir $name $repo $size $description
    ;;
  backup)
    do_backup $work_dir $name $repo $size $description
    ;;
  mount)
    do_need_mount $work_dir $name $repo $size $description
    ;;
  *)
    echo "hook called with unknown argument \`$1'" >&2
    exit 1
    ;;
esac

exit 0

Make it executable and set safer permissions as it contains sensitive information
```
chmod u=rwx,go= /etc/yunohost/hooks.d/backup_method/05-myrestic
```

Test it

rm -rf /tmp/test-backup/; mkdir /tmp/test-backup; yunohost backup create --system conf_ldap -n conf_ldap --methods myrestic --debug -r -o /tmp/test-backup; rm -rf /tmp/test-backup

Proceed to the next step only if this ran without errors.

Some warnings may be ignored as this one:

Fatal: unable to open config file: Lstat: file does not exist
Is there a repository at the following location?
sftp:mytargetserver:/home/backup/mysourceserver.mysourcedomain.tld/xxxx

this is generated when restic is trying to access the remote repository the first time.

There are still some changes left to be made in that script:

Make some use of all the parameters passed to the script or remove them
Tune cleanup policy, I kept the one used in the original borg script
Understand the use off need_mount and mount arguments, I kept them from the original script as I guess this is a required interface but I am not sure of how to use them and why
Add a restic check to make sure backups are consitent but maybe add it to a separate script planed weekly or monthly as this is resource and time consuming you may not want to do it daily. Check restic documentation about that

Scheduled script

Since we know our backup script works, we can schedule it to run daily. The first backup can take a long time to finish depending on the amount of data to be processed. The following backups should be a lot faster since changes will be incremental and take advantage of deduplication for snapshots.

Create a file /root/yunohost-99-backup with this content:

#!/bin/bash
LOCK_FILE=/tmp/yunohost-99-backup.lock
if [ -f "$LOCK_FILE" ];then
    echo "Backup already launched by process $(grep '.*' $LOCK_FILE), canceling this one" >&2
    exit 1
fi
echo $$ > "$LOCK_FILE"
if yunohost -v | grep "version: 2." > /dev/null; then
    ignore_apps="--ignore-apps"
    ignore_system="--ignore-system"
else
    ignore_apps=""
    ignore_system=""
fi
filter_hooks() {
    ls /usr/share/yunohost/hooks/backup/ /etc/yunohost/hooks.d/backup/ | grep "\-$1_" | cut -d"-" -f2 | uniq
}

# Backup system part conf
yunohost backup create $ignore_apps -n auto_conf --methods myrestic --system $(filter_hooks conf)

# Backup system data
yunohost backup create $ignore_apps -n auto_data --methods myrestic --system $(filter_hooks data)

# Backup all apps independently
for app in $(yunohost app list --installed -b | grep id: | cut -d: -f2); do
    backup_methods=$(yunohost app setting $app backup_methods)

    if [ -z "$backup_methods" ]; then
        backup_methods=myrestic
    fi

    if [ "$backup_methods" != "none" ]; then
        yunohost backup create $ignore_system -n auto_$app --methods $backup_methods --apps $app
    fi
done
rm "$LOCK_FILE"

Make it executable

chmod +x /etc/cron.daily/yunohost-99-backup

I made some changes to the original script:

Used a lock file to make sure multiple backups don’t run at the same time
I did not directly schedule this script, the reason is given in the note below

I noticed that the gitlab backup spawned a question and required an interaction (a "y" answer) for the backup to proceed. I looked into the yunohost backup --help with no luck. I tried some tricks using the yes command or a basic piped echo "y", nothing worked. Since I want a totally unattented backup script I had to use the expect program to answer this question for me. This has the advantage of being able to select which question I want to answer.

Note that this question is whether you are okay with taking a little more space temporarily to make the backup. This could be a problem in a low space environment.

Here’s how to do it

Install the expect program
```
apt install expect -y
```

Create a file /etc/cron.daily/yunohost-99-backup-answerbot with this content

#!/usr/bin/expect -f
set timeout -1
spawn /root/yunohost-99-backup
expect -re "Some files couldn't be prepared.*Do you agree?"
send -- "y\r"
expect eof

This will launch the backup script and automatically answer "y" when prompted.

Restore from backup

You may want or need to restore data from your backups someday.

See official restic documentation for details about the commands.

Here is a simple example for piwigo application restoration.

List existing snapshots

restic -r sftp:mytargetserver:/home/backup/mysourceserver.mysourcedomain.tld/auto_piwigo snapshots

# enter password for repository:
# repository xxxxxx opened successfully, password is correct
# created new cache in /home/mytargetuser/.cache/restic
# ID        Time                 Host                								Tags        Paths
# -------------------------------------------------------------------------------------------------------------------
# yyyyyyyy  2020-02-10 22:03:22  mysourceserver.mysourcedomain.tld              /home/yunohost.backup/tmp/auto_piwigo
# -------------------------------------------------------------------------------------------------------------------
# 1 snapshots

Restore files from snapshot yyyyyyyy

mkdir /tmp/restore
restic -r sftp:mytargetserver:/home/backup/mysourceserver.mysourcedomain.tld/auto_piwigo restore yyyyyyyy --target /tmp/restore

# enter password for repository:
# repository xxxxxx opened successfully, password is correct
# restoring <Snapshot yyyyyyyy of [/home/yunohost.backup/tmp/auto_piwigo] at 2020-02-10 22:03:22.602905984 +0100 CET by root@mysourceserver.mydomain.tld to /tmp/restore

Check that you have the files restored to temporary directory, choose what to restore and how according to yunohost or package documentation.

tree /tmp/restore/ -L 4
# /tmp/restore/
# ├── apps
# │   └── piwigo
# │       ├── backup
# │       │   ├── db.sql
# │       │   ├── etc
# │       │   ├── home
# │       │   └── var
# │       └── settings
# │           ├── conf
# │           ├── manifest.json
# │           ├── scripts
# │           ├── settings.yml
# │           └── status.json
# ├── backup.csv
# └── info.json
#
# 9 directories, 6 files