Borg backup server on TrueNAS SCALE

Technical howto

2023-09-17 6 min read Informatique
Screenshot of TrueNAS Applications, with my custom borg-server app running.

Introduction

Backups are an integral part of my organization system. See How I organize my home directory. I’ve been using Borg to protect my personal and work files for a very long time. I used to have a Borg server hosted as a FreeBSD Jail on TrueNAS CORE, but since I upgraded my file server to TrueNAS SCALE, which is based on Debian Linux, I had to migrate my borg server to a Docker container.

There is currently no Borg application available in TrueNAS Applications or in the popular 3rd party TrueCharts repository.

There is also no officially supported borg server Docker image.

In this document, I describe how I used an OpenSSH server Docker image to set up a Borg server on TrueNAS SCALE.

Table of Contents

Summary

Borg repositories are accessed through SSH and the borg serve command. The authorized_keys allows users to login via public key and restricts access to safe operations only.

The Docker image we’ll use is linuxserver/openssh-server. It’s “a sandboxed environment that allows ssh access without giving keys to the entire server.”

LinuxServer.io advantages:

  • Larger user base than the unofficial Borg server images created by the community.
  • All LinuxServer.io images provide standard image customization methods that don’t require forking the Dockerfile and maintaining a Docker build pipeline.

Documentation

Here is a list of links to external documentation relevant to this project.

Architecture

On the client side, I use Borgmatic to configure and automate backups. The borg archive backup task is started by a Systemd timer. SSH is used when borg connects to a remote repository.

On the server side, borg serve is started by the SSH server inside the custom Docker container when a connection from a client is established.

Create datasets

Create the following datasets:

  • <pool>/applications/borg/config
  • <pool>/applications/borg/custom-container-init-scripts
  • <pool>/backups/borg

The applications dataset

On my TrueNAS ZFS pool, I create an applications dataset to hold local custom applications data.

The applications/borg/* datasets

This one contains datasets for volume mounts on the borg server container.

  • <pool>/applications/borg/config mounted on /config.
  • <pool>/applications/borg/custom-container-init-scripts mounted on /custom-cont-init.d, read-only.

The backups dataset

On my TrueNAS ZFS pool, I create a backups dataset to hold backups for other systems. These backups are either flat filesystems updated with Rsync tasks, or borg backup repositories.

The backups dataset is replicated to an off-site TrueNAS server.

The backups/borg dataset

The dataset containing borg repositories.

<pool>/backups/borg is mounted on /var/local/borg on the borg server container.

Create a custom init script to modify the container at startup time

We need to install Borg and all of its dependencies inside the container. We achieve this the most easily by injecting an init script which installs Borg from the Alpine Linux package repository when the container starts.

Create the file <pool>/applications/borg/custom-container-init-scripts/install-borg.sh with this content:

echo "**** installing borgbackup ****"
apk add --no-cache borgbackup

Make sure the directory and script file are owned by root.

LinuxServer.io images can be easily customized by mounting a directory of custom scripts on /custom-cont-init.d inside the container.

This customization mechanism is described here:

In the background, LinuxServer images are all built with s6-overlay Which is a way of installing s6 in Docker images. s6 “is a collection of utilities revolving around process supervision and management, logging, and system initialization.” The /custom-cont-init.d directory is documented in LinuxServer’s custom base images:
https://github.com/linuxserver/docker-baseimage-alpine/blob/master/root/etc/s6-overlay/s6-rc.d/init-custom-files/run

Create a borg user on TrueNAS

In Credentials > Local users, create a user like this:

  • Full Name: borg
  • Username: borg
  • Disable password: yes
  • UID: leave the default value. For me, the default was 3000.
  • Home directory: /mnt/pool-01/backups/borg
  • Shell: nologin (default)

Configure SSH authorized_keys

LinuxServer images always store their persistent configs in /config, which is a directory inside the container that should be backed by a mounted persistent volume.

The openssh-server image supports a few environment variables which can be used to set the content of authorized_keys. I don’t really use those. I just edit the /config/.ssh/authorized_keys file directly.

Here is an example authorized_keys:

command="borg serve --restrict-to-path /var/local/borg/<user or machine>",restrict <key type> <key> <key host>

Refer to Borg: Hosting repositories for information on how to securely provide Borg repository storage, and to the sshd(8) man page for more details on SSH options.

Deploy a Docker container

linuxserver/openssh-serveris a sandboxed environment that allows ssh access without giving keys to the entire server.

In the TrueNAS Applications screen, launch a Docker image with the following settings:

  • Image: lscr.io/linuxserver/openssh-server
  • Tag: latest
  • Volumes:
    • Host path /mnt/pool-01/application/borg/config mounted on /config.
    • Host path /mnt/pool-01/application/borg/custom-container-init-scripts mounted on /custom-cont-init.d.
    • Host path /mnt/pool-01/backups/borg mounted on /var/local/borg.
  • Environnement:
    • PUID = 3000
    • PGID = 3000
    • USER_NAME = borg
    • LISTEN_PORT = 22 (default 2222)
  • Network: up to you. I like to create a network interface for the container and give it a static IPv4 address in my network.

For more information on these values, see the documentation:

Pod startup logs

These logs show that our container has started and applied our custom configurations and ran our custom script.

Initializing Borg repositories

Create a directory for the user or machine in the backups/borg dataset.

For example: sudo -u borg mkdir -p /mnt/pool-01/backups/borg/<machine>/<repository>

Add the client key to applications/borg/config/.ssh/authorized_keys with secure restrictions.

Pull the host keys from the server.

sudo ssh borg@<borg-server>

Initialize the repository. This will ask for a passphrase.

sudo borg init -e repokey borg@<borg-server>:<repository>

Save the repository key and the passphrase in your password manager. Export the repository key like this:

sudo borg key export borg@<borg-server>:<repository> borg.key

Alternatives considered

  • Here is a GitHub issue discussion on Docker images for borg:
    https://github.com/borgbackup/borg/issues/4364
  • AnotherStranger/docker-borg-backup
    • I tried it. Well designed.
    • One issue is that TrueNAS Docker containers environment variable values are limited to 1024 characters. When I tried to pass a list of SSH keys in the BORG_AUTHORIZED_KEYS environment variable, it failed because the string was longer than 1024 characters.
  • Setting up a local borg repo on the system being protected, and sync the repo to a TrueNAS ZFS dataset using an Rsync task.
    • Removes any dependency on borg running on TrueNAS.
    • Requires a bit more storage space on the protected system to hold the borg repository.
  • Using Restic
    • It doesn’t require special software on the backup server.
    • In this thread on r/selfhosted Reddit, Restic is mentioned often.
    • This thread on r/BorgBackup Reddit, there is a very detailed comparison of Restic and Borg.
  • Using a 3rd party hosting provider.
truenas backups
Alexandre de Verteuil
Alexandre de Verteuil
Senior Solutions Architect

I teach people how to see the matrix metrics.
Monkeys and sunsets make me happy.