Disaster recovery

This page covers restoring aerialis or stormwing from the offsite borg backups. For reinstalling the system disk of aerialis, follow the host rebuild steps first, then continue here.

Learnings from the past

• Have faith in the backups and start the restore immediately. Do not spend hours trying to rescue a broken disk or a half-working system. The backups are checked regularly, so restoring is the faster and safer path.
• Only borgmatic database dumps are restorable. Databases are restored from the pg_dumpall dump that borgmatic streams into every archive.
• Docker images and Docker state are not backed up. Images are pulled again and Docker stacks need a manual start after the restore.
• The system drive has no RAID yet. A single disk failure means a full restore. Adding RAID needs a reinstall and is tracked as an open task.

What is backed up

Both hosts back up to the Hetzner storage box u342919.your-storagebox.de (port 23), each into its own repository. Configuration: garuda.backup.borgmatic in nixos/hosts/<host>.nix, module nixos/services/backup.nix.

/data_1/persistent holds the SSH host keys (sops-nix decrypts with them), the Tailscale node state, machine-id, vnstat, the NixOS UID/GID map and the borgmatic check state. Left out are /home and the chaotic-v4 dir.

Monitoring of the backups

• borgmatic runs daily (borgmatic.timer) and runs repository and archives checks weekly and a full data check monthly.
• Every run pings healthchecks.io (start, finish, fail, with logs). healthchecks.io sends an email when a run fails or when no run arrives in time. This works even when the whole server is gone, because the email is sent from outside our infrastructure.
• Prometheus alerts (BorgmaticBackupMissed, BorgmaticBackupStale, …) are a second channel, but they run on aerialis and fail together with it.

Restore steps

1. Prepare credentials

From a workstation with sops access (repo root):

host=aerialis # or stormwing
sops -d --extract '["backup"]["ssh_'$host'"]' secrets/$host.yaml > ssh_key
sops -d --extract '["backup"]["repo_key"]' secrets/$host.yaml > repo_key
scp ssh_key repo_key root@<server>:/root/
rm ssh_key repo_key

2. Extract from borg

Run this from the rescue system or the freshly installed host, with the target filesystems mounted below /mnt.

mkdir -p /root/.ssh
echo '[u342919.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs' >> /root/.ssh/known_hosts
export BORG_RSH="ssh -i /root/ssh_key -o IdentitiesOnly=yes"
export BORG_PASSPHRASE="$(cat /root/repo_key)"
export BORG_REPO=ssh://[email protected]:23/./$host

borg list --last 3
ARCHIVE=<archive name from the list>

# Shows the archive size, so the duration can be estimated
borg info ::$ARCHIVE

# --progress shows the current file and percentage
# --numeric-ids keeps the UIDs/GIDs of the archive, the rescue system has different user names
# Only extract what was lost, e.g. data_1 after a system disk failure. Never overwrite an intact data_2.
cd /mnt && borg extract --progress --numeric-ids ::$ARCHIVE data_1

Borg paths have no leading slash, so always extract from /mnt. Run long extracts inside tmux or screen, so a dropped SSH connection does not abort them.

3. Check permissions

Borg restores owner and mode when it runs as root with --numeric-ids. Check the security-relevant paths anyway, a wrong mode on the host keys stops sshd and sops-nix:

chmod 700 /mnt/data_1/persistent/etc/ssh
chmod 600 /mnt/data_1/persistent/etc/ssh/ssh_host_*_key
chmod 644 /mnt/data_1/persistent/etc/ssh/ssh_host_*_key.pub
chown -R 0:0 /mnt/data_1/persistent/etc/ssh
stat -c '%a %U:%G %n' /mnt/data_1/persistent/etc/ssh/* /mnt/data_1/persistent/var/lib/tailscale

/etc itself is generated by NixOS. Only the parts in /data_1/persistent (host keys, machine-id) come from the backup. The NixOS UID/GID map in /data_1/persistent/var/lib/nixos must be restored before the first boot, so the restored files keep matching users.

Also ensure that /etc is not 700. This will cause dbus to fail, and logging in via SSH will not be possible. This has already costed us too much time once.

4. Install and boot

Install NixOS with nixos-install --flake, then boot into the system. sops-nix decrypts the secrets with the restored host key.

5. Restore the databases (aerialis)

The postgres container starts with an empty data directory, because the live data directory is not backed up. NixOS creates the databases and roles on start. Then load the dump:

# Stop everything that writes to the database
systemctl stop container@docker-proxied container@docker container@mastodon container@chaotic-backend

# Shows progress per database with --verbosity 1
borgmatic restore --archive latest --verbosity 1

systemctl start container@docker-proxied container@docker container@mastodon container@chaotic-backend

6. Start Docker stacks manually

Docker images and volumes are not part of the backup. On the first start, every compose stack pulls its images, which can hit the start limit of compose-runner-<name>.service. Start each stack manually and watch the pull:

nixos-container root-login docker
systemctl reset-failed
cd /var/garuda/compose-runner/docker && docker compose pull
systemctl restart compose-runner-docker

Repeat for docker-proxied, mastodon, chaotic-backend and forum (./launcher rebuild app in /var/discourse).

7. Restore Nextcloud AIO

The AIO master container volume nextcloud_aio_mastercontainer is part of the backup, so the AIO interface on port 8080 of the docker container keeps its configuration. If the public URL is not working, access it via SSH port forwarding:

ssh -L 8080:10.0.5.60:8080 aerialis.garudalinux.org -p666

It will then be available in https://localhost:8080. In the AIO interface:

1. Start the containers.
2. If the data is missing, use “Restore from backup” with the AIO borg backup at /var/garuda/backups/nextcloud (host path /data_2/backup/nextcloud-aio) and the AIO backup password shown in the interface before the incident.

8. Verify

• systemctl --failed on the host and in each container.
• tailscale status shows the old node name and IP.
• Grafana shows all targets up, healthchecks.io receives the next ping.
• Run systemctl start borgmatic once to confirm backups work again.

Open items

• RAID for the system drive. Needs a reinstall with a new disk layout.