immortalis (Hetzner dedicated)

General

This system utilizes a NixOS host which uses nixos-containers to build declarative systemd-nspawn machines for different purposes. To make the best use of the available resources, common directories are shared between containers. This includes /home (home-manager / NixOS configurations writing to home are generated by the host and disabled for the containers), Pacman and Chaotic cache, the /nix directory, and a few others. Further details can be found in the Nix expression of the host.

All directories containing important data were mapped to /data_1 and /data_2 to have them all in one place. The first mostly contains web services' files, the latter only builds related directories such as the Pacman cache.

The current line-up looks as follows:

nico@immortalis ~> machinectl
MACHINE        CLASS     SERVICE        OS    VERSION ADDRESSES
chaotic-v4     container systemd-nspawn nixos 24.11   10.0.5.140…
docker         container systemd-nspawn nixos 24.11   10.0.5.100…
docker-proxied container systemd-nspawn nixos 24.11   10.0.5.110…
forum          container systemd-nspawn nixos 24.11   10.0.5.70…
github-runner  container systemd-nspawn nixos 24.11   10.0.5.130…
iso-runner     container systemd-nspawn nixos 24.11   10.0.5.40…
mastodon       container systemd-nspawn nixos 24.11   10.0.5.80…
mongodb        container systemd-nspawn nixos 24.11   10.0.5.60…
postgres       container systemd-nspawn nixos 24.11   10.0.5.50…
web-front      container systemd-nspawn nixos 24.11   10.0.5.10…

We are seeing:

  • 1 ISO builder (iso-runner)
  • 1 reverse proxy serving all the websites and services (web-front)
  • 2 Docker dedicated nspawn containers (docker & docker-proxied)
  • 3 Chaotic-AUR builders ( chaotic-v4, github-runner )
  • 5 app dedicated containers (forum, mastodon, mongodb & postgres)

Connecting to the server

After connecting to the host via ssh -p 666 [email protected], containers can generally be entered by running nixos-container login $containername, eg. nixos-container login web-front. Some containers may also be connected via SSH using the following ports:

  • 224: forum
  • 225: docker
  • 227: iso-runner
  • 228: web-front
  • 229: postgres (access the database in 127.0.0.1 via ssh -p 229 [email protected] -L 5432:127.0.0.1:5432)
  • 210: chaotic-v4

Docker containers

Some services not packaged in NixOS or are easier to deploy this way are serviced via the Docker engine. This contains services like Piped, Whoogle, and Matrix. We use a custom NixOS module to deploy those with the rest of the system. Secrets are handled via our secret management which consists of a git submodule secret (private repo with ansible-vault encrypted files) and garuda-lib (see secrets section). Those contain a docker-compose directory in which the .env files for the docker-compose.yml are stored.

Squid proxy

Squid is being installed on the host machine to proxy outgoing requests via random IPv6 addresses of the /64 subnet Hetzner provides for services that need it, eg. Piped, the Chaotic-AUR builders, and other services that are getting rate limited quickly. The process is not entirely automated, which means that we currently have a pool of IPv6 addresses active and need to switch them whenever those are getting rate-limited again. Since we supplied an invalid IPv4 to force outgoing IPv6, the log files were somewhat cluttered by (expected) errors. Systemd-unit logging has been set to LogLevelMax=1 to un-clutter the journal and needs to be increased again if debugging needs to be done.

Backups

Backups are provided by daily Borg runs. Only the /data_1 directory is backed up ( minus /data_1/{dockercache,dockerdata}) as the rest are either Nix-generated or build-related files that can easily recovered from another repository mirror. The corresponding systemd-unit is named borgbackup-job-backupToHetzner.

Tailscale / mesh network

While Tailscale was commonly used to connect multiple VMs before, this server only has it active on the host. However, we are leveraging Tailscale's subnet router feature to serve the 10.0.5.0/24 subnet via Tailscale, which means that other Tailscale clients may access the nixos-containers via their IP if tailscale up --accept-routes was used to set up the service.

Known issues and their solution

System update gets stuck

For some reason, while running nixos-rebuild switch, the system update gets stuck and doesn't continue. In this case, re-running the command will fix the issue. It is unclear, what the exact reason is, yet.

Nix expression

{{#include ../../../nixos/hosts/immortalis.nix}}