immortalis (Hetzner dedicated)
General
This system utilizes a NixOS host which uses nixos-containers to build
declarative systemd-nspawn
machines for different purposes. To make the best use of the available resources, common
directories are shared between containers. This includes /home
(home-manager / NixOS configurations writing to home
are generated by the host and disabled for the containers), Pacman and Chaotic cache, the /nix
directory, and a few
others. Further details can be found in
the Nix expression of
the host.
All directories containing important data were mapped to /data_1
and /data_2
to have them all in one place. The
first mostly contains web services' files, the latter only builds related directories such as the Pacman cache.
The current line-up looks as follows:
nico@immortalis ~> machinectl
MACHINE CLASS SERVICE OS VERSION ADDRESSES
chaotic-v4 container systemd-nspawn nixos 24.11 10.0.5.140…
docker container systemd-nspawn nixos 24.11 10.0.5.100…
docker-proxied container systemd-nspawn nixos 24.11 10.0.5.110…
forum container systemd-nspawn nixos 24.11 10.0.5.70…
github-runner container systemd-nspawn nixos 24.11 10.0.5.130…
iso-runner container systemd-nspawn nixos 24.11 10.0.5.40…
mastodon container systemd-nspawn nixos 24.11 10.0.5.80…
mongodb container systemd-nspawn nixos 24.11 10.0.5.60…
postgres container systemd-nspawn nixos 24.11 10.0.5.50…
web-front container systemd-nspawn nixos 24.11 10.0.5.10…
We are seeing:
- 1 ISO builder (
iso-runner
) - 1 reverse proxy serving all the websites and services (
web-front
) - 2 Docker dedicated nspawn containers (
docker
&docker-proxied
) - 3 Chaotic-AUR builders (
chaotic-v4
,github-runner
) - 5 app dedicated containers (
forum
,mastodon
,mongodb
&postgres
)
Connecting to the server
After connecting to the host via ssh -p 666 [email protected]
, containers can generally be entered by
running nixos-container login $containername
, eg. nixos-container login web-front
. Some containers may also be
connected via SSH using the following ports:
- 224:
forum
- 225:
docker
- 227:
iso-runner
- 228:
web-front
- 229:
postgres
(access the database in127.0.0.1
viassh -p 229 [email protected] -L 5432:127.0.0.1:5432
) - 210:
chaotic-v4
Docker containers
Some services not packaged in NixOS or are easier to deploy this way are serviced via the Docker engine. This contains
services like Piped, Whoogle, and Matrix. We use a
custom NixOS module
to deploy those with the rest of the system. Secrets are handled via our secret management which consists of a git
submodule secret
(private repo with ansible-vault
encrypted files) and garuda-lib
(see secrets section). Those
contain a docker-compose
directory in which the .env
files for the docker-compose.yml
are stored.
Squid proxy
Squid is being installed on the host machine to proxy outgoing requests via random IPv6 addresses of the /64 subnet
Hetzner provides for services that need it, eg. Piped, the Chaotic-AUR builders, and other services that are getting
rate limited quickly. The process is not entirely automated, which means that we currently have a pool of IPv6 addresses
active and need to switch them whenever those are getting rate-limited again.
Since we supplied an invalid IPv4 to force outgoing IPv6, the log files were somewhat cluttered by (expected) errors.
Systemd-unit logging has been set to LogLevelMax=1
to un-clutter the journal and needs to be increased again if
debugging needs to be done.
Backups
Backups are provided by daily Borg runs. Only the /data_1
directory is backed up (
minus /data_1/{dockercache,dockerdata}
) as the rest are either Nix-generated or build-related files that can easily
recovered from another repository mirror. The corresponding systemd-unit is named borgbackup-job-backupToHetzner
.
Tailscale / mesh network
While Tailscale was commonly used to connect multiple VMs before, this server only has it active on the host. However,
we are leveraging Tailscale's subnet router feature to serve the 10.0.5.0/24
subnet via Tailscale, which means that other Tailscale clients may access the nixos-containers
via their IP
if tailscale up --accept-routes
was used to set up the service.
Known issues and their solution
System update gets stuck
For some reason, while running nixos-rebuild switch
, the system update gets stuck and doesn't continue.
In this case, re-running the command will fix the issue.
It is unclear, what the exact reason is, yet.
Nix expression
{{#include ../../../nixos/hosts/immortalis.nix}}