immortalis (Hetzner dedicated)
General
This system utilizes a NixOS host which uses nixos-containers to build declarative systemd-nspawn machines for different purposes. To make the best use of the available resources, common directories are shared between containers. This includes /home (home-manager / NixOS configurations writing to home are generated by the host and disabled for the containers), Pacman and Chaotic cache, the /nix directory, and a few others. Further details can be found in the Nix expression of the host.
All directories containing important data were mapped to /data_1 and /data_2 to have them all in one place. The first mostly contains web services' files, the latter only builds related directories such as the Pacman cache.
The current line-up looks as follows:
nico@immortalis ~> machinectl
MACHINE CLASS SERVICE OS VERSION ADDRESSES
chaotic-kde container systemd-nspawn nixos 24.05 10.0.5.90
chaotic-v4 container systemd-nspawn nixos 24.05 10.0.5.140
docker container systemd-nspawn nixos 24.05 10.0.5.100
docker-proxied container systemd-nspawn nixos 24.05 10.0.5.110
forum container systemd-nspawn nixos 24.05 10.0.5.70
github-runner container systemd-nspawn nixos 24.05 10.0.5.130
iso-runner container systemd-nspawn nixos 24.05 10.0.5.40
lemmy container systemd-nspawn nixos 24.05 10.0.5.120
mastodon container systemd-nspawn nixos 24.05 10.0.5.80
postgres container systemd-nspawn nixos 24.05 10.0.5.50
temeraire container systemd-nspawn nixos 24.05 10.0.5.20
web-front container systemd-nspawn nixos 24.05 10.0.5.10
We are seeing:
- 1 ISO builder (
iso-runner) - 1 reverse proxy serving all the websites and services (
web-front) - 2 Docker dedicated nspawn containers (
docker&docker-proxied) - 4 Chaotic-AUR builders (
chaotic-kde,chaotic-v4,github-runner&temeraire) - 5 app dedicated containers (
forum,lemmy,mastodon&postgres)
Connecting to the server
After connecting to the host via ssh -p 666 [email protected], containers can generally be entered by running nixos-container login $containername, eg. nixos-container login web-front. Some containers may also be connected via SSH using the following ports:
- 22:
temeraire(needs to be 22 to allow pushing packages to the main Chaotic-AUR node via rsync) - 224:
forum - 225:
docker - 226:
chaotic-kde - 227:
iso-runner - 228:
web-front - 229:
postgres(access the database in127.0.0.1viassh -p 229 [email protected] -L 5432:127.0.0.1:5432) - 400:
chaotic-v4
Docker containers
Some services not packaged in NixOS or are easier to deploy this way are serviced via the Docker engine. This contains services like Piped, Whoogle, and Matrix. We use a custom NixOS module to deploy those with the rest of the system. Secrets are handled via our secret management which consists of a git submodule secret (private repo with ansible-vault encrypted files) and garuda-lib (see secrets section). Those contain a docker-compose directory in which the .env files for the docker-compose.yml are stored.
Chaotic-AUR / repository
Our repository leverages Chaotic-AUR's toolbox to provide the main node for the [chaotic-aur] repository as well as two more instances building the [garuda] and [chaotic-kde] repositories. Users of the chaotic_op group may build packages on the corresponding nixos-container via the chaotic command:
chaotic get $package # pull PKGBUILD
chaotic mkd $package # build package in the previously cloned directory
chaotic bump $package # increment pkgver of $package by 0.1 to allow a rebuild
chaotic rm $package # remove the package from the repository
Further information may be obtained by clicking chaotic seen above. The corresponding builders are:
[chaotic-aur]:temeraire[garuda]:repo[chaotic-kde]:chaotic-kde
Squid proxy
Squid is being installed on the host machine to proxy outgoing requests via random IPv6 addresses of the /64 subnet Hetzner provides for services that need it, eg. Piped, the Chaotic-AUR builders, and other services that are getting rate limited quickly. The process is not entirely automated, which means that we currently have a pool of IPv6 addresses active and need to switch them whenever those are getting rate-limited again.
Since we supplied an invalid IPv4 to force outgoing IPv6, the log files were somewhat cluttered by (expected) errors. Systemd-unit logging has been set to LogLevelMax=1 to un-clutter the journal and needs to be increased again if debugging needs to be done.
Backups
Backups are provided by daily Borg runs. Only the /data_1 directory is backed up (minus /data_1/{dockercache,dockerdata}) as the rest are either Nix-generated or build-related files that can easily recovered from another repository mirror. The corresponding systemd-unit is named borgbackup-job-backupToHetzner.
Tailscale / mesh network
While Tailscale was commonly used to connect multiple VMs before, this server only has it active on the host. However, we are leveraging Tailscale's subnet router feature to serve the 10.0.5.0/24 subnet via Tailscale, which means that other Tailscale clients may access the nixos-containers via their IP if tailscale up --accept-routes was used to set up the service.
Nix expression
{ garuda-lib
, pkgs
, config
, ...
}: {
imports = [
../modules
./immortalis/containers.nix
./immortalis/hardware-configuration.nix
];
# Increase /tmp & /run size to make better use of RAM
boot = {
kernelPackages = pkgs.linuxPackages_6_6;
loader.systemd-boot.enable = true;
runSize = "50%";
tmp = {
tmpfsSize = "95%";
useTmpfs = true;
};
};
# Network configuration with a bridge interface
networking = {
defaultGateway = "116.202.208.65";
defaultGateway6 = {
address = "fe80::1";
interface = "eth0";
};
hostName = "immortalis";
interfaces = {
"eth0" = {
ipv4.addresses = [
{
address = "116.202.208.112";
prefixLength = 26;
}
];
ipv6.addresses = [
# Random outgoing
{
address = "2a01:4f8:2200:30ac:cf2d:7d73:eddf:8871";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:5b38:dbde:e5a7:91b2";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:fa33:0d97:0755:6833";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:8f15:81f6:355c:d9d6";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:4436:e5e7:2236:0d77";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:1ea4:1794:1963:b8da";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:5628:7e9f:d8ec:544d";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:d830:ce99:e2b7:3e43";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:edc9:2d08:2b32:e532";
prefixLength = 64;
}
{
address = "2a01:4f8:2200:30ac:a833:0fd7:29d4:5309";
prefixLength = 64;
}
];
};
};
# Specify these here to allow containers to access
# our services from the internal network via NAT reflection
nat.forwardPorts = [
{
# web-front (HTTP)
destination = "10.0.5.10:80";
loopbackIPs = [ "116.202.208.112" ];
proto = "tcp";
sourcePort = 80;
}
{
# web-front (HTTPS)
destination = "10.0.5.10:443";
loopbackIPs = [ "116.202.208.112" ];
proto = "tcp";
sourcePort = 443;
}
{
# web-front (HTTPS)
destination = "10.0.5.10:443";
loopbackIPs = [ "116.202.208.112" ];
proto = "udp";
sourcePort = 443;
}
{
# web-front (Matrix)
destination = "10.0.5.10:8448";
loopbackIPs = [ "116.202.208.112" ];
proto = "tcp";
sourcePort = 8448;
}
{
# iso-runner (SSH)
destination = "10.0.5.40:22";
loopbackIPs = [ "116.202.208.112" ];
proto = "tcp";
sourcePort = 227;
}
{
# chaotic-v4 (SSH)
destination = "10.0.5.140:22";
loopbackIPs = [ "116.202.208.112" ];
proto = "tcp";
sourcePort = 400;
}
];
firewall.trustedInterfaces = [ "br0" ];
};
# OpenSSH on another port to keep Chaotic's main node working
services.openssh.ports = [ 666 ];
# Make use of all threads!
security.allowSimultaneousMultithreading = true;
# Raise limits to support many containers
# (from LXC's recommendedSysctlSettings)
boot.kernel.sysctl = {
"fs.inotify.max_user_instances" = 1048576;
"fs.inotify.max_user_watches" = 1048576;
"kernel.dmesg_restrict" = 1;
"kernel.keys.maxkeys" = 2000;
"kernel.pid_max" = 4194303;
"net.ipv4.neigh.default.gc_thresh3" = 8192;
"net.ipv6.neigh.default.gc_thresh3" = 8192;
};
# Improve nspawn container performance since we grant all capabilities anyway
# https://github.com/systemd/systemd/issues/18370#issuecomment-768645418
environment.variables.SYSTEMD_SECCOMP = "0";
# Custom tailscale configuration to advertise our bridge's subnet route
systemd.services.tailscale-autoconnect.script = with pkgs; ''
sleep 2
status="$(${tailscale}/bin/tailscale status -json | ${jq}/bin/jq -r .BackendState)"
if [ $status = "Running" ]; then
exit 0
fi
${tailscale}/bin/tailscale up --authkey ${garuda-lib.secrets.tailscale.authkey} \
--advertise-routes=10.0.5.0/24
'';
# We want to have same UID's in all containers to allow sharing home directories
garuda-lib.unifiedUID = true;
# Monitor a few services of the containers
services = {
netdata.configDir = {
"go.d/postgres.conf" = pkgs.writeText "postgres.conf" ''
jobs:
- name: postgres
dsn: 'postgres://netdata:[email protected]:5432/'
'';
"go.d/squidlog.conf" = pkgs.writeText "squidlog.conf" ''
jobs:
- name: squid
path: /var/log/squid/access.log
log_type: csv
csv_config:
format: '- resp_time client_address result_code resp_size req_method - - hierarchy mime_type'
'';
"go.d/web_log.conf" = pkgs.writeText "web_log.conf" ''
jobs:
- name: nginx
path: /var/log/nginx/access.log
'';
};
smartd = {
enable = true;
extraOptions = [ "-A /var/log/smartd/" "--interval=600" ];
};
};
# Fix permissions of nginx log files to allow Netdata to read it (gets reset frequently)
system.activationScripts.netdata = ''chown 60:netdata -R /var/log/nginx'';
# Backup configurations to Hetzner storage box
programs.ssh.macs = [ "hmac-sha2-512" ];
services.borgbackup.jobs = {
backupToHetzner = {
compression = "auto,zstd";
doInit = true;
encryption = {
mode = "repokey-blake2";
passCommand = "cat /var/garuda/secrets/backup/repo_key";
};
environment = {
BORG_RSH = "ssh -i /var/garuda/secrets/backup/ssh_immortalis -p 23";
};
exclude = [ "/data_1/dockercache" "/data_1/dockerdata" ];
paths = [ "/data_1" ];
prune.keep = {
within = "1d";
daily = 3;
weekly = 2;
monthly = 2;
};
repo = "[email protected]:./immortalis";
startAt = "daily";
};
};
# A proxy server making use of our IPv6 IP addresses
# traffic sent through the proxy is only allowing IPv6 connections
services.squid = {
enable = true;
extraConfig = ''
forwarded_for delete
dns_nameservers 2606:4700:4700::1111
acl tenth random 1/10
acl ninth random 1/9
acl eighth random 1/8
acl seventh random 1/7
acl sixth random 1/6
acl fifth random 1/5
acl fourth random 1/4
acl third random 1/3
acl half random 1/2
# Invalid IP
tcp_outgoing_address 10.254.254.254
tcp_outgoing_address 2a01:4f8:2200:30ac:cf2d:7d73:eddf:8871 tenth
tcp_outgoing_address 2a01:4f8:2200:30ac:5b38:dbde:e5a7:91b2 ninth
tcp_outgoing_address 2a01:4f8:2200:30ac:fa33:0d97:0755:6833 eighth
tcp_outgoing_address 2a01:4f8:2200:30ac:8f15:81f6:355c:d9d6 seventh
tcp_outgoing_address 2a01:4f8:2200:30ac:4436:e5e7:2236:0d77 sixth
tcp_outgoing_address 2a01:4f8:2200:30ac:1ea4:1794:1963:b8da fifth
tcp_outgoing_address 2a01:4f8:2200:30ac:5628:7e9f:d8ec:544d fourth
tcp_outgoing_address 2a01:4f8:2200:30ac:d830:ce99:e2b7:3e43 third
tcp_outgoing_address 2a01:4f8:2200:30ac:edc9:2d08:2b32:e532 half
tcp_outgoing_address 2a01:4f8:2200:30ac:a833:0fd7:29d4:5309
# Invalid IP
udp_outgoing_address 10.254.254.254
udp_outgoing_address 2a01:4f8:2200:30ac:cf2d:7d73:eddf:8871 tenth
udp_outgoing_address 2a01:4f8:2200:30ac:5b38:dbde:e5a7:91b2 ninth
udp_outgoing_address 2a01:4f8:2200:30ac:fa33:0d97:0755:6833 eighth
udp_outgoing_address 2a01:4f8:2200:30ac:8f15:81f6:355c:d9d6 seventh
udp_outgoing_address 2a01:4f8:2200:30ac:4436:e5e7:2236:0d77 sixth
udp_outgoing_address 2a01:4f8:2200:30ac:1ea4:1794:1963:b8da fifth
udp_outgoing_address 2a01:4f8:2200:30ac:5628:7e9f:d8ec:544d fourth
udp_outgoing_address 2a01:4f8:2200:30ac:d830:ce99:e2b7:3e43 third
udp_outgoing_address 2a01:4f8:2200:30ac:edc9:2d08:2b32:e532 half
udp_outgoing_address 2a01:4f8:2200:30ac:a833:0fd7:29d4:5309
# This does not rotate the logs, but asks squid to reopen the log file so that logrotate can rotate it
logfile_rotate 0
'';
proxyAddress = "10.0.5.1";
};
systemd.services.squid = {
serviceConfig = {
Restart = "always";
RestartSec = 10;
# Shut off all logging but level 1 errors as we get spamming a lot due to
# not being able to use our invalid address 10.254.254.254
LogLevelMax = 1;
};
startLimitIntervalSec = 80;
startLimitBurst = 6;
};
services.logrotate.settings.squid = {
files = "/var/log/squid/*.log";
frequency = "daily";
su = "squid squid";
rotate = 5;
compress = true;
delaycompress = true;
postrotate = "${config.systemd.package}/bin/systemctl kill --signal=SIGUSR1 squid";
};
# Can't really instantly remove this, need to find an alternative first
nixpkgs.config.permittedInsecurePackages = [ "squid-6.8" ];
# Adapt Nix to our core-count
nix.settings.max-jobs = 8;
system.stateVersion = "23.05";
}