Garuda Linux server configurations

General information

• Our current infrastructure is hosted in two of these.
• The servers are being backed up to Hetzner storage boxes via Borg, driven by borgmatic.
• After multiple different setups, we settled on NixOS as our main OS as it provides reproducible and atomically updated system states
• Cloudflare protects most (sub)domains while also making use of its caching feature. Exemptions are services such as our mail server and parts violating Cloudflares rules such as proxying Mastodon video content.
• Cloudflare Access in combination with Cloudflared is used to secure access to high-risk services such as admin panels.
• Monitoring and alerting are self-hosted, based on Prometheus, Grafana, Loki and Alertmanager (see Monitoring).

Quick links

• Common maintenance tasks
• Host: aerialis
• Host: stormwing

Devshell and how to enter it

This NixOS flake provides a devshell which contains all deployment tools as well as handy aliases for common tasks. The only requirement for using it is having the Nix package manager available. It can be installed on various distributions via the package manager or the following script (click me for more information):

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix -o nix-install.sh # Check its content afterwards
sh ./nix-install.sh install --diagnostic-endpoint=""

This installs the Nix packages with flakes already pre-enabled. After that, the shell can be invoked as follows:

nix develop # The intended way to use the devshell
nix-shell # Legacy, non-flakes way if flakes are not available for some reason

This also sets up pre-commit-hooks and shows the currently implemented tasks, which can be executed by running the command.

🔨 Welcome to Garuda's infra-nix shell ❄️

[[general commands]]

  clean           - Runs the garbage collection on the servers
  colmena         - Simple, stateless NixOS deployment tool
  deploy          - Builds and switches the servers to the local configuration (args go to colmena apply, e.g. --on aerialis)
  dnscontrol      - Synchronize your DNS to multiple providers from a simple DSL
  menu            - prints this menu
  prek            - Better `pre-commit`, re-engineered in Rust
  restart         - Restarts all physical servers
  sops            - Simple and flexible tool for managing secrets
  topology        - Renders the infrastructure diagrams (nix-topology) to docs/src/topology
  update          - Bumps flake.lock and deploys it to the servers for the next boot

[infra-nix]

  buildiso-local  - Spawns a local buildiso shell to build to ./buildiso (needs Docker)
  buildiso-remote - Spawns a buildiso shell on the iso-runner builder