Garuda Linux server configurations

General information

Our current infrastructure is hosted in two of these.
The servers are being backed up to Hetzner storage boxes via Borg, driven by borgmatic.
After multiple different setups, we settled on NixOS as our main OS as it provides reproducible and atomically updated system states
Cloudflare protects most (sub)domains while also making use of its caching feature. Exemptions are services such as our mail server and parts violating Cloudflares rules such as proxying Mastodon video content.
Cloudflare Access in combination with Cloudflared is used to secure access to high-risk services such as admin panels.
Monitoring and alerting are self-hosted, based on Prometheus, Grafana, Loki and Alertmanager (see Monitoring).

This NixOS flake provides a devshell which contains all deployment tools as well as handy aliases for common tasks. The only requirement for using it is having the Nix package manager available. It can be installed on various distributions via the package manager or the following script (click me for more information):

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix -o nix-install.sh # Check its content afterwards
sh ./nix-install.sh install --diagnostic-endpoint=""

This installs the Nix packages with flakes already pre-enabled. After that, the shell can be invoked as follows:

nix develop # The intended way to use the devshell
nix-shell # Legacy, non-flakes way if flakes are not available for some reason

This also sets up pre-commit-hooks and shows the currently implemented tasks, which can be executed by running the command.

🔨 Welcome to Garuda's infra-nix shell ❄️

[[general commands]]

  ansible-core    - Radically simple IT automation
  apply           - Applies the infra-nix configuration pushed to the servers
  clean           - Runs the garbage collection on the servers
  deploy          - Deploys the local NixOS configuration to the servers
  menu            - prints this menu
  prek            - Better `pre-commit`, re-engineered in Rust
  restart         - Restarts all physical servers
  rsync           - Fast incremental file transfer utility
  sops            - Simple and flexible tool for managing secrets
  update          - Performs a full system update on the servers bumping flake lock

[infra-nix]

  buildiso-local  - Spawns a local buildiso shell to build to ./buildiso (needs Docker)
  buildiso-remote - Spawns a buildiso shell on the iso-runner builder

General structure

A general overview of the folder structure can be found below:

├── ansible
│   ├── host_vars
│   │   ├── aerialis
│   │   └── stormwing
│   └── playbooks
├── assets
├── compose
│   ├── chaotic-backend
│   ├── chaotic-v4
│   ├── docker
│   │   └── configs
│   ├── docker-proxied
│   ├── firedragon-runner
│   ├── github-runner
│   ├── gitlab-runner
│   └── mastodon
├── docs
│   ├── src
│   │   ├── hosts
│   │   │   ├── aerialis
│   │   │   └── stormwing
│   │   ├── repositories
│   │   ├── services
│   │   ├── users
│   │   └── websites
├── home-manager
├── nixos
│   ├── hosts
│   │   ├── aerialis
│   │   └── stormwing
│   ├── modules
│   │   ├── special
│   │   └── static
│   └── services
│       ├── compose-runner
│       ├── monitoring
│       └── mk.nix # garuda-lib construction helpers
├── pkgs
├── scripts
└── secrets

Secrets in this repository

Secrets are managed via the sops-nix module, which allows us to encrypt sensitive files and supply them in an encrypted way to our hosts. They will then be decrypted at runtime by using the hosts ed25519 SSH host key. This is done by using the sops tool, which encrypts files using a key stored in the ~/.config/sops/ directory. The submodule is available in the secrets directory once it has been set up for the first time. It can be initialized by running:

git submodule init
git submodule update

To view or edit any of these files, one can use the following commands:

sops secrets/filename.yaml # opens editor for the file
sops -e secrets/filename.yaml # encrypts the file
sops -d secrets/filename.yaml # decrypts the file

This assumes a fitting sops key is available in the ~/.config/sops/ directory. It is important to keep the secrets directory in the latest state before deploying a new configuration as misconfigurations might happen otherwise.

Passwords in general

Our mission-critical passwords that maintainers and team members need to have access to are stored in our Bitwarden instance. After creating an account, maintainers need to be invited to the Garuda Linux organisation in order to access the stored credentials.

Linting and formatting

We utilize pre-commit-hooks to automatically set up the pre-commit-hook with all the tools once nix-shell or nix develop is run for the first time. Checks can then be executed by running one of the following configs:

nix flake check # checks flake outputs and runs pre-commit at the end
pre-commit run --all-files # only runs the pre-commit tools on all files

Its configuration can be found in the flake.nix file. (click me). At the time of writing, the following is being run:

The pre-commit hooks generated into .pre-commit-config.yaml:

check-json
check-yaml
detect-private-keys
ripsecrets

Formatting is handled by treefmt, which runs:

It is recommended to run pre-commit run --all-files before trying to commit changes. Then use cz commit to generate a commitizen complying commit message.

CI/CD

We have used pull-/push-based mirroring for this git repository, which allows easy access to Renovate without having to run a custom instance of it. The following tasks have been implemented as of now:

nix flake check runs for every labeled PR and commit on main.
Renovate periodically checks docker-compose.yml and other supported files for version updates. It has a dependency dashboard as well as the developer interface to check logs of individual runs. Minor updates appear as grouped PRs while major updates are separated from those. Note that this only applies to the GitHub side.
Deployment of our mdBook-based documentation to Cloudflare pages.
Deployment of our Website to Cloudflare pages.

Workflows will generally only be executed if a relevant file has been changed, eg. nix flake check won’t run if only the README was changed.

Monitoring

Our monitoring stack is self-hosted and defined in nixos/services/monitoring, replacing the previous Netdata setup. It consists of Prometheus for metrics, Loki for logs, Grafana for dashboards and Alertmanager for alerting, plus Fluent Bit and node_exporter agents that hosts and containers opt into via garuda.monitoring.

Because both hosts run their own 10.0.5.0/24 container bridge, the monitoring container cannot reach stormwing’s containers directly - stormwing proxies their exporters and relays their logs over the Tailnet instead. See Monitoring for details.

Where configuration lives

nixos/hosts/<host>.nix - host-specific networking, NAT forwards, the container list and host-level tunnels.
nixos/hosts/<host>/<container>.nix - one file per container.
nixos/services/ - shared service modules, plus mk.nix which holds the garuda-lib helpers used to build repetitive structures.

Common tasks

Common maintenance tasks

Rebuilding / updating the forum container

Sometimes Discourse needs its container to build rebuild via cli rather than the webinterface. This can be done with:

ssh -p 666 [email protected]
sudo nixos-container root-login forum
cd /var/discourse
./launcher rebuild app

Building ISO files

To build Garuda ISO, one needs to connect to the iso-runner container and execute the buildiso command, which opens a shell containing the needed environment:

ssh -p 220 [email protected] # if one ran nix develop before, this can be skipped
buildiso
buildiso -i # updates the iso-profiles repo
buildiso -p dr460nized

Further information on available commands can be found in the garuda-tools repository. After the build process is finished, builds can be found on iso.builds.garudalinux.org. No automatic pushing to Sourceforge and Cloudflare R2 happens by default, see below for more information on how to achieve this.

Deploying a new ISO release

We are assuming all ISOs have been tested for functionality before executing any of those commands.

ssh -p 220 [email protected]
buildall # builds all ISO provided in the buildall command
deployiso -FS # sync to Cloudflare R2 and Sourceforge
deployiso -FSR # sync to Cloudflare R2 and Sourceforge while also updating the latest (stable, non-nightly) release
deployiso -Sd # to delete the old ISOs on Sourceforge once they aren't needed anymore
deployiso -FSRd # oneliner for the above-given commands

Updating the system

One needs to have the infra-nix repo cloned locally. Then proceed by updating the flake.lock file, pushing it to the server & building the configurations:

nix flake update
ansible-playbook garuda.yml -l $servername # Eg. aerialis
deploy # Skip using the above command and use this one in case nix develop was used

Then you can either apply it via Ansible or connect to the host to view more details about the process while it runs:

ansible-playbook apply.yml -l $servername # Ansible

apply # Nix develop shell

ssh -p 666 [email protected]
sudo nixos-rebuild switch

Keep in mind that this will restart every service whose files changed since the last system update. On our Hetzner server, this includes a restart of every declarative nixos-container if needed, causing a small downtime.

Changing system configurations

Most system configurations are contained in individual Nix files in the nix directory of this repo. This means changing anything must not be done manually but by editing the corresponding file and pushing/applying the configuration afterward.

ansible-playbook garuda.yml -l $servername # Eg. aerialis
deploy # In case nix develop is used

As with the system update, one can either apply via Ansible or manually:

ansible-playbook apply.yml -l $servername # Ansible

apply # Nix develop shell

ssh -p 666 [email protected]
sudo nixos-rebuild switch

Adding a user

Adding users needs to be done in users.nix:

Add a new user here
Add the SSH public key to flake inputs
Add the specialArgs keys.user as seen here
Deploy & apply the configuration

Changing Docker configurations

If configurations of services running in Docker containers need to be altered, one needs to edit the corresponding compose.yml (./compose/$name) file or .env entry of our sops file in the secrets directory (see the secrets section for details on that topic). The deployment is done the same way as with normal system configuration.

Updating Docker containers

Docker containers sometimes use the latest tag in case no current tag is available or in the case of services like Piped and Searx, where it is often crucial to have the latest build to bypass Google’s restrictions. Containers using the latest tag are automatically updated via watchtower daily. The remaining ones can be updated by changing their version in the corresponding compose.yml and then running deploy & apply. If containers are to be updated manually, this can be achieved by connecting to the host, running nixos-container root-login $containername, and executing:

cd /var/garuda/compose-runner/$name/ # replace $name with the actual docker-compose.yml or autocomplete via tab
sudo docker compose pull
sudo docker compose up -d

The updated containers will be pulled and automatically recreated using the new images.

Checking whether backups were successful

Backups run through borgmatic, which is configured per host with garuda.backup.borgmatic (see nixos/services/backup.nix). To check whether backups to Hetzner are still working as expected, connect to the server and execute the following:

systemctl status borgmatic.service
journalctl -u borgmatic

This should yield a successful unit state. The only exception is having an exit code != 0 due to files having changed during the run. The “Borg backup age” and “Borg backup count” panels in Grafana are a quicker way to spot a stalled backup.

Checking monitoring and alerts

The monitoring stack can be reached at:

grafana.garudalinux.net for dashboards
prometheus.garudalinux.net for queries and scrape targets
alertmanager.garudalinux.net to review and silence alerts

These are all secured by Cloudflare Zerotrust.

Alerts are additionally delivered to any configured Target. Alert rules live in nixos/services/monitoring/prometheus-rules/ and dashboards are committed as JSON in nixos/services/monitoring/dashboards/. See Monitoring for details.

Updating Chaotic-AUR toolbox

This needs to be done by updating the flake input (git repo URL of the website) src-chaotic-toolbox:

cd nix
nix flake lock --update-input src-chaotic-toolbox # toolbox

After that deploy as usual by running deploy and apply. The commit and corresponding hash will be updated and NixOS will use it to build the toolbox using the new revision automatically.

Important links

This is a collection of important links when working with the infrastructure:

Most important

The infrastructure-nix repository

Tools documentation

Web interfaces

Services to be administrated

Users

Multiple kinds of users can make use of our infrastructure. A current list of users is available here.

Adding new users

New users can be added by supplying a fitting configuration in the users.nix module. In case of a password being required, its hash needs to be generated as follows:

nix-shell -p mkpasswd --run 'mkpasswd -sm bcrypt' > /path/to/hashedPasswordFile

The file then needs to be added to our sops file and committed to our secrets repository. This one is only available to members of our GitLab org and usually is cloned as git submodule to ./secrets.

Onboarding a new admin

After confirming the trustworthiness of a new admin, the following actions need to be executed:

Add them to the admin users
Add their ssh public key to the flake inputs and specialArgs
Make them an owner of the GitLab organization
Add them to our Vaultwarden organization to allow access to passwords and email accounts
Add them to the Cloudflare Account
Make them an admin of Discourse

Users

These are the people who are currently allowed to use our servers.

Admins

Admins have root access to all servers and may therefore change everything. They are responsible for the well-being of the infrastructure and its development.

    users.nico = {
      extraGroups = [
        "wheel"
        "docker"
        "chaotic_op"
      ];
      home = "/home/nico";
      isNormalUser = true;
      openssh.authorizedKeys.keyFiles = [ keys.nico ];
      hashedPasswordFile = config.sops.secrets."passwords/nico".path;
      uid = lib.mkIf garuda-lib.unifiedUID 1001;
    };
    users.sgs = {
      extraGroups = [ "wheel" ];
      home = "/home/sgs";
      isNormalUser = true;
      openssh.authorizedKeys.keys = [
        "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxBY8TX0iEQkf3Bym+3XVlrk8OLOwHOrj7Uy+WxjncOkkutyZ1WsY9liF4j9yjptyQG7Lx8OM8q44NE6+Rk1OXJXMF7CZ4Jq/WvMVnh2zKyNnF8wHBcspsAdG90wCxo6OmNpnY/rRRlNwwnore7raF2PrERtSlsEvLsUgvspYQ8cnLwerJP43QeETlpE1oR0FrbXWQet0I63Ky6UDEp07x0yee21VHnAG74rjGeFGwJBmCPSxnfGVNhCaR0zyu9+hh222liBrlilYm8nqLlsYGZCXiVdOxXJbBy89EVpHds7Lutf+TAYwsPGZf7U4k+g2Jx8N0JHXyzVZa0zS+I48+tqBBflEOqU9oEfGuz4cU/qWys5soLcRX2p9td+RF3OEdBKlTW4UYsINJUri6QSEUrsGaXqQZy8Ds2FBdUpb4pmFVlo9+4qRouiI80a5xVa7a1E5eS5xK5BzWH4fNg5SqtT5L9i2i1ocZp7FA0oa+ixnXNiC1umPZaY/9s+5fh1s= [email protected]" # pragma: allowlist secret,
      ];
      hashedPasswordFile = config.sops.secrets."passwords/sgs".path;
      uid = lib.mkIf garuda-lib.unifiedUID 1002;
    };
    users.tne = {
      extraGroups = [
        "wheel"
        "docker"
        "chaotic_op"
      ];
      home = "/home/tne";
      isNormalUser = true;
      openssh.authorizedKeys.keyFiles = [ keys.tne ];
      hashedPasswordFile = config.sops.secrets."passwords/tne".path;
      uid = lib.mkIf garuda-lib.unifiedUID 1003;
    };

Maintainers

Maintainers have restricted access, which allows them to use buildiso to build new ISO files via the iso-runner container.

    users.frank = {
      home = "/home/frank";
      isNormalUser = true;
      openssh.authorizedKeys.keyFiles = lib.mkIf config.services.garuda-iso.enable [ keys.frank ];
      shell = lib.mkIf (!config.services.garuda-iso.enable) "${pkgs.util-linux}/bin/nologin";
      uid = lib.mkIf garuda-lib.unifiedUID 1007;
    };

Chaotic-AUR maintainers

Chaotic-AUR maintainers have access to the builder containers of our infrastructure. They may operate the repository by doing all kinds of packaging-related tasks such as adding or removing those.

    users.technetium = {
      extraGroups = lib.mkIf garuda-lib.chaoticUsers [ "chaotic_op" ];
      home = "/home/technetium";
      isNormalUser = true;
      openssh.authorizedKeys.keyFiles = lib.mkIf garuda-lib.chaoticUsers [ keys.technetium1 ];
      shell = lib.mkIf (!garuda-lib.chaoticUsers) "${pkgs.util-linux}/bin/nologin";
      uid = lib.mkIf garuda-lib.unifiedUID 1004;
    };
    users.alexjp = {
      extraGroups = lib.mkIf garuda-lib.chaoticUsers [ "chaotic_op" ];
      home = "/home/alexjp";
      isNormalUser = true;
      openssh.authorizedKeys.keyFiles = lib.mkIf garuda-lib.chaoticUsers [ keys.alexjp ];
      shell = lib.mkIf (!garuda-lib.chaoticUsers) "${pkgs.util-linux}/bin/nologin";
      uid = lib.mkIf garuda-lib.unifiedUID 1005;
    };
    users.xiota = {
      extraGroups = lib.mkIf garuda-lib.chaoticUsers [ "chaotic_op" ];
      home = "/home/xiota";
      isNormalUser = true;
      openssh.authorizedKeys.keyFiles = lib.mkIf garuda-lib.chaoticUsers [ keys.xiota ];
      shell = lib.mkIf (!garuda-lib.chaoticUsers) "${pkgs.util-linux}/bin/nologin";
      uid = lib.mkIf garuda-lib.unifiedUID 1006;
    };

aerialis

This is one of the two main infrastructure hosts (see also: stormwing). All services and containers for aerialis are defined in nixos/hosts/aerialis.nix and its submodules.

Host configuration

{
  garuda-lib,
  ...
}:
let
  mon = garuda-lib.monitoring;
in
{
  imports = [
    ../modules
    ./../modules/special/hetzner-ex44.nix
  ];

  garuda =
    garuda-lib.mkMonitoring {
      host = "aerialis";
      hostInfo = true;
      units = [
        "borgmatic.service"
        "borgmatic.timer"
        "sshd.service"
        "tailscaled.service"
      ];
      exporters = [
        "smartctlExporter"
        "borgmaticExporter"
      ];
    }
    // {
      backup.borgmatic = {
        enable = true;
        name = "aerialis";
        user = "u342919";
        host = "u342919.your-storagebox.de";
        port = 23;
        label = "hetzner";
        sshKeySecret = "backup/ssh_aerialis";
        knownHosts = [
          {
            name = "storagebox-ed25519";
            publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs"; # pragma: allowlist secret
          }
          {
            name = "storagebox-rsa";
            publicKey = "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5EB5p/5Hp3hGW1oHok+PIOH9Pbn7cnUiGmUEBrCVjnAw+HrKyN8bYVV0dIGllswYXwkG/+bgiBlE6IVIBAq+JwVWu1Sss3KarHY3OvFJUXZoZyRRg/Gc/+LRCE7lyKpwWQ70dbelGRyyJFH36eNv6ySXoUYtGkwlU5IVaHPApOxe4LHPZa/qhSRbPo2hwoh0orCtgejRebNtW5nlx00DNFgsvn8Svz2cIYLxsPVzKgUxs8Zxsxgn+Q/UvR7uq4AbAhyBMLxv7DjJ1pc7PJocuTno2Rw9uMZi1gkjbnmiOh6TTXIEWbnroyIhwc8555uto9melEUmWNQ+C+PwAK+MPw=="; # pragma: allowlist secret
          }
          {
            name = "storagebox-ecdsa";
            publicKey = "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAGK0po6usux4Qv2d8zKZN1dDvbWjxKkGsx7XwFdSUCnF19Q8psHEUWR7C/LtSQ5crU/g+tQVRBtSgoUcE8T+FWp5wBxKvWG2X9gD+s9/4zRmDeSJR77W6gSA/+hpOZoSE+4KgNdnbYSNtbZH/dN74EG7GLb/gcIpbUUzPNXpfKl7mQitw=="; # pragma: allowlist secret
          }
        ];
        sourceDirectories = [
          "/data_1/containers"
          "/data_1/persistent/etc/ssh"
          "/data_2/backup/nextcloud-aio/"
          "/data_2/containers/chaotic-backend/chaotic/database"
          "/data_2/containers/mastodon"
          "/data_2/containers/postgres"
        ];
        excludePatterns = [
          "/data_1/dockercache"
          "/data_1/dockerdata"
        ];
      };
    };

  networking = {
    defaultGateway = "157.180.57.65";
    defaultGateway6 = {
      address = "fe80::1";
      interface = "eth0";
    };
    hostName = "aerialis";
    interfaces."eth0".ipv4.addresses = [
      {
        address = "157.180.57.100";
        prefixLength = 26;
      }
    ];
    nat.forwardPorts = [
      (garuda-lib.mkNatForward {
        sourcePort = 80;
        destination = "10.0.5.10:80";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 443;
        destination = "10.0.5.10:443";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 443;
        destination = "10.0.5.10:443";
        proto = "udp";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 465;
        destination = "10.0.5.80:465";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 993;
        destination = "10.0.5.80:993";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 25;
        destination = "10.0.5.80:25";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 4190;
        destination = "10.0.5.80:4190";
      })
    ];
  };

  # Can't set this inside the containers
  boot.kernel.sysctl."vm.overcommit_memory" = "1";

  services.garuda-nspawn = {
    dockerCache = "/data_1/dockercache/";

    containers = garuda-lib.mkContainers {
      dir = ./aerialis;
      ips = mon.aerialisContainers;
      containers = {
        chaotic-backend = {
          mounts = [
            {
              name = "chaotic";
              hostPath = "/data_2/containers/chaotic-backend/chaotic";
              mountPoint = "/var/garuda/compose-runner/chaotic-backend";
            }
            {
              name = "chaotic-redis";
              hostPath = "/data_2/containers/chaotic-backend/redis";
              mountPoint = "/var/lib/redis-chaotic";
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 270;
            }
          ];
          nspawn = {
            enableTun = true;
          };
          needsDocker = true;
        };

        docker = {
          mounts = [
            {
              name = "compose";
              hostPath = "/data_1/containers/docker/";
              mountPoint = "/var/garuda/compose-runner/docker";
            }
            {
              name = "nextcloud-local-backup";
              hostPath = "/data_2/backup/nextcloud-aio";
              mountPoint = "/var/garuda/backups/nextcloud";
            }
          ];
          needsDocker = true;
        };

        docker-proxied = {
          mounts = [
            {
              name = "compose";
              hostPath = "/data_1/containers/docker-proxied/";
              mountPoint = "/var/garuda/compose-runner/docker-proxied";
            }
          ];
          needsDocker = true;
        };

        forum = {
          mounts = [
            {
              name = "forum";
              hostPath = "/data_1/containers/forum/";
              mountPoint = "/var/discourse";
            }
          ];
          needsDocker = true;
        };

        mastodon = {
          mounts = [
            {
              name = "mastodon";
              hostPath = "/data_2/containers/mastodon/mastodon";
              mountPoint = "/var/lib/mastodon";
            }
            {
              name = "compose";
              hostPath = "/data_1/containers/mastodon/compose";
              mountPoint = "/var/garuda/compose-runner/mastodon";
            }
          ];
          needsDocker = true;
        };

        mail = {
          mounts = [
            {
              name = "acme";
              hostPath = "/data_2/containers/web-front/acme";
              mountPoint = "/var/lib/acme";
            }
            {
              name = "dkim";
              hostPath = "/data_1/containers/mail/dkim";
              mountPoint = "/var/dkim";
            }
            {
              name = "index";
              hostPath = "/data_1/containers/mail/index";
              mountPoint = "/var/lib/dovecot/indices";
            }
            {
              name = "rspamd";
              hostPath = "/data_1/containers/mail/rspamd";
              mountPoint = "/var/lib/redis-rspamd";
            }
            {
              name = "vmail";
              hostPath = "/data_1/containers/mail/vmail";
              mountPoint = "/var/vmail";
            }
          ];
        };

        postgres = {
          mounts = [
            {
              name = "data";
              hostPath = "/data_1/containers/postgres/data";
              mountPoint = "/var/lib/postgresql";
            }
            {
              name = "postgres_backup";
              hostPath = "/data_2/containers/postgres/backup";
              mountPoint = "/var/garuda/backups/postgres";
            }
            {
              name = "acme";
              hostPath = "/data_2/containers/web-front/acme";
              mountPoint = "/var/lib/acme";
              readOnly = true;
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 220;
            }
            { containerPort = 5432; }
          ];
        };

        web-front = {
          mounts = [
            {
              name = "acme";
              hostPath = "/data_2/containers/web-front/acme";
              mountPoint = "/var/lib/acme";
            }
            {
              name = "nginx";
              hostPath = "/data_2/containers/web-front/nginx";
              mountPoint = "/var/log/nginx";
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 210;
            }
          ];
        };

        n8n = {
          mounts = [
            {
              name = "n8n";
              hostPath = "/data_1/containers/n8n/var-lib";
              mountPoint = "/var/lib";
            }
          ];
        };

        monitoring = {
          mounts = [
            {
              name = "alertmanager";
              hostPath = "/data_1/containers/monitoring/alertmanager";
              mountPoint = "/var/lib/private/alertmanager";
            }
            {
              name = "prometheus";
              hostPath = "/data_1/containers/monitoring/prometheus";
              mountPoint = "/var/lib/prometheus2";
            }
            {
              name = "grafana";
              hostPath = "/data_1/containers/monitoring/grafana";
              mountPoint = "/var/lib/grafana";
            }
            {
              name = "loki";
              hostPath = "/data_1/containers/monitoring/loki";
              mountPoint = "/var/lib/loki";
            }
            {
              name = "tailscale";
              hostPath = "/data_1/containers/monitoring/tailscale";
              mountPoint = "/var/lib/tailscale";
            }
          ];
          nspawn = {
            enableTun = true;
          };
        };
      };
    };
  };

  systemd.services =
    garuda-lib.mkContainerOrdering {
      root = "postgres";
      containers = [
        "docker-proxied"
        "docker"
        "mastodon"
        "chaotic-backend"
      ];
    }
    // {
      loki-tailnet-proxy = garuda-lib.mkTunnel {
        bind = mon.tailnetIPs.aerialis;
        listen = mon.ports.loki;
        target = "${mon.loki.containerAddress}:${toString mon.ports.loki}";
        after = [
          "tailscaled.service"
          "[email protected]"
        ];
        wants = [ "[email protected]" ];
        description = "Expose Loki to Tailnet only";
      };
    };

  networking.firewall.interfaces."${mon.bridge.interface}".allowedTCPPorts = [
    mon.ports.nodeExporter
    mon.ports.smartctlExporter
    mon.ports.borgmaticExporter
  ];
}

Shared host configuration

The disk layout, SSH configuration, container bridge (br0) and impermanence are shared with stormwing and live in nixos/modules/special/newgen.nix. Hardware-specific settings (kernel, microcode, SMT, smartd) come from nixos/modules/special/hetzner-ex44.nix.

_: {
  # We want to have same UID's in all containers to allow sharing home directories
  garuda-lib.unifiedUID = true;

  fileSystems."/" = {
    device = "none";
    fsType = "tmpfs";
    options = [
      "defaults"
      "size=50%"
      "mode=755"
    ];
  };

  fileSystems."/data_1" = {
    device = "/dev/disk/by-label/NIXROOT";
    fsType = "ext4";
    neededForBoot = true;
    options = [
      "defaults"
      "noatime"
      "nodiratime"
      "errors=remount-ro"
    ];
    depends = [ "/" ];
  };

  fileSystems."/data_2" = {
    device = "/dev/disk/by-label/NIXDATA";
    fsType = "btrfs";
    options = [
      "defaults"
      "noatime"
      "nodiratime"
      "compress=zstd:1"
    ];
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-label/NIXBOOT";
    fsType = "vfat";
  };

  fileSystems."/nix" = {
    device = "/data_1/nix";
    fsType = "none";
    options = [ "bind" ];
    depends = [
      "/data_1"
    ];
  };

  services.openssh = {
    ports = [ 666 ];
    hostKeys = [
      {
        type = "ed25519";
        path = "/data_1/persistent/etc/ssh/ssh_host_ed25519_key";
      }
      {
        type = "rsa";
        bits = 4096;
        path = "/data_1/persistent/etc/ssh/ssh_host_rsa_key";
      }
    ];
  };

  services.garuda-nspawn = {
    bridgeInterface = "br0";
    hostInterface = "eth0";
    hostIp = "10.0.5.1";
    defaults = {
      maxMemorySoft = 48318382080; # 45 GiB
      maxMemoryHard = 53687091200; # 50 GiB
      maxCpu = 18;
    };
  };

  networking.firewall.trustedInterfaces = [ "br0" ];

  garuda-lib.sshkeys = {
    ed25519 = "/data_1/persistent/etc/ssh/ssh_host_ed25519_key";
    rsa = "/data_1/persistent/etc/ssh/ssh_host_rsa_key";
  };

  environment.persistence."/data_1/persistent" = {
    enable = true;
    hideMounts = true;
    directories = [
      "/home"
      "/var/cache/netdata"
      "/var/cache/tailscale"
      "/var/lib/netdata"
      "/var/lib/nixos"
      "/var/lib/tailscale"
      "/var/lib/vnstat"
      "/var/log"
    ];
    files = [
      "/etc/machine-id"
    ];
  };

  security.sudo.extraConfig = ''
    Defaults lecture = never
  '';
}

Containers/services

chaotic-backend: Backend services for Chaotic-AUR, including API and job processing.
docker: General-purpose Docker container runner for services not packaged in Nix.
docker-proxied: Docker runner for services that require special proxying or network setup.
forum: Hosts the Discourse forum for the Garuda Linux community.
mail: Handles mail-related services and relays for the infrastructure.
mastodon: Runs the Mastodon social network instance.
monitoring: Runs the monitoring stack (Prometheus, Grafana, Loki, Alertmanager).
n8n: Workflow automation and incoming webhooks.
postgres: Provides PostgreSQL database services for other containers.
web-front: Acts as the main reverse proxy and web frontend for hosted services.

See the respective documentation pages for up-to-date configuration and details.

chaotic-backend (aerialis)

This container provides backend services for Chaotic-AUR, including API endpoints and job processing for the repository.

Nix expression

{
  config,
  garuda-lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [
    ../../modules
    ../../modules/special/ssh-allow-chaotic.nix
  ];

  garuda =
    garuda-lib.mkMonitoring {
      host = "aerialis";
      units = [
        "compose-runner-chaotic-backend.service"
        "docker.service"
        "redis-chaotic.service"
      ];
      exporters = [ "redisExporter" ];
      extraPrometheus.redisExporter.passwordFile = config.sops.secrets."redis/exporter".path;
    }
    // {
      services.compose-runner.chaotic-backend = {
        envfile = config.sops.secrets."compose/chaotic-backend".path;
        source = ../../../compose/chaotic-backend;
        extraEnv = {
          "SSH_KEY" = config.sops.secrets."keypairs/chaotic/private".path;
        };
      };
    };

  services.redis = {
    servers."chaotic" = {
      enable = true;
      port = 6379;
      requirePassFile = config.sops.secrets."redis/chaotic".path;
      save = [
        [
          20
          1
        ]
      ];
    };
  };

  sops.secrets =
    garuda-lib.mkSecrets [
      "compose/chaotic-backend"
      "keypairs/chaotic/private"
      "redis/chaotic"
    ]
    // {
      "redis/exporter" = {
        owner = "redis-exporter";
        group = "redis-exporter";
        mode = "0400";
      };
    };

  system.stateVersion = "25.05";
}

Docker containers

services:
  chaotic-backend:
    image: ghcr.io/chaotic-cx/chaotic-next:2.9.0-pre
    container_name: chaotic-backend
    deploy:
      restart_policy:
        condition: always
        delay: 30s
    environment:
      AUR_USERNAME: ${AUR_USERNAME:-?err}
      AUR_PASSWORD: ${AUR_PASSWORD:-?err}
      AUTH_COOKIE_DOMAIN: .chaotic.cx
      BETTER_AUTH_SECRET: ${BETTER_AUTH_SECRET:-?err}
      BETTER_AUTH_TRUSTED_ORIGINS: https://aur.chaotic.cx
      BETTER_AUTH_URL: https://backend.chaotic.cx
      CAUR_BUILDER_CLAMAV_IMAGE: 'clamav/clamav:latest'
      CAUR_BUILDER_CPU_LIMIT: 8
      CAUR_BUILDER_DOCKER_SOCKET: /var/run/docker.sock
      CAUR_BUILDER_EXTRA_PACMAN_KEYRINGS: https://secret-mirror.chaotic.cx/chaotic-aur/chaotic-keyring.pkg.tar.zst
      CAUR_BUILDER_EXTRA_PACMAN_REPOS: |
        [chaotic-aur]
        Server = https://secret-mirror.chaotic.cx/chaotic-aur/x86_64
      CAUR_BUILDER_IDLE_TIMEOUT: 600
      CAUR_BUILDER_IMAGE: registry.gitlab.com/garuda-linux/tools/chaotic-manager/builder:latest
      CAUR_BUILDER_PIDS_LIMIT: 1024
      CAUR_BUILDER_POLL_INTERVAL: 15000
      CAUR_BUILDER_PUBLIC_URL: https://backend.chaotic.cx
      CAUR_BUILDER_RAM_LIMIT: 8192
      CAUR_BUILDER_TIMEOUT: 3600
      CAUR_BUILDER_WORKDIR: /var/garuda/compose-runner/chaotic-backend/portable-builder
      CAUR_DB_KEY: ${CAUR_DB_KEY:-?err}
      CAUR_GITLAB_TOKEN: ${TEMERAIRE_BOT_TOKEN:-?err}
      CAUR_GITLAB_WEBHOOK_TOKEN: ${CAUR_GITLAB_WEBHOOK_TOKEN:-?err}
      CAUR_MANAGER_API_TOKEN: ${CAUR_MANAGER_API_TOKEN:-?err}
      CAUR_TRUST_PROXY: true
      CAUR_VAPID_PRIVATE: ${CAUR_VAPID_PRIVATE:-?err}
      CAUR_VAPID_PUBLIC: BFMijCyTcLZu3HRJyMfoB4vtnDcgl7E2SVZgd4uWB3XjO2VvRh_LjYHcY_oJLvMIhdp5Yvuz0gudRpY-PFI9iWA
      GITHUB_ISSUES_REPO_NAME: packages
      GITHUB_ISSUES_REPO_OWNER: chaotic-aur
      GITHUB_TOKEN: ${GITHUB_TOKEN:-?err}
      GITHUB_WEBHOOK_SECRET: ${GITHUB_WEBHOOK_SECRET:-?err}
      GITLAB_CLIENT_ID: ${GITLAB_CLIENT_ID:-?err}
      GITLAB_CLIENT_SECRET: ${GITLAB_CLIENT_SECRET:-?err}
      GITLAB_REDIRECT_URI: https://backend.chaotic.cx/api/auth/callback/gitlab
      LOG_LEVEL: debug
      NODE_ENV: production
      OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-?err}
      PG_DATABASE: chaotic-aur
      PG_HOST: 10.0.5.20
      PG_PASSWORD: ${PG_PASSWORD:-?err}
      PG_USER: chaotic-aur
      REDIS_PASSWORD: ${REDIS_PASSWORD:-?err}
      REDIS_SSH_HOST: host.docker.internal
      REDIS_SSH_USER: package-deployer
      REPOMANAGER_ABI_DRY_RUN: false
      REPOMANAGER_MIRROR_URL: https://builds.garudalinux.org/arch-mirror
      REPOMANAGER_SIGNAL_SCAN_ENABLED: true
      VIRUSTOTAL_API_KEY: ${VIRUSTOTAL_API_KEY:-?err}
    ports: ["3000:3000"]
    volumes:
      - "${SSH_KEY:-?err}:/app/sshkey"
      - "./backend-config:/app/config"
      - "/var/garuda/compose-runner/chaotic-backend/portable-builder:/var/garuda/compose-runner/chaotic-backend/portable-builder"
      - "/var/run/docker.sock:/var/run/docker.sock"
    extra_hosts: ["host.docker.internal:host-gateway"]

docker (aerialis)

This container runs general-purpose Docker workloads for services that are not packaged natively in NixOS.

General

This container is used to run regular Docker containers. Recently, the compose-runner module has been replaced by native Nix expressions.

Nextcloud AIO

This container also runs a Nextcloud AIO master container, which administrates its containers by itself. Consult its extensive documentation for more information. Since this container requires a Nextcloud volume at a fixed place, without being able to change it, it is not included in the regular data directory.

Instead, backups are regularly performed via the inbuilt backup function in the admin interface. They can be found at /var/garuda/compose-runner/docker/nextcloud-aio and are included in the offsite system backups.

Nix expression

{
  config,
  garuda-lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda =
    garuda-lib.mkMonitoring {
      host = "aerialis";
      units = [
        "compose-runner-docker.service"
        "docker.service"
      ];
    }
    // {
      services.compose-runner.docker = {
        envfile = config.sops.secrets."compose/docker".path;
        source = ../../../compose/docker;
        extraEnv = {
          "MATTERBRIDGE_CONFIG" = config.sops.secrets."compose/matterbridge".path;
        };
      };
    };

  sops.secrets = {
    "compose/docker" = {
      restartUnits = [ "compose-runner-docker.service" ];
    };
    "compose/matterbridge" = {
      restartUnits = [ "compose-runner-docker.service" ];
    };
  };

  system.stateVersion = "25.05";
}

Docker containers

services:
  # Nextcloud AIO (self-managed containers)
  # The dummy mounts are for creating the required volumes, even
  # though the container doesn't use them. The actual containers
  # making use of these volumes are started by the master container.
  # Do *not* change container and volume names!
  nextcloud-aio-mastercontainer:
    image: nextcloud/all-in-one:latest
    restart: always
    container_name: nextcloud-aio-mastercontainer
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - nextcloud_aio_clamav:/dummy/clamav
      - nextcloud_aio_database:/dummy/database
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config
      - nextcloud_aio_nextcloud:/dummy/nextcloud
      - nextcloud_aio_nextcloud_data:/dummy/nextcloud_data
      - nextcloud_aio_redis:/dummy/redis
    ports: ["8080:8080"]
    environment:
      APACHE_PORT: 11000
      APACHE_IP_BINDING: 10.0.5.60
      NEXTCLOUD_DATADIR: /var/garuda/compose-runner/docker/nextcloud-aio/nextcloud_data
  # Firefox syncserver
  syncserver:
    container_name: syncserver
    image: crazymax/firefox-syncserver:edge # newest, versioned one 3 years old
    volumes: ["./syncserver:/data"]
    ports: ["5001:5000"]
    environment:
      FF_SYNCSERVER_ACCESSLOG: true
      FF_SYNCSERVER_FORCE_WSGI_ENVIRON: true
      FF_SYNCSERVER_FORWARDED_ALLOW_IPS: "*"
      FF_SYNCSERVER_PUBLIC_URL: https://ffsync.garudalinux.org
      FF_SYNCSERVER_SECRET: ${FF_SYNCSERVER_SECRET:-?err}
      FF_SYNCSERVER_SQLURI: sqlite:////data/syncserver.db
      TZ: Europe/Berlin
    restart: always
  # Password vault
  vaultwarden:
    image: vaultwarden/server:1.37.3-alpine
    container_name: vaultwarden
    volumes: ["./bitwarden:/data"]
    ports: ["8081:80"]
    environment:
      ADMIN_TOKEN: ${BW_ADMIN_TOKEN:-?err}
      DOMAIN: https://bitwarden.garudalinux.org
      SIGNUPS_ALLOWED: true
      SMTP_FROM: [email protected]
      SMTP_HOST: mail.garudalinux.org
      SMTP_PASSWORD: ${BW_SMTP_PASSWORD:-?err}
      SMTP_PORT: 465
      SMTP_SSL: true
      SMTP_USERNAME: [email protected]
      WEBSOCKET_ENABLED: true
      YUBICO_CLIENT_ID: ${BW_YUBICO_CLIENT_ID:-?err}
      YUBICO_SECRET_KEY: ${BW_YUBICO_ADMIN_SECRET:-?err}
    restart: always
  # Secure pastebin
  privatebin:
    image: privatebin/nginx-fpm-alpine:2.0.6
    container_name: privatebin
    volumes:
      - ./privatebin:/srv/data
      - ./configs/privatebin.cfg.php:/srv/cfg/conf.php
    ports: ["8082:8080"]
    restart: always
  # WikiJs
  wikijs:
    image: requarks/wiki:2.5
    container_name: wikijs
    volumes: ["./wikijs/assets:/wiki/assets/favicons"]
    ports: ["3001:3000"]
    environment:
      DB_TYPE: postgres
      DB_HOST: 10.0.5.20
      DB_PORT: 5432
      DB_USER: wikijs
      DB_PASS: ${DB_PASS:-?err}
      DB_NAME: wikijs
    restart: always
  # IRC/Discord/Telegram relay
  matterbridge:
    image: registry.gitlab.com/garuda-linux/tools/matterbridge:latest
    container_name: matterbridge
    volumes:
      - ${MATTERBRIDGE_CONFIG:-?err}:/etc/matterbridge/matterbridge.toml:ro
    deploy:
      restart_policy:
        condition: always
        delay: 120s
  # Automated container updates
  watchtower:
    image: nickfedor/watchtower:1.22.1
    container_name: watchtower
    command: --cleanup matterbridge wikijs privatebin vaultwarden thelounge syncserver
    volumes: ["/var/run/docker.sock:/var/run/docker.sock"]
    restart: always
volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer
    driver_opts:
      type: none
      device: /var/garuda/compose-runner/docker/nextcloud-aio/mastercontainer
      o: bind
  nextcloud_aio_clamav:
    name: nextcloud_aio_clamav
    driver_opts:
      type: none
      device: /var/garuda/compose-runner/docker/nextcloud-aio/clamav
      o: bind
  nextcloud_aio_database:
    name: nextcloud_aio_database
    driver_opts:
      type: none
      device: /var/garuda/compose-runner/docker/nextcloud-aio/database
      o: bind
  nextcloud_aio_nextcloud:
    name: nextcloud_aio_nextcloud
    driver_opts:
      type: none
      device: /var/garuda/compose-runner/docker/nextcloud-aio/nextcloud
      o: bind
  nextcloud_aio_nextcloud_data:
    name: nextcloud_aio_nextcloud_data
    driver_opts:
      type: none
      device: /var/garuda/compose-runner/docker/nextcloud-aio/nextcloud_data
      o: bind
  nextcloud_aio_redis:
    name: nextcloud_aio_redis
    driver_opts:
      type: none
      device: /var/garuda/compose-runner/docker/nextcloud-aio/redis
      o: bind

docker-proxied

General

Here, all the Docker containers that need to have proxied outgoing requests are being deployed. This is mainly for privacy-focused or alternative frontends and search engines that benefit from outgoing proxying.

Container explanations

whoogle: A self-hosted, ad-free, privacy-respecting metasearch engine that proxies Google Search results.
searx: SearxNG, a privacy-respecting metasearch engine aggregating results from various sources.
librey: Librey, a metasearch engine with a focus on privacy and alternative search sources.
lingva: Lingva Translate, a privacy-friendly alternative frontend for Google Translate.
redlib: Redlib, a privacy-respecting alternative frontend for Reddit.
watchtower: Automatically updates running containers to the latest image versions.
autoheal: Monitors containers and restarts them if they become unhealthy (e.g., Whoogle).

Restarting containers

This can happen via the following command:

sudo systemctl restart compose-runner-docker-proxied

Nix expression

{
  config,
  garuda-lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda =
    garuda-lib.mkMonitoring {
      host = "aerialis";
      units = [
        "compose-runner-docker-proxied.service"
        "docker.service"
      ];
    }
    // {
      services.compose-runner.docker-proxied = {
        envfile = config.sops.secrets."compose/docker-proxied".path;
        source = ../../../compose/docker-proxied;
      };
    };

  # Let Docker use squid as outgoig proxy
  # Fails to pull images if *.docker.io is not excluded from proxy
  # systemd.services.docker = {
  #   environment = {
  #     HTTPS_PROXY = "http://10.0.5.1:3128";
  #     HTTP_PROXY = "http://10.0.5.1:3128";
  #     NO_PROXY = "localhost,127.0.0.1,*.docker.io,ghcr.io";
  #   };
  # };

  sops.secrets."compose/docker-proxied" = { };

  system.stateVersion = "25.05";
}

Docker containers

services:
  # Whoogle search engine
  whoogle:
    image: benbusby/whoogle-search:latest # It tends do be important to stay current
    container_name: whoogle
    user: whoogle
    security_opt: [no-new-privileges]
    cap_drop: [ALL]
    tmpfs:
      - /var/lib/tor/:size=10M,uid=927,gid=927,mode=1700
      - /run/tor/:size=1M,uid=927,gid=927,mode=1700
    volumes: ['./whoogle:/config']
    ports: ['5000:5000']
    environment:
      WHOOGLE_AUTOCOMPLETE: 1
      WHOOGLE_CONFIG_LANGUAGE: lang_en
      WHOOGLE_CONFIG_NEW_TAB: 1
      WHOOGLE_CONFIG_SEARCH_LANGUAGE: lang_en
      WHOOGLE_CONFIG_STYLE: |
        :root {--whoogle-logo: #4c4f69;--whoogle-page-bg: #eff1f5;--whoogle-element-bg: #bcc0cc;--whoogle-text: #4c4f69;--whoogle-contrast-text: #5c5f77;--whoogle-secondary-text: #6c6f85;
        --whoogle-result-bg: #ccd0da;--whoogle-result-title: #7287fd;--whoogle-result-url: #dc8a78;--whoogle-result-visited: #e64553;--whoogle-dark-logo: #cdd6f4;
        --whoogle-dark-page-bg: #1e1e2e;--whoogle-dark-element-bg: #45475a;--whoogle-dark-text: #cdd6f4;--whoogle-dark-contrast-text: #bac2de;--whoogle-dark-secondary-text: #a6adc8;
        --whoogle-dark-result-bg: #313244;--whoogle-dark-result-title: #b4befe;--whoogle-dark-result-url: #f5e0dc;--whoogle-dark-result-visited: #eba0ac;}
        #whoogle-w {fill: #89b4fa;} #whoogle-h {fill: #f38ba8;}#whoogle-o-1 {fill: #f9e2af;}#whoogle-o-2 {fill: #89b4fa;}#whoogle-g {fill: #a6e3a1;}#whoogle-l {fill: #f38ba8;}
        #whoogle-e {fill: #f9e2af;}
      WHOOGLE_CONFIG_THEME: dark
      WHOOGLE_CONFIG_URL: https://search.garudalinux.org
      WHOOGLE_CONFIG_VIEW_IMAGE: 1
      WHOOGLE_RESULTS_PER_PAGE: 15
    healthcheck:
      test: [CMD-SHELL, 'wget --spider -q --tries=1 http://127.0.0.1:5000']
      interval: 30s
      timeout: 10s
      start_period: 30s
      retries: 1
    restart: always
  # Searxng search engine
  searx:
    image: searxng/searxng:latest # It tends do be important to stay current
    container_name: searx
    volumes: ['./searxng:/etc/searxng']
    ports: ['8080:8080']
    environment:
      BASE_URL: https://searx.garudalinux.org/
      BIND_ADDRESS: 0.0.0.0:8080
      INSTANCE_NAME: Garuda's SearxNG
      NO_PROXY: '*.garudalinux.org'
    cap_drop: [ALL]
    cap_add: [CHOWN, SETGID, SETUID, DAC_OVERRIDE]
    healthcheck:
      test:
        - CMD
        - wget
        - --no-verbose
        - --tries=1
        - --spider
        - http://127.0.0.1:8080/info/en/about
      interval: 2m
      timeout: 5s
    restart: always
  # Librey search engine
  librey:
    image: ghcr.io/ahwxorg/librey:latest # It tends do be important to stay current
    container_name: librey
    ports: ['8081:8080']
    environment:
      - CONFIG_CACHE_TIME=20
      - CONFIG_DISABLE_BITTORRENT_SEARCH=false
      - CONFIG_GOOGLE_DOMAIN=com
      - CONFIG_HIDDEN_SERVICE_SEARCH=true
      - CONFIG_INSTANCE_FALLBACK=true
      - CONFIG_INVIDIOUS_INSTANCE=https://invidious.snopyta.org
      - CONFIG_LANGUAGE=en
      - CONFIG_NUMBER_OF_RESULTS=10
      - CONFIG_RATE_LIMIT_COOLDOWN=25
      - CONFIG_TEXT_SEARCH_ENGINE=google
    healthcheck:
      test:
        - CMD
        - wget
        - --no-verbose
        - --tries=1
        - --spider
        - http://127.0.0.1:8080
      interval: 2m
      timeout: 5s
    restart: always
  # Lingva
  lingva:
    image: thedaviddelta/lingva-translate:latest # Only latest tag is available
    container_name: lingva
    environment:
      DARK_THEME: 'true'
      DEFAULT_SOURCE_LANG: auto
      DEFAULT_TARGET_LANG: en
      SITE_DOMAIN: lingva.garudalinux.org
    ports: ['3002:3000']
    restart: always
  # Reddit frontend
  redlib:
    image: quay.io/redlib/redlib:latest
    container_name: redlib
    environment:
      REDLIB_BANNER_: Garuda's Redlib
      REDLIB_DEFAULT_AUTOPLAY_VIDEOS: true
      REDLIB_DEFAULT_BLUR_NSFW: true
      REDLIB_DEFAULT_COMMENT_SORT: confidence
      REDLIB_DEFAULT_DISABLE_VISIT_REDDIT_CONFIRMATION: false
      REDLIB_DEFAULT_FIXED_NAVBAR: true
      REDLIB_DEFAULT_FRONT_PAGE: popular
      REDLIB_DEFAULT_HIDE_AWARDS: true
      REDLIB_DEFAULT_HIDE_HLS_NOTIFICATION=: true
      REDLIB_DEFAULT_HIDE_SCORE: false
      REDLIB_DEFAULT_LAYOUT: card
      REDLIB_DEFAULT_POST_SORT: hot
      REDLIB_DEFAULT_SHOW_NSFW: false
      REDLIB_DEFAULT_THEME: dracula
      REDLIB_DEFAULT_USE_HLS: true
      REDLIB_DEFAULT_WIDE: false
      REDLIB_PUSHSHIFT_FRONTEND: undelete.pullpush.io
      REDLIB_ROBOTS_DISABLE_INDEXING: true
      REDLIB_SFW_ONLY: false
    ports: ['8082:8080']
    user: nobody
    read_only: true
    security_opt: ['no-new-privileges:true']
    cap_drop: [ALL]
    healthcheck:
      test:
        - CMD
        - wget
        - --spider
        - -q
        - --tries=1
        - http://127.0.0.1:8080/settings
      interval: 5m
      timeout: 3s
    restart: always
  # Automated container updates
  watchtower:
    image: nickfedor/watchtower:1.22.1
    container_name: watchtower
    command: --cleanup searx lingva whoogle librey redlib
    volumes: ['/var/run/docker.sock:/var/run/docker.sock']
    restart: always
  # Auto-restart unhealthy containers (looking at you, Whoogle)
  autoheal:
    image: willfarrell/autoheal:latest
    container_name: autoheal
    environment:
      AUTOHEAL_CONTAINER_LABEL: all
    network_mode: none
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always

forum (aerialis)

This container hosts the Discourse forum for the Garuda Linux community, providing discussion and support.

Nix expression

{ garuda-lib, sources, ... }:
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda = garuda-lib.mkMonitoring {
    host = "aerialis";
    units = [ "docker.service" ];
  };

  # Enable Docker since we use the official Docker image in /var/discourse
  virtualisation.docker.enable = true;

  # Open required port
  networking.firewall.allowedTCPPorts = [ 80 ];

  system.stateVersion = "25.05";
}

mail (aerialis)

This container handles mail-related services and relays for the infrastructure, such as SMTP/IMAP relaying or filtering.

Nix expression

Configuration for the mail container on aerialis.

{
  config,
  garuda-lib,
  lib,
  pkgs,
  sources,
  ...
}:
let
  mon = garuda-lib.monitoring;

  authres_status = pkgs.roundcubePlugins.roundcubePlugin rec {
    pname = "authres_status";
    version = "0.7.1";
    src = pkgs.fetchzip {
      url = "https://github.com/pimlie/authres_status/archive/refs/tags/${version}.zip";
      hash = "sha256-vQGllx3NzVkjQdnIosvQtI3iQ02HrGIF7sSL7G2E5c4=";
    };
  };
in
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda = lib.recursiveUpdate (garuda-lib.mkMonitoring {
    host = "aerialis";
    units = [
      "dovecot2.service"
      "nginx.service"
      "postfix.service"
      "rspamd.service"
    ];
    exporters = [ "postfixExporter" ];
  }) { monitoring.rspamd.enable = true; };

  # NixOS Mailserver
  mailserver = {
    x509.useACMEHost = "garudalinux.net";
    dmarcReporting.enable = true;
    domains = [
      "garudalinux.org"
      "chaotic.cx"
    ];
    enable = true;
    enableManageSieve = true;
    fqdn = "mail.garudalinux.net";
    fullTextSearch = {
      enable = true;
      memoryLimit = 512;
    };

    # To create the password hashes, use nix-shell -p mkpasswd --run 'mkpasswd -sm bcrypt'
    accounts = {
      # garudalinux.org
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/cloudatgl".path;
        sendOnly = true;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/complaintsatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/dr460nf1r3atgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/filoatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/gitlabatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/mastodonatgl".path;
        sendOnly = true;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/noreplyatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/securityatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/sgsatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/spam-reportsatgl".path;
      };
      "[email protected]" = {
        aliases = [
          "[email protected]"
          "[email protected]"
          "[email protected]"
          "[email protected]"
        ];
        hashedPasswordFile = config.sops.secrets."mail/teamatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/tneatgl".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/yorperatgl".path;
      };
      # chaotic.cx
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/a0xzatchaotic".path;
      };
      "[email protected]" = {
        aliases = [
          "[email protected]"
          "[email protected]"
          "[email protected]"
          "[email protected]"
          "[email protected]"
        ];
        hashedPasswordFile = config.sops.secrets."mail/dr460nf1r3atchaotic".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/hatchaotic".path;
      };
      "[email protected]" = {
        hashedPasswordFile = config.sops.secrets."mail/wilburatchaotic".path;
      };
    };
    forwards = {
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.pedrohlc ];
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.pedrohlc ];
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.pedrohlc ];
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.islandc0der ];
      "[email protected]" = [ ];
      "[email protected]" = [ ];
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.pedrohlc ];
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.pedrohlc ];
      "[email protected]" = [ config.garuda-lib.secrets.mail.forwards.xstefen ];
    };
    indexDir = "/var/lib/dovecot/indices";
    systemDomain = "garudalinux.org";
    systemName = "Garuda Linux";
    # SMTP on port 587 is deprecated and disabled by default
    # Explicitly disable it to discourage misconfiguration
    enableSubmission = false;
  };

  # Fix dovecot errors caused by failed scudo allocations
  environment.memoryAllocator.provider = lib.mkForce "libc";

  # Set up push notifications
  services.dovecot2.settings = {
    mail_plugins = {
      notify = true;
      push_notification = true;
    };

    "service stats" = {
      "inet_listener http" = {
        port = mon.ports.dovecotMetrics;
      };
    };
    "metric auth_success" = {
      filter = "event=auth_request_finished AND success=yes";
    };
    "metric auth_failure" = {
      filter = "event=auth_request_finished AND success=no";
    };
    "metric imap_command" = {
      filter = "event=imap_command_finished";
      "group_by cmd_name" = { };
      "group_by tagged_reply_state" = { };
    };
    "metric mail_delivery" = {
      filter = "event=mail_delivery_finished";
    };
    "metric push_notification" = {
      filter = "event=push_notification_finished";
    };
  };

  networking.firewall.interfaces."eth0".allowedTCPPorts = [ mon.ports.dovecotMetrics ];

  # Postmaster alias
  services.postfix.postmasterAlias = "[email protected]";

  # Web UI
  services.roundcube = {
    enable = true;
    # this is the url of the vhost, not necessarily the same as the fqdn of
    # the mailserver
    hostName = "mail.garudalinux.net";
    extraConfig = ''
      $config['default_host'] = "ssl://mail.garudalinux.net";
      $config['smtp_host'] = "ssl://mail.garudalinux.net";
      $config['managesieve_host'] = "tls://mail.garudalinux.net:4190";
      $config['smtp_user'] = "%u";
      $config['smtp_pass'] = "%p";
    '';
    package = pkgs.roundcube.withPlugins (plugins: [
      authres_status
      plugins.carddav
      plugins.contextmenu
      plugins.custom_from
      plugins.persistent_login
      plugins.thunderbird_labels
    ]);
    plugins = [
      "attachment_reminder" # Roundcube internal plugin
      "authres_status"
      "carddav"
      "contextmenu"
      "custom_from"
      "managesieve" # Roundcube internal plugin
      "newmail_notifier" # Roundcube internal plugin
      "persistent_login"
      "thunderbird_labels"
      "zipdownload" # Roundcube internal plugin
    ];
  };
  services.nginx.virtualHosts."mail.garudalinux.net" = {
    forceSSL = lib.mkForce false;
  };

  sops.secrets = garuda-lib.mkSecrets [
    "backup/repo_key"
    "backup/ssh_aerialis"
    "mail/rspamd_controller"
    "mail/a0xzatchaotic"
    "mail/cloudatgl"
    "mail/complaintsatgl"
    "mail/dr460nf1r3atchaotic"
    "mail/dr460nf1r3atgl"
    "mail/filoatgl"
    "mail/gitlabatgl"
    "mail/hatchaotic"
    "mail/mastodonatgl"
    "mail/noreplyatgl"
    "mail/securityatgl"
    "mail/sgsatgl"
    "mail/spam-reportsatgl"
    "mail/teamatgl"
    "mail/tneatgl"
    "mail/wilburatchaotic"
    "mail/yorperatgl"
  ];

  system.stateVersion = "22.05";

  # https://nixos-mailserver.readthedocs.io/en/latest/migrations.html
  mailserver.stateVersion = 5;
}

mastodon (aerialis)

This container runs the Mastodon social network instance for Garuda Linux, providing decentralized microblogging.

Nix expression

Configuration for the mastodon container on aerialis.

{
  config,
  garuda-lib,
  lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda =
    garuda-lib.mkMonitoring {
      host = "aerialis";
      units = [
        "compose-runner-mastodon.service"
        "mastodon-sidekiq.service"
        "mastodon-streaming.service"
        "mastodon-web.service"
        "nginx.service"
      ];
    }
    // {
      services.compose-runner.mastodon = {
        source = ../../../compose/mastodon;
      };
    };

  # Our Mastodon
  services.mastodon = {
    configureNginx = true;
    database = {
      createLocally = false;
      host = "10.0.5.20";
      name = "mastodon";
      passwordFile = config.sops.secrets."mastodon/db_password".path;
      user = "mastodon";
    };
    enable = true;
    extraConfig = {
      "LOCAL_DOMAIN" = "garudalinux.org";
      "SMTP_DOMAIN" = "social.garudalinux.org";
      "WEB_DOMAIN" = "social.garudalinux.org";
    };
    extraEnvFiles = [ config.sops.secrets."mastodon/env".path ];
    localDomain = "social.garudalinux.org";
    mediaAutoRemove = {
      enable = true;
      startAt = "daily";
      olderThanDays = 7;
    };
    smtp = {
      authenticate = true;
      fromAddress = "[email protected]";
      host = "mail.garudalinux.net";
      passwordFile = config.sops.secrets."mastodon/smtp_password".path;
      port = 465;
      user = "[email protected]";
      createLocally = false;
    };
    streamingProcesses = 16;
    redis = {
      createLocally = false;
      enableUnixSocket = false;
      host = "localhost";
      port = 6379;
    };
  };

  # This disables HTTPS certificates and forced redirects
  garuda-lib.behind_proxy = true;

  services.nginx = {
    recommendedProxySettings = lib.mkForce false;
    virtualHosts."social.garudalinux.org" = {
      enableACME = lib.mkForce false;
      forceSSL = lib.mkForce false;
      extraConfig = ''
        real_ip_header          X-Real-IP;
        set_real_ip_from        10.0.5.10;
        proxy_redirect          off;
        proxy_connect_timeout   60s;
        proxy_send_timeout      60s;
        proxy_read_timeout      60s;
        proxy_http_version      1.1;
        proxy_set_header        Upgrade $http_upgrade;
        proxy_set_header        Connection $connection_upgrade;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $remote_addr;
        # I'm a filthy liar
        proxy_set_header        X-Forwarded-Proto https;
        proxy_set_header        X-Forwarded-Host $http_x_forwarded_host;
        proxy_set_header        X-Forwarded-Server $http_x_forwarded_server;
      '';
      locations = {
        "@proxy" = {
          proxyWebsockets = lib.mkForce false;
          extraConfig = ''
            real_ip_header          X-Real-IP;
            set_real_ip_from        10.0.5.10;
            proxy_redirect          off;
            proxy_connect_timeout   60s;
            proxy_send_timeout      60s;
            proxy_read_timeout      60s;
            proxy_http_version      1.1;
            proxy_set_header        Upgrade $http_upgrade;
            proxy_set_header        Connection $connection_upgrade;
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $remote_addr;
            # I'm a filthy liar
            proxy_set_header        X-Forwarded-Proto https;
            proxy_set_header        X-Forwarded-Host $http_x_forwarded_host;
            proxy_set_header        X-Forwarded-Server $http_x_forwarded_server;
          '';
        };
        "/api/v1/streaming/" = {
          proxyWebsockets = lib.mkForce false;
        };
      };
    };
  };

  sops.secrets = {
    "mastodon/db_password" = {
      owner = "mastodon";
      group = "mastodon";
    };
    "mastodon/env" = {
      owner = "mastodon";
      group = "mastodon";
    };
    "mastodon/smtp_password" = {
      owner = "mastodon";
      group = "mastodon";
    };
  };

  system.stateVersion = "25.05";
}

monitoring (aerialis)

This container runs our complete monitoring stack: Prometheus, Alertmanager, Grafana and Loki, together with the exporters that feed them.

General

It is the counterpart to the agents that the hosts and containers run themselves (node_exporter and Fluent Bit). Unlike the other containers it runs its own tailscaled, so that Prometheus can scrape the Chaotic-AUR mirrors over the Tailnet using MagicDNS names, and it listens on 10.0.5.100 on the host’s container bridge.

See Monitoring for the full picture, including how the stormwing containers are reached and how alerts are routed.

Nix expression

{
  config,
  garuda-lib,
  sources,
  ...
}:
let
  mon = garuda-lib.monitoring;

  units = [
    "grafana.service"
    "prometheus.service"
    "alertmanager.service"
    "prometheus-tailscale-exporter.service"
    "loki.service"
  ];
in
{
  imports = sources.defaultModules ++ [
    ../../modules
  ];

  # Own Tailnet identity so Prometheus scrapes mirrors/hosts directly
  # via MagicDNS instead of socat hairpins on the host.
  services.garuda-tailscale.enable = true;

  garuda.monitoring = {
    enable = true;

    domain = "grafana.garudalinux.net";
    baseDomain = "garudalinux.net";

    buildQueue.enable = true;

    grafana.enable = true;

    prometheus = {
      enable = true;
      port = mon.ports.prometheus;

      alertmanager = {
        enable = true;
        port = mon.ports.alertmanager;
        environmentFile = config.sops.templates."alertmanager-env".path;
        telegram.enable = true;
      };

      inherit (mon) smartctlTargets;

      inherit (mon) borgmaticTargets;

      nodeExporter = {
        enable = true;
        port = mon.ports.nodeExporter;
      };

      postgresExporter = {
        enable = true;
        environmentFile = config.sops.templates."postgres-exporter-env".path;
      };

      tailscaleExporter = {
        enable = true;
        port = 9813;
        environmentFile = config.sops.templates."tailscale-exporter-env".path;
      };

      cloudflareExporter = {
        enable = true;
        port = mon.ports.cloudflareExporter;
        environmentFile = config.sops.templates."cloudflare-exporter-env".path;
      };

      applicationTargets = [
        {
          name = "dovecot";
          address = mon.aerialisContainers.mail;
          port = mon.ports.dovecotMetrics;
        }
        {
          name = "postfix";
          address = mon.aerialisContainers.mail;
          port = mon.ports.postfixExporter;
        }
        {
          name = "redis";
          address = mon.aerialisContainers.chaotic-backend;
          port = mon.ports.redisExporter;
        }
      ];

      scrapeConfigs = [
        {
          job_name = "node-hosts";
          static_configs = map (t: {
            targets = [ t.target ];
            labels = {
              instance = t.name;
            };
          }) mon.nodeHostTargets;
        }
        {
          job_name = "node-aerialis-containers";
          static_configs = map (t: {
            targets = [ t.target ];
            labels = {
              inherit (t) instance;
            };
          }) mon.nodeContainerTargets;
        }
        {
          job_name = "node-stormwing-containers";
          static_configs = map (t: {
            targets = [ t.target ];
            labels = {
              inherit (t) instance;
            };
          }) mon.stormwingNodeContainerTargets;
        }
        {
          job_name = "nginx";
          static_configs = [
            {
              targets = mon.nginxTargets;
            }
          ];
        }
        {
          job_name = "cloudflared";
          static_configs = [
            {
              targets = mon.cloudflaredTargets;
            }
          ];
        }
        {
          job_name = "gitlab-runner";
          static_configs = [
            {
              targets = mon.gitlabRunnerTargets;
            }
          ];
          scrape_interval = "30s";
        }
        {
          job_name = "node-chaotic-mirrors";
          static_configs = map (m: {
            targets = [ m.target ];
            labels = {
              instance = m.name;
            };
          }) mon.chaoticMirrorTargets;
        }
        {
          job_name = "loki";
          static_configs = [
            {
              targets = [ "127.0.0.1:${toString mon.ports.loki}" ];
            }
          ];
        }
        {
          job_name = "alertmanager";
          static_configs = [
            {
              targets = [ "127.0.0.1:${toString mon.ports.alertmanager}" ];
            }
          ];
        }
      ];
    };

    loki = {
      enable = true;
      port = mon.ports.loki;
      dataDir = "/var/lib/loki";
    };

    fluent-bit = {
      enable = true;
      lokiAddress = "127.0.0.1";
      lokiPort = mon.ports.loki;
      extraSystemdUnits = units;
    };
  };

  networking.firewall = {
    allowedTCPPorts = with mon.ports; [
      grafana
      loki
      prometheus
      alertmanager
    ];
    interfaces."eth0".allowedTCPPorts = [ mon.ports.nodeExporter ];
  };

  sops.secrets."monitoring/telegram_bot_token" = { };
  sops.templates."alertmanager-env" = {
    mode = "0400";
    content = ''
      TELEGRAM_BOT_TOKEN=${config.sops.placeholder."monitoring/telegram_bot_token"}
    '';
  };

  sops.secrets."grafana/gitlab_client_secret" = {
    owner = "grafana";
  };
  sops.secrets."grafana/smtp_password" = {
    owner = "grafana";
  };
  sops.secrets."grafana/secret_key" = {
    owner = "grafana";
  };

  sops.secrets."postgres/netdata" = { };
  sops.templates."postgres-exporter-env" = {
    owner = "postgres-exporter";
    group = "postgres-exporter";
    mode = "0400";
    content = "PGPASSWORD=${config.sops.placeholder."postgres/netdata"}";
  };

  sops.secrets."tailscale/oauth_client_id" = { };
  sops.secrets."tailscale/oauth_client_secret" = { };
  sops.templates."tailscale-exporter-env" = {
    mode = "0400";
    content = ''
      TAILSCALE_TAILNET=${mon.tailnetDomain}
      TAILSCALE_OAUTH_CLIENT_ID=${config.sops.placeholder."tailscale/oauth_client_id"}
      TAILSCALE_OAUTH_CLIENT_SECRET=${config.sops.placeholder."tailscale/oauth_client_secret"}
    '';
  };

  sops.secrets."cloudflare/exporter_api_token" = { };
  sops.templates."cloudflare-exporter-env" = {
    mode = "0400";
    content = ''
      CF_API_TOKEN=${config.sops.placeholder."cloudflare/exporter_api_token"}
    '';
  };

  system.stateVersion = "26.11";
}

n8n (aerialis)

This container runs our n8n workflow automation instance.

General

n8n is used to automate recurring tasks and glue our services together. The editor and API are reachable at n8n.garudalinux.net, while incoming webhooks use the separate n8n-webhooks.garudalinux.net host, which only proxies /webhook and returns 404 for everything else. Both hosts are proxied by the aerialis web-front.

Nix expression

{
  garuda-lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [
    ../../modules
  ];

  garuda = garuda-lib.mkMonitoring {
    host = "aerialis";
    units = [ "n8n.service" ];
  };

  services.n8n = {
    enable = true;
    environment = {
      N8N_PORT = "5678";
      N8N_PROXY_HOPS = "1";
      N8N_HOST = "n8n.garudalinux.net";
      N8N_EDITOR_BASE_URL = "https://n8n.garudalinux.net";
      WEBHOOK_URL = "https://n8n-webhooks.garudalinux.net";
      N8N_PUBLIC_API_DISABLED = true;
      N8N_PUBLIC_API_SWAGGERUI_DISABLED = true;
      N8N_DIAGNOSTICS_ENABLED = false;
      N8N_VERSION_NOTIFICATIONS_ENABLED = false;
      N8N_TEMPLATES_ENABLED = false;
    };
    openFirewall = true;
  };

  system.stateVersion = "25.11";
}

postgres (aerialis)

This container provides PostgreSQL database services for other containers and applications on aerialis.

General

This container houses our Postgres database. Multiple services access it:

Mastodon
Matrix
Matrix bridges
WikiJs

Admin interface

The admin interface powered by Pgadmin can be accessed here. Authentication happens via Cloudflare Access.

Nix expression

{
  garuda-lib,
  pkgs,
  sources,
  config,
  lib,
  ...
}:
let
  server_config = pkgs.writeText "server-config" ''
    {
      "Servers": {
        "1": {
          "Name": "Main",
          "Group": "Garuda",
          "Username": "pgadmin",
          "Host": "/var/run/postgresql",
          "Port": 5432,
          "SSLMode": "prefer",
          "MaintenanceDB": "postgres",
          "PassFile": "/dev/null",
          "Shared": true,
          "SharedUsername": "pgadmin"
        }
      }
    }
  '';
in
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda = garuda-lib.mkMonitoring {
    host = "aerialis";
    units = [ "postgresql.service" ];
  };

  services.postgresql = {
    enable = true;
    ensureDatabases = [
      "chaotic-aur"
      "mastodon"
      "wikijs"
    ];
    ensureUsers = [
      {
        name = "mastodon";
        ensureDBOwnership = true;
      }
      {
        name = "wikijs";
        ensureDBOwnership = true;
      }
      {
        name = "pgadmin";
        ensureClauses.superuser = true;
      }
      {
        name = "chaotic-router";
      }
      {
        name = "chaotic-aur";
        ensureDBOwnership = true;
      }
      {
        name = "netdata";
        ensureClauses.login = true;
      }
    ];
    extensions = with pkgs.postgresql_18.pkgs; [
      pg_hll # chaotic-aur uses this for the useragent/ip statistics
      pg_repack # this lets us run the vacuum online
    ];
    authentication = lib.mkForce ''
      local all all peer
      hostssl chaotic-aur chaotic-router 0.0.0.0/0 scram-sha-256
      host all netdata 10.0.5.1/32 scram-sha-256
      # Reject anything else coming from the outside world somehow someway
      host all all 10.0.5.1/32 reject
      # Allow connections from the internal network
      host all all 10.0.5.0/24 scram-sha-256
      # Allow localhost connections
      host all all 127.0.0.1/32 scram-sha-256
      # Block the rest of the internet
      host all all 0.0.0.0/0 reject
    '';
    # This is publically accessible now through port 5432, however only the chaotic-router user can access
    # the database through the internet, and only over TLS
    enableTCPIP = true;
    package = pkgs.postgresql_18;
    settings = {
      ssl = true;
      ssl_cert_file = "${config.services.postgresql.dataDir}/server.crt";
      ssl_key_file = "${config.services.postgresql.dataDir}/server.key";

      # Defaults (128MB / 4GB) are far too small for a cluster shared by multiple services
      shared_buffers = "4GB";
      effective_cache_size = "12GB";
    };
  };

  # Regular backups for our database (every 6h)
  services.postgresqlBackup = {
    compression = "zstd";
    enable = true;
    location = "/var/garuda/backups/postgres";
  };

  services.pgadmin = {
    enable = true;
    initialEmail = "[email protected]";
    initialPasswordFile = config.sops.secrets."postgres/pg_admin".path;
    openFirewall = true;
    settings = {
      DEFAULT_SERVER = "10.0.5.20";
      FIXED_BINARY_PATHS = {
        "pg" = "${config.services.postgresql.package}/bin";
      };
      SUPPORT_SSH_TUNNEL = false;
      AUTHENTICATION_SOURCES = [ "webserver" ];
      WEBSERVER_REMOTE_USER = "X-Forwarded-User";
      MASTER_PASSWORD_REQUIRED = false;
    };
  };

  systemd.services.pgadmin = {
    preStart = lib.mkAfter ''
      EMAIL=${lib.escapeShellArg config.services.pgadmin.initialEmail}
      FILE=${lib.escapeShellArg server_config}
      ${config.services.pgadmin.package}/bin/pgadmin4-cli load-servers "$FILE" --user "$EMAIL"
    '';
  };

  # Serve the live ACME wildcard cert to Postgres for external TLS
  system.activationScripts.postgresServerCert = ''
    install -o postgres -g postgres -m 0600 \
      /var/lib/acme/garudalinux.org/key.pem \
      ${config.services.postgresql.dataDir}/server.key
    install -o postgres -g postgres -m 0644 \
      /var/lib/acme/garudalinux.org/fullchain.pem \
      ${config.services.postgresql.dataDir}/server.crt
  '';

  # Open up ports for Postgres
  networking.firewall.allowedTCPPorts = [ 5432 ];

  sops.secrets."postgres/pg_admin" = { };
  sops.secrets."postgres/netdata" = {
    owner = "postgres";
    group = "postgres";
    mode = "0400";
  };

  systemd.services.postgresql.postStart = lib.mkAfter ''
    PASS=$(cat ${config.sops.secrets."postgres/netdata".path})
    ${config.services.postgresql.package}/bin/psql -U postgres -c "ALTER USER netdata WITH LOGIN PASSWORD '$PASS'"
    ${config.services.postgresql.package}/bin/psql -U postgres -c "GRANT pg_monitor TO netdata"
  '';

  system.stateVersion = "23.05";
}

web-front (aerialis)

This container acts as the main reverse proxy and web frontend for hosted services on aerialis, handling HTTPS termination and routing.

General

This container is used as a reverse proxy for all of our public facing services. It also contains a Cloudflared instance, which a few services are only being exposed to, instead of being reverse proxied by Nginx itself.

Nix expression

{
  garuda-lib,
  pkgs,
  sources,
  ...
}:
let
  inherit (garuda-lib) allowOnlyCloudflareZerotrust;
  inherit (garuda-lib) mkCatchAllVhost;
  inherit (garuda-lib) mkCloudflaredVhost;
  inherit (garuda-lib) mkProxyVhost;
  inherit (garuda-lib) mkZerotrustVhost;

  website = pkgs.garuda-website;
  startpage = pkgs.garuda-startpage;

  vhosts = {
    "garudalinux.org" = {
      addSSL = true;
      http3 = true;
      locations = {
        "/" = {
          index = "index.html";
          root = website;
          extraConfig = ''
            # First attempt to serve request as file, then
            # as directory, then redirect to index.html (Angular) if no file found.
            try_files $uri $uri/ /index.html;
            expires 5m;
          '';
        };
        "/discord" = {
          extraConfig = "expires 12h;";
          return = "307 https://discord.gg/w5jbhq3juh";
        };
        "/telegram" = {
          extraConfig = "expires 12h;";
          return = "307 https://t.me/+TAZWHgryP6elOyS8";
        };
        "/os/" = {
          extraConfig = ''
            try_files $uri =404;
            expires 5m;

            location = /os/garuda-update/remote-update {
              expires 12h;
              return 301 https://gitlab.com/garuda-linux/themes-and-settings/settings/garuda-common-settings/-/snippets/2147440/raw/main/remote-update;
            }
            location = /os/garuda-diag/diagnostic {
              expires 12h;
              return 301 https://gitlab.com/garuda-linux/themes-and-settings/settings/garuda-common-settings/-/snippets/4892890/raw/main/diagnostics;
            }
            location = /os/garuda-update/hotfix {
              expires 5m;
              return 307 https://gitlab.com/garuda-linux/themes-and-settings/settings/garuda-common-settings/-/snippets/4899885/raw/main/hotfix;
            }
            location = /os/garuda-update/hotfix-check {
              expires 5m;
              return 200 '7';
            }
            location = /os/garuda-update/garuda-hotfixes-version {
              expires 12h;
              return 410 'Feature removed in Garuda System Maintenance 3.1.0';
            }
          '';
        };
        "/.well-known/webfinger" = {
          extraConfig = "expires 12h;";
          return = "301 https://social.garudalinux.org$request_uri";
        };
      };
      quic = true;
      serverAliases = [ "www.garudalinux.org" ];
      useACMEHost = "garudalinux.org";
    };
    "start.garudalinux.org" = {
      addSSL = true;
      http3 = true;
      locations = {
        "/" = {
          index = "index.html";
          root = startpage;
          extraConfig = ''
            # First attempt to serve request as file, then
            # as directory, then redirect to index.html (Angular) if no file found.
            try_files $uri $uri/ /index.html;
            expires 5m;
          '';
        };
      };
      quic = true;
      useACMEHost = "garudalinux.org";
    };
    "cloud-aio.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.60:11000";
      locationExtraConfig = ''
        client_body_buffer_size 512k;
        proxy_read_timeout 86400s;
        client_max_body_size 0;

        # Allow accessing through trusted domain
        set_real_ip_from      172.0.0.0/16;
      '';
    };
    "cloud-temp.garudalinux.org" = mkZerotrustVhost {
      upstream = "https://10.0.5.60:8080";
      locationExtraConfig = ''
        client_body_buffer_size 512k;
        proxy_read_timeout 86400s;
        client_max_body_size 0;

        # Allow accessing through trusted domain
        set_real_ip_from      172.0.0.0/16;
      '';
    };
    "search.garudalinux.org" = mkCloudflaredVhost {
      upstream = "http://10.0.5.50:5000";
    };
    "searx.garudalinux.org" = mkCloudflaredVhost {
      upstream = "http://10.0.5.50:8080";
    };
    "librey.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.50:8081";
    };
    "ffsync.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.60:5001";
    };
    "bin.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.60:8082";
    };
    "bitwarden.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.60:8081";
      serverAliases = [ "vault.garudalinux.org" ];
    };
    "forum.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.40:80";
      prologue = "client_max_body_size 100M;";
      extraLocations = {
        "/c/announcements/announcements-maintenance/45.json" = {
          proxyPass = "http://10.0.5.40:80";
          extraConfig = "expires 2m;";
        };
      };
    };
    "social.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.30";
      prologue = "client_max_body_size 100M;";
      extraLocations = {
        "/.well-known/webfinger" = {
          proxyPass = "http://10.0.5.30";
          extraConfig = ''
            if ($args ~* "resource=acct:(.*)@(chaotic.cx|social.garudalinux.org)$") {
              set $w1 $1;
              rewrite .* /.well-known/webfinger?resource=acct:[email protected]? break;
            }
          '';
        };
      };
    };
    "social-video.garudalinux.org" = {
      addSSL = true;
      extraConfig = ''
        client_max_body_size 100M;
        ${garuda-lib.setRealIpFromConfig}
        ${garuda-lib.nginxReverseProxySettings}
        location ~* .(mp4|webm)$ {
          proxy_pass http://10.0.5.30;
        }
      '';
      locations = {
        "/" = {
          return = "301 https://social.garudalinux.org$request_uri";
        };
      };
      http3 = true;
      quic = true;
      useACMEHost = "garudalinux.org";
    };
    "element.garudalinux.org" = {
      addSSL = true;
      extraConfig = ''
        ${garuda-lib.setRealIpFromConfig}
        ${garuda-lib.nginxReverseProxySettings}
      '';
      http3 = true;
      locations = {
        # Redirect to forum post
        "/" = {
          return = "301 https://forum.garudalinux.org/t/39538";
        };
      };
      quic = true;
      useACMEHost = "garudalinux.org";
    };
    "matrix.garudalinux.org" = {
      addSSL = true;
      http3 = true;
      listen = [
        {
          addr = "0.0.0.0";
          port = 443;
          ssl = true;
        }
      ];
      locations = {
        "/" = {
          # Redirect to forum post
          return = "301 https://forum.garudalinux.org/t/39538";
        };
      };
      quic = true;
      useACMEHost = "garudalinux.org";
    };
    "lingva.garudalinux.org" = mkCloudflaredVhost {
      upstream = "http://10.0.5.50:3002";
    };
    "reddit.garudalinux.org" = mkCloudflaredVhost {
      upstream = "http://10.0.5.50:8082";
    };
    "pgadmin.garudalinux.net" = allowOnlyCloudflareZerotrust {
      locations = {
        "/" = {
          extraConfig = ''
            ${garuda-lib.nginxReverseProxySettings}

            proxy_pass http://10.0.5.20:5050;
            proxy_set_header X-Forwarded-User $http_cf_access_authenticated_user_email;

            proxy_hide_header Cache-Control;
            proxy_hide_header Expires;
            add_header Cache-Control 'no-store';
          '';
        };
      };
    };
    "n8n-webhooks.garudalinux.net" = {
      addSSL = true;
      locations = {
        "/" = {
          return = "404";
        };
        "/webhook" = {
          extraConfig = ''
            ${garuda-lib.nginxReverseProxySettings}

            proxy_pass http://10.0.5.90:5678;
          '';
        };
      };
      useACMEHost = "garudalinux.net";
    };
    "n8n.garudalinux.net" = allowOnlyCloudflareZerotrust {
      locations = {
        "/" = {
          extraConfig = ''
            ${garuda-lib.nginxReverseProxySettings}

            proxy_pass http://10.0.5.90:5678;
          '';
        };
      };
    };
    "wiki.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.60:3001";
    };
    "backend.chaotic.cx" = {
      addSSL = true;
      http3 = true;
      locations = {
        "~ ^/(sse|metrics/live/traffic|logs/[^/]+/[^/]+|api/manager/logs|gitlab/(aur-scan|pipelines)/)" = {
          proxyPass = "http://10.0.5.70:3000";
          recommendedProxySettings = false;
          extraConfig = ''
            proxy_http_version      1.1;
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;
            proxy_set_header        X-Forwarded-Host $host;
            proxy_set_header        X-Forwarded-Server $host;
            proxy_set_header        Connection "";
            proxy_set_header        Upgrade $http_upgrade;

            proxy_redirect          off;
            proxy_connect_timeout   60s;
            proxy_read_timeout      3600s;
            proxy_send_timeout      3600s;
          '';
        };
        "/" = {
          proxyPass = "http://10.0.5.70:3000";
          recommendedProxySettings = false;
          extraConfig = ''
            proxy_http_version      1.1;
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;
            proxy_set_header        X-Forwarded-Host $host;
            proxy_set_header        X-Forwarded-Server $host;
            proxy_set_header        Upgrade $http_upgrade;
            proxy_set_header        Connection $connection_upgrade;

            proxy_redirect          off;
            proxy_buffering         off;
            proxy_connect_timeout   60s;
            proxy_read_timeout      60s;
            proxy_send_timeout      60s;
          '';
        };
      };
      quic = true;
      useACMEHost = "backend.chaotic.cx";
    };
    "mail.garudalinux.net" = mkProxyVhost {
      upstream = "http://10.0.5.80:80";
      acmeHost = "garudalinux.net";
      realIp = false;
    };
    "grafana.garudalinux.net" = mkCloudflaredVhost {
      upstream = "http://10.0.5.100:3010";
      acmeHost = "garudalinux.net";
      extraLocations = {
        "= /static/fly-regions.geojson" = {
          alias = ../../services/monitoring/static/fly-regions.geojson;
        };
      };
    };
    "prometheus.garudalinux.net" = mkZerotrustVhost {
      upstream = "http://10.0.5.100:9090";
      acmeHost = "garudalinux.net";
    };
    "alertmanager.garudalinux.net" = mkZerotrustVhost {
      upstream = "http://10.0.5.100:9093";
      acmeHost = "garudalinux.net";
    };
    "_" = mkCatchAllVhost { };
  };
in
{
  imports = sources.defaultModules ++ [ ../../modules ];

  inherit
    (garuda-lib.mkWebFront {
      host = "aerialis";
      inherit vhosts;
    })
    garuda
    networking
    services
    sops
    systemd
    ;

  system.stateVersion = "25.05";
}

stormwing

This is one of the two main infrastructure hosts (see also: aerialis). All services and containers for stormwing are defined in nixos/hosts/stormwing.nix and its submodules.

Host configuration

{
  config,
  garuda-lib,
  lib,
  ...
}:
let
  mon = garuda-lib.monitoring;

  mkTailnetProxy =
    {
      name,
      container,
      port,
      targetPort ? port,
      description,
    }:
    {
      inherit name;
      bind = mon.tailnetIPs.stormwing;
      listen = port;
      target = "${mon.stormwingContainers.${container}}:${toString targetPort}";
      after = [
        "tailscaled.service"
        "container@${container}.service"
      ];
      wants = [ "container@${container}.service" ];
      inherit description;
    };
in
{
  imports = [
    ../modules
    ./../modules/special/hetzner-ex44.nix
  ];

  garuda = garuda-lib.mkMonitoring {
    host = "stormwing";
    lokiAddress = mon.loki.tailnetAddress;
    hostInfo = true;
    units = [
      "sshd.service"
      "tailscaled.service"
    ];
    exporters = [ "smartctlExporter" ];
  };

  swapDevices = [
    {
      device = "/data_1/swapfile";
      size = 32 * 1024;
    }
  ];

  networking = {
    defaultGateway = "157.180.57.1";
    defaultGateway6 = {
      address = "fe80::1";
      interface = "eth0";
    };
    hostName = "stormwing";
    interfaces."eth0".ipv4.addresses = [
      {
        address = "157.180.57.51";
        prefixLength = 26;
      }
    ];
    nat.forwardPorts = [
      (garuda-lib.mkNatForward {
        sourcePort = 210;
        destination = "10.0.5.10:22";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 220;
        destination = "10.0.5.20:22";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 80;
        destination = "10.0.5.40:80";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 443;
        destination = "10.0.5.40:443";
      })
      (garuda-lib.mkNatForward {
        sourcePort = 443;
        destination = "10.0.5.40:443";
        proto = "udp";
      })
    ];
  };

  services.garuda-nspawn = {
    dockerCache = "/data_2/dockercache/";

    containers = garuda-lib.mkContainers {
      dir = ./stormwing;
      ips = mon.stormwingContainers;
      containers = {
        chaotic-v4 = {
          mounts = [
            {
              name = "arch-mirror";
              hostPath = "/data_2/containers/arch-mirror/mirror";
              mountPoint = "/srv/http/arch-mirror";
            }
            {
              name = "chaotic";
              hostPath = "/data_2/containers/chaotic-v4/chaotic";
              mountPoint = "/var/garuda/compose-runner/chaotic-v4";
            }
            {
              name = "syncthing";
              hostPath = "/data_2/containers/chaotic-v4/syncthing";
              mountPoint = "/var/lib/syncthing";
            }
            {
              name = "chaotic-v4";
              hostPath = "/data_1/chaotic-v4/";
              mountPoint = "/srv/http/repos";
            }
            {
              name = "iso-builds";
              hostPath = "/data_1/iso/iso";
              mountPoint = "/srv/http/iso";
            }
            {
              name = "garuda-nix-builds";
              hostPath = "/data_1/iso/garuda-nix";
              mountPoint = "/srv/http/garuda-nix";
            }
            {
              name = "misc-files";
              hostPath = "/data_2/containers/chaotic-v4/misc";
              mountPoint = "/srv/http/misc";
            }
          ];
          forwardPorts = [
            { containerPort = 873; }
            {
              containerPort = 21027;
              protocol = "udp";
            }
            { containerPort = 22000; }
            {
              containerPort = 22000;
              protocol = "udp";
            }
          ];
          nspawn = {
            enableTun = true;
          };
          needsDocker = true;
          # Only entitled to 1/5 of the CPU resources in case of contention
          cpuWeight = 20;
          ioWeight = 20;
        };

        arch-mirror = {
          mounts = [
            {
              name = "arch-mirror";
              hostPath = "/data_2/containers/arch-mirror/mirror";
              mountPoint = "/srv/http/arch-mirror";
            }
          ];
        };

        github-runner = {
          mounts = [
            {
              name = "token";
              hostPath = config.sops.secrets."compose/github-runner".path;
              mountPoint = "/var/.github-runner.env";
              readOnly = true;
            }
            {
              name = "gitlab-config";
              hostPath = "/data_2/containers/github-runner/gitlab-runner";
              mountPoint = "/etc/gitlab-runner";
            }
            {
              name = "ssh-keys";
              hostPath = "/data_2/containers/github-runner/ssh";
              mountPoint = "/etc/ssh";
            }
            {
              name = "github-cache";
              hostPath = "/data_2/cache/github-runner";
              mountPoint = "/var/cache/github-runner";
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 230;
            }
          ];
          nspawn = {
            ephemeral = lib.mkForce true;
          };
          defaults = false;
          needsDocker = true;
          cpuWeight = 20;
          ioWeight = 20;
        };

        firedragon-runner = {
          mounts = [
            {
              name = "firedragon-runner";
              hostPath = "/data_2/containers/firedragon-runner";
              mountPoint = "/var/garuda/compose-runner/firedragon-runner";
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 250;
            }
          ];
          nspawn = {
            ephemeral = lib.mkForce true;
          };
          defaults = false;
          needsDocker = true;
          cpuWeight = 10;
          ioWeight = 10;
        };

        gitlab-runner = {
          mounts = [
            {
              name = "nix-cache";
              hostPath = "/data_2/containers/gitlab-runner/nix";
              mountPoint = "/nix";
            }
            {
              name = "gitlab-runner";
              hostPath = "/data_2/containers/gitlab-runner/gitlab-runner";
              mountPoint = "/var/lib/private/gitlab-runner";
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 260;
            }
          ];
          needsDocker = true;
          needsKvm = true;
          cpuWeight = 20;
          ioWeight = 20;
        };

        iso-runner = {
          mounts = [
            {
              name = "iso";
              hostPath = "/data_1/iso/";
              mountPoint = "/var/garuda/buildiso";
            }
            {
              name = "cache";
              hostPath = "/data_1/cache/iso-runner";
              mountPoint = "/var/garuda/buildiso/cache";
            }
            {
              name = "pacman_cache";
              hostPath = "/data_1/cache/pacman-cache";
              mountPoint = "/var/cache/pacman/pkg";
            }
          ];
          needsDocker = true;
        };

        web-front = {
          mounts = [
            {
              name = "acme";
              hostPath = "/data_1/containers/web-front/acme";
              mountPoint = "/var/lib/acme";
            }
            {
              name = "nginx";
              hostPath = "/data_1/containers/web-front/nginx";
              mountPoint = "/var/log/nginx";
            }
          ];
          forwardPorts = [
            {
              containerPort = 22;
              hostPort = 240;
            }
          ];
        };
      };
    };
  };

  systemd.services = garuda-lib.mkTunnels (
    [
      (mkTailnetProxy {
        name = "nginx-tailnet-proxy";
        container = "web-front";
        port = mon.ports.nginxExporter;
        description = "Expose web-front nginx exporter to Tailnet only";
      })
      {
        name = "loki-relay";
        bind = mon.bridge.address;
        listen = mon.stormwingLokiRelay.listenPort;
        target = mon.stormwingLokiRelay.target;
        after = [ "tailscaled.service" ];
        description = "Relay stormwing container logs to Loki on aerialis";
      }
    ]
    ++ map (
      p:
      mkTailnetProxy {
        name = "node-exporter-${p.name}-tailnet-proxy";
        container = p.name;
        inherit (p) port;
        targetPort = mon.ports.nodeExporter;
        description = "Expose ${p.name} node exporter to Tailnet only";
      }
    ) mon.stormwingNodeProxies
    ++ map (
      p:
      mkTailnetProxy {
        name = "service-${p.name}-tailnet-proxy";
        inherit (p) container;
        inherit (p) port;
        inherit (p) targetPort;
        description = "Expose ${p.container} ${toString p.targetPort} to Tailnet only";
      }
    ) mon.stormwingServiceProxies
  );

  networking.firewall.interfaces."tailscale0".allowedTCPPorts = [
    mon.ports.nginxExporter
    32041
    39252
  ];

  sops.secrets."compose/github-runner" = { };
}

Shared host configuration

The disk layout, SSH configuration, container bridge (br0) and impermanence are shared with aerialis and live in nixos/modules/special/newgen.nix. Hardware-specific settings (kernel, microcode, SMT, smartd) come from nixos/modules/special/hetzner-ex44.nix.

_: {
  # We want to have same UID's in all containers to allow sharing home directories
  garuda-lib.unifiedUID = true;

  fileSystems."/" = {
    device = "none";
    fsType = "tmpfs";
    options = [
      "defaults"
      "size=50%"
      "mode=755"
    ];
  };

  fileSystems."/data_1" = {
    device = "/dev/disk/by-label/NIXROOT";
    fsType = "ext4";
    neededForBoot = true;
    options = [
      "defaults"
      "noatime"
      "nodiratime"
      "errors=remount-ro"
    ];
    depends = [ "/" ];
  };

  fileSystems."/data_2" = {
    device = "/dev/disk/by-label/NIXDATA";
    fsType = "btrfs";
    options = [
      "defaults"
      "noatime"
      "nodiratime"
      "compress=zstd:1"
    ];
  };

  fileSystems."/boot" = {
    device = "/dev/disk/by-label/NIXBOOT";
    fsType = "vfat";
  };

  fileSystems."/nix" = {
    device = "/data_1/nix";
    fsType = "none";
    options = [ "bind" ];
    depends = [
      "/data_1"
    ];
  };

  services.openssh = {
    ports = [ 666 ];
    hostKeys = [
      {
        type = "ed25519";
        path = "/data_1/persistent/etc/ssh/ssh_host_ed25519_key";
      }
      {
        type = "rsa";
        bits = 4096;
        path = "/data_1/persistent/etc/ssh/ssh_host_rsa_key";
      }
    ];
  };

  services.garuda-nspawn = {
    bridgeInterface = "br0";
    hostInterface = "eth0";
    hostIp = "10.0.5.1";
    defaults = {
      maxMemorySoft = 48318382080; # 45 GiB
      maxMemoryHard = 53687091200; # 50 GiB
      maxCpu = 18;
    };
  };

  networking.firewall.trustedInterfaces = [ "br0" ];

  garuda-lib.sshkeys = {
    ed25519 = "/data_1/persistent/etc/ssh/ssh_host_ed25519_key";
    rsa = "/data_1/persistent/etc/ssh/ssh_host_rsa_key";
  };

  environment.persistence."/data_1/persistent" = {
    enable = true;
    hideMounts = true;
    directories = [
      "/home"
      "/var/cache/netdata"
      "/var/cache/tailscale"
      "/var/lib/netdata"
      "/var/lib/nixos"
      "/var/lib/tailscale"
      "/var/lib/vnstat"
      "/var/log"
    ];
    files = [
      "/etc/machine-id"
    ];
  };

  security.sudo.extraConfig = ''
    Defaults lecture = never
  '';
}

Containers/services

arch-mirror: Mirrors the Arch Linux repositories and pushes them to Cloudflare R2.
chaotic-v4: Main Chaotic-AUR builder and repository sync container.
firedragon-runner: CI runner for building and testing the Firedragon browser.
github-runner: GitHub Actions runner for CI/CD tasks related to Garuda Linux projects.
gitlab-runner: GitLab CI runners for building our packages.
iso-runner: Dedicated builder for Garuda Linux ISO images.
web-front: Reverse proxy and web frontend for services running on stormwing.

See the respective documentation pages for up-to-date configuration and details.

arch-mirror (stormwing)

This container mirrors the Arch Linux repositories.

General

The garuda-arch-mirror service syncs the mirror from its configured upstream and serves it through nginx. Synced packages are additionally pushed to Cloudflare R2 with rclone, so users can also fetch them from there. The mirror directory is shared with chaotic-v4, which serves it as /srv/http/arch-mirror as well.

Nix expression

{
  config,
  garuda-lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda = garuda-lib.mkMonitoring {
    host = "stormwing";
    units = [
      "garuda-arch-mirror.service"
      "nginx.service"
    ];
  };

  services.garuda-arch-mirror = {
    enable = true;
    upstreamUrl = "rsync://f.matthieul.dev/mirror/archlinux/";
    lastupdateUrl = "https://f.matthieul.dev/mirror/archlinux/lastupdate";
    tls = false;
    rcloneConfig = config.sops.secrets."cloudflare/r2_rclone".path;
    rcloneDest = "r2:/mirror/arch";
  };

  sops.secrets = garuda-lib.mkSecrets [ "cloudflare/r2_rclone" ];

  system.stateVersion = "25.05";
}

chaotic-v4 (stormwing)

This container is the main Chaotic-AUR builder and repository sync node, responsible for building and distributing packages.

General

This is the nspawn container used to run Chaotic-AUR’s new build system, infra 4.0.

Restarting the Docker stack, in case it is needed, can happen via sudo chaotic-restart. For information on how to use the new build system, please refer to the documentation.

In general, manual intervention should not be needed, as the system is designed to be fully automated via GitLab CI or GitHub actions.

Nix expression

{
  config,
  garuda-lib,
  sources,
  pkgs,
  ...
}:
let
  wrapperScript = pkgs.writeScriptBin "chaotic-restart" ''
    echo "Restarting Chaotic-AUR containers..."
    systemctl restart compose-runner-chaotic-v4.service
    echo "Done."
  '';
in
{
  imports = sources.defaultModules ++ [
    ../../modules
    "${sources.chaotic-portable-builder}/nix/nixos.nix"
    ../../modules/special/iso-upload.nix
    ../../modules/special/ssh-allow-chaotic.nix
  ];

  garuda =
    garuda-lib.mkMonitoring {
      host = "stormwing";
      units = [
        "compose-runner-chaotic-v4.service"
        "nginx.service"
        "rsyncd.service"
        "syncthing.service"
      ];
    }
    // {
      services.compose-runner.chaotic-v4 = {
        envfile = config.sops.secrets."compose/chaotic-v4".path;
        source = ../../../compose/chaotic-v4;
        extraEnv = {
          "REDIS_SSH_HOST" = garuda-lib.dns.aerialis;
          "REDIS_SSH_PORT" = "270";
        };
      };
    };

  networking.extraHosts = "10.0.5.40 builds.garudalinux.org";

  # Allow controlling infra 4.0's containers without root
  environment.systemPackages = [ wrapperScript ];
  security.sudo.extraRules = [
    {
      users = [ "xiota" ];
      commands = [
        {
          command = "${wrapperScript}/bin/chaotic-restart";
          options = [ "NOPASSWD" ];
        }
      ];
    }
  ];

  # Expose raw /proc for podman
  systemd.services.expose-raw-proc = {
    description = "Expose clean /proc for podman";
    wantedBy = [ "multi-user.target" ];
    serviceConfig.Type = "oneshot";
    script = ''
      mkdir /tmp/raw_proc
      ${pkgs.mount}/bin/mount --bind /proc /tmp/raw_proc
    '';
  };

  networking.firewall.allowedTCPPorts = [
    config.services.rsyncd.port # Rsync
    8384 # Syncthing web interface
  ];

  # Enable the user accounts of chaotic maintainers
  garuda-lib.chaoticUsers = true;

  # Syncthing setup
  services.syncthing = {
    enable = true;
    openDefaultPorts = true;
    configDir = config.services.syncthing.dataDir;
    cert = config.sops.secrets."keypairs/syncthing/cert".path;
    key = config.sops.secrets."keypairs/syncthing/private".path;
    overrideFolders = false;
    overrideDevices = false;
    user = "root";
    group = "chaotic-op";
    settings = {
      gui = {
        apikey = "garudalinux";
        insecureSkipHostcheck = true;
        inherit (garuda-lib.secrets.syncthing.esxi-build.credentials) user password;
      };
    };
    guiAddress = "10.0.5.10:8384";
  };

  # Auto reset syncthing stuff
  systemd.services.syncthing-reset = {
    serviceConfig.Type = "oneshot";
    script = ''
      "${pkgs.curl}/bin/curl" -X POST -H "X-API-Key: garudalinux" http://10.0.5.10:8384/rest/db/override?folder=${garuda-lib.secrets.syncthing.folders.chaotic-aur}
    '';
  };
  systemd.timers.syncthing-reset = {
    wantedBy = [ "timers.target" ];
    timerConfig.OnCalendar = [ "hourly" ];
  };

  # This disables HTTPS certificates and forced redirects
  garuda-lib.behind_proxy = true;

  # Nginx
  services.nginx = {
    enable = true;
    virtualHosts = {
      "builds.garudalinux.org" = {
        extraConfig = ''
          # Disable index.html
          index fully_disabled.html;
          # Our beautiful autoindex theme
          autoindex on;
          autoindex_exact_size off;
          autoindex_format xml;

          # Security
          add_header X-XSS-Protection          "1; mode=block" always;
          add_header X-Content-Type-Options    "nosniff" always;
          add_header Referrer-Policy           "no-referrer-when-downgrade" always;
          add_header Content-Security-Policy   "default-src 'self' http: https: data: blob: 'unsafe-inline'; frame-ancestors 'self' https://aur.chaotic.cx;" always;
          add_header Permissions-Policy        "interest-cohort=()" always;

          # Locations
          location ~* ^.+\.log {
              default_type text/plain;
          }
          #location ~* /repos/(chaotic-aur|garuda)/x86_64/(?!.*(chaotic-aur|garuda)\.(db|files)).+\.tar.* {
          #    return 301 https://cf-builds.garudalinux.org$request_uri;
          #    expires 2d;
          #}
          location /api/ {
              proxy_pass http://127.0.0.1:8080/api/;
          }
          location /backend/ {
              proxy_pass http://10.0.5.30:3000/;
          }
          location /logs/ {
              proxy_pass http://127.0.0.1:8080/;
              proxy_buffering off;
              proxy_read_timeout 330s;
          }
          location / {
              xslt_string_param path $uri;
              xslt_string_param hostname "Chaotic-AUR main node - Stormwing 🐉";
              xslt_stylesheet "${garuda-lib.xslt_style}";
              location /iso {
                  expires 2d;
                  return 301 https://iso.builds.garudalinux.org$request_uri;
              }
              location = /repos/.stignore {
                  return 404;
              }
              location /repos/.stfolder {
                  return 404;
              }
          }
        '';
        http3 = true;
        root = "/srv/http/";
      };
      "cf-builds.garudalinux.org" = {
        /*
          extraConfig = ''
            location ~* /repos/(chaotic-aur|garuda)/x86_64/(?!.*(chaotic-aur|garuda)\.(db|files)).+\.tar.* {
                add_header Cache-Control "max-age=150, stale-while-revalidate=150, stale-if-error=86400";
            }
            location ~* /repos/(chaotic-aur|garuda)/x86_64/(chaotic-aur|garuda)\.db.* {
                add_header Cache-Control 'no-cache';
            }
            location /repos/chaotic-aur {
                expires 5m;
                error_page 403 =301 https://builds.garudalinux.org$request_uri;
                error_page 404 =301 https://builds.garudalinux.org$request_uri;
            }
            location /repos/garuda {
                expires 5m;
                error_page 403 =301 https://builds.garudalinux.org$request_uri;
                error_page 404 =301 https://builds.garudalinux.org$request_uri;
            }
            location / {
                expires 2d;
                return 301 https://builds.garudalinux.org$request_uri;
            }
          '';
          root = "/srv/http/";
        */
        extraConfig = ''
          location / {
              return 301 https://builds.garudalinux.org$request_uri;
              expires 2d;
          }
        '';
        http3 = true;
      };
      "iso.builds.garudalinux.org" = {
        extraConfig = ''
          autoindex on;
          autoindex_format xml;
          xslt_string_param path $uri;
          xslt_string_param hostname "Garuda Linux ISO Builds";
          # Used by garuda-os-reset
          location = /iso/latest/garuda/listeditions/ {
            autoindex_format json;
            alias "/srv/http/iso/latest/garuda/";
            break;
          }
        '';
        locations."/".return = "307 https://builds.garudalinux.org";
        locations."/iso" = {
          root = "/srv/http/";
          extraConfig = ''
            xslt_stylesheet "${garuda-lib.xslt_style}";
            if ($symlink_target_rel != "") {
              rewrite ^ https://$server_name/iso/$symlink_target_rel redirect;
            }
            if ($arg_sourceforge) {
              rewrite ^/iso/(.*)$ https://sourceforge.net/projects/garuda-linux/files/$1? permanent;
            }
            if ($arg_r2) {
              set $args "";
              rewrite ^/iso/(.*)$ https://r2.garudalinux.org/iso/$1?r2request permanent;
            }
            break;
          '';
        };
      };
    };
  };

  # Rsyncd
  services.rsyncd = {
    enable = true;
    settings = {
      sections = {
        chaotic = {
          "read only" = "yes";
          comment = "Chaotic-AUR repository";
          exclude = "/.stignore /.stfolder/*** /chaotic-aur/archive/*** /garuda/archive/***";
          path = "/srv/http/repos/";
        };
        chaotic-minimal = {
          "read only" = "yes";
          comment = "Chaotic-AUR repository minus largest packages";
          exclude = "/.stignore /.stfolder/*** /chaotic-aur/archive/*** /garuda/archive/*** /chaotic-aur/x86_64/quartus* /chaotic-aur/x86_64/unrealtournament4* /chaotic-aur/x86_64/urbanterror*";
          path = "/srv/http/repos/";
        };
        iso = {
          path = "/srv/http/iso/";
          comment = "ISO downloads";
          "read only" = "yes";
        };
        arch-mirror = {
          path = "/srv/http/arch-mirror/";
          comment = "Arch Linux mirror";
          "read only" = "yes";
        };
      };
      globalSection = {
        "max connections" = 80;
        "max verbosity" = 3;
        "transfer logging" = true;
        "use chroot" = false;
        gid = "nobody";
        uid = "nobody";
      };
    };
  };

  # Push chaotic to r2 hourly automatically
  services.garuda-rclone.chaotic = {
    src = "/srv/http/repos/";
    dest = "r2:/mirror/repos";
    config = config.sops.secrets."cloudflare/r2_rclone".path;
    args = "--s3-upload-cutoff 5G --s3-chunk-size 4G --fast-list --s3-no-head --s3-no-check-bucket --ignore-checksum --s3-disable-checksum -u --use-server-modtime --delete-during --delete-excluded --include /*/x86_64/*.pkg.tar.zst --include /*/lastupdate --order-by modtime,ascending --stats-log-level NOTICE";
    startAt = "hourly";
  };
  systemd.services.chaotic-rclone-inotify = {
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    wants = [ "network-online.target" ];
    # Get all file changes, upload pkg.tar.zst. Not more than 5 per 5 seconds queued and only one uploaded at the same time. Queue dropped if uploading takes longer than 15 seconds.
    # This prevents the queue from getting overloaded with nonsense requests if that ever were to happen. The hourly sync should take care of this.
    script = ''
      upload() {
        operation="''${1%%|*}"
        path="''${1#*|}"
        relative="$(realpath --relative-to="." "$path")"
        relative="''${relative#./}"
        destpath="r2:/mirror/$relative"
        if [ "$operation" != "MOVED_FROM" ]; then
        ${pkgs.flock}/bin/flock -w 30 /tmp/chaotic-rclone-inotify.lock \
          ${pkgs.rclone}/bin/rclone copyto "$path" "$destpath" --s3-upload-cutoff 5G --s3-chunk-size 4G --s3-no-head --no-check-dest --s3-no-check-bucket --ignore-checksum --s3-disable-checksum --config "${
            config.sops.secrets."cloudflare/r2_rclone".path
          }" --stats-one-line -v
        else
          ${pkgs.flock}/bin/flock -w 30 /tmp/chaotic-rclone-inotify.lock ${pkgs.rclone}/bin/rclone deletefile "$destpath" --s3-no-head --no-check-dest --s3-no-check-bucket --config "${
            config.sops.secrets."cloudflare/r2_rclone".path
          }" --stats-one-line -v
          (
            ${pkgs.flock}/bin/flock -w 200 -s 200
            ${pkgs.curl}/bin/curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_GARUDALINUX_ORG/purge_cache" -H "Authorization: Bearer $CF_CACHE_API_TOKEN" -H "Content-Type:application/json" --data "{\"files\":[\"https://r2.garudalinux.org/''${relative}\"]}"
            sleep 0.5
          ) 200>/tmp/chaotic-rclone-inotify-invalidate.lock
        fi
      }
      export -f upload
      ${pkgs.inotify-tools}/bin/inotifywait -m ./repos/*/x86_64 -e CLOSE_WRITE,MOVED_TO,MOVED_FROM --format "%e|%w%f" | \
        ${pkgs.gawk}/bin/awk '/\.pkg\.tar\.zst$/ { print $0; fflush(); }' | \
        xargs -rP 0 -I % ${pkgs.bash}/bin/bash -c 'upload "%"'
    '';
    serviceConfig = {
      EnvironmentFile = config.sops.secrets."cloudflare/api_keys".path;
      Restart = "always";
      WorkingDirectory = "/srv/http";
    };
  };

  sops.secrets = garuda-lib.mkSecrets [
    "cloudflare/api_keys"
    "cloudflare/r2_rclone"
    "compose/chaotic-v4"
    "keypairs/syncthing/cert"
    "keypairs/syncthing/private"
  ];

  system.stateVersion = "25.05";
}

Docker containers

services:
  chaotic-builder-1:
    image: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager:latest
    container_name: chaotic-builder
    command: builder
    deploy:
      restart_policy:
        condition: always
        delay: 60s
    tty: true
    environment:
      BUILDER_CLASS: 9
      BUILDER_HOSTNAME: stormwing-1
      BUILDER_TIMEOUT: 7200
      REDIS_PASSWORD: ${REDIS_PASSWORD:-?err}
      REDIS_SSH_HOST: ${REDIS_SSH_HOST:-?err}
      REDIS_SSH_PORT: ${REDIS_SSH_PORT:-270}
      REDIS_SSH_USER: package-deployer
      SHARED_PATH: /var/garuda/compose-runner/chaotic-v4/shared
      # Override the default database host
      DATABASE_HOST: host.docker.internal
      DATABASE_PORT: 22
    volumes:
      - ./shared:/shared
      - ./sshkey:/app/sshkey
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts: ["host.docker.internal:host-gateway"]
  chaotic-builder-2:
    image: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager:latest
    container_name: chaotic-builder-2
    command: builder
    deploy:
      restart_policy:
        condition: always
        delay: 60s
    tty: true
    environment:
      BUILDER_CLASS: 6
      BUILDER_HOSTNAME: stormwing-2
      BUILDER_TIMEOUT: 7200
      REDIS_PASSWORD: ${REDIS_PASSWORD:-?err}
      REDIS_SSH_HOST: ${REDIS_SSH_HOST:-?err}
      REDIS_SSH_PORT: ${REDIS_SSH_PORT:-270}
      REDIS_SSH_USER: package-deployer
      SHARED_PATH: /var/garuda/compose-runner/chaotic-v4/shared-2
      BUILDER_SRCDEST_CACHE_OVERRIDE: /var/garuda/compose-runner/chaotic-v4/shared/srcdest_cache
      # Override the default database host
      DATABASE_HOST: host.docker.internal
      DATABASE_PORT: 22
    volumes:
      - ./shared-2:/shared
      - ./shared/srcdest_cache:/shared/srcdest_cache
      - ./sshkey:/app/sshkey
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts: ["host.docker.internal:host-gateway"]
  chaotic-builder-3:
    image: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager:latest
    container_name: chaotic-builder-3
    command: builder
    deploy:
      restart_policy:
        condition: always
        delay: 60s
    tty: true
    environment:
      BUILDER_CLASS: 2
      BUILDER_HOSTNAME: stormwing-3
      REDIS_PASSWORD: ${REDIS_PASSWORD:-?err}
      REDIS_SSH_HOST: ${REDIS_SSH_HOST:-?err}
      REDIS_SSH_PORT: ${REDIS_SSH_PORT:-270}
      REDIS_SSH_USER: package-deployer
      SHARED_PATH: /var/garuda/compose-runner/chaotic-v4/shared-3
      BUILDER_SRCDEST_CACHE_OVERRIDE: /var/garuda/compose-runner/chaotic-v4/shared/srcdest_cache
      # Override the default database host
      DATABASE_HOST: host.docker.internal
      DATABASE_PORT: 22
    volumes:
      - ./shared-3:/shared
      - ./shared/srcdest_cache:/shared/srcdest_cache
      - ./sshkey:/app/sshkey
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts: ["host.docker.internal:host-gateway"]
  chaotic-manager:
    image: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager:latest
    container_name: chaotic-manager
    command: database --web-port 8080
    # This is to ensure the database gets the shutdown signal first, and can save all jobs before the
    # builders get killed, which would then potentially end up in an inconsistent state and no saved
    # jobs (when restarting the entire compose runner service).
    depends_on:
      - chaotic-builder-1
      - chaotic-builder-2
      - chaotic-builder-3
    deploy:
      restart_policy:
        condition: always
        delay: 60s
    tty: true
    environment:
      API_TOKEN: ${API_TOKEN:-?err}
      # Address published to outside world
      DATABASE_HOST: builds.garudalinux.org
      DATABASE_PORT: 210
      CI_CODE_SKIP: 123
      DATABASE_USER: package-deployer
      GPG_PATH: /var/garuda/compose-runner/chaotic-v4/gnupg
      LANDING_ZONE_PATH: /var/garuda/compose-runner/chaotic-v4/landing-zone
      LOGS_URL: https://builds.garudalinux.org/logs/logs.html
      REDIS_PASSWORD: ${REDIS_PASSWORD:-?err}
      REDIS_SSH_HOST: ${REDIS_SSH_HOST:-?err}
      REDIS_SSH_PORT: ${REDIS_SSH_PORT:-270}
      REDIS_SSH_USER: package-deployer
      REPO_PATH: /srv/http/repos
      TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN:-?err}
      TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID:-?err}
      PACKAGE_REPOS: >-
        {
            "chaotic-aur": {
                "url": "https://gitlab.com/chaotic-aur/pkgbuilds"
            },
            "garuda": {
                "url": "https://gitlab.com/garuda-linux/pkgbuilds"
            }
        }
      PACKAGE_TARGET_REPOS: >-
        {
            "chaotic-aur": {
                "extra_repos": [
                    {
                        "name": "chaotic-aur",
                        "servers": [
                            "https://builds.garudalinux.org/repos/chaotic-aur/x86_64"
                        ]
                    }
                ],
                "extra_keyrings": [
                    "https://builds.garudalinux.org/repos/chaotic-aur/chaotic-keyring.pkg.tar.zst"
                ]
            },
            "garuda": {
                "extra_repos": [
                    {
                        "name": "garuda",
                        "servers": [
                            "https://builds.garudalinux.org/repos/garuda/x86_64"
                        ]
                    },
                    {
                        "name": "chaotic-aur",
                        "servers": [
                            "https://builds.garudalinux.org/repos/chaotic-aur/x86_64"
                        ]
                    }
                ],
                "extra_keyrings": [
                    "https://builds.garudalinux.org/repos/chaotic-aur/chaotic-keyring.pkg.tar.zst"
                ]
            }
        }
      PACKAGE_REPOS_NOTIFIERS: >-
        {
            "chaotic-aur": {
                "id": "54867625",
                "token": "${GITLAB_TOKEN_CX:-?err}",
                "check_name": "chaotic-aur: %pkgbase%"
            },
            "garuda": {
                "id": "48461689",
                "token": "${GITLAB_TOKEN:-?err}",
                "check_name": "garuda: %pkgbase%"
            }
        }
    volumes:
      - ./sshkey:/app/sshkey
      - /var/run/docker.sock:/var/run/docker.sock
      - /srv/http/repos:/repo_root
    extra_hosts: ["host.docker.internal:host-gateway"]
    ports: ["127.0.0.1:8080:8080", "127.0.0.1:3030:3030"]
  # Automated container updates
  watchtower:
    image: nickfedor/watchtower:latest
    container_name: watchtower
    deploy:
      restart_policy:
        condition: always
        delay: 60s
    command: --cleanup chaotic-builder chaotic-builder-2 chaotic-builder-3 chaotic-manager watchtower --interval 3600
    volumes: ["/var/run/docker.sock:/var/run/docker.sock"]

firedragon-runner (stormwing)

This container is a CI runner for building and testing the Firedragon browser in an isolated environment. It is separate from the other GitLab runner to ensure only one build runs at a time, while the others can run in parallel.

Nix expression

{
  garuda-lib,
  inputs,
  keys,
  ...
}:
{
  # No default modules, untrusted container!
  imports = [
    inputs.sops-nix.nixosModules.sops
    ../../modules/garuda-lib.nix
    ../../modules/hardening.nix
    ../../modules/motd.nix
    ../../services/compose-runner/compose-runner.nix
    ../../services/mk.nix
    ../../services/monitoring
  ];

  inherit
    (garuda-lib.mkUntrustedRunner {
      user = "stefan";
      home = "/home/stefan";
      key = keys.stefan;
      units = [
        "compose-runner-firedragon.service"
        "docker.service"
      ];
      runners = {
        firedragon-runner = {
          source = ../../../compose/firedragon-runner;
        };
      };
    })
    garuda
    nix
    security
    services
    systemd
    users
    virtualisation
    ;

  system.stateVersion = "25.05";
}

Docker containers

services:
  firedragon-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: firedragon-runner
    volumes:
      - ./firedragon-runner:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    restart: 'no'
  firedragon-runner-dind:
    image: gitlab/gitlab-runner:alpine
    container_name: firedragon-runner-dind
    volumes:
      - ./firedragon-runner-dind:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    restart: 'no'

github-runner (stormwing)

This container is a GitHub Actions runner for CI/CD tasks related to Garuda Linux projects.

General

With this container, we provide a GitHub runner as well as (more recently), a GitLab runner. This container does not have the regular Garuda configurations because it is considered untrusted. Access needs to happen by running nixos-container root-login on stormwing.

Restarting containers

This can happen via the following commands:

sudo systemctl restart compose-runner-github-runner
sudo systemctl restart compose-runner-gitlab-runner

Watchtower additionally keeps the containers up to date.

Nix expression

{
  garuda-lib,
  inputs,
  keys,
  ...
}:
{
  # No default modules, untrusted container!
  imports = [
    inputs.sops-nix.nixosModules.sops
    ../../modules/garuda-lib.nix
    ../../modules/hardening.nix
    ../../modules/motd.nix
    ../../services/compose-runner/compose-runner.nix
    ../../services/mk.nix
    ../../services/monitoring
  ];

  inherit
    (garuda-lib.mkUntrustedRunner {
      user = "pedrohlc";
      home = "/home/pedrohlc";
      key = keys.pedrohlc;
      units = [
        "compose-runner-github-runner.service"
        "docker.service"
      ];
      runners = {
        github-runner = {
          envfile = "/var/.github-runner.env";
          source = ../../../compose/github-runner;
        };
        gitlab-runner = {
          source = ../../../compose/gitlab-runner;
        };
      };
    })
    garuda
    nix
    security
    services
    systemd
    users
    virtualisation
    ;

  system.stateVersion = "25.05";
}

Docker containers (GitHub)

services:
  github-runner:
    image: myoung34/github-runner:latest
    container_name: github-runner
    privileged: true
    environment:
      ACCESS_TOKEN: ${ACCESS_TOKEN:-?err}
      EPHEMERAL: true
      LABELS: nyxbuilder
      ORG_NAME: chaotic-cx
      RUNNER_NAME: immortalis
      RUNNER_SCOPE: org
      RUNNER_WORKDIR: /var/cache/github-runner/work
    volumes: ['/var/cache/github-runner/work:/var/cache/github-runner/work']
    restart: 'no'

Docker containers (GitLab)

services:
  gitlab-runner-garuda:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner-garuda
    volumes:
      - /etc/gitlab-runner:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    restart: 'no'
  gitlab-runner-chaotic:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner-chaotic
    volumes:
      - /etc/gitlab-runner/chaotic:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    restart: 'no'
  gitlab-runner-dind:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner-dind
    volumes:
      - /etc/gitlab-runner/dind:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    restart: 'no'
  # Automated container updates
  watchtower:
    image: nickfedor/watchtower:1.22.1
    container_name: watchtower
    command: --cleanup watchtower gitlab-runner-chaotic gitlab-runner-garuda gitlab-runner-dind
    volumes: ['/var/run/docker.sock:/var/run/docker.sock']
    restart: always

gitlab-runner (stormwing)

This container runs GitLab CI runners that build our packages.

General

Two runners are configured on it, stormwing-nix-caching and stormwing-nix-dind. Both make use of the shared Nix cache and are allowed to use KVM, which is why the container has /dev/kvm passed through. The main selling point of these is obviously the shared Nix store, which is the reason why these should in the long run be preferred over the other Docker-based runner.

For these to pick up any builds, the tags nix or nix-dind must be added to the CI configuration.

Its metrics are exposed on port 9252 and published on stormwing’s Tailnet IP for Prometheus (see Monitoring).

Nix expression

{
  config,
  garuda-lib,
  sources,
  ...
}:
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda = garuda-lib.mkMonitoring {
    host = "stormwing";
    units = [
      "docker.service"
      "gitlab-runner.service"
    ];
  };

  networking.firewall.interfaces."eth0".allowedTCPPorts = [ 9252 ];

  services.garuda-gitlab-runner = {
    enable = true;
    runners = {
      stormwing-nix-caching = {
        authenticationTokenConfigFile =
          config.sops.secrets."gitlab-runner/runners/stormwing-nix-caching".path;
        nix-caching.enable = true;
        kvm.enable = true;
      };
      stormwing-nix-dind = {
        authenticationTokenConfigFile = config.sops.secrets."gitlab-runner/runners/stormwing-nix-dind".path;
        nix-caching.enable = true;
        kvm.enable = true;
      };
    };
  };

  sops.secrets = garuda-lib.mkSecrets [
    "gitlab-runner/runners/stormwing-nix-caching"
    "gitlab-runner/runners/stormwing-nix-dind"
  ];

  system.stateVersion = "26.11";
}

iso-runner (stormwing)

This container is a dedicated builder for Garuda Linux ISO images, providing a reproducible build environment.

General

This container is used to build our ISO via a Docker container. It has been used to provide a GitHub runner as well, though this one got moved to its own container recently.

Nix expression

{
  garuda-lib,
  lib,
  pkgs,
  sources,
  ...
}:
let
  # Simple wrapper to dispatch SSH commands to NixOS
  ci-trigger = pkgs.writeShellScriptBin "ci-trigger" ''
    echo $SSH_ORIGINAL_COMMAND
    _FLAVOUR=$(echo "$SSH_ORIGINAL_COMMAND" | cut -d' ' -f2)
    _KERNEL=$(echo "$SSH_ORIGINAL_COMMAND" | cut -d' ' -f3)

    case "$SSH_ORIGINAL_COMMAND" in
      "ci-trigger buildall")
        echo "Ensuring container and garuda-tools are up-to-date.."
        docker exec buildiso pacman -Syu --noconfirm || exit 1
        echo "Building all ISO Garuda currently offers.."
        docker exec buildiso bash -c "umask 0022; buildall" || exit 1
        ;;
      "ci-trigger release "*)
        echo "Ensuring container and garuda-tools are up-to-date.."
        docker exec buildiso pacman -Syu --noconfirm || exit 1
        echo "Building and deploying release ISOs.."
        docker exec buildiso bash -c "umask 0022; buildall; deployiso -SCv" || exit 1
        echo "Deployed all ISOs to the mirror, to push it as latest run 'deployiso -R' on the server after testing!"
        ;;
      "ci-trigger "* )
        echo "Ensuring container and garuda-tools are up-to-date.."
        docker exec buildiso pacman -Syu --noconfirm || exit 2
        echo "Building $_FLAVOUR.."
        docker exec buildiso bash -c "umask 0022; buildiso -i" || exit 2
        [[ $_KERNEL != "" ]] && (docker exec buildiso bash -c "umask 0022; buildiso -p "$_FLAVOUR" -k "$_KERNEL"" || exit 3)
        docker exec buildiso bash -c "umask 0022; buildiso -p "$_FLAVOUR"" || exit 3
        ;;
      *)
        echo "Access only allowed for building purposes!"
        exit 4
    esac
  '';
in
{
  imports = sources.defaultModules ++ [ ../../modules ];

  garuda = garuda-lib.mkMonitoring {
    host = "stormwing";
    units = [
      "buildiso.service"
      "docker.service"
    ];
  };

  # Lets build Garuda ISO here, serving is done via
  # Temeraire already
  services = {
    garuda-iso.enable = true;
    nginx.enable = lib.mkForce false;
    rsyncd.enable = lib.mkForce false;
  };

  # Create a locked down user for GitLab CI who can only access our wrapper
  users.users.gitlab = {
    extraGroups = [ "docker" ];
    isNormalUser = true;
    openssh.authorizedKeys.keys = [
      "restrict,pty,command=\"${ci-trigger}/bin/ci-trigger\"  ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN7W5KtNH5nsjIHBN1zBwEc0BZMhg6HfFurMIJoWf39p"
    ];
  };

  # Let maintainers use buildiso (which is a wrapper around the Docker container)
  # without having to enter a password - our devshell should work just like that
  security.sudo.extraRules = [
    {
      users = [ "frank" ];
      commands = [
        {
          command = "/run/current-system/sw/bin/buildiso";
          options = [ "NOPASSWD" ];
        }
      ];
    }
  ];
  users.users.frank.extraGroups = [ "docker" ];

  system.stateVersion = "25.05";
}

web-front (stormwing)

This container acts as the reverse proxy and web frontend for services running on stormwing, handling HTTPS and routing.

Nix expression

Configuration for the web-front container on stormwing.

{
  garuda-lib,
  sources,
  ...
}:
let
  inherit (garuda-lib) allowOnlyCloudflareZerotrust;
  inherit (garuda-lib) mkCatchAllVhost;
  inherit (garuda-lib) mkProxyVhost;

  vhosts = {
    "builds.garudalinux.org" = mkProxyVhost {
      upstream = "http://10.0.5.10:80";
      prologue = "proxy_buffering off;";
      serverAliases = [
        "cf-builds.garudalinux.org"
        "iso.builds.garudalinux.org"
      ];
      extraLocations = {
        "/logs/" = {
          proxyPass = "http://10.0.5.10:80";
          extraConfig = ''
            proxy_buffering off;
            proxy_read_timeout 330s;
          '';
        };
      };
    };
    "syncthing-build.garudalinux.net" = allowOnlyCloudflareZerotrust {
      extraConfig = ''
        ${garuda-lib.nginxReverseProxySettings}
      '';
      locations = {
        "/" = {
          extraConfig = ''
            proxy_pass http://10.0.5.10:8384;
            proxy_set_header Authorization "Basic ${garuda-lib.secrets.syncthing.esxi-build.credentials.base64}";
          '';
        };
      };
    };
    "_" = mkCatchAllVhost { };
  };
in
{
  imports = sources.defaultModules ++ [ ../../modules ];

  inherit
    (garuda-lib.mkWebFront {
      host = "stormwing";
      inherit vhosts;
    })
    garuda
    networking
    services
    sops
    systemd
    ;

  system.stateVersion = "25.05";
}

Repositories

Notifications for new events at GitLab

Since GitLab has an inbuilt Telegram integration, we can leverage this feature to send notifications to our dedicated Telegram development updates channel. Posts are sent for all kinds of relevant, but non-confidential events like commits, comments or new merge requests. Failed pipelines would also be reported here.

Backing up current repositories

Current repositories may be backed up using ghorg. To use ghorg, one needs a GitLab access token and the application itself. To generate a fitting token, follow these instructions.

ghorg clone --scm gitlab --token "glpat-1234567890" garuda-linux # regular system
nix run nixpkgs#ghorg -- clone --scm gitlab --token "glpat-1234567890" garuda-linux # oneliner on Nix

PKGBUILDs

Types of PKGBUILDs

There are two types of repo packaging-wise:

The ones that have all required files in the new pkgbuilds repo and don’t reference any external repo in PKGBUILDs source()
The ones requiring external repositories as a source. These are listed in the SOURCES files below, packages not listed here are automatically packages of the first category:

This file provides the needed information to check for the new version with the scheme $repourl $pkgbuildPathInPkgbuildsRepo $GitlabProjectId

Releasing a new version

Releasing only needs a new tag or a push - the pipelines (in particular the chaotic-repository-template) take care of the rest.

Packages that have all required files in the pkgbuilds repo: bump pkgver in the PKGBUILD and push. The pipeline picks the change up and deploys the package.
Packages that build from an external repository (listed in the SOURCES file above): push the corresponding tag to that repository (omitting “v”, adding v breaks the PKGBUILD!). Nothing else is needed as long as no packaging changes are required. The half-hourly pipeline of the PKGBUILD repo then detects the new tag, updates the PKGBUILD and deploys the package. If PKGBUILD changes need to be implemented as well, this would of course indicate doing it as described in 1. This would increase pkgrel only and not the actual version.

There are currently two bash scripts responsible for CI/CD:

Past pipeline runs may be reviewed by visiting the pipelines page.

Chaotic-AUR infra 4.0

This is a manual for handling our new Chaotic-AUR infrastructure, which is based on GitLab CI and GitHub Actions. It is powering the garuda repository, which contains all PKGBUILDs and other necessary files to build packages for Garuda Linux. Content has mostly been pasted from the original documentation for visibility.

Reasoning

Our previous build tools, the so-called toolbox was initially created by @pedrohlc to deal with one issue: having a lot of packages to compile while not having many maintainers for all of the packages. Additionally, Chaotic-AUR has quite inhomogeneous builders: servers, personal devices, and one HPC which all need to be integrated somehow. The toolbox had a nice approach to this - keeping things as KISS as possible and using Git to distribute package builds between builders. These would then grab builds according to their activated routines. While this works fairly well, it had a few problems which we tried to get rid of in the new version. A few key ideas about this new setup:

Since we like working with CI a lot besides it providing great enhancement for automating boring tasks as well as making the whole process more transparent to the public as well, it was clear CI should be a major part of it.
The system should have a scheduler that distributes build tasks to nodes, which prevents useless build routines and enables nodes to grab jobs whenever they are queued.
The tools should be available as Docker containers to make them easy to use on other systems than Arch.
All logic besides the scheduler (which is written in TypeScript using BullMQ) should be written in Bash

How it works

The new system consists of three integral parts:

The CI (which can be both GitLab CI and GitHub Actions!) handles PKGBUILDs, their changes, and figuring out what to build, utilizing a Chaotic Manager container to schedule packages via the central Redis instance.
The central Redis instance storing information about currently scheduled builds.
The Chaotic Manager which is used to add new builds to the queue and execute them via the main manager container. All containers have SSH-tunneled access to the Redis instance, enabling the build containers to grab new builds whenever they enter the queue.

Compared to Infra 3.0, this means we have the following key differences:

We no longer have package lists but a repository full of PKGBUILD folders. These PKGBUILDs are getting pulled either from AUR once a package has been updated or updated manually in case a Git repository and its tags serve as a source.
No more dedicated builders (might change in the future, eg. for heavy builds?) but a common build queue.
Routines are no longer necessary - CI determines and adds packages to the schedule as needed. The only “routine-like” thing we have is the CI schedule, executing tasks like PKGBUILD or version updates.
The actual logic behind the build process (like interfere.sh or database management) was moved to the builder container of Chaotic Manager - this one updates daily/on-commit and gets pulled regularly by the Manager instance.
Live-updating build logs will be available via CI - multiple revisions instead of only the latest.
The interfere repo is no longer needed, instead, package builds can be configured via the .CI folder in their respective PKGBUILD folders. All known interfere types can be put here (eg. PKGBUILD.append or prepare.sh), keeping existing interferes working.
The CI’s behavior concerning each package can be configured via a config file in the .CI folder: this file stores information like PKGBUILD source (it can be AUR or something different), PKGBUILD timestamp on AUR, most recent Git commit as well as settings like whether to push a PKGBUILD change back to AUR.
PKGBUILD changes can now be reviewed in case of major (all changes other than pkgver, hashes, pkgrel) updates - CI automatically creates a PR containing the changes for human review.
Adding and removing packages is entirely controlled via Git - after adding a new PKGBUILD folder via commit, the corresponding package will automatically be deployed. Removing it has the opposite effect.

Workflows and information

Adding packages

Adding packages is as easy as creating a new folder named after the $pkgbase of the package. Put the PKGBUILD and all other required files in here. Adding AUR packages is therefore as simple as cloning its repo and removing the .git folder. CI relies on .SRCINFO files to parse most information, therefore, it is important to have them in place and up-to-date in case of self-managed packages. Finally, add a .CI folder containing the basic config (CI_PKGBUILD_SOURCE is required in case its external package, self-managed PKBUILDs don’t need it), commit any changes, and push the changes back to the main branch. Please follow the conventional commit convention while doing so (cz-cli can help with that!). This means commits like:

feat($pkgname): init
fix($pkgname): fix xyz
chore($pkgname): update PKGBUILD
ci(config): update

This not only helps with having a uniform commit history, it also allows automatic changelog generation.

Removing packages

This can be done by removing the folder containing a package’s PKGBUILD. A cleanup job will then automatically remove any obsolete package via the on-commit pipeline run. This will also consider any split packages that a package might produce. Renaming folders does also count as removing packages.

On-commit pipeline

Whenever pushing a new commit, the CI pipeline will carry out the following actions:

Checking when the last scheduled tag was created. This is used to determine which packages need to be scheduled.
The changed files are parsed, which also includes removed packages. Any changed relevant folder content causes a deployment of the corresponding package.
The final action is to build the schedule parameters (handing it over to the scheduled job via artifacts) and remove all obsolete packages in case an earlier step is detected.
In case all of these actions succeed, the scheduled tag gets updated, so we can refer to it on a later pipeline run.

On-schedule pipeline

Half-hourly

Every half an hour, the on-schedule pipeline will carry out a few tasks:

Updating the CI template from the template repository (in case this is enabled via .ci/config)
Check if the scheduled tag does not exist or scheduled does not point to HEAD (in this case abort mission!)
Check whether the .state worktree containing the state of the packages exists, if it does, it sets it up. Otherwise, it re-creates it from scratch (e.g., on force push)
Check whether the last commit is automated (containing “chore(packages): update packages [skip ci]”), if yes, the commit resulting from the schedule will overwrite it to keep the commit history clean.
Collect AUR timestamps of packages to determine whether a PKGBUILD changed
Loop through each valid package and carry out the following actions:
- Read the .CI/config file to gain information about the package configuration (e.g., whether to manage the AUR repository, the source of the PKGBUILD, etc.)
- Update PKGBUILD in the following cases:
  - CI_PKGBUILD_SOURCE is set to gitlab: Updates the PKGBUILD from the GitLab repository tags
  - CI_PKGBUILD_SOURCE is set to aur: Updates the PKGBUILD from the AUR repository, pulling in the git repo and replacing the existing files with the new ones. If the AUR timestamp could not be collected earlier, the package update gets skipped.
  - CI_PKGBUILD_SOURCE is not set to gitlab or aur: tries to update the PKGBUILD by pulling the repository specified in CI_PKGBUILD_SOURCE. In case cloning was not successful after 2 tries, the update process gets skipped.
- In case CI_GIT_COMMIT is set in the packages configuration variables, the latest commit of the git URL set in the source section of the PKGBUILD is updated. If it differs, schedule a build.
- In case a custom hook exists (.CI/update.sh inside the package directory), it gets executed - this can be used for updating PKGBUILDs with a custom script.
- Writing needed variables back to .CI/config (eg. Git hash)
Either update the PKGBUILD silently in case of minor changes, create a PR for review in case of major updates (and only if CI_HUMAN_REVIEW is true)
- Updates are only considered if diff actually reports changes between current PKGBUILD folder and AUR PKGBUILD repo
- Any change made to the source files is detected, this however does not detect malicious changes in the upstream project source that the package builds
The state worktree gets updated with new information
Schedule parameters are getting built and handed over to the scheduled job via artifact
Obsolete branches (eg. merged review PRs) are getting pruned
The scheduled tag gets updated again

Daily

A daily pipeline schedule has been added for specific packages which generate their pkgver dynamically. To make use of it, set CI_ON_TRIGGER=daily inside the .CI/config file of the package.

Manual scheduling

Scheduling packages without git commits

Packages can be added to the schedule manually by going to the pipeline runs page, selecting “Run pipeline” and adding PACKAGES as a variable with the package names as its value. The pipeline will then pick up the packages and schedule them. PACKAGES can also be set to all to schedule all packages. In case one or many packages are getting scheduled, it needs to follow the format pkgname1:pkgname2:pkgname3.

Running scheduled pipelines on-demand

This can be done by going to the pipeline runs page, selecting “Run pipeline” (the play symbol). A link to the pipeline page will be provided, where the pipeline logs can be obtained.

Adding interfere

Put the required interfere file in the .CI folder of a PKGBUILD folder:

prepare: A script that is executed after the building chroot has been set up. It can be used to source environment variables or modify other things before compilation starts.
- If something needs to be set up before the actual compilation process, commands can be pushed by inserting eg. $CAUR_PUSH 'source /etc/profile'. Likewise, package conflicts can be solved, eg. as follows: $CAUR_PUSH 'yes | pacman -S nftables' (single quotes are important because we want the variables/pipes to evaluate in the guest’s runtime and not while interfering)
interfere.patch: a patch file that can be used to fix multiple files when many changes are required. All changes need to be added to this file.
PKGBUILD.prepend: contents of this file are added to the beginning of PKGBUILD. This can be used to set configuration variables.
PKGBUILD.append: contents of this file are added to the end of PKGBUILD. This can be used for all kinds of fixes. To fix build(), include the replacement in this file. To add an item to an array, makedepend+=(somepackage).

To skip build, return $CI_CODE_SKIP. This can be used to conditionally skip builds based on upstream check-in results. See kicad-git for a GitLab example. See openvino-git and scummvm-git for GitHub examples.
on-failure.sh: A script that is executed if the build fails.
on-success.sh: A script that is executed if the build succeeds.

Bumping pkgrel

This is now carried out by adding the required variable CI_PACKAGE_BUMP to .CI/config. See below for more information.

Dependency trees

The CI builds dependency trees automatically. They are passed to the Chaotic manager as a CI artifact and read whenever a schedule command is being executed. No manual intervention is needed.

.CI/config

The .CI/config file inside each package directory contains additional flags to control the pipelines and build processes with.

CI_MANAGE_AUR: By setting this variable to true, the CI will update the corresponding AUR repository at the end of a pipeline run if changes occur (omitting CI-related files)
CI_PACKAGE_BUMP: Controls package bumps for all packages which don’t have CI_MANAGE_AUR set to true. The format this needs to follow is either 1:1.2.3-1/1 (full current version and bump count after the slash) or 1.2.3 (full current package version, resolves to bump count 1).
CI_PKGBUILD_SOURCE: Sets the source for all PKGBUILD-related files, used for pulling updated files from remote repositories. Valid values as of now are:
- gitlab: Pulls the PKGBUILD from the GitLab repository tags. It needs to follow the format gitlab:$PROJECT_ID. The ID can be obtained by browsing the repository settings general section.
- aur: Pulls the PKGBUILD from the AUR repository, pulling in the git repo and replacing the existing files with the new ones.
CI_ON_TRIGGER: Can be provided in case a special schedule trigger should schedule the corresponding package. This can be used to schedule packages daily, by setting the value to daily. Since this checks whether “$TRIGGER == $CI_ON_TRIGGER”, any custom schedule can be created using pipeline schedules and setting TRIGGER to midnight, adding a fitting schedule and setting CI_ON_TRIGGER for any affected package to midnight. Packages having this variable set will not be scheduled via the regular on-schedule pipeline, hence this one can also be used to prevent wasting builder resources, e.g. useful for huge -git packages with a lot of commit activity, like llvm-git.
CI_REBUILD_TRIGGERS: Add packages known to be causing rebuilds to this variable. A list of repositories to track package versions for is provided via the repositories’ CI_LIB_DB parameter. Each package version is hashed and dumped to .ci/lib.state. Each scheduled pipeline run compares versions by checking hash mismatches and will bump each each affected package via CI_PACKAGE_BUMP.
BUILDER_CACHE_SOURCES: Can be set to true in case the sources should be cached between builds. This can be useful in case of slow sources or sources that are not available all the time. Sources will be cleared automatically after 1 month, which is important in case packages are getting removed or the source changes.

Known state variables

State will be kept in the .state worktree. It can be viewed by browsing the state branch of a PKGBUILD repository. Each package will have their own file named after the package name. The following variables are known to be stored:

CI_GIT_COMMIT: Used by CI to determine whether the latest commit changed. Used by fetch-gitsrc to schedule new builds. Needs to be provided in case the package should be treated as a git package. CI will automatically update the latest available commit of the git URL set in the source section of the PKGBUILD. If it differs, schedule a build. -CI_PKGBUILD_TIMESTAMP: The last modified date of the PKGBUILD on AUR. This is used to determine whether the PKGBUILD has changed. If it differs, schedule a build. Will be maintained automatically.

CI pipeline variables

These variables can be set in in the repo root’s.ci/config to configure the pipeline behavior globally as follows:

BUILD_REPO: The target repository that will be the deploy target
GIT_AUTHOR_EMAIL: The email of the user that will be used to commit
GIT_AUTHOR_NAME: The name of the user that will be used to commit
REDIS_SSH_HOST: The Redis SSH host for the target repository (for SSH tunneling)
REDIS_SSH_PORT: The Redis SSH port for the target repository (for SSH tunneling)
REDIS_SSH_USER: The Redis SSH user for the target repository (for SSH tunneling)
REDIS_PORT: The redis port for the target repository (inside the SSH tunnel)
REPO_NAME: The name that this repository is referred to in Chaotic Manager’s config
CI_HUMAN_REVIEW: If merge/pull requests should be created for non pkgver changes
CI_MANAGE_AUR: This should be set to true in case select AUR repositories should be managed by CI
CI_OVERWRITE_COMMITS: If we should overwrite existing automated commits to reduce the size of the git history
CI_CLONE_DELAY: How long to wait between every executed git clone command for rate limits
CI_AUR_PROXY: Proxy to use for AUR requests

Managing AUR packages

AUR packages can also be managed via this repository in an automated way using .CI_CONFIG. This means that after each scheduled and on-commit pipeline, the AUR repository will be updated to reflect the changes done to the PKGBUILD folder’s files. Files not relevant to AUR maintenance (e.g. .CI folders) will be omitted. The commit message reflects the fact that the commit was created by a CI pipeline and contains the link to the source repository’s commit history and the pipeline run which triggered the update commit.

Updating the CI’s scripts

This is done automatically via the CI pipeline. Once changes have been detected on the template repository, all files will be updated to the current version.

Issues and pipeline failures

Last on-commit pipeline failed

This can happen in case of a few reasons, for example having provided an invalid package name. This causes the scheduled tag to not be updated. In this case, the on-schedule pipeline will not be able to run. The last on-commit pipeline needs to be fixed before the on-schedule pipeline can run again. Build failures however are not accounted as the scheduled tag would be updated already as soon as the scheduling parameters were generated. Force pushing a fixed up commit is actively encouraged in such a case, as pushing another commit will cause the CI to evaluate the previous commits it missed, leading to noticing the same issue again and bailing out instead of silently continuing. This has been a design decision to prevent failures from being overlooked.

Resetting the build queue

There might be rare cases in which a reset of the build queue is needed. This can be done by shutting down the central Redis instance, removing its dump, and restarting its service.

Deploying to different repos using the same infrastructure

This is now an officially supported use case. The only thing required is to use another repository that is going to store PKGBUILDs and execute CI pipelines. The environment variables passed to the main Chaotic Manager instance control which repositories are available to use while scheduling packages. See below for more information.

Chaotic Manager

This tool is distributed as Docker containers and consists of a pair of manager and builder instances.

Manager: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager
- Manages builds by adding them to the schedule, used e.g. in the schedule step of CI pipelines
- Provides log management and the live-updating logs
- Manages any existing builds by spinning up build containers, picking from the available BullMQ builder / database queues
- Picks up already built package archives from the landing zone (builder containers push finished build archives here) to add them to the database of the target repository
Builder: registry.gitlab.com/garuda-linux/tools/chaotic-manager/builder
- This one contains the actual logic behind package builds ( seen here) known from infra 3.0 like interfere.sh, database.sh etc.
- This one is used by an executing manager instance to run the build processes with. It runs jobs present in the builder BullMQ queue.

An example of a valid config can be found in the Garuda Linux infrastructure repository. The following variables can be set in Docker environment:

DATABASE_HOST: database address published to the outside world
DATABASE_PORT: the port behind packages can be deployed to
DATABASE_USER: the user to use to deploy packages
GPG_PATH: where the .gnupg folder resides (holding the key for signing packages)
LANDING_ZONE_PATH: where the landing zone is (here packages get deployed and later picked up by the database job before getting into the final repository)
LOGS_URL: the URL that serves the logfiles (we get sent here when clicking CI’s external stages)
PACKAGE_REPOS_NOTIFIERS: needed configs to provide external CI stages for GitLab CI/GitHub Actions
PACKAGE_REPOS: the source repositories containing PKGBUILD folders
PACKAGE_TARGET_REPOS: the repository a package is getting deployed to (including its URL and extra keyrings/repos needed)
REDIS_PASSWORD: password for accessing the Redis instance
REDIS_SSH_HOST: where to access the Redis instance
REDIS_SSH_USER: the user who can access the Redis instance
REPO_PATH: the path where the final package deployment happens
TELEGRAM_BOT_TOKEN: the token for the Telegram bot, used for notifications
TELEGRAM_CHAT_ID: the chat ID for the Telegram bot to send deployment or failure notifications to

The following variables are only relevant for builder instances:

BUILDER_HOSTNAME: the hostname of the builder will be displayed in package logs to determine which builder built a package
BUILDER_TIMEOUT: the timeout for a package build, 3600 seconds by default. Should be increased on slow builders

Setting up

Requirements

The base requirements for running this kind of setup are as follows:

Docker/Podman must be installed in the target system, docker-/podman-compose are good to have as well. We will use it in our following examples.
A Redis instance must be available, e.g. installed on the host system or added to ´docker-compose.yml`:
```
chaotic-redis:
  image: redis:alpine
  container_name: chaotic-redis
  restart: always
  ports:
    - "6379:6379"
  command: redis-server --save 60 1 --loglevel warning --requirepass verysecurepassword
  volumes:
    - ./redis-data:/data
```
The following examples assume Redis to be installed on the host system. In case it is added to docker-compose.yml, replace any occurances of host.docker.internal with chaotic-redis.
A reverse proxy like Nginx to expose the Chaotic Manager’s logs to the public in a secure way should be available. E.g., using Nginx it is sufficient to proxy_pass the specified --web-port value to the Manager instance container. Additionally, the following settings might be usedful:
```
proxy_buffering off;
proxy_read_timeout 330s;
```

Repository setup

The repository needs to be derived from the repository template. On GitHub, the “Use this template” feature may be used. Afterward, customize the .ci/config file according to your needs. This file contains global configuration for pipeline runs and CI behaviour. The following options exist as of today:

BUILD_REPO: The target repository that will be the deploy target
GIT_AUTHOR_EMAIL: The email of the user that will be used to commit
GIT_AUTHOR_NAME: The name of the user that will be used to commit
REDIS_SSH_HOST: The redis host for the target repository
REDIS_SSH_PORT: The redis port for the target repository
REDIS_SSH_USER: The redis user for the target repository
REDIS_PORT: The redis port for the target repository
REPO_NAME: The name that this repository is referred to in chaotic-manager’s config
CI_HUMAN_REVIEW: Whether merge/pull requests should be created for non pkgver changes (false/true)
CI_MANAGE_AUR: This should be set to true in case select AUR repositories should be managed by CI. A fitting SSH key needs to be deployed as AUR_KEY via secret CI variable.
CI_OVERWRITE_COMMITS: Whether we should overwrite existing automated commits to reduce the size of the git history ( false/true)
CI_CLONE_DELAY: How long to wait between every executed git clone command for ratelimits (false/true)
CI_AUR_PROXY: Proxy to use for AUR requests
CI_LIB_DB: Archlinux / Chaotic-AUR repo mirror to use for pulling db files from, in the following format: https://arch.mirror.constant.com/core/os/x86_64/core.db https://arch.mirror.constant.com/community/os/x86_64/community.db ...

Exemplary manager instance setup

chaotic-manager:
  image: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager:latest
  container_name: chaotic-manager
  command: database --web-port 8080
  environment:
    DATABASE_HOST: sub.domain.tld
    DATABASE_PORT: 22
    DATABASE_USER: package-deployer
    GPG_PATH: /var/awesome-repo/gnupg
    LANDING_ZONE_PATH: /var/awesome-repo/landing-zone
    LOGS_URL: https://sub.domain.tld/logs/logs.html
    REDIS_PASSWORD: verysecurepassword
    REDIS_SSH_HOST: host.docker.internal
    REDIS_SSH_USER: package-deployer
    REPO_PATH: /srv/http/repos
    TELEGRAM_BOT_TOKEN: 1234567890
    TELEGRAM_CHAT_ID: 0987654321
    PACKAGE_REPOS: >-
      {
          "awesome-repo": {
              "url": "https://gitlab.com/awesome-repo/pkgbuilds"
          }
      }
    PACKAGE_TARGET_REPOS: >-
      {
          "awesome-repo": {
              "extra_repos": [
                  {
                      "name": "awesome-repo",
                      "servers": [
                          "https://sub.domain.tld/awesome-repo/x86_64"
                      ]
                  }
              ],
              "extra_keyrings": [
                  "https://sub.domain.tld/awesome-repo/awesome-keyring.pkg.tar.zst"
              ]
          }
      }
    PACKAGE_REPOS_NOTIFIERS: >-
      {
          "awesome-repo": {
              "id": "123456",
              "token": "GITLABAPITOKENWITHAPIACCESS",
              "check_name": "awesome-repo: %pkgbase%"
          }
      }
  volumes:
    - ./sshkey:/app/sshkey
    - /var/run/docker.sock:/var/run/docker.sock
    - /srv/http/repos:/repo_root
  extra_hosts:
    - "host.docker.internal:host-gateway"
  ports: [8080:8080]

The following things are to note:

PACKAGE_REPOS, PACKAGE_TARGET_REPOS and PACKAGE_REPOS_NOTIFIERS are JSON values and need to be valid JSON in order to be processed.
The above setup assumes the docker-compose.yml to be present in var/awesome-repo.
LOGS_URL needs to match the address which the reverse proxy publishes --web-port 8080 to the outside world.
REPO_PATH is the path of the repository on the Docker host. The same path must be mapped to /repo_root inside the container via volumes.
/app/sshkey is assumed to be the private SSH key
Ports don’t have to explicitly exposed if using an Nginx Docker container, in this setup however, our Nginx and Redis instance are present on the host system.
PACKAGE_REPOS_NOTIFIERS and TELEGRAM_* variables are optional but provide additional functionality of they are set.
DATABASE_HOST refers to the address published to the outside world, e.g. for additional builders an other servers.

Examplary builder instance setup

---
services:
  chaotic-builder:
    image: registry.gitlab.com/garuda-linux/tools/chaotic-manager/manager:latest
    container_name: chaotic-builder
    command: builder
    environment:
      BUILDER_TIMEOUT: 7200
      BUILDER_HOSTNAME: awesome-builder
      REDIS_PASSWORD: verysecurepassword
      REDIS_SSH_HOST: host.docker.internal
      REDIS_SSH_USER: package-deployer
      SHARED_PATH: /var/chaotic/shared
      DATABASE_HOST: host.docker.internal
      DATABASE_PORT: 22
    volumes:
      - ./shared:/shared
      - ./sshkey:/app/sshkey
      - /var/run/docker.sock:/var/run/docker.sock
    extra_hosts:
      - "host.docker.internal:host-gateway"

The following things are to note:

The above setup assumes the docker-compose.yml to be present in var/awesome-repo.
The SHARED_PATH variable needs to match the directory mapped to /shared inside the container.
DATABASE_HOST can in theory be any other host, but can be set to host.docker.internal in case the Redis instance runs on the Docker host.
The Docker socket needs to be mounted as the builder instance will use it to spin up build container instances.
/app/sshkey is assumed to be the private SSH key used for pushing finished package builds to the manager instance’s landing zone.
BUILDER_TIMEOUT only needs to be set in case it is a slower build machine which does not finish heaver tasks in one hour.
As many instances of this container can be added the setup as wanted. Each of them will allow processing another build at the same time in total.

Features

Chaotic-Manager container commands

schedule: Schedules a new package build by adding it to the Redis instance. It takes the following arguments:
- arch: The architecture to build the package for
- target-repo: The target repository to deploy the package to, referring to the PACKAGE_TARGET_REPOS variable set in the Docker environment variables.
- source-repo: The source repository to pull the package from, referring to the PACKAGE_REPOS variable set in the Docker environment variables.
- commit: The commit hash which the schedule call originates from
- deptree: the dependency tree built by the CI pipeline. This parameter is omitted in CI pipelines and instead passed as file, reading from /.ci/deptree.txt. The reason is that the parameter will be to huge to be processed by the shell if 100+ packages are scheduled at the same time. It contains information about the build order of packages and their dependencies.
builder: Starts the build job, which then grabs any available build jobs from the build queue.
auto-repo-remove: Removes obsolete packages from the target repository. Further parameters must include the pkgbases to be removed.
database: Starts the manager instance, which is responsible for managing queues, logs and database jobs. It additionally spins up a web server to serve logs from if --web-port is passed as argument.
web: Starts the web server to serve logs from. This is only needed in case the manager instance does not run the web server.

Web server

Available routes on the port set up be the --web-port parameter are as follows:

/api/logs/:id/:timestamp: Returns the log file of a package build. The id is the package’s ID, the timestamp is the timestamp of the build.
/api/logs/:id: Returns the latest log file of a package build. The id is the package’s ID.
/api/queue/stats: Returns a JSON object containing the current queue stats.
/api/queue/packages: Returns a JSON object containing information the currently scheduled packages.
/metrics: Returns collected Prometheus metrics.

Notifications

Notifications about relevant events can be sent to a Telegram channel or chat via a Bot. This requires a valid Bot token and the Chat ID to be set. The following events are currently supported:

Build failures: additionally contains links to full build logs and the originating commit.
```
🚨 Failed deploying to awesome-repo:
> freecad-git - logs- commit
```

Build success:

📣 New deployment to awesome-repo:
> freecad-git

Timed out build: Contains links to full build logs and the originating commit.

⏳ Build for awesome-repo failed due to a timeout:
> freecad-git - logs - commit

Successful repo-remove jobs:

✅ Repo-remove job for awesome-repo finished successfully

Failed repo-remove jobs:

🚫 Repo-remove job for awesome-repo failed

Build order

The build order is determined by the dependency tree built by the CI pipeline. This tree is passed to the manager and is then used to determine the correct build order automatically. No further intervention is needed to achieve this.

Live-updating logs

Logs are live-updating and can be viewed in real-time via the web server. In case GitLab is used and PACKAGE_REPOS_NOTIFIERS is set, an external CI stage will be created for every package scheduled during the CI run, linking to the log.

Prometheus metrics

Prometheus metrics are available at the /metrics endpoint of the web server. Currently, we collect default prom-client metrics as well as statistics about total event count of each build status (failed, successful, already-built, timed out) as well as metrics about overall build times. These can be collected via a Prometheus instance and then be visualized using Grafana.

Discourse

Discourse is the application we use to host our forum.

Documentation

Building it

The documentation is created by using mdBook, which generates Markdown files and generates HTML pages for them. The documentation can be build by running:

nix build .#docs # plain simple

The files can then be found at ./result/, which is a symlink to the corresponding path in /nix/store. mdBook is also able to automatically serve the current content and update it automatically whenever a change is detected. This makes testing and previewing content easy.

mdbook serve --open # the latter additionally opens the website in a browser

Useful information

mdBook syntax

While the general syntax for writing Markdown applies to mdBook, it has several extensions beyond the standard CommonMark specification.

Especially importing code blocks as Markdown is really handy to keep content always up-to-date and helps providing a full text searchable code documentation.

Updating mdBook plugins contents

Some of the mdBook parts are plugins that need their content to be updated from time to time. Namely, thats:

mdbook-admonish: run mdbook-admonish inside the docs folder
mdbook-emojicodes: works without CSS, so no updates needed
mdbook-catppuccin: follow instructions in the mdbook-catppuccin repository

Deployment

Deployment to Cloudflare pages automated and happens whenever a commit to main occurs. A GitHub actions workflow builds and pushes it to the cf-pages branch, which will then be used by the Cloudflare pages app to deploy the new version from.

{ {#include ../../../.github/workflows/pages.yml}}

Issues and their solution

Chances are that the custom CSS parts need to be rebased to a newer version. They can be found in ./docs/theme/css and the only addition we made here is to use the Fira Sans font instead of the default one. To rebase against a newer version comment out dditional-css in ./docs/book.toml and move the css folder somewhere else temporarily. After that, run mdbook build inside the docs folder. The new CSS files can now be found inside the ./docs/book/css folder. Copy those to the ./docs/theme/css folder and alter the occurrences of font settings to include Fira Sans (or run a diff to find out where). After uncommenting additional-css in book.toml, run mdbook build again to verify nothing got broken along the way.

Monitoring

Our monitoring stack is self-hosted and lives in nixos/services/monitoring. It replaced the previous Netdata setup, which kept only a few GB per host and offered no central alerting.

Components

Component	Purpose	Port
Prometheus	Metric storage and alert evaluation	9090
Alertmanager	Alert routing (Telegram)	9093
Grafana	Dashboards	3010
Loki	Log storage	3030
node_exporter	Host and container metrics	3021
Fluent Bit	Ships container/host logs to Loki	-
smartctl_exporter	Disk health	9633
borgmatic_exporter	Backup age, size and archive count	9996
nginx_exporter	Web frontend metrics	9113
redis_exporter	Chaotic-AUR Redis	9121
postfix_exporter	Mail queue and delivery	9154
dovecot (OpenMetrics)	Mail auth/IMAP/delivery metrics	9900
GitLab runner metrics	CI runner	9252
cloudflared metrics	Tunnel metrics	20241
cloudflare_exporter	Cloudflare zone/worker analytics	9333

Everything runs in the monitoring container on aerialis (10.0.5.100). Hosts and containers opt in with garuda.monitoring; the port numbers above are all defined once in garuda-lib.monitoring.ports.

Not every endpoint belongs to a dedicated exporter: Dovecot is scraped through its built-in stats listener, which serves OpenMetrics on port 9900. Endpoints like this are declared with garuda.monitoring.prometheus.applicationTargets, which also covers the postfix and redis exporters.

Prometheus retention is bounded by garuda.monitoring.prometheus.retentionTime and retentionSize. Loki is bounded by garuda.monitoring.loki.retentionPeriod.

How the pieces reach each other

Both hosts run their own 10.0.5.0/24 container bridge, so the monitoring container cannot reach stormwing’s containers at all. Stormwing therefore publishes them on its own Tailnet IP:

each stormwing container’s node_exporter is proxied on 31000 + last octet of the container IP (garuda-lib.monitoring.stormwingNodeProxies)
the GitLab runner and web-front cloudflared metrics are proxied on 39252 and 32041 respectively (stormwingServiceProxies)
stormwing container logs are shipped to the bridge address and relayed by the loki-relay unit to aerialis:3030, where loki-tailnet-proxy forwards them into the monitoring container

The Chaotic-AUR mirrors are scraped over the Tailnet using MagicDNS short names, which is why the monitoring container runs its own tailscaled instead of going through a proxy on the host.

Grafana

Dashboards live in nixos/services/monitoring/dashboards and are provisioned into the main organization. Datasource uids are pinned, because the dashboards reference them by uid.

Fly.io’s managed Prometheus is provisioned as an extra datasource in the main organization, authenticated with the grafana/fly_token sops secret.

Grafana has no provisioning support for organizations, and it refuses to start when provisioning targets one that does not exist yet. The grafana-organizations unit therefore creates any missing organization listed in nixos/services/monitoring/grafana.nix through the Admin API.

Alerting

Alert rules live in nixos/services/monitoring/prometheus-rules/ (node-exporter, nginx-exporter, postgres-exporter, smartctl-exporter, borgmatic and self-monitoring rules). Alertmanager delivers firing and resolved alerts to Telegram as of right now. It can of course be extended to other channels as well.

nspawn containers share the host kernel, so kernel-global metrics (/proc/meminfo, /proc/vmstat, /proc/stat, /sys, …) are identical inside a container and its host. Rules backed by those metrics are tagged scope = "host", and Alertmanager inhibits the duplicate alert from the node-*-containers jobs while the host’s own alert keeps firing. Container-local rules (filesystem, systemd, network, conntrack) deliberately carry no such label, so a genuine container-only problem is still reported.

Adding a host or container

Set garuda.monitoring on the host or container. garuda-lib.mkMonitoring also adds the units to the the MOTD and into Fluent Bit’s log collection:

garuda = garuda-lib.mkMonitoring {
  host = "aerialis";
  units = [ "nginx.service" ];
  exporters = [ "nginxExporter" ];
};

exporters adds exporters on top of the always-enabled node_exporter. hostInfo = true additionally exports NixOS build and version information, which feeds the “Host fleet” panel, and is only set on the two hosts themselves.

Tailscale

Our current access policies look as follows:

// This tailnet's ACLs are maintained in https://gitlab.com/garuda-linux/infra-nix
{
	// Define access control lists for users, groups, autogroups, tags,
	// Tailscale IP addresses, and subnet ranges
	"acls": [
		// All servers can connect to each other, use exit nodes and oracle-dragon as DNS
		{
			"action": "accept",
			"src":    ["tag:infra"],
			"dst":    ["tag:infra:*", "autogroup:internet:*", "100.86.102.115:*"],
		},
		// Tailscale admins can access every device
		{
			"action": "accept",
			"src":    ["autogroup:admin"],
			"dst":    ["*:*"],
		},
		// Shared out nodes can be accessed on SSH / Mosh ports
		{
			"action": "accept",
			"src":    ["autogroup:shared"],
			"dst":    ["*:22,222-230,666,60000-61000"],
		},
		// Let the chaotic nodes connect to chaotic-v4's Redis (build distribution)
		{
			"action": "accept",
			"src":    ["tag:chaotic-node"],
			"dst":    ["100.75.227.149:22,6379"],
		},
		// Monitoring (Prometheus scrapes exporters on infra, fluent-bit pushes to Loki)
		{
			"action": "accept",
			"src":    ["tag:monitoring"],
			"dst":    ["tag:infra:3021,9113,9633,9996,31010,31020,31030,31040,31050,31060,31070,32041,39252"],
		},
		{
			"action": "accept",
			"src":    ["tag:infra"],
			"dst":    ["tag:monitoring:3030"],
		},
		// Mirrors: Prometheus scrapes their exporters, they push logs to Loki
		{
			"action": "accept",
			"src":    ["tag:monitoring"],
			"dst":    ["tag:mirror:3021,9113,9633,9996"],
		},
		{
			"action": "accept",
			"src":    ["tag:mirror"],
			"dst":    ["tag:monitoring:3030"],
		},
	],
	// Current infra maintainers
	"groups": {
		"group:admins": ["dr460nf1r3@github", "JustTNE@github"],
	},
	// Define a tag to use as destinations
	"tagOwners": {
		// Admins may apply the "infra" tag
		"tag:infra":        ["group:admins"],
		"tag:chaotic-node": ["group:admins"],
		"tag:monitoring":   ["group:admins"],
		"tag:mirror":       ["group:admins"],
	},
}

Contributor Covenant 3.0 Code of Conduct

Our Pledge

We pledge to make our community welcoming, safe, and equitable for all.

We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant.

Encouraged Behaviors

While acknowledging differences in social norms, we all strive to meet our community’s expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language.

With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including:

Respecting the purpose of our community, our activities, and our ways of gathering.
Engaging kindly and honestly with others.
Respecting different viewpoints and experiences.
Taking responsibility for our actions and contributions.
Gracefully giving and accepting constructive feedback.
Committing to repairing harm when it occurs.
Behaving in other ways that promote and sustain the well-being of our community.

Restricted Behaviors

We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct.

Harassment. Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop.
Character attacks. Making insulting, demeaning, or pejorative comments directed at a community member or group of people.
Stereotyping or discrimination. Characterizing anyone’s personality or behavior on the basis of immutable identities or traits.
Sexualization. Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community.
Violating confidentiality. Sharing or acting on someone’s personal or private information without their permission.
Endangerment. Causing, encouraging, or threatening violence or other harm toward any person or group.
Behaving in other ways that threaten the well-being of our community.

Other Restrictions

Misleading identity. Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions.
Failing to credit sources. Not properly crediting the sources of content you contribute.
Promotional materials. Sharing marketing or other commercial content in a way that is outside the norms of the community.
Irresponsible communication. Failing to responsibly present content which includes, links or describes any other restricted behaviors.

Reporting an Issue

Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm.

When an incident does occur, it is important to report it promptly. To report a possible violation, [NOTE: describe your means of reporting here.]

Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution.

Addressing and Repairing Harm

[NOTE: The remedies and repairs outlined below are suggestions based on best practices in code of conduct enforcement. If your community has its own established enforcement process, be sure to edit this section to describe your own policies.]

If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident’s impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped.

Warning
1. Event: A violation involving a single incident or series of incidents.
2. Consequence: A private, written warning from the Community Moderators.
3. Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations.
Temporarily Limited Activities
1. Event: A repeated incidence of a violation that previously resulted in a warning, or the first incidence of a more serious violation.
2. Consequence: A private, written warning with a time-limited cooldown period designed to underscore the seriousness of the situation and give the community members involved time to process the incident. The cooldown period may be limited to particular communication channels or interactions with particular community members.
3. Repair: Examples of repair may include making an apology, using the cooldown period to reflect on actions and impact, and being thoughtful about re-entering community spaces after the period is over.
Temporary Suspension
1. Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation.
2. Consequence: A private written warning with conditions for return from suspension. In general, temporary suspensions give the person being suspended time to reflect upon their behavior and possible corrective actions.
3. Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted.
Permanent Ban
1. Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member.
2. Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior.
3. Repair: There is no possible repair in cases of this severity.

This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community.

Scope

This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event.

Attribution

This Code of Conduct is adapted from the Contributor Covenant, version 3.0, permanently available at https://www.contributor-covenant.org/version/3/0/.

Contributor Covenant is stewarded by the Organization for Ethical Source and licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/

For answers to common questions about Contributor Covenant, see the FAQ at https://www.contributor-covenant.org/faq. Translations are provided at https://www.contributor-covenant.org/translations. Additional enforcement and community guideline resources can be found at https://www.contributor-covenant.org/resources. The enforcement ladder was inspired by the work of Mozilla’s code of conduct team.

Privacy policy for Garuda Linux

About this document

This Privacy Policy governs the manner in which Garuda Linux collects, uses, maintains and discloses information collected from users (each, a “User”) of our website and web services..

What information do we collect?

We collect information from you when you register on our site and gather data when you participate in the forum by reading, writing, and evaluating the content shared here.

When registering on our site, you may be asked to enter your name and e-mail address. You may, however, visit our site without registering. Your e-mail address will be verified by an email containing a unique link. If that link is visited, we know that you control the e-mail address. Your IP address will be checked against a database of known spammers to prevent such actions.

If you contact us directly, we may receive additional information about you such as your name, email address, the contents of the message and/or attachments you may send us, and any other information you may choose to provide.

When registered and posting, we record the IP address that the post originated from. We also may retain server logs which include the IP address of every request to our server, which will be purged after 30 days.

What do we use your information for?

Any of the information we collect from you may be used in one of the following ways:

To provide, operate, and maintain our infrastructure
To allow using our services that require a login, as well as to provide convenience features such as staying logged in or keeping personally chosen settings.
To send periodic emails that are generated by our services such as the forum, which may however be turned off if desired.

We have no interest in your data and only store the minimum needed to operate the services we provide to our users.

How do we protect your information?

We implement a variety of security measures to maintain the safety of your personal information when you enter, submit, or access your personal information.

What is your data retention policy?

We will make a good faith effort to:

Retain server logs containing the IP address of all requests to this server no more than 90 days.
Retain the IP addresses associated with registered users and their posts no more than 5 years.

Third Party Privacy Policies

Garuda Linux’s Privacy Policy does not apply to some of the services we utilize in our infrastructure. Thus, we are advising you to consult the respective Privacy Policies of these third-party services for more detailed information.

This includes, but may not be limited to:

Cloudflare to protect against common threats and enhance our infrastructure
Hetzner as server and backup storage provider (located in Germany)
Google Translate to offer translations on our website
OpenCollective, Liberapay and Paypal to allow the collection of donations that sustain our infrastructure

Cookies

Our Site may use “cookies” to enhance User experience. User’s web browser places cookies on their hard drive for record-keeping purposes and sometimes to track information about them. The user may choose to set their web browser to refuse cookies or to alert you when cookies are being sent. If they do so, note that some parts of the Site may not function properly.

We do not sell, trade, or rent User’s personal identification information to others.

How long do we retain your data

If you leave a comment, the comment and its metadata are retained indefinitely. This is so we can recognize and approve any follow-up comments automatically instead of holding them in a moderation queue.

For users that register on our website (if any), we also store the personal information they provide in their user profile. All users can see, edit, or delete their personal information at any time (except they cannot change their username). Website administrators can also see and edit that information.

Embedded content from other websites

Articles on this site may include embedded content (e.g. videos, images, articles, etc.). Embedded content from other websites behaves in the exact same way as if the visitor has visited the other website.

These websites may collect data about you, use cookies, embed additional third-party tracking, and monitor your interaction with that embedded content, including tracing your interaction with the embedded content if you have an account and are logged in to that website.

Free software

Garuda Linux develops free software. All our tools are and will always be free software. Garuda Linux is part of OIN since November 2020. The current license can be viewed here. Additional information about packages covered by this license can be viewed here.

If you want to check the license of a package, you can do so with Pacman.

What rights you have over your data

If you have an account on this site or have left comments, you can request to receive an exported file of the personal data we hold about you, including any data you have provided to us. You can also request that we erase any personal data we hold about you. This does not include any data we are obliged to keep for administrative, legal, or security purposes.

Children’s Information

Another part of our priority is adding protection for children while using the internet. We encourage parents and guardians to observe, participate in, and/or monitor and guide their online activity.

Garuda Linux does not knowingly collect any personal identifiable information from children under the age of 13. If you think that your child provided this kind of information on our website, we strongly encourage you to contact us immediately and we will do our best efforts to promptly remove such information from our records.

Changes to This Privacy Policy

We may update our Privacy Policy from time to time. Thus, we advise you to review this page periodically for any changes. We will notify you of any changes by posting the new Privacy Policy on this page. These changes are effective immediately, after they are posted on this page.

Your acceptance of these terms

By using this Site, you signify your acceptance of this policy. If you do not agree to this policy, please do not use our services. Your continued use of them following the posting of changes to this policy will be deemed your acceptance of those changes.

Contact Us

If you have any questions about this Privacy Policy, the practices of this site, or your dealings with this site, please contact us via email.

This privacy policy has been updated in September 2023.

Security Policy

If any vulnerability or security flaw is discovered, please contact us directly via [email protected].

We will try to respond within 24–48 hours on a best-effort basis.

Keyboard shortcuts

Garuda's infra documentation