Monitoring

Our monitoring stack is self-hosted and lives in nixos/services/monitoring. It replaced the previous Netdata setup, which kept only a few GB per host and offered no central alerting.

Components

Everything runs in the monitoring container on aerialis (10.0.5.100). Hosts and containers opt in with garuda.monitoring; the port numbers above are all defined once in garuda-lib.monitoring.ports.

Not every endpoint belongs to a dedicated exporter: Dovecot is scraped through its built-in stats listener, which serves OpenMetrics on port 9900. Endpoints like this are declared with garuda.monitoring.prometheus.applicationTargets, which also covers the postfix and redis exporters.

Prometheus retention is bounded by garuda.monitoring.prometheus.retentionTime and retentionSize. Loki is bounded by garuda.monitoring.loki.retentionPeriod.

How the pieces reach each other

Both hosts run their own 10.0.5.0/24 container bridge, so the monitoring container cannot reach stormwing’s containers at all. Stormwing therefore publishes them on its own Tailnet IP:

• each stormwing container’s node_exporter is proxied on 31000 + last octet of the container IP (garuda-lib.monitoring.stormwingNodeProxies)
• the GitLab runner and web-front cloudflared metrics are proxied on 39252 and 32041 respectively (stormwingServiceProxies)
• stormwing container logs are shipped to the bridge address and relayed by the loki-relay unit to aerialis:3030, where loki-tailnet-proxy forwards them into the monitoring container

The Chaotic-AUR mirrors are scraped over the Tailnet using MagicDNS short names, which is why the monitoring container runs its own tailscaled instead of going through a proxy on the host.

Grafana

Dashboards live in nixos/services/monitoring/dashboards and are provisioned into the main organization. Datasource uids are pinned, because the dashboards reference them by uid.

Fly.io’s managed Prometheus is provisioned as an extra datasource in the main organization, authenticated with the grafana/fly_token sops secret.

Grafana has no provisioning support for organizations, and it refuses to start when provisioning targets one that does not exist yet. The grafana-organizations unit therefore creates any missing organization listed in nixos/services/monitoring/grafana.nix through the Admin API.

The Grafana role for GitLab logins is mapped from group membership in nixos/services/monitoring/grafana.nix. GitLab access levels are not available to Grafana - its connector only fetches group full paths from /api/v4/groups - so roles cannot be derived from Owner/Maintainer. Matching is exact, so a member of a subgroup that is not listed falls back to Viewer.

Alerting

Alert rules live in nixos/services/monitoring/prometheus-rules/ (node-exporter, nginx-exporter, postgres-exporter, smartctl-exporter, borgmatic and self-monitoring rules). Alertmanager delivers firing and resolved alerts to Telegram as of right now. It can of course be extended to other channels as well.

nspawn containers share the host kernel, so kernel-global metrics (/proc/meminfo, /proc/vmstat, /proc/stat, /sys, …) are identical inside a container and its host. Rules backed by those metrics are tagged scope = "host", and Alertmanager inhibits the duplicate alert from the node-*-containers jobs while the host’s own alert keeps firing. Container-local rules (filesystem, systemd, network, conntrack) deliberately carry no such label, so a genuine container-only problem is still reported.

Adding a host or container

Set garuda.monitoring on the host or container. garuda-lib.mkMonitoring also adds the units to the the MOTD and into Fluent Bit’s log collection:

garuda = garuda-lib.mkMonitoring {
  host = "aerialis";
  units = [ "nginx.service" ];
  exporters = [ "nginxExporter" ];
};

exporters adds exporters on top of the always-enabled node_exporter. hostInfo = true additionally exports NixOS build and version information, which feeds the “Host fleet” panel, and is only set on the two hosts themselves.