Fediversity

Feature Specification: Components telemetry and otel-collector

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/components/hosting-provider/telemetry/default.nix, core/components/hosting-provider/telemetry/oidc.nix, core/components/hosting-provider/telemetry/alert-rules.nix, core/components/hosting-provider/telemetry.nix, core/components/hosting-provider/otel-collector/default.nix, core/components/hosting-provider/otel-collector.nix, core/components/hosting-provider/otel-collector-test.nix, core/components/hosting-provider/otel-wiring-test.nix, core/components/hosting-provider/node-logs.nix, core/components/hosting-provider/node-metrics.nix, core/components/hosting-provider/dashboards/, core/components/hosting-provider/monitoring-test.nix, core/components/hosting-provider/monitoring-test.py, core/setups/hosts-common.nix, core/checks/default.nix, core/checks/ci.nix), pull request #1690.

User Scenarios & Testing (mandatory)

User Story 1 - Collect the signals of the fleet (Priority: P1)

The hosting provider wants the metrics and logs of every node in one place. The otel-collector component declares provides = [ "otel" ]. Each node of a group with a node that provides otel runs two shippers: otel-node-metrics and otel-node-logs. They push over OTLP/HTTP to the collector of the group. The collector exports each signal to its store on the telemetry node: traces to Tempo, metrics to Mimir, logs to Loki.

Why this priority: without the collection, the dashboards and the alert rules have no data.

Independent Test: test-otel-collector-service posts a synthetic OTLP trace and finds the span in the journal through the debug exporter. test-monitoring-storage posts a trace, a metric and a log, and queries each back from its store.

Acceptance Scenarios:

  1. Given a group with a node that provides otel, When the setups evaluate, Then each node imports node-metrics.nix and node-logs.nix with endpoint = "http://<collector>:4318", and the collector node itself uses http://127.0.0.1:4318.
  2. Given the node that provides otel, When the setups evaluate, Then the collector listens on 0.0.0.0:4317 and 0.0.0.0:4318, and the firewall opens both ports.
  3. Given backends, When the collector exports, Then logs go to <endpoint>/otlp/v1/logs, metrics to <endpoint>/otlp/v1/metrics and traces to <endpoint>/v1/traces, each with the Authorization header read from authHeaderFile.
  4. Given a node whose roster network.name is an IPv4 literal, When the setups evaluate, Then both shippers take instanceId from machine.network.name, not the hostname nixos.

User Story 2 - Store the signals in Garage (Priority: P1)

The telemetry node runs Loki, Tempo and Mimir on loopback, with their data in three Garage buckets. nginx on domain gates the ingest with basic auth and routes each OTLP path to its store.

Why this priority: the stores keep the history that the dashboards and the alert rules query.

Independent Test: test-monitoring-storage checks the auth_basic gate of the ingest vhost and queries each signal back.

Acceptance Scenarios:

  1. Given the telemetry-buckets request, When the garage resource applies it, Then the buckets loki, tempo and mimir exist with the tenant prefix, and the key telemetry can read, write and own them.
  2. Given a request to domain without the ingest password, When nginx checks it, Then nginx refuses it.
  3. Given new S3 key files, When the systemd.paths unit sees the change, Then fediversity-telemetry-s3-env writes /run/fediversity-telemetry/s3.env again and runs try-restart on the stores.
  4. Given the default retentionDays = 90, When the stores run, Then Loki, Tempo and Mimir drop data older than 2160h. retentionDays has no value that keeps data without a limit.

User Story 3 - Read dashboards and get alerts (Priority: P2)

An administrator opens Grafana on uiDomain and logs in with OIDC. Grafana shows the provisioned datasources, the dashboards and the alert rules. A rule that fires sends an email.

Why this priority: the alerts tell the administrator about failed backups, expiring certificates and full disks.

Independent Test: test-monitoring-storage checks the datasources, the dashboards and the rules. It drives the rule of a failing restic-backups-* unit to Alerting. No test runs the OIDC login.

Acceptance Scenarios:

  1. Given a user in fediversity-admins, When the user logs in, Then Grafana gives the role Admin. Other users get Viewer.
  2. Given a deployment email in default-configuration, When the setups evaluate, Then alertEmails holds each ;-delimited address of it.
  3. Given no email and no group data alertEmails, When the otelBackend node evaluates, Then the evaluation stops with an error that names ADMIN_EMAIL.
  4. Given a firing rule, When Grafana evaluates it, Then the contact point fediversity-email sends an email through the telemetry-smtp account.

Edge Cases

  • An empty authHeaderFile fails the ExecStartPre guard of the collector. The unit retries with RestartSec=10s and no start limit.
  • Without backends, the collector exports only to debug.
  • An empty SMTP password file fails the guard of Grafana. Grafana retries with RestartSec=30s and no start limit.
  • A new admin password restarts Grafana. A new secretKey does not restart it.
  • node-cert is on only when security.acme.certs is not empty.
  • The windmill OTLP exporters are stubs. Grafana is the only span source.
  • The backup copies the SQLite database of grafana with .backup, in one read transaction, thus grafana continues to write during the backup. The restore stops grafana.service, replaces the database, removes its -wal and -shm files, and starts the service again.
  • A restore needs the secretKey of telemetry-grafana that encrypted the secrets in the database. The key is in OpenBao KV, not in the backup.

Requirements (mandatory)

Functional Requirements

  • FR-001: The otel-collector component MUST request spiffe, declare provides = [ "otel" ] and import otel-collector.nix through code.
  • FR-002: otel-collector.nix MUST declare services.fediversity-otel-collector with listenAddress (default loopback), grpcPort (4317), httpPort (4318), exporters, backends (endpoint, authHeaderFile, caFile) and scrapeTargets.
  • FR-003: Each signal MUST go only to its own store: traces to otlphttp/tempo, metrics to otlphttp/mimir, logs to otlphttp/loki.
  • FR-004: Each entry of scrapeTargets MUST become one scrape_configs job of a prometheus receiver in the metrics pipeline.
  • FR-005: The setups MUST give the collector authHeaderFile through contracts.credentials from vars/secret/telemetry-ingest/authHeader, and caFile from the deployment PKI.
  • FR-006: The setups MUST add the scrape target incus at incus.<domain>:443/1.0/metrics on an Incus deployment.
  • FR-007: node-metrics.nix MUST run otel-node-metrics with the host_metrics receiver every 30s, the systemd and node-cert exporters on loopback, and diagnostics on port 8889.
  • FR-008: node-logs.nix MUST run otel-node-logs with the journald receiver at priority info, key each record on _SYSTEMD_UNIT, map PRIORITY to a severity, and serve diagnostics on port 8890.
  • FR-009: Both shippers MUST set service.instance.id from instanceId and run as DynamicUser. otel-node-logs MUST be in the group systemd-journal.
  • FR-010: The telemetry component MUST set certAudience = "bootstrap", the default subdomain telemetry and uiDomain with default monitoring.<apex>, and request ssl for both names.
  • FR-011: The telemetry component MUST generate telemetry-ingest (htpasswd and authHeader), telemetry-grafana (secretKey, adminPassword) and telemetry-s3 (accessKey, secretKey).
  • FR-012: The telemetry-oidc request MUST use the client ID grafana, the redirect https://<uiDomain>/login/generic_oauth and the scopes openid, profile, email and groups.
  • FR-013: telemetry.nix MUST provision the datasources mimir, loki and tempo, the dashboards in dashboards/ in the folder Fediversity, and the rule group fediversity from telemetry/alert-rules.nix.
  • FR-014: The ingest vhost on domain MUST use basicAuthFile, route /otlp/v1/logs to Loki, /v1/traces to Tempo and /otlp/v1/metrics to Mimir, and set client_max_body_size 0.
  • FR-015: The telemetry node MUST open only port 443. Grafana, Loki, Tempo and Mimir MUST listen on loopback.
  • FR-016: When the group has a node that provides otel, the setups MUST set tracingEndpoint to <collector>:4317.
  • FR-017: The telemetry component MUST declare provides = [ "otelBackend" ]. The setups MUST point the collector exporters at the host of the otelBackend provider, MUST give the otel provider the trust of its CA, and MUST give tracingEndpoint and alertEmails to the otelBackend provider node.
  • FR-018: A group that has no node that provides otel and sets centralTelemetry = true in its group data MUST ship its telemetry to the otel provider of the central group (the one group whose roster provides varsBackend). The hypervisor group sets centralTelemetry. A group that sets neither ships no telemetry.
  • FR-019: The telemetry component MUST request garage through mkBackupsGarageRequest with the bucket grafana-backups, and telemetry/backup.nix MUST add a streamingBackup of the SQLite database of grafana to that bucket with the restic provider, as the user grafana.
  • FR-020: alertEmails MUST have no default. When the deployment email is set, the setups MUST set alertEmails on the otelBackend provider node to the ;-delimited addresses of email. When email is empty, the setups MUST set it from the group data alertEmails, which the alertEmails argument of core/default.nix gives for an in-tree evaluation. When neither gives a value, the evaluation MUST stop with an error that names ADMIN_EMAIL.

Key Entities

  • Ingest credential: telemetry-ingest. The htpasswd gates nginx. The authHeader is Basic base64(otel-collector:<password>).
  • Store buckets: loki, tempo and mimir in Garage, one per store.
  • Alert rules: maddy-failed-dsn, unit-failed, unit-stuck, backup-failed, backup-stale, renewal-failed, cert-expiring, cert-expiring-critical, disk-filling, disk-filling-trend, disk-full, node-absent and ingest-silent.
  • Dashboards: fediversity-infra-overview, fediversity-logs-traces and fediversity-node-detail.
  • Offsite backup: the restic repository grafana_sqlite in the grafana-backups bucket, with the SQLite database of grafana: the users, their preferences, the service accounts, the silences, the annotations and the dashboards that are not provisioned.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: test-otel-collector-service, test-otel-wiring and test-monitoring-storage pass.
  • SC-002: nix-unit-setups-hypervisor-config passes, so the hypervisor ships its telemetry under an actionable instance.
  • SC-003: nix-unit-setups-vars-reload passes, so a new telemetry-grafana/adminPassword restarts Grafana.

Assumptions

  • The garage node of the group serves the buckets (see garage).
  • Authelia serves the OIDC client (see authelia).
  • The app nodes reach the collector over the internal network. The collector receivers have no auth.

Known Deviations

None.