Fediversity

Hosting provider guide

This guide is for hosting providers: people who run the shared services that the deployments of their operators use. It tells how to find and use the web UIs of a hosting provider deployment. To use the panel, see the panel user guide. The concepts page defines the terms that this guide uses.

In this guide, <domain> is the domain of your hosting provider deployment.

Find the web UIs

Open https://<domain>/. This overview page shows a tile for each web UI of the deployment, with its address. The same page is at https://overview.<domain>/. The page itself has no login: each UI that it links to does its own login.

The page shows only the UIs that the deployment deploys. For example, with an external NetBox, it shows no NetBox tile.

Log in

The UIs use single sign-on (SSO), with Authelia at https://auth.<domain>/. The first login asks for the user name and the password from the lldap directory of the deployment.

The deployment makes one administrator account, hosting-provider, if its configuration sets an administrator email address. This account starts without a password:

  1. Open https://users.<domain>/reset-password/step1.
  2. Enter hosting-provider.
  3. Open the link in the email that the deployment sends to the administrator address.
  4. Set a password.

This account is a member of lldap_admin. Authelia cannot set the password of an lldap_admin member, so its "Reset password?" does not work for this account. lldap and Authelia limit how frequently you can ask for a reset. If the deployment has no SMTP server, Authelia writes the email to the file notifications.txt in its state directory, on the authelia node.

Manage users and groups

The users are in lldap, at https://users.<domain>/. Add a user there, and give the user a group to give access:

Group What a member can do
authelia-users Log in through SSO. A user without this group cannot log in to any UI.
fediversity-admins Administer: the Admin role in Grafana, write access to the domain claims in OpenBao, and access to Windmill and to the Valkey UI.
lldap_admin Manage users and groups in lldap.

A new user sets a password with "Reset password?" at https://auth.<domain>/. A member of lldap_admin uses https://users.<domain>/reset-password/step1.

Monitor the deployments

Grafana is at https://monitoring.<domain>/. Log in with "Sign in with Fediversity SSO". Members of fediversity-admins get the Admin role. Other users get the Viewer role.

The "Fediversity" folder has these dashboards:

  • "Infrastructure overview": the state of all nodes.
  • "Node detail": the metrics of one node.
  • "Logs and traces": the logs and the traces of the nodes.

Grafana sends alerts by email to the addresses in services.fediversity-telemetry.alertEmails.

Know which personal data the logs keep

The logs and the telemetry keep personal data for a limited time:

Store Retention Personal data
The journal of each node 30 days, and at most 512 MB Client IP addresses and user names (sshd, the nginx error log, Authelia), email addresses (maddy), and what the applications write to their log.
Loki (logs) services.fediversity-telemetry.retentionDays, 90 days by default A copy of the journal of each node at level info and higher.
Tempo (traces) The same as Loki The spans of Grafana. A span can contain a request path.
Mimir (metrics) The same as Loki None known. The labels identify nodes and units.
The nginx access log of each node 26 weeks of rotated files The network of the client (the first three octets of an IPv4 address, the first three groups of an IPv6 address), the request line with its query string, the referrer, the user agent, and the HTTP basic auth user name.
The audit_event table of api/ services.fediversity-api.auditRetentionDays, 365 days by default. api/ removes the older events when it starts and each time that it records an event. 0 keeps all events. The actor of each action on a deployment: with the panel, the primary key of the panel account. The event also holds the action, the deployment, the result, the time, and the X-Request-ID that the nginx of api/ sets for the request. The table keeps the events of a deleted deployment until they are older than the retention.

The journal keeps the full client IP address. The nginx access log does not, see core/setups/nixos/nginx.nix.

api/ also writes each audit event as one JSON line on the fediversity_api.audit logger. These lines go to the journal of the node, so the retention of the journal and of Loki applies to them, not auditRetentionDays.

OpenBao writes each request and each response as one JSON line to the journal of its node (openbao.service), so the retention of the journal and of Loki applies to these lines. A line holds the client IP address, the path and the operation of the request, and the display name and the entity of the token, for example the name that an OIDC login gives to an administrator. Secret values and tokens are HMAC-SHA256 hashes. To find the hash of a known value, use bao write sys/audit-hash/journal input=<value>.

Know how long the backups keep personal data

All the backups of an application (Mastodon, Pixelfed, PeerTube) keep its data for 30 days. The backups of the databases of lldap, Authelia and the panel keep their data for the same period, because these databases hold the accounts of the users and of the operators. Thus data that one of these services deletes, for example a deleted account, leaves all these backups after the same period. The value is backupRetention.application in core/components/operator/app-resources.nix. A component gives the retention of its database backup in fediversity.postgresBackup.<name>.retention.

Backup Retention Personal data
The database of the application, in its <app>-backups bucket 30 days All that the database holds, for example the accounts, their email addresses, their posts and their messages.
The local storage of PeerTube, in the peertube-backups bucket 30 days The files of PeerTube that are not in the object storage, for example avatars, thumbnails and captions.
The valkey cache of the application, in the cache backups bucket of its group 30 days The cache of the application, for example queued jobs and cached timelines.
The media of the application, in the prefix media/ of its <app>-backups bucket 30 days of history, fediversity.mediaBackup.instances.<app>.retentionDays. The media backup is off until the operator enables it. The uploads of the users, for example images, videos and avatars.
The databases of lldap, Authelia and the panel, in the lldap-backups, authelia-backups and panel-backups buckets 30 days The users of lldap and their email addresses, the sessions, the consents and the second factors of Authelia, and the accounts, the sessions and the deployments of the panel.
The databases of api/, NetBox and Windmill, the database of Grafana, the raft storage of OpenBao, the states of the terraform backend and the zone of BIND 7 daily, 4 weekly and 6 monthly snapshots, thus approximately 6 months For example the audit_event table of api/, the users of Grafana and of NetBox, and the names that OIDC logins give to the administrators in OpenBao.

A daily timer makes each backup. After each backup, restic removes the snapshots that are more than 30 days older than the newest snapshot (restic forget --keep-within 30d), and the media backup removes the history that is older than 30 days. Thus deleted data leaves the backups 30 days after the last backup that holds it, at the next backup. When the backups stop, the expiry stops too, and the backups keep their data until a backup runs again.

Erase personal data

An erasure removes the data from the live service. The backups that were made before the erasure keep the data until they expire. Until then, the data in these backups is out of use:

  • Use a backup only to restore a service, and for no other purpose. Do not read personal data from a backup, and do not copy it to another system.
  • After a restore from a backup that is older than an erasure, the erasure must be done again, see Do the erasures again after a restore.

These erasures remove data from the live services:

Data How to erase it What stays in the backups
An account of Mastodon, Pixelfed or PeerTube The account deletion of the application, by the user or by a moderator. The operator settings ACCOUNT_DELETION and ACCOUNT_DELETE_AFTER of Pixelfed apply to it. The account in the database backups of the application, for 30 days.
An object in a media bucket The application deletes the object. For an erasure request, media-backup-<app> erase <bucket> <key> also removes the object from the media backup and from all of its history at once. Nothing in the media backup after erase. Without erase, the history keeps the object for 30 days.
A user of lldap A member of lldap_admin deletes the user at https://users.<domain>/. The user in the database backups of lldap, for 30 days.
An account of the panel A panel administrator removes the account. The panel first queues the destroy of each deployment of the account. The account in the database backups of the panel, for 30 days.
A deployment The panel, or DELETE /deployments/{id} of api/. The destroy removes the nodes, drops the databases, removes the KV entries, and empties and deletes the buckets of the deployment, its <app>-backups buckets too. Nothing in the buckets of the deployment. The database backups of the panel keep the deployment for 30 days. The database backups of api/ and its audit_event table keep what api/ records about the deployment, for approximately 6 months.

The restic backups (the databases, the local storage of PeerTube and the valkey caches) have no erase command. Thus an erased record stays in them until they expire.

Do the erasures again after a restore

Who does an erasure again depends on who controls the data:

  • The operator controls the data of its deployment, for example the accounts and the media of its applications and the users of its lldap. The operator does the erasures of this data. After a restore of a service of a deployment, tell the operator at once which service you restored and the date of the backup. The operator then does again each erasure that it did after this date.
  • The hosting provider controls the accounts of the panel and the users of its own lldap. After a restore of one of these services, and before the service is open to its users again, do again each erasure that is newer than the backup. The record of the erasure requests that the hosting provider answered tells which erasures to do.

For a deleted deployment, the audit_event table of api/ is the record. The table records each deployment_delete with the deployment, the actor and the time. api/ also writes each event as a JSON line to the journal, and Loki keeps a copy for services.fediversity-telemetry.retentionDays, 90 days by default. A restore of the database of api/ loses the events after the backup, thus read them from Loki.

See the containers

Incus is at https://incus.<domain>/. It shows each node of each deployment as an instance. The name of an instance is <namespace>-<node>, where <namespace> identifies the deployment.

Warning: Incus gives each user who logs in through SSO full administrator access. Incus does not limit its access to a group. Give authelia-users only to people that you trust with all containers.

Read the logs of a deploy

Windmill runs the deploys that the panel starts. It is at https://jobs.<domain>/, for members of fediversity-admins only. In the fediversity workspace, the runs of the scripts f/fediversity/deploy and f/fediversity/destroy show the log of each deploy and each destroy.

Assign a domain to a deployment

When a hosting provider has a registrar account, a deployment can register its domain through it. OpenBao records which deployment has each domain. These records are the domain claims, at vars/public/domain-claims/<domain> in the secret store of OpenBao, at https://secrets.<domain>/. The deployment field of a claim is the namespace of the deployment.

When a deployment deploys a domain:

  • If the domain is free, the deployment writes its claim, and then registers the domain. The registration costs at most the configured maximum price. The deployment does not register a premium domain.
  • If the registrar account has the domain, and the claim names this deployment, the deployment uses the domain.
  • If the registrar account has the domain, and the claim names a different deployment or there is no claim, the deploy fails.

When a deployment is destroyed, its claim is removed, but the registration stays in the registrar account.

To give a domain that the registrar account has to a deployment:

  1. Log in to OpenBao at https://secrets.<domain>/ with OIDC, as a member of fediversity-admins.
  2. Go to secret/vars/public/domain-claims/.
  3. Create or edit the entry of the domain, and set its deployment field to the namespace of the deployment.
  4. Deploy the deployment again.