Feature Specification: Secret Rotation

Created: 2026-10-22

Status: Partially implemented

Input: Reconstructed from the code at 1f4c46e3d (nix/contracts/nixos/modules/services/security/vars-openbao.nix, core/components/hosting-provider/openbao/, core/setups/wiring/postgresql-roster.nix, core/components/operator/app-resources.nix, core/resources/secrets/README.md) and the generators that the table below names.

User Scenarios & Testing (mandatory)

A deployment has three kinds of secrets:

Engine credentials: an OpenBao secrets engine holds them and changes them on a schedule. The consumers read them through openbao-agent.
Generated secrets: a generateFiles instance makes them once and uploads them to OpenBao KV v2, at secret/<prefix>/{secret,public}/<instance>/<file>. The upload does not write over a value that exists (vars-openbao.nix). Thus a generated secret changes only when an operator runs upload-vars-openbao-force <instance>[/<file>], or when the KV entry is lost and the next deploy makes a new value.
Stored secrets: a person encrypts them in the repository with agenix (the secrets resource), or keeps them outside the deployment. A person rotates them by hand, as core/resources/secrets/README.md ("Rotating") tells.

Some generated secrets encrypt data that the offsite backups hold. A backup restores only with the key that encrypted it.

User Story 1 - Know how each secret rotates (Priority: P1)

An operator or an auditor wants to know, for each secret, how it changes and which backups need it. This specification is that list.

Why this priority: a rotation of a key that a backup needs makes that backup unreadable. An operator must know this before a rotation.

Independent Test: the table in Key Entities agrees with the generators and the engine roles that it names.

Acceptance Scenarios:

Given a secret in the table, When an operator reads its row, Then the row tells the rotation (schedule, manual or never), the backups that depend on it, and how the consumers get the new value.

User Story 2 - Rotate the engine credentials on a schedule (Priority: P1)

The static roles of the postgres, valkey and LDAP engines rotate every 24h. openbao-agent renders the new value and reloads or restarts the consumer.

Why this priority: a long-lived database or directory password that leaks stays usable for at most one rotation period.

Independent Test: core/checks/integrations/postgresql/postgres-rotation.nix, and test-valkey-service.

Acceptance Scenarios:

Given a postgres static role, When the engine rotates it, Then the agent renders the new password and restarts the reloadUnits of the consumer, and the consumer connects with the new password.
Given the authelia-bind LDAP role, When a bind fails with code 49, Then the LDAP engine unit rotates the role (rotate-role).

User Story 3 - Restore a backup after a change of a key that it depends on (Priority: P2)

The keys that a backup depends on do not rotate. An operator can change such a key by hand, for example by a force upload. The backups from before the change must then stay usable for as long as the retention keeps them. KV keeps the old value, thus the operator gets it back with bao kv rollback.

Why this priority: a restore with a different key gives data that the application cannot read, and the error comes only later, when the application reads the data.

Independent Test: test-mastodon-service changes a key in KV, checks that the restore stops and names the version of the key, does a rollback of the key, and checks that the restore works.

Acceptance Scenarios:

Given a backup from before a key change, When the operator restores it, Then the restore stops before it changes data, and names each key that is different and the KV version that the backup needs.
Given the operator did a rollback of each key to the version that the backup needs, When the operator restores again, Then the restore works.

Requirements (mandatory)

Functional Requirements

FR-001: An engine static role MUST rotate on the schedule of its roster (24h now). openbao-agent MUST render the new value and reload or restart the units that the consumer names.
FR-002: A generateFiles upload MUST NOT write over a value that exists in KV. Only upload-vars-openbao-force changes a generated secret.
FR-003: A secret that encrypts data in a backup MUST NOT change on a deploy, a redeploy or a recreate of its node.
FR-004: The repository password of each restic backup MUST be a generated secret. A change MUST keep the old password as previousPasswordFile until the repository has the new key (mkPasswordMigration, nix/contracts/nixos/modules/services/backup/restic.nix).
FR-005: A key that a backup depends on MUST NOT rotate on a schedule. Each backup MUST record a digest and the KV version of each such key. A restore MUST stop on a snapshot whose keys are different from the keys of the node, and MUST name the KV version that the snapshot needs.

Key Entities

"force" means: the secret changes only with upload-vars-openbao-force. "agent" means: openbao-agent renders the value and runs try-reload-or-restart on the reloadUnits.

Secret	Source	Rotation	Backups that depend on it	Consumer refresh
postgres static roles, `postgresql/static-roles/<role>`	`openbao/database-init.nix`	every 24h	none	agent
valkey `default` user, `database/static-roles/<namespace>`	`openbao/valkey-engine.nix`; the first value is the `valkey-<namespace>` generated secret	every 24h	none	agent; the instance applies the password when it starts
LDAP `authelia-bind`	`openbao/ldap-engine.nix`	every 24h, and on a bind error 49	none	agent
postgres admin, `postgresql-admin/password`	`core/resources/postgresql/default.nix`	force	none	agent
lldap admin, `ldap-bind-<group>`	`core/resources/ldap/default.nix`	force	none	`lldap-provision`
lldap seed, `ldap-key-<group>`	`core/resources/ldap/default.nix`	force	lldap database (the OPAQUE password data)	agent
authelia `<instance>`: JWT, `storageEncryptionKey`, HMAC, issuer key, client secrets	`core/resources/oidc/default.nix`	force	authelia database (`storageEncryptionKey`)	the database password reloads `authelia-main`
grafana `secretKey`, `adminPassword`	`core/components/hosting-provider/telemetry/default.nix`	force	`grafana-backups` (`secretKey`)	only `adminPassword` reloads
`mastodon-keys`: `secretKeyBase`, three Active Record encryption keys, VAPID keypair	`core/components/operator/mastodon/default.nix`	force	mastodon database (the encrypted columns)	`mastodon-init-dirs`
pixelfed `pixelfed-app/appKey`	`core/components/operator/pixelfed/default.nix`	force	pixelfed database (values that Laravel encrypts)	agent, then `pixelfed-data-setup`
pixelfed OAuth keypair, `pixelfed-passport/{privateKey,publicKey}`	`core/components/operator/pixelfed/default.nix`	force	none; a change ends the OAuth tokens that it signed	agent; Passport reads the files at each use
peertube `peertube-secret/secret`	`core/components/operator/peertube/default.nix`	force	peertube database (the two-factor secrets)	`peertube.service`
garage S3 keys, `<key>-s3/{accessKey,secretKey}`	`core/resources/garage/mint-keys.nix`	never	all restic and rclone backups (access to the bucket)	agent or `envSecrets`
garage `adminToken`, `rpcSecret`	`core/resources/garage/server.nix`	force	none	`garage.service`
restic `<app>-backups-restic/password`, `<namespace>-cache-backups-restic/password`	`core/components/operator/app-resources.nix`, `core/components/shared/valkey/valkey.nix`	force, with FR-004	the restic repository	agent
media backup password and salt, `<app>-media-backup`	`core/components/operator/app-resources.nix`	force	the media backup (rclone `crypt`)	agent
SMTP submission passwords, smarthost	`core/resources/smtp/maddy-provider.nix`	force	none	agent
DKIM keypair	`core/resources/smtp/maddy-provider.nix`	never	none	agent
TSIG secret	`core/setups/wiring/vars-plumbing.nix`	force	none	agent
DNSSEC CSK	`core/setups/wiring/vars-plumbing.nix`	never (`lifetime unlimited`)	none; the bind backup does not hold it	agent
OpenBao PKI root CA	`openbao/pki-init.nix`	never (TTL 10 years, leaf certificates at most 30 days)	none	n/a
SPIRE CA	`core/components/hosting-provider/spire/spire.nix`	the default `ca_ttl` of SPIRE	none	SPIRE
windmill superadmin, API token	`core/components/hosting-provider/windmill/`	force; the database role every 24h	none	bootstrap unit
netbox `secretKey`, peppers, API token	`core/components/hosting-provider/netbox.nix`	force; the token is made again when it is not valid	none	agent
panel `SECRET_KEY`, `panel-authz/token`	`panel/nix/`	force	none	agent
agenix secrets (forgejo, runner token, security PGP key)	`core/resources/secrets/secrets.nix`	by hand (`core/resources/secrets/README.md`)	none	rebuild
CD Incus client certificate	`notes/cd-credentials.md`	by hand, each 365 days	none	by hand
terraform backend `kmsKey`	`core/components/hosting-provider/terraform-backend/terraform-backend.nix`, on the node	never	the states, and their offsite backup	n/a
age recipients of the bootstrap backups, `fediversity.bootstrapBackupRecipients`	`core/components/hosting-provider/backup-recipients.nix`, `core/keys`	by hand (`core/keys/README.md`)	`unseal-key.age`, `kmsKey.age`	rebuild; the next backup encrypts again
restic password of the bootstrap backups	`openbao/backup.nix`, `terraform-backend/backup.nix`	never (fixed and public)	the raft snapshot, the terraform states	rebuild
OpenBao unseal key	`openbao/openbao.nix`, sealed to the TPM of the node, and in `unseal-key.age`	never	the raft snapshot	n/a
OpenBao root token	`openbao/openbao.nix`, `/var/lib/openbao-init/root-token`	never	none; the snapshot job uses it	n/a

The paths that start with openbao/ are under core/components/hosting-provider/.

Success Criteria (mandatory)

Measurable Outcomes

SC-001: Each secret that a generateFiles instance or an engine role makes has a row in the table.
SC-002: The postgres rotation checks, test-valkey-service and test-mastodon-service pass.

Assumptions

KV v2 keeps older versions of a value. The secret mount has no max_versions, thus the default of OpenBao applies, and bao kv get -version=<n> reads an older version after a force upload.
The backups keep data for the periods of backupRetention in core/components/operator/app-resources.nix.

Known Deviations

The lldap seed cannot change, also not by hand: lldap derives the OPAQUE data of each password from it, thus a new seed makes each password of the live database invalid too.
The garage S3 keys do not rotate.
The DKIM keypair and the DNSSEC CSK do not rotate. The code says that a DKIM rotation is a procedure with two selectors, but no procedure is written down.
The restic password of the bootstrap backups is fixed and public. Age protects only the unseal key and the kmsKey in these backups.
A change of fediversity.bootstrapBackupRecipients does not revoke a recipient that it removes. The backups from before the change hold the unseal key and the kmsKey encrypted to that recipient, and these keys do not rotate.
A raft snapshot restore gives back the engine credentials of the time of the snapshot. The roles that rotated after the snapshot rotate again within seconds, and the consumers get the new values when their agents log in again, which can take about 1h.