Fediversity

Feature Specification: Component openbao

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/components/hosting-provider/openbao/default.nix, core/components/hosting-provider/openbao/openbao.nix, core/components/hosting-provider/openbao/loopback-api.nix, core/components/hosting-provider/openbao/pki-init.nix, core/components/hosting-provider/openbao/spire-auth.nix, core/components/hosting-provider/openbao/deployer-policies.nix, core/components/hosting-provider/openbao/cd-oidc.nix, core/components/hosting-provider/openbao/oidc.nix, core/components/hosting-provider/openbao/database-init.nix, core/components/hosting-provider/openbao/valkey-engine.nix, core/components/hosting-provider/openbao/ldap-engine.nix, core/checks/default.nix, core/checks/ci.nix, core/checks/nix-unit-tests.nix), pull requests #1820, #1822, #1853 and #1956, and issues #1759, #1760 and #1817.

User Scenarios & Testing (mandatory)

User Story 1 - Run an unattended secrets backend (Priority: P1)

A hosting provider deploys the openbao component on one node. The node initializes OpenBao, seals the unseal key to the TPM, and unseals itself at every boot. No person types an unseal key. The secret KV v2 mount then holds the vars of the deployment. The component declares provides = [ "varsBackend" "pki" ]: the node is the store of the generateFiles secrets and the CA of the openbao-pki provider of ssl.

Why this priority: every other node reads its secrets from this backend. Without it, no credential reaches any consumer.

Independent Test: integrations-spire-openbao boots the spire and openbao components together. integrations-openbao-pki boots an openbao whose openbao-init seals its unseal key to a TPM, and shows that the audit device writes each request to the journal.

Acceptance Scenarios:

  1. Given a fresh node with a TPM, When openbao-init runs, Then it initializes OpenBao with one key share, seals that key to the TPM under the persistent handle 0x81000001, and enables the secret KV v2 mount.
  2. Given an initialized node that reboots, When openbao-init runs, Then it reads the unseal key back from the TPM and unseals OpenBao.
  3. Given an initialized OpenBao whose sealed key or root token is missing, When openbao-init runs, Then it fails with "the unseal seed is lost" or "the root token is lost and cannot be recovered", and does not initialize again.
  4. Given an unsealed OpenBao, When a client writes and reads a secret, Then the journal of openbao.service holds one JSON request entry and one JSON response entry for each operation, and the secret value is only an hmac-sha256: hash.

User Story 2 - Let each node log in by its SPIFFE identity (Priority: P1)

A node of the deployment presents its SPIFFE X.509 SVID to OpenBao. The openbao-spire-auth unit writes a host-<name> policy per consumer host, and the openbao-spire-cert-auth unit registers a cert auth role per consumer host. The role binds to the SPIFFE ID of that host and attaches its host-<name> policy. The policy grants the KV paths of the vars instances that host owns or reads. For a host in namespacedHosts (a node of a tenant deployment), the KV, valkey and LDAP grants are on the objects of its own deployment only. The policy names that deployment with the cert-auth alias name of the login, which is the deployment namespace in the CN of the SVID.

Why this priority: this login is how every consumer reads its secrets without a shared token.

Independent Test: integrations-spire-openbao. The nix-unit test nix-unit-setups-ldap-grant asserts which cert-auth policy gets read on which group's rotated LDAP bind credential. The nix-unit test nix-unit-components-openbao-spire-auth asserts the policy text and the cert-auth role of each roster entry kind for a small roster.

Acceptance Scenarios:

  1. Given a host in consumerHosts, When openbao-spire-cert-auth runs, Then it writes auth/cert/certs/<role> with allowed_uri_sans set to the SPIFFE ID of the host, token_ttl=1h and token_max_ttl=24h.
  2. Given a host in garageConsumerStems that is on no roster, When the node evaluates, Then an assertion fails and names that host.
  3. Given the SPIRE trust bundle is not on disk yet, When openbao-spire-cert-auth starts, Then ConditionPathExists skips it, openbao-spire-auth logs a warning, and the openbao-spire-auth timer restarts openbao-spire-cert-auth through openbao-spire-auth-refresh (OnBootSec=5m, OnUnitActiveSec=15m).
  4. Given a host that leaves the roster, When openbao-spire-auth and openbao-spire-cert-auth run, Then they delete its host-<name> policy and its cert auth role.
  5. Given an agent on this node whose Workload API fails or returns a bundle with no CA that is valid now, When openbao-spire-ca-bundle runs, Then it keeps the current bundle and the cert auth roles, and exits 0.
  6. Given a host in namespacedHosts that logs in with an SVID whose CN is tenant42, When it writes vars/operator-deployments/tenant42/secret/<instance>/<key>, Then OpenBao accepts the write, and it refuses the same write under tenant43, under vars/secret/ and under secret/<host>/.
  7. Given an agent bundle at the trust_bundle_path that is newer than the current bundle and has no CA in common with the Workload API (the SPIRE server has a new CA), When openbao-spire-ca-bundle runs, Then it polls the Workload API for at most 180 seconds until the agent attests to that CA, and then installs the bundle and re-seeds the cert auth roles. When the agent does not attest in that time, it keeps the current bundle and the roles, and the next run tries again.

User Story 3 - Issue internal TLS certificates (Priority: P2)

A node needs a certificate for node-to-node TLS. When the deployment wires the openbao-pki ssl provider, openbao-pki-init creates an internal root CA and a role that limits issuance to the deployment's domains. OpenBao then serves an mTLS listener on :8200 with a leaf from its own CA.

Why this priority: internal TLS is optional. Public application vhosts use ACME.

Independent Test: integrations-openbao-pki.

Acceptance Scenarios:

  1. Given services.openbao-pki-init.enable, When openbao-pki-init runs, Then it enables the pki mount, writes the role internal, generates the root CA Fediversity Internal CA once, and writes the CA to /var/lib/openbao-init/pki-ca.pem.
  2. Given a config without the pki mount, When openbao-pki-init runs, Then the apply fails on prevent_destroy, and the mount and its root CA stay.
  3. Given a fresh consumer without the CA, When it fetches over plain HTTP on port 8083, Then it gets the CA certificate.
  4. Given the fediversity.openbao-pki option exists, When the node evaluates, Then OpenBao listens on 0.0.0.0:8200 with tls_client_ca_file=/var/lib/openbao-init/spire-ca.pem, and the firewall opens port 8200.

User Story 4 - Mint deployer and database credentials (Priority: P2)

CD, a workstation deploy, and the operator deploys need tokens. Applications need database passwords that rotate. OpenBao mints these through its jwt and oidc auth methods and its database and ldap secrets engines.

Why this priority: deploys and rotated credentials depend on it, but a node can boot and serve without them.

Independent Test: integrations-cd-jwt-login (CD login on the jwt role). deployment-cd-env-oidc (CD and workstation token exchange against a stub OpenBao). test-ldap-engine (rotation of authelia-bind). test-valkey-service (rotation of the valkey app user by openbao-valkey-engine, a second apply that changes nothing, and an optional namespace whose instance does not answer).

Acceptance Scenarios:

  1. Given services.openbao-cd-oidc.issuer and repository are set, When openbao-spire-auth runs, Then it writes the jwt role fediversity-deployer, bound to that repository on refs/heads/main, with a TTL of 30m and a maximum of 2h.
  2. Given a person in the fediversity-admins group, When that person logs in on the oidc-deployer mount, Then the token carries only the fediversity-deployer policy.
  3. Given a role in services.openbao-database-init.roles with mode = "rotating", When openbao-database-init runs, Then it writes a static role, and deletes a dynamic role of the same name.
  4. Given a dynamic role under <mount>/roles/ that services.openbao-database-init.roles does not state as dynamic, When openbao-database-init runs, Then it revokes the leases of that role and deletes it.
  5. Given ldapEngine.enable, When openbao-ldap-engine runs, Then it mounts the ldap engine per directory and rotates the authelia-bind account every 24h.

Edge Cases

  • The TPM is not ready at boot. openbao-init waits for it, and seals the key before bao operator init, so a failed seal never loses the key of an initialized OpenBao.
  • A transient TPM read error occurs at unseal. tpm_unseal_key_retrying retries the read.
  • The storage is the old file backend. The ExecStartPre step openbao-migrate-file-to-raft moves it to raft at /var/lib/openbao/raft.
  • A node of an earlier release has no audit device. restartIfChanged = false keeps openbao.service up on a switch, and a reload reads the old config again, so OpenBao adds the device at the next start of openbao.service.
  • The options of the journal/ audit device in the config differ from the stored device. OpenBao refuses to unseal. To change an option, the config gives the device a new name, and OpenBao then adds the new device and removes the old one.
  • journald does not accept the write of an audit entry. OpenBao refuses the request, or waits until journald accepts the write.
  • The OIDC issuer is not up yet. openbao-oidc-auth restarts every 30s, and StartLimitIntervalSec = 0 keeps it from start-limit-hit.
  • The OIDC client secret arrives empty. openbao-oidc-auth rejects the empty value and fails the start.
  • The OIDC issuer chains to the private PKI CA. openbao-oidc-auth sets oidc_discovery_ca_pem to the system anchors plus that CA.
  • The postgresql-admin password is not rendered yet. The systemd credential of openbao-database-init fails the start, and Restart= retries it.
  • A shell oneshot of an earlier release created a mount, an auth mount, an identity group or a group alias, and no state holds it. The reconciler writes an import block for it before the apply, so the apply adopts it and does not fail on a create.
  • An object that a shell oneshot of an earlier release created, and that no config names, stays: no state holds it, so no apply deletes it.
  • The valkey node of one required namespace has not published its address. openbao-valkey-engine configures the other namespaces, then fails, so that it starts again.
  • One optional namespace (an operator deployment's) has a password in KV, but no address, or nothing answers at its address. This is also the case when the deployment is gone and its KV records stay. openbao-valkey-engine keeps the objects of that namespace that its state holds, or configures none, and does not fail. The timer openbao-valkey-engine runs the engine again every 2 minutes.
  • The valkey node of one namespace is not deployed, so KV holds no password for it. openbao-valkey-engine configures no objects for that namespace, deletes those that it configured before, and does not fail.
  • The operator deployer token in KV still renews. openbao-operator-deployer-token keeps it and mints no new token.
  • The node or its TPM is lost. The raft snapshots in openbao-backups do not restore, because the unseal key is sealed only to that TPM and no copy of it exists elsewhere. The secrets that only KV holds are then lost: the app encryption keys of mastodon and pixelfed, the DNSSEC and TSIG keys, and the PKI CA. Thus a restore of the app databases cannot decrypt their encrypted columns.

Requirements (mandatory)

Functional Requirements

  • FR-001: The component MUST register as components.openbao with name = "OpenBao", package = pkgs.openbao and certAudience = "bootstrap".
  • FR-002: The component MUST take the shared options with the default subdomain openbao, and the option uiDomain with the default secrets.<apex>.
  • FR-003: When enabled, the component MUST request a SPIFFE identity (see ../../007-resources/spiffe.md), a TLS certificate for domain and uiDomain (see ../../007-resources/ssl.md), an OIDC client openbao (see ../../007-resources/oidc.md), and a nixos-module that imports openbao.nix (see ../../007-resources/nixos-module.md).
  • FR-004: The OIDC request MUST allow the UI callback on both domain and uiDomain, and http://localhost:8250/oidc/callback, with the scopes openid, profile, email and groups.
  • FR-005: OpenBao MUST serve its admin API on the loopback address http://127.0.0.1:8250, and every engine unit MUST use that address as VAULT_ADDR.
  • FR-006: OpenBao MUST use raft storage at /var/lib/openbao/raft with cluster_addr https://127.0.0.1:8251.
  • FR-007: nginx MUST proxy domain and uiDomain to the loopback API. The implementation sets both settings as the options fediversity.openbao.domain and fediversity.openbao.uiDomain that openbao.nix declares.
  • FR-008: openbao-init MUST initialize with one key share, seal the key to the TPM, keep the root token at /var/lib/openbao-init/root-token, unseal, and enable the secret KV v2 mount.
  • FR-009: openbao-init MUST fail, and MUST NOT initialize again, when OpenBao is initialized but the sealed key or the root token is missing.
  • FR-010: When the fediversity.openbao-pki option exists, the node MUST serve an mTLS listener on 0.0.0.0:8200 that trusts the SPIRE CA at /var/lib/openbao-init/spire-ca.pem.
  • FR-011: openbao-pki-init MUST create the pki mount, the root CA and the role internal, limited to allowedDomains, and MUST serve the CA over plain HTTP on caFetchPort.
  • FR-012: openbao-spire-auth MUST write one host-<name> policy per roster host, and openbao-spire-cert-auth MUST write one cert auth role per roster host, bound to the SPIFFE ID of that host, with that policy.
  • FR-013: Each host that garageConsumerStems names MUST be on the roster, also when it owns no instance, so that its policy carries the read grant.
  • FR-014: openbao-spire-cert-auth MUST skip, not fail, when the SPIRE trust bundle is missing, and the openbao-spire-auth timer MUST restart it. openbao-spire-auth MUST NOT wait for the bundle.
  • FR-015: The policies fediversity-deployer and fediversity-operator-deployer MUST come from deployer-policies.nix, the same text the check deployment-tf-incus-operator uses.
  • FR-016: openbao-operator-deployer-token MUST keep a periodic token with the fediversity-operator-deployer policy at vars/secret/deployer/openbaoToken, with a period of 768h, and renew it daily.
  • FR-017: When services.openbao-cd-oidc.issuer is set, OpenBao MUST admit a forge Actions token as fediversity-deployer only for repository on refs/heads/main.
  • FR-018: openbao-oidc-auth MUST configure the oidc auth method with the role operator, map the groups claim value fediversity-admins to the fediversity-admins policy, and grant that policy write only on secret/data/vars/public/domain-claims/*.
  • FR-019: openbao-oidc-auth MUST configure a separate oidc-deployer mount whose role gives only fediversity-deployer, for members of fediversity-admins.
  • FR-020: openbao-database-init MUST configure the postgres connection on services.openbao-database-init.host port 5432 with sslMode = "require", and MUST write a static role for rotating and a dynamic role for dynamic roles.
  • FR-021: openbao-valkey-engine MUST write database/config/valkey-<key> and database/static-roles/<key> for each instance whose password and address are in KV, and MUST delete them for each other instance. For an entry with required = true, <key> is <ns> and KV is at the prefix of the hosting provider. For an entry with required = false, there is one instance per deployment <id> under deploymentVarsRoot that has a password for it, <key> is <ns>-<id>, and KV is at <deploymentVarsRoot>/<id>. For an entry with required = false, it MUST first open a TCP connection to the address. When KV has no address or the connection fails, it MUST keep the objects as its state holds them, MUST NOT fail, and a timer MUST run it again.
  • FR-022: openbao-ldap-engine MUST mount ldap/ for the directory of the central group (the one group whose roster provides varsBackend) and ldap-<group>/<namespace>/ per deployment, and rotate the roleName account every rotationPeriod. It MUST delete the mount of each namespace whose KV under deploymentsKV is gone. It MUST keep a live mount that its state holds while the inputs of that directory are missing, except for a deployment mount. The management-bind password MUST go to OpenBao as a write-only value, so the state holds only a digest of it. An LDAP engine mount with no config MUST be made again, not imported.
  • FR-023: A host-<name> policy MUST read the submission passwords of a relay only when the setup names that relay's instance in varsReadInstances for it. The setup names it for each node of the relay's own group that sends mail: a node whose component requests smtp, and the providers of oidc and ldap. Only the hosts in services.openbao-spire-auth.smtpRelayHosts MUST read the upstream submission password of the smarthost.
  • FR-023a: A host-<name> policy MUST read a vars instance that another host writes only when varsReadInstances, oidcConsumerInstances or s3PairReadInstances names that instance for it, or a dedicated roster (incusStoreConsumerHosts, operatorDnsConsumerHosts, acmeDnsConsumerHosts) names the host. An instance MUST NOT be readable by every host.
  • FR-023b: For a host in namespacedHosts, the host-<name> policy MUST grant KV only under <deploymentVarsRoot>/<deployment>, valkey credentials only at database/static-creds/<ns>-<deployment> and LDAP credentials only on the mount of <deployment>, where <deployment> is the policy template of the cert-auth alias name. It MUST NOT grant the per-host stem secret/<host>/, the shared vars/ instances, or the objects of another deployment. The one exception is the read on the upstream submission password of the smarthost that FR-023 gives to a relay host.
  • FR-024: The component MUST declare provides = [ "varsBackend" "pki" ], so the setups route the secrets store and the PKI CA of the group to this node.
  • FR-025: Each entry of services.openbao.runtime MUST be an openbao-<name> oneshot that applies its collections with OpenTofu and the hashicorp/vault provider against services.openbao.runtimeApi.address, with local state at /var/lib/openbao-runtime/<name>/declarative-terraform.
  • FR-026: A reconciler MUST delete from OpenBao each object that leaves its config, except an object with lifecycle.prevent_destroy. The pki mount of openbao-pki-init and the database mount of openbao-database-init MUST have prevent_destroy.
  • FR-027: Before each apply, a reconciler MUST import each mount, auth mount, identity group and identity group alias of its config that OpenBao holds and its state does not.
  • FR-028: The setups MUST give the directories of the operator groups to openbao-ldap-engine only on the node of the central group.
  • FR-029: The implementation MUST request garage through mkBackupsGarageRequest with the bucket openbao-backups, and backup.nix MUST add a streamingBackup of a raft snapshot to that bucket with the restic provider, as the user root, with the root token over the loopback API.
  • FR-030: OpenBao MUST enable the audit device journal/ from its config, of type file with file_path = "stdout" and elide_list_responses = "true", and MUST keep the default HMAC of secret values. openbao.service MUST set LogRateLimitIntervalSec = 0, so that journald does not remove audit entries.
  • FR-031: On a node with a SPIRE agent, openbao-spire-ca-bundle MUST read the trust bundle from the Workload API of that agent, and from the agent's trust_bundle_path only when the current bundle has no CA that is valid now. It MUST NOT install a bundle in which a block does not parse or no CA is valid now. When the agent's trust_bundle_path is newer than the current bundle and has no CA in common with the Workload API, it MUST wait at most 180 seconds for the agent to attest to that CA. It MUST NOT install the agent's trust_bundle_path in place of a current bundle that has a CA that is valid now. It MUST restart openbao-spire-cert-auth only when the bundle changes.

Key Entities

  • openbao-init: the oneshot that initializes, seals to the TPM, and unseals OpenBao. It holds the root token on disk.
  • Reconciler: an entry of services.openbao.runtime (runtime/module.nix), the oneshot openbao-<name>. The reconcilers are openbao-spire-auth, openbao-spire-cert-auth, openbao-oidc-auth, openbao-pki-init, openbao-database-init, openbao-valkey-engine and openbao-ldap-engine. Each has local state, because the HTTP state back-end tf-backend is served with a certificate from this OpenBao's PKI and is deployed after it. The shell oneshots openbao-init and openbao-operator-deployer-token stay shell.
  • Roster: services.openbao-spire-auth.consumerHosts, varsProviderHosts, varsReadInstances, garageConsumerStems, s3PairReadInstances and oidcConsumerInstances. Each key becomes a cert auth role and a host-<name> policy. consumerHosts is empty by default: the setup derives the other rosters from the components that each node runs. The setup adds to varsReadInstances the readers that no component request states: the deploy runners read incus-deploy and the deploy credential instances, and the database-engine node reads postgresql-admin. s3PairReadInstances gives the garage node read on the key pairs of its own group that another node generates.
  • SMTP relays: services.openbao-spire-auth.smtpRelayHosts (listOf str, default []), the roster labels of the SMTP relays. The setup sets it to the label of each node whose component provides smtp, in each group that it collects (supportingRequests.smtpRelayRoster).
  • fediversity-deployer: the policy of a CD or workstation deploy token.
  • fediversity-operator-deployer: the policy of the periodic token for operator deploys.
  • fediversity-admins: the identity group and policy of a person who logs in to the UI.
  • Engines: pki (internal CA), database (postgres and valkey), ldap (bind account rotation).

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: integrations-spire-openbao passes.
  • SC-002: integrations-openbao-pki passes.
  • SC-003: integrations-cd-jwt-login passes, including the refusals of a pull_request run, another branch, another repository and another audience.
  • SC-004: deployment-cd-env-oidc and test-ldap-engine pass.
  • SC-005: nix-unit-setups-ldap-grant, nix-unit-components-openbao-spire-auth and nix-unit-components-openbao-init pass.
  • SC-006: After a reboot, OpenBao is unsealed with no human step.

Assumptions

  • The node has a TPM 2.0 with the storage hierarchy enabled.
  • The setup supplies the roster options from the requests of each component, as described in ../../005-setups-and-wiring/spec.md.
  • The spire component (spire.md) gives this node its SVID. openbao-spire-ca-bundle reads the trust bundle from the Workload API of the agent on this node, or from the agent's trust_bundle_path that the deploy writes, and keeps the last valid bundle when both fail.
  • The postgresql component (postgresql.md), ../shared/valkey.md and ../shared/lldap.md serve the backends the engines manage.
  • The authelia component (../shared/authelia.md) is the OIDC issuer for the UI login.
  • deploy-credentials.nix and operator-curation.nix are not part of this component. windmill/curation.nix imports both, so their spec belongs with the windmill component.

Known Deviations

  • Hidden state: the root token persists at /var/lib/openbao-init/root-token, and the engine units read it. The state of each reconciler persists at /var/lib/openbao-runtime/<name>/declarative-terraform and holds the secrets its config writes, such as the OIDC client secret. A TODO in openbao.nix (notes/bootstrap-seeds.md) plans scoped tokens instead.
  • The audit log is not tamper-evident. A user with root on the node can change the journal before the node log shipper sends it to Loki, and Loki is not write-once storage. The audit log has the retention of the journal and of Loki, not a retention of its own (#2270). An audit entry of a list response holds the count of the keys, not the keys.
  • FR-023a: without the LDAP engine (ldapEngine.enable = false), each roster host reads each ldap-bind-<group> instance. The setups always enable the engine, so only a use of the component outside the setups has this grant (#2291).
  • The backup of this component uses the fixed and public restic password ReallySafePasswordTrustMe39, because a restore of this backup comes before the secret store can give a password (#2102). Thus the encryption of this repository gives no protection.