Fediversity

Feature Specification: Component terraform-backend

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/components/hosting-provider/terraform-backend/default.nix, core/components/hosting-provider/terraform-backend/terraform-backend.nix, core/effects/tf/common/terraform-backend.nix).

User Scenarios & Testing (mandatory)

User Story 1 - Keep the OpenTofu state of the group (Priority: P1)

A hosting provider enables terraform-backend on the tf-backend node. The node runs the nimbolus/tf-backend HTTP state backend. The OpenTofu deployment methods store their state and take their locks there.

Why this priority: Every tf deployment program reads and writes its state through this backend. Without it, no steady-state deploy can run. See Stateless Deployment Programs.

Independent Test: nix-unit-setups-terraform-backend-config evaluates the node of the component alone. It checks that the nginx front sends requests to the port that the backend binds and serves only TLS, that the port is open but not published, that the kmsKey is the file of generateFiles from the on-machine provider, that the service waits for generate-vars.service and not for an OpenBao agent, and that a switch does not restart the service. deployment-tf-hosts, deployment-tf-incus-hosts, deployment-tf-incus-operator, apps-tf and apps-tf-incus deploy it, and deployment-tf-self-state-reachable reaches it.

Acceptance Scenarios:

  1. Given the component is enabled, When the node activates, Then terraform-backend.service listens on :8080 and nginx serves https://<domain>/ as a proxy to http://127.0.0.1:8080.
  2. Given the node is up, When a deployer runs tofu init against https://<domain>/, Then the backend stores the state encrypted with the node's kmsKey.

User Story 2 - Bootstrap a fresh provider before public DNS and TLS (Priority: P2)

On a fresh provider, public DNS and a public certificate do not exist yet. A BOOTSTRAP=1 migrate reaches the backend over the Incus bridge on port 8080.

Why this priority: The state of the graph that repairs the cluster must not depend on public DNS, a public CA, OpenBao or SPIRE.

Independent Test: deployment-tf-hosts-bootstrap and integrations-migrate-after-failed-apply.

Acceptance Scenarios:

  1. Given a fresh provider, When the deployer migrates with BOOTSTRAP=1, Then it reaches the backend at <node>:8080 from an internalSources address.
  2. Given the TLS policy of the group, When it reads certAudience = "bootstrap", Then it does not issue the certificate of this node from a public CA. See nix-unit-setups-tls-routing.
  3. Given OpenBao attestation is broken, When the node starts, Then generate-vars still renders the kmsKey on local disk and the backend starts.

User Story 3 - Recover a lock that a stopped client left (Priority: P3)

An operator finds a stale lock after a client stopped before its UNLOCK, for example a tofu that got a SIGKILL during an apply. The operator runs the force-unlock verb of the effects library or of the CD workflows.

Why this priority: Only the client that holds a lock releases it, so a lock stays until an operator releases it. A restart of the backend clears all locks, because the lock table is in memory.

Independent Test: integrations-reclaim-own-lock and integrations-steady-backend-freshness.

Acceptance Scenarios:

  1. Given a lock that no client holds, When the operator runs force-unlock, Then the backend accepts it because FORCE_UNLOCK_ENABLED is true.
  2. Given a change to the backend unit, When the node switches, Then systemd does not restart terraform-backend.service.

Edge Cases

  • A restart of the service during an apply clears the in-memory lock table. Each later state write returns 400 with an empty body, and the state after the apply is lost.
  • A change to the unit takes effect only after a manual restart, when no apply is in flight.
  • The kmsKey is generated once if it is absent. A new key makes the existing encrypted state unreadable. The offsite backup holds the key with the states, thus a restore brings back both. A person who can read the backup repository can thus read the states.
  • A state write replaces the old file in place. If a state changes during the copy of the backup, the backup copies the states again, at most 5 times, and then fails.
  • The restore stops and starts terraform-backend.service, which clears the lock table. Do a restore only when no apply runs.
  • The backup unit gets its S3 key from OpenBao KV. If OpenBao is broken, the backup fails, but the backend continues to serve the states.
  • Port 8080 is never published. The firewall resource admits it only from internalSources.

Requirements (mandatory)

Functional Requirements

  • FR-001: The component MUST declare name = "Terraform backend", description = "OpenTofu HTTP state backend.", no package, and certAudience = "bootstrap".
  • FR-002: The component options MUST come from sharedOptions.nix with a default subdomain of tf-backend, read from terraform-backend/node-name.nix.
  • FR-003: When enabled, the component MUST request ssl.domains = [domain] in group tls and spiffe.enable in group spiffe. See ssl and spiffe.
  • FR-004: When enabled, the component MUST request a nixos-module in group code that imports terraform-backend.nix and serves domain through nginx with a proxy to http://127.0.0.1:8080.
  • FR-005: terraform-backend.nix MUST enable services.terraform-backend with LISTEN_ADDR defaulting to :8080 and openFirewall = true.
  • FR-006: terraform-backend.nix MUST set systemd.services.terraform-backend.restartIfChanged = false.
  • FR-007: core/effects/tf/common/terraform-backend.nix MUST import the service module from the nixpkgs-terraform-backend pin.
  • FR-008: It MUST mint kmsKey through generateFiles as 32 random bytes in base64, owned by root, and pass it as kmsKeyFile.
  • FR-009: It MUST force contracts.varsBackend.defaultProvider to the on-machine provider, so that no generated file of the node depends on OpenBao.
  • FR-010: It MUST set FORCE_UNLOCK_ENABLED = true.
  • FR-011: terraform-backend.service MUST order after and want generate-vars.service, and MUST NOT wait for openbao-agent-default.
  • FR-012: The implementation MUST request garage through mkBackupsGarageRequest with the bucket terraform-backend-backups, and backup.nix MUST add a streamingBackup to that bucket with the restic provider, as the user root. The stream MUST be a tar of a copy of the states that is the same as the states after the copy, and of the kmsKey.
  • FR-013: The restore MUST stop terraform-backend.service, replace the states and the kmsKey, give the states the owner of the state directory, and start the service again.

Key Entities

  • State backend: nimbolus/tf-backend as services.terraform-backend, on :8080, fronted by nginx on :443.
  • kmsKey: the state-at-rest key, local to the node, from contracts.generateFiles.results.terraform-backend.files.kmsKey.path.
  • Lock table: in memory, in the backend process.
  • Offsite backup: the restic repository terraform-backend_states in the terraform-backend-backups bucket, with the states and the kmsKey.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: deployment-tf-hosts, deployment-tf-incus-hosts, deployment-tf-incus-operator, apps-tf and apps-tf-incus pass with state on this backend.
  • SC-002: deployment-tf-hosts-bootstrap and integrations-migrate-after-failed-apply pass over the :8080 lane.
  • SC-003: integrations-reclaim-own-lock, integrations-steady-backend-freshness, integrations-steady-refuses-lost-state and deployment-tf-self-state-reachable pass.
  • SC-004: integrations-tf-backend-state-id passes: a state written with the default credentials of the clients has the ID that the existing states on a live deployment have.

Assumptions

  • The node that runs the component has the network.name tf-backend, which the hosting-provider group reads from terraform-backend/node-name.nix. The zone of the group publishes a served name equal to the node label with no alias.
  • The firewall resource limits port 8080 to internalSources.
  • The nixpkgs-terraform-backend pin carries the branch of nixpkgs pull request 554986 until the module is in the nixpkgs pin.

Known Deviations

  • The lock table lives in memory in the backend process. This is hidden state against principle III, and a restart during an apply loses the state written after it.
  • restartIfChanged = false means that a change to the unit needs a manual restart.
  • terraform-backend.nix has a TODO: nimbolus/tf-backend has no OpenTelemetry support, so the component sends no traces or metrics.
  • The forced on-machine vars provider applies to every generated file on the node, not only to the kmsKey.
  • The basic authentication of the backend accepts every request. The password only selects the ID of the state, which is sha256("<password>:" + sha256("<project>-<name>")). Thus each client that can reach port 8080 or the nginx front can read, write and unlock each state whose key and password it knows, and all clients use the same fixed password. FORCE_UNLOCK_ENABLED = true adds no access to this: a LOCK on a locked state answers 423 with the record of the holder, and an UNLOCK with that record releases the lock. A change of the password or of the authentication method changes each ID, and the existing states then need a migration.
  • The backup of this component uses the fixed and public restic password ReallySafePasswordTrustMe39, because a restore of this backup comes before the secret store can give a password (#2102). Thus the encryption of this repository gives no protection.