Fediversity

Feature Specification: Deployment method tf-incus-operator

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code at 093e68b6f (core/effects/tf/incus/operator/, core/setups/tf-incus-operator-hosts.nix, core/setups/hosts-common.nix), PRs #1856, #1869

The tf-incus-operator method deploys one operator deployment of the hosting provider. It is the tf-incus method (see tf-incus.md) with a per-deployment tenancy on the provider's shared nodes in the same OpenTofu graph. Shared method behavior is in spec.md. The program grammar, the verbs and the state rules are in ../004-deployment-programs/spec.md. The registrar lane is in ../010-effects/registrars.md. The zone publish of the provider is in ../010-effects/octodns.md.

This file describes only what the operator method adds to tf-incus or changes in it.

User Scenarios & Testing (mandatory)

User Story 1 - Mint a deployment's tenancy on the shared nodes (Priority: P1)

An operator deploys a group of applications. The graph creates the garage buckets and keys, the postgres databases and roles, and the OpenBao entries that the applications consume. A destroy removes them again.

Why this priority: Without the tenancy, no application node can start. The shared nodes converge once for all deployments, so they cannot create or drop a per-deployment name themselves.

Independent Test: deployment-tf-incus-operator runs the garage and OpenBao half. It mints a cache-backup bucket and key on a shared garage node and publishes the pair into OpenBao KV over the :8200 TLS listener. Then it destroys, and checks that the buckets and the deployment's KV are gone. No check runs the postgres half of the graph.

Acceptance Scenarios:

  1. Given a garageRequests entry with a bucket and a key, When the operator applies, Then the graph creates the bucket, the key and the permission, and writes the key pair under the deployment's KV prefix.
  2. Given a postgresqlRequests entry, When the operator applies, Then the graph creates the login role, the database and an OpenBao static role on the named database mount.
  3. Given a deployed tenancy with objects in a bucket, When the operator destroys, Then the graph empties each bucket before it deletes the bucket.
  4. Given a namespaced deployment whose nodes wrote entries under its KV prefix, When the operator destroys, Then the graph deletes each entry under that prefix after the instances are gone, and keeps the entries of other deployments and of the shared prefix.
  5. Given a garageRequests bucket with enabled corsRules, When the operator applies, Then a preflight request to the bucket from an allowed origin gets the Access-Control-Allow-* headers.

User Story 2 - Publish the names the nodes serve (Priority: P1)

An operator deploys on a domain. The graph writes an address record for each name a node serves before that node switches, so that ACME validation resolves.

Why this priority: A node that orders a certificate for a name that does not resolve fails its activation.

Independent Test: integrations-publish-records-named runs the publisher over its RFC2136 lane against the provider's own named primary and secondary. nix-unit-setups-operator-publish holds what the setup does with the uplink and the record lane.

Acceptance Scenarios:

  1. Given publishDns is set and a node has published names and a NetBox address, When the operator applies, Then the publisher writes the records, reads them back and waits until the names resolve, before the node switches.
  2. Given a name in reservedNames, When the publisher runs, Then it refuses the name.
  3. Given a published record, When the operator destroys, Then the publisher deletes the record only while it still holds this deployment's address.
  4. Given an own zone and a node that provides smtp, When that node has switched, Then the publisher writes the SPF, DKIM and DMARC records of the zone.
  5. Given a published record that the publisher cannot release, When the operator destroys, Then the destroy completes, names the record and exits 4.

User Story 3 - Hold and register an operator's own domain (Priority: P2)

An operator submits a domain that is not under the provider's apex. On the bind lane, the graph adds the domain to the provider's catalog. When the provider sets domainRegistration, the graph also registers the domain, or adopts it, at the provider's registrar account. When domainRegistration also sets zones, deSEC holds the domain in place of the provider's named: the api creates the zone at deSEC and a token that can write only that zone, and the graph delegates the domain to deSEC's nameservers and gives the registry the DS records of the keys that deSEC signs the zone with.

Why this priority: Without the zone and the delegation, no record of the deployment resolves on the public internet.

Independent Test: deployment-domain-claim-probe and deployment-tld-handles-probe and deployment-zone-keys-probe run the three probes. nix-unit-setups-registrar, nix-unit-setups-domain-claim-grant and nix-unit-render-env hold the projection, the claim ledger grant and the coordinates the provider's curation fixes. No check runs a registration against a registrar.

Acceptance Scenarios:

  1. Given the bind provider and a zone, When the operator applies, Then the publisher adds a catalog member and waits until every nameserver answers for the zone.
  2. Given a free domain that no deployment claims, When the operator applies, Then the graph writes a claim with cas = 0 and registers the domain for one period.
  3. Given a domain that the account holds and the claim names this deployment, When the operator applies, Then the graph imports the claim and the domain instead of a new registration.
  4. Given a domain that the account holds with no claim for this deployment, When the operator applies, Then a precondition fails the plan.
  5. Given a registered domain, When the operator destroys, Then the graph keeps the registration and removes the catalog member.

User Story 4 - Give each node a SPIFFE identity (Priority: P2)

The graph allow-lists the TPM endorsement key of each container on the provider's spire node after the switch, and retires it on destroy.

Why this priority: An operator node logs in to the provider's OpenBao with its SVID. Without a registration, the node waits forever for an agent socket.

Independent Test: nix-unit-setups-operator-config (core/setups/operator-config-test.nix) holds that each realised operator node carries the agent and get_tpm_pubhash. No check runs register_ek for this method.

Acceptance Scenarios:

  1. Given a spire node on the group or the hosting-provider roster, When the setup evaluates, Then spireNode names it with the incus-exec transport.
  2. Given no spire node on either roster and no spireNode, When the setup evaluates, Then evaluation fails with a message that names the missing spire node.

User Story 5 - Isolate the deployment on the shared bridge (Priority: P1)

The nodes of all deployments on an incus host share its managed bridge. The graph gives this deployment a /28 of its own from the NetBox prefix of the operator deployments, gives each node a static address in it, and puts an incus network ACL on the eth0 of each node (bridge-isolation.tf, module core/effects/tf/incus/bridge-acl/). The ACL drops the traffic between the /28 and the rest of that prefix, in the two directions.

Why this priority: The firewall resource of each node trusts the whole bridge subnet. Without the ACL, a node of one operator deployment can send to all ports of the nodes of the other operator deployments.

Independent Test: integrations-incus-bridge-acl applies the bridge-acl module for two deployments on a real incus bridge, and sends ICMP and TCP between containers with the NIC keys of instance.tf. No check applies the NetBox allocation of bridge-isolation.tf.

Acceptance Scenarios:

  1. Given two deployments A and B, When a node of A sends to a node of B, or B to A, Then the ACL drops it.
  2. Given a deployment, When its node sends to a node of the same deployment, to a node of the hosting provider, to the bridge gateway or out of the bridge, Then the ACL passes it, and the bridge dnsmasq answers its DNS.
  3. Given a node of the hosting provider, When it sends to a node of an operator deployment, Then the ACL passes it.
  4. Given a deployment with nodes and no NetBox URL or token, When the graph plans, Then it stops with an error, and puts no node on the bridge without isolation.

Edge Cases

  • registration stays null when the zone is the apex, is under the apex, or the provider is not bind. Then registrar-none.hcl replaces registrar.tf, and await_delegation has count 0.
  • A premium domain fails a precondition, so the graph does not spend more than maxCost.
  • The claim probe fails the plan on any OpenBao answer other than 200 or 404. The handles probe fails the plan on any Openprovider failure.
  • A name that does not resolve within the poll ceiling fails the apply.
  • A DKIM read that fails publishes no mail record and fails the apply.
  • A catalog member that the provider's render refuses times out in hold.
  • delegated waits at most delegation_seconds (default 900) for the parent's authoritative servers.
  • Garage refuses to delete a non-empty bucket (409 BucketNotEmpty). The destroy step mints a temporary admin key to empty each bucket. A key that it cannot delete is reported, not raised.
  • The v2 admin API has no verb for CORS, and garage accepts a CORS write only from a key with owner on the bucket. The apply mints a temporary admin key to set the rules over S3. A failure to set them fails the apply, and the next apply sets them again.
  • With public_uplink.mode = "bridged", a postcondition requires the uplink parent to be a bridge.
  • A node of an earlier deployment without isolation keeps its old DHCP address until it renews its lease, but the NIC filter accepts only the new static address immediately. Restart such nodes after the first apply with isolation.
  • The NetBox selector of the bridge prefix must match exactly one IPv4 prefix, else the plan fails. When all /28s are in use, NetBox refuses the allocation and the apply fails.
  • When the selector of the bridge prefix selects a different prefix, the next apply replaces the /28 and the bridge address of each node, and the ACL. Unlike the public addresses, the bridge addresses do not ignore a change of their pool: the ACL is computed from the /28, so the nodes must be in it. Restart the nodes after such an apply.
  • The isolation is IPv4 only. The managed bridge has no IPv6 subnet, so the NIC filters drop all IPv6 from eth0.

Requirements (mandatory)

Functional Requirements

  • FR-001: The method MUST run one OpenTofu graph, tf/incus/operator, through tfRun, with tf/incus/host and tf/incus/bridge-acl as dependent directories, and share instance.tf, instance-variables.tf and netbox.tf with tf-incus.
  • FR-002: The graph MUST read the hosting provider's coordinates from its state at hpStateAddress through terraform_remote_state, with basic auth (tfHttpUsername, tfHttpPassword).
  • FR-003: The graph MUST address OpenBao, garage and postgres at the provider node's bridge address first, then its public address, then the *_host variable.
  • FR-004: The graph MUST derive the NetBox URL from the provider state, and read the NetBox token from KV <prefix>/secret/netbox/apiToken when it is not set.
  • FR-005: For each garageRequests entry, the graph MUST create the buckets, the keys and the bucket permissions, and write <kvName>-s3/accessKey under public/ and <kvName>-s3/secretKey under secret/ of <openbao_kv_prefix>/operator-deployments/<namespace>.
  • FR-006: The graph MUST create the garage names exactly as it gets them. The setup MUST hand them in with the namespace prefix already applied.
  • FR-007: For each postgresqlRequests entry, the graph MUST create a login role, a database and a vault_database_secret_backend_static_role on mount and connection with rotationSeconds, and drop them on destroy.
  • FR-008: The graph MUST read the postgres admin password ephemerally from KV <prefix>/secret/postgresql-admin/password and connect with sslmode = require.
  • FR-009: Before it deletes a bucket, the destroy MUST empty the bucket through empty_buckets_runner.
  • FR-010: When publishDns is set, the graph MUST publish each node's names before module.nixos-rebuild of that node, and retire them on destroy.
  • FR-011: The publisher MUST find the zone by a walk up the labels of the served name, and MUST refuse names in reservedNames.
  • FR-012: When mail_publish is set, the graph MUST publish the mail records after publish_app_records and after the switch of the node that provides smtp (mail_publish.relay_node).
  • FR-013: On the bind provider with a zone, the graph MUST hold the zone in the catalog before it publishes, and release it on destroy. On the desec provider with a zone, the api MUST create the zone at deSEC before the deploy, the graph MUST make sure that the zone is there before it publishes, and MUST keep it on destroy: the delegation stays, and another deSEC account can create a deleted domain.
  • FR-014: When registration is set, the graph MUST order the steps as hold_zone, domain_claim, openprovider_domain, await_delegation, then publish_app_records.
  • FR-015: The graph MUST register a domain only when it is free and no deployment claims it, and adopt a held domain only when the claim names this namespace.
  • FR-016: The registration MUST use one period, the provider's ns_group and max_cost, and MUST NOT delete the domain on destroy (on_destroy = "retain"). When zones is set, the registration MUST use the ns_group of zones in place of the provider's, and MUST give the key-signing keys (flags 257) of the zone at deSEC as dnssec_keys. A deploy that cannot read these keys MUST fail before the registration. The graph MUST create the ns_group of zones when the registrar account does not have it, adopt it when it does, and MUST NOT delete it on destroy.
  • FR-017: When spireNode is set, the graph MUST run register_ek for each container after its switch, skip the spire node itself, and retire the registration on destroy.
  • FR-018: The setup MUST fail evaluation when no spire node is published and spireNode is not set.
  • FR-019: The graph MUST run the deploy verdict for each node after register_ek and module.nixos-rebuild.
  • FR-020: Credentials MUST come from the environment: TF_VAR_dns_api_token, TF_VAR_openbao_token, TF_VAR_garage_admin_token, OPENPROVIDER_USERNAME, OPENPROVIDER_PASSWORD, and TF_VAR_incus_config_dir for the Incus client.
  • FR-021: The publisher MUST pass TF_VAR_dns_api_token to curl through a config file on a pipe, and to nsupdate on stdin. When zones is set, the publisher and the key read MUST read the deSEC token from <deployment prefix>/secret/desec-zone/token in KV with TF_VAR_openbao_token, and MUST pass both tokens to curl through a config file on a pipe. The graph MUST NOT take a deSEC token as a variable.
  • FR-022: For a namespaced deployment, the destroy MUST delete each entry under the deployment KV prefix, with all of its versions, after every instance is gone, through reap_kv_runner. It MUST NOT delete under any other prefix.
  • FR-023: For each bucket, the apply MUST set the union of the corsRules of its requests through bucket_cors_runner, MUST remove the CORS rules of a bucket that has none, and MUST set them again when they change or the bucket is created again.
  • FR-024: When the deployment has nodes, the graph MUST allocate one /28 from the NetBox prefix that netbox_bridge_prefix_selector selects (default: tag fediversity-bridge), and one address in it per node.
  • FR-025: When the deployment has nodes and NetBox is not available, the plan MUST fail.
  • FR-026: The graph MUST create one incus network ACL <instance prefix>isolation, which drops egress to and ingress from each sibling subnet on the path from the bridge prefix to the /28, and nothing else.
  • FR-027: The eth0 of each node MUST set ipv4.address to its NetBox address, security.acls to the ACL, and security.acls.default.egress.action and security.acls.default.ingress.action to allow.
  • FR-028: The method MUST set centralServices = true, so that the routing gives the postgresql and redis resources of the operator group to the providers of the central group (the group whose roster provides varsBackend).
  • FR-029: A failed destroy step for records, zones or SPIRE registrations MUST NOT stop the destroy, so that a destroy of a half-built deployment, or of one whose DNS primary is gone, completes. The step MUST add the object that it left to a list. At the end of the run, bootstrap-apply.sh MUST print the list. For TF_VERB=destroy, it MUST exit 4 when the run otherwise succeeded. For an apply, where a replace or a node removal runs these steps, the list MUST be a warning, and the exit status MUST NOT change. A retire of a name that no zone holds is not a failure.

Key Entities

  • Tenancy request: garageRequests (buckets, keys, access) and postgresqlRequests (mount, connection, rotation, databases and roles). The setup folds them from every node of the deployment.
  • Deployment KV prefix: <openbao_kv_prefix>/operator-deployments/<namespace>. Without a namespace, it is openbao_kv_prefix.
  • Record lane: publishDns, with provider powerdns or bind, the zone, the TTL, the reserved names and the optional mail part. When domainRegistration sets zones, the publisher writes to deSEC on the desec provider in place of bind.
  • Domain claim: the KV entry <prefix>/public/domain-claims/<zone> with {deployment = <namespace>}. It tells a registration from an adoption.
  • Registration lane: domainRegistration, with apex, nameservers, maxCost in EUR cents, ownerHandle and the optional zones, set by the provider's curation. zones has provider (desec), serverUrl, nameservers and nsGroup.
  • Bridge prefix: the NetBox prefix of all operator deployments on one incus bridge (operatorPrefix in core/components/hosting-provider/incus-bridge.nix, 10.0.104.0/21). Each deployment holds one /28 in it.
  • Spire node: spireNode, with transport incus-exec or ssh, on the hosting provider's deployment.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After an apply, each requested key pair is in KV under the deployment prefix, and deployment-tf-incus-operator passes.
  • SC-002: After an apply, each published name resolves to its node's NetBox address before that node switches.
  • SC-003: After a destroy, no bucket, key, database, role or KV entry of the deployment remains on the shared nodes.
  • SC-004: A deployment without a spire node fails at evaluation, not at apply.
  • SC-005: A held domain without a matching claim fails at plan, before the graph spends money.
  • SC-006: integrations-incus-bridge-acl passes: the nodes of two deployments do not reach each other, and they reach their own deployment, the provider's nodes, the gateway and its DNS.

Assumptions

  • The hosting provider is deployed first, and its state at hpStateAddress exposes ipv4 and bridge_ipv4 per node.
  • The shared postgres node converges once for all deployments and never drops a database. This is why the graph creates and drops the per-deployment objects.
  • Valkey tenancy is declarative on the valkey and OpenBao nodes. This graph does not provision it.
  • Each database and role name carries the deployment's namespace as a suffix before it reaches the graph.
  • Last writer wins on a published name.
  • The hosting provider seeds the bridge prefix in NetBox, with the tag fediversity-bridge. Incus hosts that share one NetBox need one prefix and one selector each.
  • The nodes of an operator deployment can send to all ports of the nodes of the hosting provider and of the incus host, because these trust the whole bridge subnet (internalSources). This includes ports that the deployment does not use, such as the HTTP state backend and the incus metrics location (/1.0/metrics). The ACL does not narrow this.

Known Deviations

  • tfHttpPassword defaults to "fake-secret", and hpStateAddress falls back to a localhost example address (#2220).
  • No check runs the postgres half, the registration, register_ek or the NetBox allocation of the bridge prefix of this method.