Fediversity

Feature Specification: Deployment method tf-incus

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code at 093e68b6f (paths in backticks), and PRs #527, #723, #1071, #1184, #1205, #1234, #1235, #1238 and #1393.

The tf-incus method deploys the nodes of one group into Incus instances on one hypervisor. One OpenTofu run creates the instances, gives them addresses, deploys NixOS into them and judges the result. The setup is core/setups/tf-incus-hosts.nix. The effect is core/effects/tf/incus/hosts/effect.nix. The OpenTofu graph is core/effects/tf/incus/hosts/ with the per-node module core/effects/tf/incus/host/.

This file states only what is particular to tf-incus. The attribute grammar, the verbs and the state rules are in ../004-deployment-programs/spec.md. The shared deploy body (deploy.sh), its verdicts, the bootstrap wrapper and the publication of environment keys are in spec.md. The operator method tf-incus-operator.md uses the same instance and NetBox files (instance.tf, instance-variables.tf, netbox.tf). The base image comes from ../010-effects/image.md. The dev runner dev-tf-incus-bootstrap is in ../010-effects/incus-bootstrap.md. The provider's zone records come from the octodns-zone effect, not from this method.

User Scenarios & Testing (mandatory)

User Story 1 - Deploy a group into Incus containers (Priority: P1)

An operator runs <group>-tf-incus-<machine> against an Incus hypervisor. The run creates one instance per node, waits until each instance answers incus exec, deploys the node's NixOS closure into it and judges each node.

Why this priority: This is the method. Without it, no group reaches an Incus hypervisor.

Independent Test: deployment-tf-incus-hosts.

Acceptance Scenarios:

  1. Given a hypervisor with a managed bridge and a base image, When the operator applies the graph, Then each node exists as an instance named <namespace>-<network name>, and operator can run a program from the node's closure inside it.
  2. Given a deployed node, When the instance is deleted and created again by a later apply, Then null_resource.await_init gets a new id, and the deploy and both verdict passes run again for that node.
  3. Given a deployed group, When the operator reads the names, ipv4, bridge_ipv4 and ipv6 outputs, Then each output holds one entry per node.

User Story 2 - Boot containers off the hypervisor store (Priority: P2)

A node that does not need a writable store shares the hypervisor's /nix/store read-only. It gets no image and no root disk. Its closure lives on the hypervisor under a per-instance profile.

Why this priority: Shared-store mode saves a full closure copy per container on the hypervisor disk. A wrong plan in this mode destroys a live container, so the method guards it.

Independent Test: deployment-tf-incus-hosts (the shared-store subtests) and integrations-incus-shared-store. The single evaluation for the seed pass and the deploy pass has deployment-eval-cache. nix-unit-effects-tf-incus-plan-inputs covers the shared_store and host_nix_daemon values of each node and the probe of the replace guard. deployment-tf-incus-decisions covers what the graph plans from those values: the store disks, the daemon socket, no image and a stopped create only for a shared-store container, and the refusal of a shared-store VM.

Acceptance Scenarios:

  1. Given a shared-store node, When the graph creates it, Then terraform_data.seed_profile first plants the node's generation at /nix/var/nix/profiles/inx/<instance> on the hypervisor, and the instance boots /nix/var/nix/profiles/system/init from that profile.
  2. Given a shared-store node after a deploy, When the operator compares the host profile and the guest profile, Then both name the same generation, and the host profile is a GC root of that generation.
  3. Given an instance that currently boots off the hypervisor store, When a plan asks for the image path for it, Then the plan fails with a precondition error and replaces nothing.
  4. Given a shared-store node and instance_type = "virtual-machine", When the operator plans, Then the plan fails with a precondition error.
  5. Given a shared-store node, When the operator destroys the graph, Then the destroy removes the host profile with the runner and key paths of the current run.

User Story 3 - Allocate public addresses in NetBox (Priority: P2)

A node that needs a public address gets one from NetBox. The graph creates a NetBox virtual machine and interface for the node and allocates or pins its addresses. The second interface eth1 carries that address.

Why this priority: Public nodes need stable addresses that the zone can publish. A provider that hosts its own NetBox finds it through its own state.

Independent Test: deployment-tf-self-state-reachable covers the probe that gates the read of the own state. nix-unit-effects-tf-incus-plan-inputs covers which nodes ask for an address, the pinned addresses, the NetBox coordinates and the uplink mode. deployment-tf-incus-decisions covers what the graph plans from those values with mocked providers: NetBox records only the nodes with needs_public_ip, and only when NetBox has a URL and a token; eth1 per uplink mode; and the refusal of a routed uplink with no v4 address. No check applies the NetBox half of this graph against a real NetBox.

Acceptance Scenarios:

  1. Given a fresh provider with no own state yet, When the operator applies, Then seed-self-state.sh writes an empty state, the probe reports reachable = "false", and the apply continues without NetBox URLs from the own state.
  2. Given a provider whose own state has an ipv4 entry for the node that runs the netbox component, When no serverUrl is set, Then the NetBox provider dials http://<that address>:8001.
  3. Given a node with a pinned address, When the graph applies, Then NetBox holds that address on the node's interface, and the graph allocates no other address for it.

User Story 4 - Register TPM keys and judge the fleet (Priority: P2)

When the caller names a SPIRE node, the graph registers each other container's TPM endorsement key with it. A strict verdict pass then judges every node after its secret chain completes.

Why this priority: A node whose units wait on SPIRE credentials cannot pass a verdict until its key is registered.

Independent Test: integrations-spire-incus.

Acceptance Scenarios:

  1. Given spire_node is set and instance_type = "container", When the graph applies, Then null_resource.register_ek runs for each node except the SPIRE node, after both nodes are deployed.
  2. Given a deployed node, When null_resource.deploy_verdict runs, Then it runs the deploy runner with DEPLOY_VERDICT_ONLY=1 and fails the apply when a wanted unit stays failed.

Edge Cases

  • A shared-store container has no bridge lease at creation, because the graph creates it stopped. Its ipv4 output is the daemon's bridge lease for that node until the next apply refreshes the instance.
  • An empty image_fingerprint makes the graph look up image_alias. The graph does this only when some node needs an image.
  • public_uplink.mode = "nat" adds no eth1. routed adds eth1 with the bare IPv4 address and no IPv6.
  • public_uplink.mode = "bridged" requires the parent network to have the type bridge. Else the plan fails.
  • A filtered NIC sends only from its MAC and its addresses. A guest that changes the MAC of a NIC loses that NIC. A guest that sends from a different address loses those packets.
  • On eth0 without a static ipv4.address, incus reserves the current DHCP lease for the NIC when it applies the filters. The node keeps that address until its NIC is removed. The nodes of an operator deployment have a static address (see tf-incus-operator.md).
  • Under nat, the NetBox address is an alias on eth0. The filters drop traffic from that alias. No node uses it, because the nodes find each other by the dnsmasq names of their leases.
  • The filters drop IPv6 from temporary (privacy) addresses, so a node sets networking.tempAddresses = "disabled". When the managed bridge has no IPv6 subnet, the filters drop all IPv6 from eth0, link-local too.
  • A bridged eth1 with ipv6.address = none sends no IPv6. Incus never filters the traffic to a guest.
  • An unreadable hypervisor makes shared-store-state.sh report nothing. The replace guard then passes, and the apply fails against the daemon.
  • The incus remote can exist already from a concurrent deploy. run.sh accepts that and fails on any other incus remote add error.
  • The NetBox token can come from TF_VAR_netbox_api_token, from netbox.apiTokenFile, or from OpenBao KV. The KV read is ephemeral, so the token stays out of the state.

Requirements (mandatory)

Functional Requirements

  • FR-001: The setup MUST import core/setups/hosts-common.nix with deployment = "tf-incus" and bootstrap = true, and MUST give each node name = <app>, networkName and ssh.host = <host>.
  • FR-002: The effect MUST enroll the Incus client certificate with enroll-client-cert.sh before tofu runs, and MUST seed an empty own state when none exists.
  • FR-003: The graph MUST name each instance <namespace>-<network name>, and MUST fall back to the node key when the network name is null.
  • FR-004: Every instance MUST have eth0 on the managed bridge, and MUST set boot.autostart and security.nesting.
  • FR-004a: Every container MUST set security.idmap.isolated, so that each container maps to its own block of 65536 host uids and gids.
  • FR-004b: Every eth0 and every bridged eth1 MUST set security.mac_filtering, security.ipv4_filtering and security.ipv6_filtering. A bridged eth1 MUST set ipv4.address and ipv6.address to the NetBox address without its mask, or to none when NetBox holds no address of that family.
  • FR-005: The graph MUST add eth1 only for public nodes under the bridged or routed uplink.
  • FR-006: An image-path instance MUST be created running from incus:<fingerprint> and MUST wait for an eth0 IPv4 lease.
  • FR-007: A shared-store instance MUST be a container, MUST have no image, MUST be created stopped, and MUST bind /nix/store read-only and /nix/var/nix/profiles/inx/<instance> writable, both with shift.
  • FR-008: The graph MUST bind the hypervisor nix-daemon socket read-only only for nodes with hostNixDaemon, and hostNixDaemon MUST require sharedStore.
  • FR-009: The graph MUST plant the shared-store profile before it creates the instance, and MUST replace the profile only when the instance name changes.
  • FR-010: The plan MUST refuse to move an instance that the daemon reports as shared-store back to the image path.
  • FR-011: The graph MUST wait until incus exec answers in each instance before the deploy, and MUST run the deploy again when the instance is replaced.
  • FR-012: The per-node deploy MUST run core/effects/incus/run.sh over incus exec with strict_units = false.
  • FR-013: When spire_node is set and the instances are containers, the graph MUST register the TPM endorsement key of each other node after both are deployed.
  • FR-014: The graph MUST run a strict verdict pass per node after the deploy and the key registration, and MUST fail the apply on a failed wanted unit.
  • FR-015: The graph MUST read its own prior state only when self-state-reachable.sh reports reachable = "true".
  • FR-015a: The setup MUST set openbaoNode and netboxNode to the roster node that provides varsBackend and netbox (providerNode in core/setups/wiring/resource-routing.nix), and to null when the group runs no such node. The graph MUST take the OpenBao address and the NetBox URL from the own state of these nodes only, and MUST NOT find them by a node name.
  • FR-016: When netbox is set, the graph MUST create one NetBox virtual machine and interface per public node, named by its DNS label, and MUST allocate or pin its addresses there.
  • FR-017: The graph MUST dial OpenBao over HTTPS, and MUST skip TLS verification only when no CA file is given.
  • FR-018: When deployerKv is set, the graph MUST publish each given deployer value to <kvPrefix>/secret/deployer/<key>.
  • FR-019: The graph MUST write no address records. Its published-names step MUST list no nodes.
  • FR-020: The runtime overrides MUST cover host, port, network, instance_type, image_fingerprint, image_alias, the NetBox selectors and custom fields, and namespace.

Key Entities

  • Instance: One incus_instance.nix_instance per node. Its state key is the node key. Its Incus name carries the namespace.
  • Host profile: The directory /nix/var/nix/profiles/inx/<instance> on the hypervisor. It is the boot profile and the only GC root of a shared-store node's closure.
  • Recreate detector: null_resource.await_init. Its id changes when the instance is replaced. The per-node module takes it as instance_generation.
  • Own state: The HTTP-backend state of this graph, read through data.terraform_remote_state.self. Its ipv4 output locates the provider's own openbao and netbox nodes.
  • NetBox objects: A virtual machine, an interface and one IPv4 and optional IPv6 address per public node, in the cluster netbox.clusterName.
  • Deployer KV values: adminSources, netboxApiToken, dnsApiToken, zonesManagerToken, openproviderUsername and openproviderPassword under <kvPrefix>/secret/deployer/.

Run-time order

  1. enroll-client-cert.sh, then seed-self-state.sh.
  2. The reachability probe, the own-state read, the shared-store probe and the image lookup.
  3. NetBox objects and addresses for public nodes.
  4. terraform_data.seed_profile for shared-store nodes.
  5. incus_instance.nix_instance, then null_resource.await_init.
  6. module.nixos-rebuild per node.
  7. null_resource.register_ek per node.
  8. null_resource.deploy_verdict per node.
  9. data.external.bridge_ip and the outputs.

null_resource.publish_deployer_kv has no order relation with the steps above.

Eval-time and run-time failures

  • Eval time: a value that does not match the type of the netbox, publicUplink, spireNode or deployerKv option fails in the module system.
  • Plan time: the two shared-store preconditions, and the bridged parent postcondition.
  • Apply time: an unreachable Incus daemon, a failed incus exec wait, a failed deploy, a failed key registration, or a strict verdict that fails.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: deployment-tf-incus-hosts passes. It deploys an image-path node and a shared-store node, and runs a program from each.
  • SC-002: In deployment-tf-incus-hosts, the host and guest generations of the shared-store node are equal, and the host profile is a GC root.
  • SC-003: In deployment-tf-incus-hosts, a shared-store node without hostNixDaemon has no nixdaemonsocket device and no store database device.
  • SC-004: deployment-eval-cache passes, so the seed pass and the deploy pass of one node share one evaluation.
  • SC-005: deployment-tf-self-state-reachable passes, so a fresh provider falls through to DHCP-only addresses.
  • SC-006: deployment-remote-path-fallback passes, so a command run in a node finds its programs when the node's own PATH resolves nothing.
  • SC-007: The deployment-cd-env-* checks pass, so CD stages the hypervisor host key, the shared-store identity and the KV coordinates that this method reads.
  • SC-008: In apps-tf-incus, a node with a bridged eth1 reaches the incus host from its own addresses on eth0 and on eth1, and the NIC filters drop the traffic it sends from an address that it added itself.
  • SC-009: nix-unit-effects-tf-incus-plan-inputs passes, so the tfvars that drive the instance plan select the expected nodes, and each tfvar and node field that Nix renders is declared by the graph.
  • SC-010: deployment-tf-incus-decisions passes, so the graph plans the shared store only for a container, eth1 only for a public node in the form of its uplink mode, and NetBox objects only for the nodes with needs_public_ip.

Assumptions

  • The hypervisor runs Incus with a managed bridge that serves DHCP, the 10.0.100.1 resolver and the nix-serve substituter.
  • The base image exists on the hypervisor when some node uses the image path.
  • For shared-store nodes, the caller supplies an SSH identity into the hypervisor store and a Nix signing key.
  • The HTTP state backend is reachable when httpBackend is set.
  • A provider that hosts its own NetBox and OpenBao names those nodes netbox and openbao.
  • All deployments on one incus host share the managed bridge, and the firewall resource trusts its whole subnet. The NIC filters (FR-004b) stop a node from sending as a node of a different deployment. The nodes of the hosting provider get DHCP leases from the provider range (dhcpRange in core/components/hosting-provider/incus-bridge.nix) and have no ACL. The operator deployments are isolated from each other by an ACL per deployment (see tf-incus-operator.md).

Known Deviations

None.