Fediversity

Feature Specification: Component incus

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/components/hosting-provider/incus/default.nix, core/components/hosting-provider/incus/oidc.nix, core/components/hosting-provider/incus/runtime/module.nix, core/components/hosting-provider/incus.nix, core/components/hosting-provider/incus-attest.nix, core/components/hosting-provider/incus-bridge.nix, core/components/hosting-provider/incus-bridge-dnsmasq.nix, core/components/hosting-provider/incus-hwrng-udev.nix, core/components/hosting-provider/incus-tpm-udev.nix, core/components/hosting-provider/incus-store-identity.nix, core/components/hosting-provider/incus-store-identity-test.nix, core/components/hosting-provider/incus-store-serve.nix, core/components/hosting-provider/incus-test.nix).

User Scenarios & Testing (mandatory)

User Story 1 - Run the hypervisor for the guest nodes (Priority: P1)

A hosting provider enables incus on a host node. The host runs the Incus daemon with a bridge, a storage pool and a default profile. The deployment methods then create the other nodes of the group as Incus instances on this host.

Why this priority: The tf-incus deployment methods need a hypervisor. Without it, no guest node exists.

Independent Test: test-incus-service (core/components/hosting-provider/incus-test.nix). The bridge DNS answers have integrations-bridge-dns.

Acceptance Scenarios:

  1. Given a node that imports incus.nix, When it boots, Then incus list succeeds and the incusbr0 network, the default storage pool and the default profile exist.
  2. Given fediversity.incus.uiDomain is set, When the firewall loads, Then ports 8443 and 443 are in the input-allow chain.
  3. Given a guest on the bridge, When it resolves dnsmasq.<apex> or a name in bridgeCnames, Then dnsmasq on the bridge answers it.

User Story 2 - Sign in to the Incus web UI (Priority: P2)

An administrator opens https://incus.<apex>/. nginx terminates TLS and proxies to the daemon. The daemon accepts an OIDC login from the identity provider of the group.

Why this priority: The UI is the operator-visible face of the hypervisor. The overview page links to it through uiSubdomain.

Independent Test: test-incus-service checks that a client reaches https://incus/. nix-unit-setups-hypervisor-config checks the served UI port and the published UI name. nix-unit-components-incus-oidc checks the OIDC settings of the daemon. test-incus-oidc-login (core/components/hosting-provider/incus/oidc-login-test.nix) does the OIDC login through authelia and its lldap directory, with core/testers/authelia-oidc-login.py in place of the browser.

Acceptance Scenarios:

  1. Given the component is enabled, When a browser opens https://<domain>/, Then nginx proxies the request to https://127.0.0.1:8443 with websocket upgrade.
  2. Given an OIDC provider answers the incus-oidc request, When the node activates, Then the server config of virtualisation.incus.runtime carries oidc.issuer, oidc.client.id and oidc.scopes.
  3. Given a user of the directory of the identity provider, When the user logs in at https://<domain>/oidc/login and accepts the consent, Then GET /1.0 with the session cookies of incus gives auth: trusted and auth_user_method: oidc, and without them auth: untrusted.

User Story 3 - Let guests share the host store (Priority: P2)

A deployer builds a guest closure once on the host. Guests in shared-store mode read the host /nix/store. A scoped SSH identity lets the deployer copy signed closures in and set per-instance profiles.

Why this priority: Shared-store mode saves disk space and build time on the host. It is the default nixStore mode of a component.

Independent Test: integrations-incus-shared-store, nix-unit-setups-shared-store and nix-unit-components-incus-store-identity.

Acceptance Scenarios:

  1. Given fediversity.incus.storeServe.enable, When the incus-store user logs in over SSH, Then ForceCommand runs incus-store-serve, which dispatches incus-store-profile set|remove or runs nix-daemon --stdio.
  2. Given the minted public halves change, When the path unit fires, Then incus-store-serve-trust rewrites authorized_keys and the trusted signing key, and stops nix-daemon.
  3. Given the host SSH key, When incus-store-host-key-publish runs, Then OpenBao KV <prefix>/public/incus-store/sshHostPublicKey holds that key.

User Story 4 - Attest the host and publish deploy tokens (Priority: P3)

The host registers its TPM endorsement key with SPIRE. It also mints Incus trust tokens for the deploy runners and publishes them to OpenBao KV.

Why this priority: These tasks let remote deployers and guest SPIRE agents trust the host. A host without SPIRE or runners works without them.

Independent Test: integrations-spire-incus, integrations-register-ek-evidence, integrations-incus-token-publish and nix-unit-components-incus-deploy-tokens.

Acceptance Scenarios:

  1. Given spireInstance and spireHostName are set, When the host boots, Then host-swtpm provides a vTPM and host-register-ek registers spiffe://fediversity/host/<network.name>.
  2. Given deployRunners names a runner, When incus-deploy-token-publish runs and a KV entry or trust name is missing, Then it mints deployer-hosts-<runner> and deployer-image-<runner> and writes tokenHosts and tokenImage to KV.

Edge Cases

  • If the KV probe write fails, incus-deploy-token-publish exits with an error and retries every 30 seconds without a start limit.
  • A runner whose KV entries and trust names all exist gets no new token.
  • virtualisation.incus.package is pkgs.incus (not LTS). Its database migration cannot go back to an older version.
  • The host has no TPM resource manager other than tpmrm0 in use. A udev rule clears SYSTEMD_WANTS for other tpmrm devices, and dev-tpmrm0.device waits at most 10 seconds.
  • A TPM never stays the hardware RNG source. A udev rule sets rng_current to none for tpm-rng-*.
  • /run/vars is tmpfs, so the store-serve keys are minted again at boot. The KV pair then replaces them, and the path unit applies the change.
  • Incus has no fine-grained authorization. Every OIDC user is an administrator, so admission control is at the identity provider.

Requirements (mandatory)

Functional Requirements

  • FR-001: The component MUST declare name = "Incus", package = pkgs.incus, the Incus logo as the icon of its artwork, uiSubdomain = "incus", certAudience = "browser" and a default subdomain of incus.
  • FR-002: When enabled, the component MUST request ssl.domains = [domain] in group tls and spiffe.enable in group spiffe. See ssl and spiffe.
  • FR-003: When enabled, the component MUST request an oidc client incus in group incus-oidc, public, with JWT access tokens, scopes openid profile email offline_access and redirect URIs on <domain>:8443 and <domain>.
  • FR-004: The oidc request MUST import incus/oidc.nix with the issuer from contracts.oidc.results.incus.default.
  • FR-005: If the issuer is https:// and fediversity.openbao-pki.caCertFile is set, the incus service MUST order after openbao-pki-trust.service and use SSL_CERT_FILE=/etc/ssl/certs/openbao-pki-bundle.crt.
  • FR-006: The code request MUST import incus.nix with fediversity.incus.uiDomain = domain and fediversity.incus.storeServe.enable = true. See nixos-module.
  • FR-007: incus.nix MUST preseed only bridge incusbr0 at 10.0.100.1/20 with the DHCP range 10.0.100.2-10.0.100.254, NAT and no IPv6, and a dir storage pool default. Services on the bridge address need these two objects at boot, and the preseed applies them even when the reconciler fails.
  • FR-007a: incus.nix MUST declare the other runtime state through the declarative-incus reconciler of the lxc/incus declarative-runtime pairing (virtualisation.incus.runtime): dns.domain and raw.dnsmasq on incusbr0, the eth0 and root disk devices of the default profile, core.https_address = ":8443" and the oidc.* keys in the server config, and the trusted metrics certificate. The reconciler removes what is no longer declared.
  • FR-007b: The reconciler MUST set adopt = true on incusbr0 and default, and MUST declare each key that the preseed sets with the same value (bridgeBase). Otherwise the preseed and the reconciler change the key back and forth on each start of incus.
  • FR-007c: declarative-incus MUST restart on failure, with no start limit.
  • FR-008: The bridge MUST serve DNS for the apex through raw.dnsmasq from incus-bridge-dnsmasq.nix, with a cname= line per bridgeCnames entry and a host-record for uiDomain.
  • FR-009: incus-bridge.nix MUST be the single source of the bridge address and subnet. core/setups/hosts-common.nix reads it.
  • FR-010: The host MUST put the bridge address first in networking.nameservers, and dhcpcd MUST ignore veth* and incusbr*.
  • FR-011: The host MUST open TCP port 8443, and also 443 when uiDomain is not empty.
  • FR-012: The nginx vhost for uiDomain MUST proxy / to https://127.0.0.1:8443 with websockets and no body size limit.
  • FR-012a: The nginx vhost for uiDomain MUST verify the daemon with /var/lib/incus/server.crt as its only trust anchor, and it MUST check the DNS name that is in that cert. nginx MUST start after incus.service.
  • FR-013: The location = /1.0/metrics MUST admit only the bridge subnet and present the client certificate that incus-metrics-cert mints. The reconciler MUST trust it as the metrics certificate otel-collector-metrics.
  • FR-014: The host MUST run nix-serve on the bridge address, port 5000, with net.ipv4.ip_nonlocal_bind = 1. nix-serve MUST sign each narinfo with the key that incus-bridge-cache-key mints once under /var/lib/incus-bridge-cache.
  • FR-014a: The default profile MUST bind /var/lib/incus-bridge-cache/public, which holds the public key as a nix.conf fragment, read-only into each container at cacheKeyDir of incus-bridge.nix, with required=false. A node that substitutes from the bridge cache MUST !include that fragment and MUST keep require-sigs on (core/setups/nixos/incus-bridge-cache.nix).
  • FR-015: When deployRunners is not empty, incus-deploy-token-publish MUST gate on incus config trust list and a KV probe write, then mint and publish missing tokens per runner under <prefix>/secret/incus-deploy/<runner>/. It MUST skip the mint for a runner only when both KV entries and both trust names of that runner are present, and the mint MUST write the same trust names and KV paths that the check reads. incus-deploy-tokens.nix gives these checks, names and paths. It MUST start after declarative-incus, because incus issues a token only when it listens on the network, and the reconciler sets core.https_address.
  • FR-016: A timer MUST start incus-deploy-token-publish-refresh every 5 minutes.
  • FR-017: When storeServe.enable, the host MUST mint the incus-store file set through generateFiles: an ed25519 SSH key pair and a Nix signing key pair named incus-store-<hostName>.
  • FR-018: The incus-store user MUST linger, MUST NOT be a trusted Nix user, and MUST have no forwarding, TTY or tunnel in its Match block.
  • FR-019: incus-store-profile MUST validate the instance name and the store path before it sets or removes a profile under /nix/var/nix/profiles/inx.
  • FR-020: incus-store-identity.nix MUST hold the user name and key file paths. windmill/f/fediversity/_jobenv.py MUST restate the same values.
  • FR-021: When spireInstance and spireHostName are both set, the host MUST run host-swtpm, host-swtpm-perms, host-register-ek and a 5-minute host-register-ek-refresh timer.
  • FR-022: When the SPIRE agent is enabled, its failure MUST start host-register-ek-refresh.
  • FR-023: The component MUST declare provides = [ "instances" ]. The setups MUST give the public uplink, the bridge firewall, the bridge CNAMEs, the deploy-runner list, the SPIRE instance and the public UI record to the instances provider node, not to a node by name.

Key Entities

  • fediversity.incus options: uiDomain, bridgeCnames, deployRunners, spireInstance, spireHostName, storeServe.enable, storeServe.user.
  • Bridge data: address 10.0.100.1, subnet 10.0.96.0/20, the provider's DHCP range 10.0.100.2-10.0.100.254 and the prefix of the operator deployments 10.0.104.0/21, from incus-bridge.nix.
  • Deploy tokens: Incus trust names deployer-hosts-<runner> and deployer-image-<runner>, and KV keys tokenHosts and tokenImage.
  • Store-serve identity: user incus-store, keys at /run/incus-deploy/incus-store-ssh.key and /run/incus-deploy/incus-store-signing.key.
  • Host SPIFFE ID: spiffe://fediversity/host/<network.name>.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: test-incus-service passes: the daemon, network, storage pool and profile devices exist, the bridge has a dns.domain, the metrics certificate is trusted, the bridge cache signs with the key the default profile binds into the containers, a reapply after a restart of incus changes nothing, ports 22, 443 and 8443 are open, a client reaches https://incus/, and nginx verifies the daemon and proxies /1.0 and /1.0/metrics to it.
  • SC-002: integrations-bridge-dns, nix-unit-setups-hypervisor-config and nix-unit-setups-apex-resolution pass.
  • SC-003: integrations-incus-shared-store, nix-unit-setups-shared-store and nix-unit-components-incus-store-identity pass.
  • SC-004: integrations-incus-token-publish, integrations-spire-incus, integrations-register-ek-evidence and nix-unit-components-incus-deploy-tokens pass.
  • SC-005: nix-unit-components-incus-oidc and test-incus-oidc-login pass.

Assumptions

  • The node vars use the vars-openbao service, which gives the OpenBao address, token, mount, prefix and CA to the publish units.
  • The far end of the host key publish is .forgejo/workflows/cd-env.sh, which reads the key to pin the host.
  • The deploy-side overlay of the deployment method gives the API trust token and the TLS material.
  • nginx and its TLS certificate come from the ssl resource.

Known Deviations

  • deployRunners holds runner network.name labels, and the trust and KV names derive from them.
  • Incus has no fine-grained authorization, so every OIDC user is an administrator.
  • When declarative-incus fails on a fresh host, the host has the bridge and the pool, but no :8443 listener, no default profile devices, no bridge zone and no OIDC. The unit tries again every 30 seconds. A preseed of these keys would hide the failure, but it would also apply again on each start of incus, so a key removed from the configuration would come back.
  • The provider identifies a certificate by its fingerprint, so the reconciler cannot adopt a trust entry. A host whose trust store has the otel-collector-metrics entry from before the reconciler needs incus config trust remove <fingerprint> once, before the first apply.