Fediversity

Feature Specification: Deployment method k8s

Created: 2026-10-31 Status: Implemented Input: Issues #2632 and #2630, the effect core/effects/k8s/ and the node setup core/setups/nixos/k8s-pod.nix.

The k8s method runs each node of a group as the one pod of a Kubernetes Deployment, in a cluster of the hosting provider. The method does not switch a closure on a node. It builds an image of the node configuration, pushes it to a registry, and rolls the Deployment out to that image. If the new pod does not become ready, the method sets the Deployment back to the previous image.

The graph is an OpenTofu graph, with its state on the same HTTP state backend as the other OpenTofu methods (tf.md). The verbs and the state rules are in 004. The plans of the nodes, the submission and the key sets are in spec.md.

User Scenarios & Testing (mandatory)

User Story 1 - Deploy a node to a cluster as an image (Priority: P1)

An operator runs the k8s program for a group, with the kubeconfig of the cluster and the address of a registry. The program pushes the image of each node and applies its Deployment. The node then runs in the cluster, with systemd as PID 1 of the pod.

Why this priority: This is the only purpose of the method.

Independent Test: A VM test runs k3s and a registry in one VM and the state backend and the deployer in a container. It deploys one node of a group, checks that the pod becomes ready and that the Deployment names the image by its digest, checks that the pod keeps its address on eth0, checks that a pod in the namespace of the target can connect to the node and that a pod in a different namespace cannot, checks that /var/lib of the node is on its volume and that the machine ID is the container_uuid of the pod, and then checks that a plan shows no changes.

Acceptance Scenarios:

  1. Given a cluster that has the RuntimeClass of the method and a registry that it can pull from, When the operator runs the program, Then the image of each node goes to <registry>/<namespace>/<node>, and the Deployment of the node runs that image by its digest.
  2. Given a node whose configuration did not change, When the operator applies again, Then the push, the Deployment and the rollout are not replaced, and the pod is not restarted.
  3. Given a node whose configuration changed, When the operator applies, Then the program pushes a new image and the Deployment replaces the pod.
  4. Given a deployed node, When a pod in a different namespace connects to a port of the node, Then the connection fails. A pod in the namespace of the target can connect.

User Story 2 - Keep the previous image when a node does not become ready (Priority: P1)

The operator deploys a configuration with which systemd in the pod does not get to running. The rollout does not complete in its time limit. The program sets the Deployment back to the previous image and fails.

Why this priority: The method has no boot entry to revert. Without the rollback, a configuration that does not boot leaves the node down.

Independent Test: The same VM test deploys a configuration with a unit that always fails, after a good deploy. It checks that the program fails, that the output tells of the rollback, and that the Deployment runs the previous image again. It then checks that the new pod has the file that the test wrote in /var/lib before the failed deploy, and the same machine ID.

Acceptance Scenarios:

  1. Given a Deployment that runs a good image, When the operator deploys a configuration whose pod does not become ready, Then the program sets the Deployment back to the good image and fails.
  2. Given a first deploy whose pod does not become ready, When the rollout times out, Then the program fails, and there is no previous image to set back.
  3. Given a failed rollout, When the operator applies again with the same configuration, Then the program tries the rollout again.
  4. Given a node with state in /var/lib, When a rollout or a rollback replaces its pod, Then the new pod has the same /var/lib, the same machine ID and the same host keys of sshd.

User Story 3 - Remove the deployment of a group (Priority: P2)

An operator runs destroy for a group.

Why this priority: The operator must be able to remove the nodes of a group from the cluster.

Independent Test: none.

Acceptance Scenarios:

  1. Given a deployed group, When the operator runs destroy, Then the program removes the namespace of the target with all that is in it, which includes the claims of the volumes of the state. The images stay in the registry.

Edge Cases

  • The cluster does not have the RuntimeClass of the method: the plan fails, before the program pushes an image.
  • The registry is not reachable from the deployer: the push fails, and the program changes nothing in the cluster.
  • The cluster cannot pull the image: the pod does not become ready, and the program sets the Deployment back as for a failed boot.
  • A unit of the node failed, thus systemd is degraded: the pod is not ready.
  • The configuration of a node declares eth0 in networking.interfaces: systemd-networkd does not manage eth0, thus the address and the routes of the pod stay.
  • A rollout failed and the program set the Deployment back: the state holds the new image, and the cluster runs the previous one, until the next apply.
  • The cluster has no volume for the claim of a node: the pod does not get scheduled, does not become ready, and the program sets the Deployment back as for a failed boot.
  • A node is removed from the group, or gets a different name: the graph removes its claim, and the state of the node is lost when the StorageClass deletes the volume.
  • The spire node is not reachable at plan time: the plan fails, also the plan of a destroy, because the graph reads the trust bundle from the spire node.
  • The spire node is made again, with a new machine ID: the next plan replaces the hand-off, and the apply gives the new node the kubeconfig and the drop-files again.
  • The SPIRE server rotates its CA: the next plan changes the ConfigMap spire-agent, and the apply gives the pods the new bundle.
  • The roster of the group has a spiffe provider: the evaluation fails, because that spire node would be a pod of the same deployment.
  • No roster has a spiffe provider: the method registers nothing, and the unprovided assertion of the spiffe resource refuses a node that needs an identity.
  • A submission is given: the graph drops the nodes that the submission turns off, and an apply with a different submission fails (spec.md).

Requirements (mandatory)

Functional Requirements

  • FR-001: The method MUST run its OpenTofu graph through the shared OpenTofu runner of spec.md, with OpenTofu, its providers, skopeo and kubectl from the Nix store, and MUST keep its state on the HTTP state backend of tf.md.
  • FR-002: For each node, the graph MUST build the image of the node configuration on the deployer and push it to <registry>/<namespace>/<node>, with a tag from the node configuration. The push MUST be replaced when, and only when, the tag changes.
  • FR-003: The Deployment of a node MUST name the image by the digest that the registry gives for the pushed tag, so that a later change of the tag does not change the pod.
  • FR-004: The Deployment MUST hold one pod, MUST replace the pod and not run two pods at a time, and MUST give the pod the RuntimeClass of the method, its own user namespace, and the settings that systemd needs as PID 1 (see core/checks/integrations/k3s-oci/nixosTest.nix).
  • FR-005: The pod MUST be ready only when systemd in the pod is running.
  • FR-006: The apply of the Deployment MUST NOT wait for the rollout. A separate step MUST wait for the rollout, for a time that the operator can set. If the rollout does not complete, that step MUST set the Deployment back to the previous image and MUST fail.
  • FR-007: The method MUST create the namespace of the target, and a destroy MUST remove it. The namespace of the all target is K8S_NAMESPACE, fediversity when unset. The namespace of a per-app target is that name with -<app> appended, thus two targets do not hold the same Deployment.
  • FR-008: The method MUST read the address of the registry from K8S_REGISTRY, and MUST stop with an error that names it when it is unset. The method MUST read the kubeconfig of the cluster from the file that KUBE_CONFIG_PATH names, and the credentials of the registry, if any, from the file that REGISTRY_AUTH_FILE names. These are per-run values and MUST NOT be in the node plans. The hosting provider keeps the kubeconfig as a secret. The k8s profile of .forgejo/workflows/cd-env.sh writes the secrets KUBE_CONFIG and REGISTRY_AUTH to these files.
  • FR-009: The effect MUST publish its node names, as FR-016 of spec.md states.
  • FR-010: The method MUST apply a NetworkPolicy in the namespace of the target, before the Deployments. The policy MUST let the pods of the namespace get connections only from pods in the same namespace.
  • FR-011: The node MUST keep the address and the routes that the network plugin of the cluster puts on eth0. The gate of the firewall resource on the node MUST NOT filter eth0.
  • FR-012: For each node, the method MUST apply a PersistentVolumeClaim <node>-state, of a StorageClass and a size that the operator can set, and the pod MUST mount it on /var/lib. The claim MUST stay when the pod is replaced.
  • FR-013: The machine ID of a node MUST stay the same when its pod is replaced, and MUST be different for each node. The method MUST give it to systemd as container_uuid, from a random UUID in the state of the graph.
  • FR-014: The host keys of sshd on the node MUST be in /var/lib.
  • FR-015: When the method has the coordinates of a SPIRE server node, the SPIRE agent of each node MUST attest with k8s_psat, with a projected token of the service account node-<node> of the namespace, of the audience spire-server. Only root on the node MUST be able to read the token. The agent MUST read a copy of it, which a root service makes again within the life of the token.
  • FR-016: Before the Deployments, the method MUST give the spire node a kubeconfig of the cluster with a token of the service account spire-server of the namespace, which can review tokens, and a drop-file for each node (spire, FR-017). The token MUST go from kubectl to the spire node through a pipe, and MUST NOT be in an argument, in the state, or on stdout. The method MUST give the pods the trust bundle of the server in the ConfigMap spire-agent, from a read at plan time, and MUST NOT make a change at each apply. A destroy MUST retire the names of the namespace and remove its kubeconfig on the spire node.
  • FR-017: The setup MUST give the method the coordinates of the spire node of the central group: its FQDN, and the user, the key, the host key and the ssh options of the SSH_USER, SSH_PRIVATE_KEY_FILE and SSH_NODES layers, as for a node of the ssh method. The setup MUST read the address of the API server for the spire node from K8S_SPIRE_API_SERVER, and MUST use the address of the kubeconfig when it is unset. The path of the key MUST NOT be in the input or in the triggers of a resource, because each deploy stages the key to a different path. A destroy MUST use SSH_PRIVATE_KEY_FILE. When the roster of the group has a spiffe provider, the evaluation MUST fail with an error that names the group.

Key Entities

  • Node image: A docker archive with the rootfs of the node configuration as its one layer, and /sbin/init as its command.
  • Image reference: <registry>/<namespace>/<node>@<digest>.
  • Deployment: The Kubernetes Deployment of one node, in the namespace of the target.
  • State claim: The PersistentVolumeClaim <node>-state of one node. Its volume holds /var/lib of the node.
  • Cluster coordinates: The kubeconfig of the cluster and the address of the registry.
  • SPIRE coordinates: The address of the spire node, the user, the key, the host key and the ssh options to reach it, and the address of the API server of the cluster as the spire node reaches it.
  • Node service account: node-<node>, the identity in the token of the pod of one node. The SPIRE node alias is spiffe://fediversity/k8s/<namespace>/<namespace>/node-<node>, and the host entry is spiffe://fediversity/host/<namespace>-<node>.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After an apply, every node runs as a ready pod, from the image of its configuration.
  • SC-002: After an apply, a plan shows no changes.
  • SC-003: After a deploy whose pod does not become ready, the Deployment runs the previous image, and the program failed.
  • SC-004: After a pod of a node is replaced, the node has the same /var/lib and the same machine ID.
  • SC-005: With the SPIRE coordinates set, the agent of each node has the X.509 SVID of its host entry, and the server lists the agent with the attestor k8s_psat.

Assumptions

  • The hosting provider runs the cluster. The cluster has the RuntimeClass of the method, with a handler that gives systemd a writable cgroup file system.
  • The deployer and the cluster reach the registry at the same address.
  • The deployer can build every image, or substitute it.
  • One operator applies a given state key at a time. The backend lock enforces this.
  • The cluster gives a volume for each claim of the StorageClass of the method. The file system of the volume supports idmapped mounts, because each pod has its own user namespace.
  • The OpenBao and SPIRE servers stay outside of the cluster. The deployer reaches the spire node over ssh, and the spire node reaches the API server of the cluster.
  • The network plugin of the cluster enforces NetworkPolicy. The cluster accepts a NetworkPolicy also when nothing enforces it. With such a plugin, all pods of the cluster can connect to all ports of the nodes.

Known Deviations

  • The method does not run the deploy body of spec.md. The readiness of the pod takes the place of the verdict, thus the program does not name the unit that failed.
  • After a rollback, the state holds the image that failed, and the cluster runs the previous image. A plan does not show this difference.
  • Only /var/lib stays when the pod is replaced. A node that keeps state in a different path, such as /etc, loses it on each rollout. The persistentPaths of a component are under /var/lib.
  • The name of the k8s_psat cluster on the SPIRE server is the namespace. Two clusters with the same namespace cannot use one SPIRE server.
  • The service account spire-server can read all kubernetes nodes of the cluster, because the k8s_psat server reads the node of the pod.
  • A group that deploys with this method cannot have its own OpenBao node. OpenBao seals its unseal key to the TPM, and a pod has no TPM. The evaluation of the node fails with an assertion.
  • A group that deploys with this method cannot have its own spire node. A SPIRE server in the cluster needs these changes:
    • The server reads the token reviews with the service account of its pod and the in-cluster configuration, in place of a kubeconfig on the node.
    • The method gives the server the drop-files in a ConfigMap, or with kubectl exec, in place of ssh.
    • The agents reach the server at a stable Service name, in place of the address of the pod.
    • Agents outside of the cluster, such as TPM agents of the hosting provider, reach port 8081 through a NodePort or a LoadBalancer.
    • The server exports its trust bundle to OpenBao from the pod.
    • The data of the server is on a claim, and the CA key is in a KMS plugin, for example the transit engine of OpenBao.
  • The agent attests the node, not the workloads of the node by their pods. Workloads attest with the unix and systemd attestors of the agent.
  • The gate of the firewall resource does not filter the pod network. On that network, internalSources and adminSources do not apply, and the NetworkPolicy does not limit the traffic between the nodes of a target per node or per port.