Fediversity

Feature Specification: Deployment method ssh

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code at 093e68b6f (core/effects/ssh/, core/setups/ssh-hosts.nix, core/setups/hosts-common.nix, core/setups/wiring/ssh-defaults.nix, core/effects/common/effect-meta.nix, core/render/hosts.nix, core/checks/deployment/ssh-hosts/, core/checks/deployment/nixos-hosts/), issue #723

The ssh method updates NixOS hosts that already exist. It reaches each host over SSH, copies the closure and switches the host to it. It creates no machine, no address and no state. The setup is ssh-hosts, and the effect is core/effects/ssh/effect.nix.

The deploy body is shared. run.sh defines the SSH transport hooks and then sources core/effects/common/deploy.sh. The phases, the verdict and its knobs are in spec.md. The program grammar and the verbs are in ../004-deployment-programs/spec.md. The tf method (see tf.md) runs the same run.sh for each node from OpenTofu.

User Scenarios & Testing (mandatory)

User Story 1 - Update existing hosts in order (Priority: P1)

An operator has a roster of NixOS hosts that they can reach over SSH. They run the ssh-hosts program. Each host gets its new configuration, one host after the other.

Why this priority: This is the only purpose of the method.

Independent Test: deployment-ssh-hosts deploys two targets from a deployer VM. It checks that the closure's tmpfiles rules apply before the units start, that both targets serve on :443 from the deployer, and that TLS ends on the ssl resource. Each target takes a login only on a port that is not 22. The deploy reaches target1 through the sshOpts of its own SSH_NODES entry, and target2 through SSH_CONFIG_FILE. deployment-nixos-hosts boots the same kind of configuration as test nodes with no transport. It is the baseline for this check.

Acceptance Scenarios:

  1. Given two reachable hosts in the node list, When the operator runs the program, Then the script prints DEPLOYING <name> for each node in list order and switches each host to its closure.
  2. Given a node that fails, When the program runs, Then the later nodes still deploy, the script prints DEPLOY FAILED for <name> (exit N) for each failed node, and the program exits 1 at the end.
  3. Given a successful switch, When the deploy ends, Then the host's bootloader entry points at the new generation.

User Story 2 - Reach each host with its own identity (Priority: P2)

Hosts of one roster can differ in address, user and key. The operator states one shared identity and, per node, the overrides.

Why this priority: Without overrides, all hosts must share one user, one key and the roster FQDN.

Independent Test: nix-unit-setups (test setups-ssh-defaults) checks the layering in core/setups/wiring/ssh-defaults.nix. nix-unit-effects (test env-config) checks that the ssh and tf readers parse SSH_USER and SSH_NODES. nix-unit-effects (test effects-ssh-option) checks that an sshOpts item in argv form fails at evaluation, and that the form schema refuses it with a pattern. deployment-ssh-hosts deploys a target that takes a login only on the port that the sshOpts of its own SSH_NODES entry names.

Acceptance Scenarios:

  1. Given no overrides, When the setup builds a node's ssh, Then it uses user root, the SSH_PRIVATE_KEY_FILE key and the roster host.
  2. Given sshNodes.defaults and sshNodes.nodes.<name>, When the setup builds that node's ssh, Then the node entry wins over the defaults, and the defaults win over the built-in values.
  3. Given a node override with sshOpts, When the setup merges it, Then the list replaces the default list and does not append to it.
  4. Given a node entry that is null, When the setup merges it, Then the node gets no override.
  5. Given a set node with a null field, When the setup merges it, Then the field inherits sshNodes.defaults, and a null host gives the roster host.
  6. Given an sshOpts item that is not a bare Key=Value, such as -o Port=2222 or -J, When the deploy is evaluated, Then the evaluation fails with a message that names the form.
  7. Given SSH_USER, When the setup builds a node's ssh, Then it uses that user in place of root, below both sshNodes layers.
  8. Given the form schema of the ssh-hosts settings, When a front-end validates an sshOpts item, Then the pattern of the item admits the same items as the type of scenario 6.

User Story 3 - Get through a slow or broken target (Priority: P2)

A target can be slow, can hold the bootloader lock, or can lose its D-Bus during the switch. The deploy works around these faults or reports them.

Why this priority: A deploy that hangs or gives a false success hides the state of the host from the operator.

Independent Test: deployment-ssh-recovery breaks the target from outside. It covers a blocked start job, a switch that cannot be scheduled, a hanging systemd-run, a dead bus, a stuck logind, and the bootloader lock retry and its budget. The deployment-verdict-* checks read the verdict on the same transport (see spec.md).

Acceptance Scenarios:

  1. Given a switch that still holds the bootloader lock, When the bootloader install gets exit 11, Then it tries again every 5 seconds, up to DEPLOY_BOOT_LOCK_ATTEMPTS times.
  2. Given a target in a container, When the switch runs, Then the switch has a RuntimeMaxSec=180 limit. On bare metal it has no limit.
  3. Given a container target, When the bootloader install fails, Then the failure hint tells the operator to use tf-incus.

User Story 4 - Refuse a host that shows a different key (Priority: P2)

The operator can state the public host key of a node. The deploy then logs in only to a host that shows that key.

Why this priority: The deploy logs in as a user that can switch the system. A host that is not the node must not get that login.

Independent Test: deployment-ssh-host-key drives run.sh and reads with ssh -G the options that each connection gets, with and without host_key, and with a host-key option in sshOpts, in SSH_CONFIG_FILE and for a jump host. deployment-ssh-hosts gives each target a host-key and deploys to it. Before the deploy, it shows that the key of each target is the host-key, and that a different host_key stops remote_exec with Host key verification failed.

Acceptance Scenarios:

  1. Given a node with host-key, When the deploy connects, Then it accepts only that key, and a host that shows a different key stops the deploy of that node before the login.
  2. Given a node with no host-key, When the deploy connects for the first time, Then it accepts the key and saves it. A later deploy that reads the same file refuses a different key.
  3. Given a host-key option in sshOpts or in SSH_CONFIG_FILE, When the deploy connects, Then that option wins over the values of the deploy.

Edge Cases

  • No key_file, no SSH_PRIVATE_KEY_FILE and no ssh-agent: BatchMode=yes makes SSH fail at once and does not prompt.
  • SSH_CONFIG_FILE sends the host to a different address: the script logs the change.
  • A ProxyJump= option: direct ssh gets it as a ProxyCommand, and nix-copy-closure keeps the original form.
  • A remote command that hangs: the keepalive ends it after about REMOTE_EXEC_TIMEOUT seconds (default 60).
  • A target without the deployer's PATH: remote_exec adds the fallback prefix (check deployment-remote-path-fallback).
  • The combined environment: all nodes go into one configuration on the first host of the roster.
  • A changed host key: with host-key, the deploy of that node stops. Without it, the key of the first connection is saved in ssh-hosts-known-hosts, and a deploy that reads the same file refuses a different key. A fresh CI runner has a new $TMPDIR and no saved key, so it accepts the key again.
  • A node that is installed again gets a new host key: change its host-key, or remove its line from ssh-hosts-known-hosts.
  • A ProxyJump host: it gets accept-new with the same file, also when the node has a host-key. The connection to the node runs end to end through the jump host and checks the key of the node.

Requirements (mandatory)

Functional Requirements

  • FR-001: The method MUST deploy the nodes of nodes one at a time, in list order, in the foreground.
  • FR-002: A failed node MUST NOT stop the later nodes. The program MUST exit 1 when one or more nodes failed.
  • FR-003: Each node MUST run run.sh with key_file, host_key, ssh_opts, host, username and nixos_conf in the environment, and run.sh MUST source deploy.sh.
  • FR-004: nixos_conf MUST come from writeConfig with effect = "ssh-hosts" and multiNode = true.
  • FR-005: The key MUST come from key_file, else from SSH_PRIVATE_KEY_FILE, else from ssh-agent.
  • FR-006: Every SSH call MUST use BatchMode=yes. SSH_CONFIG_FILE, when set, MUST reach SSH as -F.
  • FR-007: remote_exec MUST use ConnectTimeout=10 and a keepalive of ServerAliveInterval=5 with REMOTE_EXEC_TIMEOUT / 5 probes.
  • FR-008: copy_closure MUST use nix-copy-closure --to --gzip --use-substitutes.
  • FR-009: In a container, the switch MUST run with RuntimeMaxSec=180.
  • FR-010: finalize_bootloader MUST run switch-to-configuration boot under timeout 120, and retry exit 11 every 5 seconds up to DEPLOY_BOOT_LOCK_ATTEMPTS (default 60) times.
  • FR-011: revert_bootloader MUST do nothing.
  • FR-012: The setup MUST layer a node's SSH settings as built-in defaults, then sshNodes.defaults, then sshNodes.nodes.<name>, and fall back to the roster host. The built-in user MUST be SSH_USER, else root. A null or absent field MUST inherit the layer below. hostKey MUST come only from sshNodes.nodes.<name>.
  • FR-013: The method MUST read SSH_PRIVATE_KEY_FILE at evaluation through core/render/hosts.nix, and SSH_USER, SSH_NODES and its other keys through the ssh entry of core/effects/env-config.nix.
  • FR-014: With host_key, every SSH call to the node MUST use StrictHostKeyChecking=yes, GlobalKnownHostsFile=/dev/null and a temporary known_hosts that holds only * <host_key>. The script MUST remove that file when it exits.
  • FR-015: Without host_key, every SSH call to the node MUST use StrictHostKeyChecking=accept-new with the known_hosts file ssh-hosts-known-hosts in the directory of the key file, else in $TMPDIR. When neither directory is writable, SSH MUST use its default file.
  • FR-016: The host-key options of FR-014 and FR-015 MUST come after ssh_opts, so that a host-key option in sshOpts wins. The script MUST NOT give a host-key option that SSH_CONFIG_FILE sets.
  • FR-017: A ProxyJump host MUST get StrictHostKeyChecking=accept-new with the known_hosts file of FR-015.
  • FR-018: Each item of sshOpts MUST be a bare ssh option Key=Value, such as Port=2222. run.sh MUST put -o in front of each item. The type of sshOpts in the host-ssh submodule and in the ssh-hosts settings MUST refuse an item that starts with - or has no Key= (core/effects/common/ssh-option.nix). The type MUST carry a JSON Schema pattern that admits the same items, read as ECMA-262 and as POSIX ERE, so that the form schema of each sshOpts item refuses the items that the type refuses.

Key Entities

  • Node: {deployment-name, nixos-configuration, ssh}. The list order is the deploy order.
  • Host SSH (host-ssh): host, username (default SSH_USER, else root, in the setup), key-file, host-key (one line of the ssh_host_*_key.pub of the host, default null) and sshOpts (a list of bare Key=Value options, FR-018).
  • SSH nodes map (sshNodes): {defaults, nodes.<name>}, parsed from the SSH_NODES JSON. Every field is optional. hostKey is a field of nodes.<name> only. The effect metadata settings describe it for a front end.
  • Transport hooks: remote_exec, copy_closure, switch_runtime_max_args, finalize_bootloader, revert_bootloader and activation_failure_hint.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After a run, every reachable node runs its new closure, and deployment-ssh-hosts passes.
  • SC-002: One unreachable node gives exit 1 and a DEPLOY FAILED line, and does not block the other nodes.
  • SC-003: A held bootloader lock costs at most DEPLOY_BOOT_LOCK_ATTEMPTS times 5 seconds, then the deploy fails.
  • SC-004: deployment-ssh-recovery and the four deployment-verdict-* checks pass on the SSH transport.
  • SC-005: A node with a host-key that the host does not show gives Host key verification failed and no login, and deployment-ssh-host-key passes.
  • SC-006: A deploy reaches a node on a port that only the sshOpts of the node's own SSH_NODES entry names, and a node on a port that only SSH_CONFIG_FILE names, and deployment-ssh-hosts passes.

Assumptions

  • Each host already runs NixOS and accepts SSH as a user that can switch the system.
  • The deployer can build every closure, or substitute it.
  • A node with no host-key is reached over a trusted network for its first connection, because that connection accepts any key.
  • A host that is a container is deployed with tf-incus, not with this method.

Known Deviations

  • The caller option carries a FIXME: it must also work from outside this repository.