Fediversity

Feature Specification: Method family deployment-methods

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/effects/default.nix, core/effects/common/, core/effects/ssh/run.sh, core/effects/incus/run.sh, core/effects/tf/common/, core/effects/env-config.nix, core/render/catalog.nix), pull requests #1531, #1705, #1712, #1784, #1790, #1848 and #1857.

This spec covers what the four deployment methods share. The program grammar, the verbs, the published key sets and the state rules are in 004. This spec does not repeat them. Each method has its own item spec, listed in the Index.

User Scenarios & Testing (mandatory)

User Story 1 - Deploy a node and get a verdict on it (Priority: P1)

An operator runs a deployment program for a group. Each method gets the closure onto the node in its own way. After that, all methods run the same deploy body, core/effects/common/deploy.sh. The body builds the closure, copies it, switches the node, and then decides whether the node came up. The operator reads one verdict block per node.

Why this priority: A deploy that exits 0 on a broken node hides the failure. The verdict is the only signal that a closure works on the node.

Independent Test: deployment-verdict-failed, deployment-verdict-loop, deployment-verdict-spawn and deployment-unit-parsers. The first three are NixOS tests through an SSH deploy of one host. deployment-ssh-hosts and deployment-nixos-hosts check the deploy body on a trivial host.

Acceptance Scenarios:

  1. Given a closure that leaves a unit in the failed state, When the deploy finishes, Then the verdict names the unit and the program exits non-zero.
  2. Given the same closure and DEPLOY_STRICT_UNITS=0, When the deploy finishes, Then the verdict still names the unit and the program exits 0.
  3. Given a unit that systemd never spawned because its cgroup could not be opened, When the verdict runs, Then it names the unit as a host spawn fault and does not blame the closure.
  4. Given a unit in a restart loop, When a deploy finishes, Then the verdict reports the loop and does not fail the deploy for it.

User Story 2 - Deploy the submitted configuration (Priority: P1)

The api submits a configuration through the DEPLOYMENT variable. The deploy shell reads DEPLOYMENT when it is evaluated, merges it into the baked configuration, and makes the plan of each node. The apply deploys only the nodes that the merged configuration enables.

Why this priority: If the node list and the node configurations disagree, a container gets no applications and fails at register_ek. An apply with a different DEPLOYMENT would deploy a configuration that the operator did not submit.

Independent Test: nix-unit-effects-submitted-config and nix-unit-effects-write-config, both in nix-unit-effects.

Acceptance Scenarios:

  1. Given a submission that names an application without enable, When the merge runs, Then the application is off.
  2. Given a submission that does not name an application, When the merge runs, Then the baked entry for that application stays.
  3. Given a submission with applications, domain and email inside the picked deployment-method.<method> branch, When the merge runs, Then these values move to the top level and deployment-method is removed.

User Story 3 - Redeploy a node when its inputs change, and only then (Priority: P2)

The deploy shell reads some coordinates from the environment when it is evaluated, and the plan of each node holds them. A change in one of these coordinates must redeploy the node. A change in a per-run value, such as a temporary credential path, must not.

Why this priority: A missed input leaves a node on a stale closure. A per-run input replaces every node on every apply.

Independent Test: nix-unit-env-config (the test-node-identity-covers-every-node-read test), nix-unit-effects-write-config (the test-submittedArgs-ignores-the-credential-staging-path and test-submittedArgs-still-tracks-the-coordinates tests) and deployment-eval-cache.

Acceptance Scenarios:

  1. Given a change in DNS_DOMAIN, When the deploy shell makes the plan of a node, Then the plan and thus nixos_conf change, and the node is redeployed.
  2. Given two runs that stage the NetBox token at different mktemp -d paths, When submittedArgs removes the credential paths, Then the two runs give equal args and thus equal plans.
  3. Given a getEnv read in the files that make the plan of a node, When test-node-identity-covers-every-node-read runs, Then the key of that read is in nodeIdentityKeys, thus the front-ends forward it.
  4. Given a node that the seed pass already evaluated, When the deploy pass evaluates the same node with the same inputs, Then it reads the cached derivation and does not evaluate again.

User Story 4 - Judge a running node without a switch (Priority: P3)

The incus graphs run a second pass with DEPLOY_VERDICT_ONLY=1 after register_ek. This pass does not build, copy or switch. It waits for the secret chain of the node, restarts failed units that the closure wants, and then gives the verdict.

Why this priority: The first pass judges a node before its secrets arrive. Only the second pass can say if the node converged.

Independent Test: deployment-verdict-chain, deployment-verdict-loop and deployment-verdict-failed.

Acceptance Scenarios:

  1. Given a secret chain that converges, When the verdict-only pass runs, Then it waits for the chain and sweeps only after the chain is up.
  2. Given a chain head that is in the failed state, or that shows no change for DEPLOY_CHAIN_STALL, When the gate samples it, Then the gate stops early and names that member.
  3. Given a unit that is still in a restart loop after its loop budget, When the verdict-only pass runs, Then the pass fails and names the unit and its budget.

Edge Cases

  • A node that the deployer cannot reach gets a few samples in the chain gate. Then the verdict runs and reports the node as unreachable.
  • A severed system bus leaves the unit sets empty. The report prints <not read> for them, not <none>.
  • A closure whose .link files name a PermanentMACAddress that the node does not have is refused before the copy. A closure with no such file is not gated.
  • A switch that never activates makes the transport revert the boot entry to the running generation (revert_bootloader).
  • A failed unit that the deployed configuration does not define is set aside and does not fail the deploy (#1712).
  • A unit that systemd skips on a Condition*= is not restarted by the sweep, and the verdict still names it.
  • A cached evaluation record that names a derivation the store does not hold is ignored. A cache directory that cannot be written is not an error.
  • A node whose own PATH resolves nothing still runs remote commands, because each remote command appends the sw/bin of the deployed closure to PATH.

Requirements (mandatory)

Functional Requirements

  • FR-001: The effect registry core/effects/default.nix MUST define the four methods as ssh-hosts, tf-hosts, tf-incus-hosts and tf-incus-operator-hosts. core/render/catalog.nix (hostsFor) MUST expose them as ssh, tf, tf-incus and tf-incus-operator, each with its setup module, setup arguments and description. A method MAY declare an available predicate on the group, and every consumer MUST take the methods of a group from hostsForGroup. tf-incus-operator MUST be available only for a group that runs an application (applicationComponents in core/setups/wiring/resource-routing.nix).
  • FR-002: Every method MUST deploy a node through the shared body core/effects/common/deploy.sh. A transport prologue (core/effects/ssh/run.sh for ssh and tf, core/effects/incus/run.sh for tf-incus and tf-incus-operator) MUST supply the hooks target, remote_exec, copy_closure, switch_runtime_max_args, finalize_bootloader and revert_bootloader. deploy.sh MUST NOT name a transport.
  • FR-003: The deploy body MUST run its phases in this order: build, wait for the node, push the closure, chain gate, snapshot, switch, read the switch result, never-confirmed check, owned units, sweep, settle, cycle watch, loop watch, classify, verdict, report, finalize, conclude.
  • FR-004: The deploy body MUST set the system profile and the boot entry (phase_finalize) before it exits non-zero for a failed unit, so that a node with a failed unit keeps the new boot entry.
  • FR-005: The verdict MUST name failed units, restart loops, units that cycle, start jobs that are too old, and units that systemd never spawned. It MUST exit non-zero for a failed unit unless DEPLOY_STRICT_UNITS=0.
  • FR-006: A unit MAY declare its own loop budget with DEPLOY_LOOP_BUDGET=<seconds> in its Environment=. The budget MUST delay the loop verdict and MUST NOT remove it.
  • FR-007: With DEPLOY_VERDICT_ONLY=1, the deploy body MUST NOT build, copy or switch. It MUST wait for the secret chain within DEPLOY_CHAIN_TIMEOUT (default 600) before the snapshot, and MUST restart only failed units that the closure wants.
  • FR-008: The tuning variables DEPLOY_LOOP_GRACE (default 600), DEPLOY_SETTLE_ATTEMPTS (default 12), DEPLOY_CYCLE_INTERVAL (default 5), DEPLOY_CYCLE_WINDOWS (default 4), DEPLOY_CHAIN_STALL (default 120) and DEPLOY_LOOPING_BEFORE_GAP (default 0.3) MUST have a default in deploy.sh.
  • FR-009: The deploy body MUST cache the toplevel derivation of a node under DEPLOY_EVAL_CACHE_DIR, keyed on nixos_conf alone. The cache MUST NOT be a source of truth.
  • FR-010: submitted-config.nix MUST merge a submission into the baked configuration with plain builtins. It MUST raise applications, domain and email out of the picked method branch, remove deployment-method, and default enable to false for each application that the submission names.
  • FR-011: submittedPlan (write-config.nix) MUST read DEPLOYMENT when the deploy shell is evaluated, and MUST make the plan of each node from it. core/effects/common/lib.nix MUST record the SHA-256 of that DEPLOYMENT in the shell as DEPLOYMENT_HASH. bootstrap-apply.sh MUST refuse an apply with a different DEPLOYMENT.
  • FR-012: write-config.nix MUST remove the deployer credential paths (netbox.apiTokenFile, dns.apiTokenFile, openbao.tokenFile, openbao.caCertFile) from the plan arguments.
  • FR-012a: writeConfig MUST write the plan of the node to plan.json through the writer of core/lib/plan-json.nix, and the generated configuration.nix MUST read it through the reader. The writer gives a path of a file in the operator tree as { _repoPath = "<path relative to the tree>"; }, and fails on a value that is not data. The reader gives the path back in the operator tree.
  • FR-012b: The generated configuration.nix MUST take the plan of the node from plan.json and MUST NOT evaluate the plan again. For an incus effect it MUST also import core/resources/network/guest-api.nix. The addresses that NetBox allocates MUST NOT be in the plan: an incus node reads them at run time from the incus guest API.
  • FR-013: Each effect that bakes a node map MUST give the passthru publishedNames = { nodes, disabledApplications } from the same submitted plan as its node configurations (tfRun in core/effects/common/lib.nix). nodes MUST NOT hold a disabled node. The api MUST read it with p: p.publishedNames or null. bootstrap-apply.sh MUST refuse a DEPLOYMENT with a node map that has no publishedNames.
  • FR-014: nodeIdentityKeys (node-identity.nix) MUST list every coordinate that the files that make the plan of a node read from the environment, so that the front-ends forward it. The plan holds the values of these coordinates, thus nixos_conf covers them.
  • FR-015: Each method MUST declare its coordinates in core/effects/env-config.nix with a description and a default text. env-keys.nix MUST fold them, plus self-pins.nix, hosting-env.nix and host-env.nix, into deployEnvKeys and deployEnvKeysByMethod.
  • FR-016: tfRun (core/effects/common/lib.nix) MUST give each OpenTofu method one program per verb in tf-verbs.nix plus run. Every verb MUST go through core/effects/tf/common/bootstrap-apply.sh. A verb program MUST clear BOOTSTRAP and BOOTSTRAP_SEED.
  • FR-017: tfRun MUST seed an empty self-state (seed-self-state.sh) before apply, destroy, plan and refresh, so that the self-read of a first apply reads empty and does not fail.
  • FR-018: Each operator-facing effect MUST declare its settings through the internal effect-meta option (effect-meta.nix). core/effects/default.nix MUST set its kind to deployment when the effect has a nodes roster, and to effect otherwise. An effect that the form does not offer MUST declare no effect-meta and MUST be in buildingBlocks.
  • FR-019: Each settings leaf in effect-meta.nix MUST have a default.
  • FR-019a: The node roster of effect-meta.nix (the ssh-hosts per-node settings and the tf-incus-image node enum) MUST hold the nodes of each group that runs an application, and MUST NOT name a group.

Key Entities

  • Method: One entry in hostsFor. It names a setup module under core/setups/, its arguments, its verbs, a description and optionally available.
  • Transport prologue: core/effects/ssh/run.sh or core/effects/incus/run.sh. It resolves the connection and defines the hooks of the deploy body.
  • Deploy body: core/effects/common/deploy.sh. It is sourced through DEPLOY_COMMON and runs the phases in main.
  • Verdict block: The Deploy verdict for <target>: report. It lists the switch result, the system state, the unit sets and the strict-units mode.
  • Plan file: The plan.json file of a node. writeConfig writes it from submittedPlan when the deploy shell is evaluated, and the generated configuration.nix reads it.
  • Node identity keys: The nodeIdentityKeys list in node-identity.nix. It names the coordinates that the plan of a node reads, which the front-ends forward.
  • Effect meta: The internal effect-meta option. It holds the settings schema that the form union in core/lib/data-model.nix reads.
  • Building block: An effect in buildingBlocks (core/effects/default.nix), such as a registrar or tf-netbox-* effect. A setup, a check or the catalog composes it into a program. It is not in the form union.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: A deploy that leaves a unit failed exits non-zero on that apply and on the next, as deployment-verdict-failed shows.
  • SC-002: Two applies with the same submission and different credential staging paths produce the same nixos_conf.
  • SC-003: The seed pass and the deploy pass of one node evaluate the node once, as deployment-eval-cache shows.
  • SC-004: Every environment variable that a method reads is in its declared key set, as the test-*-reads-only-declared tests in nix-unit-env-config show.
  • SC-005: A verdict-only pass on a chain head in the failed state stops before DEPLOY_CHAIN_TIMEOUT.

Assumptions

  • The nodes run systemd and NixOS. The verdict reads unit states with systemctl.
  • The caller of an incus graph schedules the verdict-only pass after the step that publishes what the units wait for (register_ek).
  • The api passes DEPLOYMENT as JSON in the shape that submitted-config.nix reads.
  • The deployer has nix-instantiate and can evaluate the source tree that the generated configuration.nix and its plan.json name by store path.

Known Deviations

  • write-config.nix cannot serialize a NixOS configuration (FIXME, NixOS/nix#4000). The plan is data, and the generated configuration.nix imports the modules that plan.json names. Each effect makes the plans of its nodes once, when the deploy shell is evaluated. The node configuration is data: attribute sets, lists, plain values, and the paths of files in the repository. It holds no function, derivation, or string with a store context (nix-unit-setups-plan-data).
  • caller is a path relative to the root of the operator tree, so a setup outside a tree that has a boundary cannot be a caller (FIXME).
  • core/effects/default.nix does not allow custom deployment types (FIXME).

Index

  • ssh.md: ssh, a deploy over SSH to machines that already exist. It provisions nothing.
  • tf.md: tf, the same SSH deploy inside an OpenTofu graph with tracked state.
  • tf-incus.md: tf-incus, which creates the Incus containers of a group and deploys onto them in one OpenTofu graph.
  • tf-incus-operator.md: tf-incus-operator, which adds the per-tenant shared resources, DNS records, the domain and SPIRE registration to tf-incus.