Fediversity

Feature Specification: Component windmill

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/components/hosting-provider/windmill/default.nix, core/components/hosting-provider/windmill/windmill.nix, core/components/hosting-provider/windmill/package.nix, core/components/hosting-provider/windmill/workspace-imports-test.nix, core/components/hosting-provider/windmill/f/fediversity/, core/components/hosting-provider/windmill-otel.nix, core/components/hosting-provider/otel-wiring-test.nix, core/resources/oidc/forward-auth-domain.nix, core/checks/nix-unit-tests.nix, core/checks/default.nix), pull requests #1622 and #1691.

User Scenarios & Testing (mandatory)

User Story 1 - Run operator deploys as jobs (Priority: P1)

A front-end submits an operator deployment as a job. The windmill component runs a Windmill server and its workers on a node of the hosting-provider group. A bootstrap oneshot uploads the scripts f/fediversity/deploy and f/fediversity/destroy. A caller names one of these scripts. It does not supply code. The script runs the deploy program of 004 with nix-shell -A devShells.<system>.<attribute> --run <executable>. The component sets nixStore = "host-daemon", so the builds land in the store of the hypervisor.

Why this priority: operator deploys from a front-end run only through this node.

Independent Test: nix-unit-components-windmill-workspace-imports holds that the bootstrap uploads each workspace import, that the runnable scripts define only main, and that deploy and destroy share one job environment. nix-unit-setups-windmill-config checks the realised node.

Acceptance Scenarios:

  1. Given a fresh node, When fediversity-windmill-bootstrap.service runs, Then it logs in as admin@windmill.dev, mints an API token labeled fediversity, creates the workspace fediversity and the folder fediversity, uploads the scripts and sets instance_python_version to 3.12.
  2. Given a script that the instance already holds with the same content, When the bootstrap runs again, Then it skips the upload. A changed script gets a new version with parent_hash.
  3. Given a deploy job, When deploy.py builds the environment, Then the keys from /etc/fediversity-windmill/deploy-env.json take priority over the keys of the payload, and the payload keys take priority over os.environ.
  4. Given the API name cfg.domain, When a client requests /, Then nginx returns 404. Only /api/ reaches 127.0.0.1:8001.

User Story 2 - Give the deploy its credentials (Priority: P1)

A deploy job needs credentials for OpenBao, NetBox, DNS, the registrar and the Incus store. The front-end does not send them. The node reads them for itself.

Why this priority: without the credentials, a deploy cannot allocate addresses, publish records or reach the hypervisor.

Independent Test: nix-unit-setups-windmill-config holds that the node reads each credential from its own OpenBao render files.

Acceptance Scenarios:

  1. Given the node has services.vars-openbao, When it renders, Then each entry of deploySecrets becomes one file under /run/fediversity-windmill/deploy-secrets with owner root:windmill and mode 0440.
  2. Given the node, When openbao-agent renders, Then /run/incus-deploy/tf-token.env holds TF_VAR_token from incus-deploy/<hostName>/tokenHosts, with mode 0440.
  3. Given the node, When openbao-agent renders, Then /run/incus-deploy/incus-store-ssh.key and /run/incus-deploy/incus-store-signing.key hold the Incus store keys.
  4. Given a minted API token, When the bootstrap finishes, Then it runs bao kv put on <prefix>/secret/windmill/apiToken.

User Story 3 - Read job logs in the browser (Priority: P2)

An administrator wants to follow a deploy. Windmill CE cannot do SSO. The component puts an Authelia forward-auth vhost on uiDomain in front of 127.0.0.1:8001. The vhost presents the API token of Windmill on behalf of the user.

Why this priority: the deploys run without the UI. The UI helps an administrator to find failures.

Independent Test: nix-unit-setups-windmill-config holds that the vhost sends a browser without a session to the portal and passes the job-log websocket.

Acceptance Scenarios:

  1. Given the policy sets deployAdminUIs, When the oidc resource applies windmill-ui, Then the vhost asks for one_factor and admits only the group fediversity-admins.
  2. Given a minted token, When the bootstrap writes /run/windmill-ui-auth/token.conf, Then nginx includes it through the glob /run/windmill-ui-auth/*.conf.

User Story 4 - Point Windmill at the collector (Priority: P3)

The component requests otel with windmill-otel.nix. This sets the OTLP variables on windmill-server, windmill-worker and windmill-worker-native.

Why this priority: no signal leaves the process. The OSS build of Windmill has stub exporters.

Independent Test: test-otel-wiring boots a node with the collector, windmill.nix and windmill-otel.nix, and checks the OTel variables on the units.

Acceptance Scenarios:

  1. Given an otel endpoint, When the node renders, Then each unit gets OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, OTEL_SERVICE_NAME=windmill-<role> and OTEL_RESOURCE_ATTRIBUTES=service.namespace=fediversity.

Edge Cases

  • POST /api/users/setpassword of Windmill CE returns 500. The bootstrap writes the argon2 hash of the superadmin password into the password table over the database DSN with SET ROLE windmill_admin.
  • The bootstrap has Restart=on-failure, RestartSec=30s and StartLimitIntervalSec=0. It retries the login 12 times.
  • A new render of windmill-superadmin/adminPassword restarts the bootstrap through reloadUnits.
  • The database request uses rotating credentials, not dynamic ones. Dynamic credentials restart the units at each SVID rotation.
  • The server and both workers have RestartSec=5s and start after openbao-agent-default.
  • On a standalone eval without services.vars-openbao, deploySecrets is empty.

Requirements (mandatory)

Functional Requirements

  • FR-001: The component MUST declare components.windmill with nixStore = "host-daemon", certAudience = "bootstrap" and the default subdomain windmill.
  • FR-002: The component MUST declare the options operatorDns (serverUrl, provider with default powerdns), operatorNetbox.rangeSelector (kind, value), operatorRegistrar (maxCost in EUR cents, ownerHandle, baseUrl, nameservers) and uiDomain (default jobs.<apex>).
  • FR-003: The implementation MUST request spiffe, ssl for cfg.domain, netbox and dns with consumerName = "windmill", and otel from windmill-otel.nix.
  • FR-004: The ipam request MUST write the operator curation to services.fediversity-windmill.deployEnv.HOSTING_CONFIG, and set deploySecrets from deploy-credentials.nix only when options.services ? vars-openbao.
  • FR-005: The windmill-db request MUST ask postgresql for database and user windmill with rotating credentials, and declare in groupRoles the group roles windmill_user, with all privileges on the database, and windmill_admin, with BYPASSRLS, a member of windmill_user and granted to the user.
  • FR-006: The windmill-db request MUST contain only data. The code module MUST render its client (database.nix) from contracts.postgresql.results.windmill.default, MUST render nothing when the result is absent, and when the result is present MUST set services.windmill.database.createLocally = false and services.windmill.database.urlPath to the result urlFile.
  • FR-007: The windmill-ui request MUST set domain = cfg.uiDomain, upstream = "127.0.0.1:8001", authorizationPolicy = "one_factor", allowedGroups = [ "fediversity-admins" ], proxyWebsockets = true and upstreamAuthorizationInclude = "/run/windmill-ui-auth/*.conf".
  • FR-008: The component MUST generate windmill-superadmin/adminPassword with openssl rand -hex 32 and declare windmill as a written instance of api-token.
  • FR-009: The windmill-smtp request MUST set SMTP_HOST, SMTP_PORT, SMTP_USERNAME, SMTP_FROM=noreply@<domain> and SMTP_TLS_IMPLICIT=false on windmill-server, and pass SMTP_PASSWORD through contracts.envSecrets.
  • FR-010: The code request MUST import windmill.nix, set services.windmill.baseUrl = "https://<uiDomain>", and set deployEnv.DEPLOYMENT_ROOT to the git-tracked repository tree and deployEnv.DEPLOYMENT_NIXPKGS to pkgs.path.
  • FR-011: windmill.nix MUST use package.nix, which keeps only the Python and Bash toolchains of Windmill.
  • FR-012: windmill.nix MUST make the user windmill a trusted Nix user, pin the registry entry nixpkgs to DEPLOYMENT_NIXPKGS, turn off the global registry and enable flakes.
  • FR-013: windmill.nix MUST give windmill-worker nix, openssh, git and bash on its PATH, and KillMode=mixed.
  • FR-014: The deploy script MUST read DEPLOYMENT_ROOT, ATTRIBUTE, EXECUTABLE, SYSTEM, NIXPKGS, SSH_PRIVATE_KEY_FILE, INTERNAL_HOSTS, PUBLISHED_HOSTS, ENVIRONMENTS and TF_HTTP_ADDRESS from its job environment.
  • FR-015: The implementation MUST request garage through mkBackupsGarageRequest with the bucket windmill-backups, and the code module MUST add a streamingBackup of the windmill-db database to that bucket with the restic provider, as the user root.

Key Entities

  • Workspace fediversity: holds the folder f/fediversity with _runner, _jobenv, deploy and destroy. core holds _runner.py and gives it to api as core.lib.components.hosting-provider.windmill-runner.
  • API token: minted on each boot, stored in OpenBao KV and in the UI auth snippet.
  • Deploy environment: /etc/fediversity-windmill/deploy-env.json, from deployEnv.
  • Deploy secrets: one file per variable under /run/fediversity-windmill/deploy-secrets.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: nix-unit-components-windmill-workspace-imports and nix-unit-setups-windmill-config pass.
  • SC-002: nix-unit-components-incus-store-identity passes, so _jobenv.py agrees with incus-store-identity.nix.
  • SC-003: test-otel-wiring passes.
  • SC-004: On a live node, fediversity-windmill-bootstrap.service is active, and <prefix>/secret/windmill/apiToken holds the current token.

Assumptions

  • The postgresql, smtp, netbox and dns resources resolve on the node (see postgresql).
  • core/render.nix sets the operator* options.
  • OpenBao and openbao-agent run on the node.

Known Deviations

  • windmill-otel.nix sets OTLP variables, but the OSS exporters of Windmill are stubs. No signal reaches the collector.