Fediversity

Feature Specification: Test Harness, Checks and Documentation

Created: 2026-09-22

Status: Implemented

Input: Reconstructed from the code at 0bf4ecd36 (core/testers/, core/checks/, core/checks/README.md, core/checks/ci.nix, core/docs/), pull requests #1210, #1790 and #1833, and the design notes of the maintainers.

User Scenarios & Testing (mandatory)

User Story 1 - Test a deployment end to end (Priority: P1)

A developer of core or of a front-end wants to prove that a deployment works. The shared harness (lib.testers) starts the machines of a deployment in a NixOS test (VMs or nspawn containers), deploys the applications with a deployment method, and checks them.

Why this priority: the deployment programs change machines. Only an end to end test shows that they work.

Independent Test: run nix build ./core#checks.x86_64-linux.apps-tf. It deploys the applications onto NixOS VMs and checks each one.

Acceptance Scenarios:

  1. Given the default configuration with the applications on, When apps-tf runs, Then each application answers on its domain.
  2. Given a front-end with its own deployment driver, When it uses lib.testers, Then it gets the same machines and the same deployment steps as core.

User Story 2 - Run one check by name, and route every check in CI (Priority: P1)

Each check has a name with a family prefix (apps-*, deployment-*, integrations-*, contracts-*, test-*, nix-unit-*), a description, and a CI runner route. A developer runs one check by name.

Independent Test: add a check without a route. The check annotation fails.

Acceptance Scenarios:

  1. Given a check name, When a developer runs nix build ./core#checks.<system>.<name>, Then only that check builds.
  2. Given a nix-unit suite, When CI runs, Then it builds the group check of the directory. The per-suite check stays available to run one suite.

User Story 3 - Read the documentation of core (Priority: P2)

A reader opens the core-docs site. It shows the concepts, the data model, the operator configuration, the hosting resources, the attribute names of the deployment programs, the module options and the reference of lib/ from its doc comments.

Independent Test: build core-docs. The flake attributes page lists the families of attributeFamilies.

Edge Cases

  • An environment variable is documented next to the code that reads it. The docs take the description from its declaration, so they cannot go out of date.

Requirements (mandatory)

Functional Requirements

  • FR-001: core MUST publish a test harness (lib.testers) that front-ends can use for their own end to end checks.
  • FR-002: The harness MUST take the modules and drivers of a front-end as parameters, and MUST NOT import a front-end.
  • FR-003: Each check MUST have a family prefix, a description and a CI runner route.
  • FR-004: core MUST build the core-docs site from the pages under docs/md/, the module options and the doc comments of lib/.
  • FR-005: The docs MUST take the names of the deployment programs from the same grammar that renders them.
  • FR-006: The docs MUST include a guide that tells a hosting provider how to find and use the web UIs of a hosting-provider deployment.
  • FR-007: The docs MUST tell a maintainer how to check a change without a deploy, how to deploy from CI and from a workstation, and where to find the deploy logs. A template MUST list each repository secret and variable that a CD job reads.
  • FR-008: A unit test under core/ MUST import core/default.nix, not the repository root entrypoint. A Nix file under core/ MUST NOT import the root entrypoint or refer to a path in api or panel (principle I). The nix-unit-testers-leaf-boundary check finds a file that does.

Key Entities

  • Check: a derivation under checks, with a name, a description and a runner route.
  • Tester: the shared harness of the NixOS tests.
  • Docs site: the core-docs package, published at core.pages.git.fediversity.eu.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The treefmt, core-docs and nix-unit-* checks pass.
  • SC-002: No check lacks a runner route.

Assumptions

  • The heavy checks run on the CI runners, not on the machine of a developer.

Known Deviations

  • The VM tests core/checks/apps/tf/nixosTest.nix, core/checks/apps/tf-incus/nixosTest.nix and core/testers/apps-deployment.nix put the root default.nix and flake.nix in their sourceFileset. This is against FR-008. The nix-unit-testers-leaf-boundary check allows these three files by name. The deployment needs the root entrypoint for these reasons (#2377):

    • The plan file of core/effects/common/write-config.nix imports the operator root and reads its boundary attribute. Only the root default.nix defines boundary.
    • core/setups/hosts-common.nix sets root-path = ../.., so the deployment refers to the repository root.
    • Windmill runs nix develop .#operator-tf-incus-all in a root flake.
    • core/components/hosting-provider/windmill/deploy-env.nix copies the whole repository.

    The three VM tests import their setups from sourceTree (core/testers/nixosTest.nix), the store copy of sourceFileset. Thus the root-path of the deployment holds only the staged files, and a change outside sourceFileset does not change the check.