Maintainer guide

This guide is for maintainers of the Fediversity code. It tells how to check a change without deploying, how to deploy from CI and from a workstation, and where to find the logs. Each step links to the notes that give more detail:

notes/bootstrap.md -- setting up Fediversity for a hosting provider.
notes/cd-credentials.md -- the deploy coordinates, and how CI uses them.
notes/private-access.md -- how to reach the admin surface.

The flake attributes page tells how the attributes below get their names. To add an application that operators deploy from the panel, read Add an application.

Before you start

A person with access to the hypervisor did the bootstrap: the Incus hypervisor runs, and it has the base image and the client certificate of the deploy.
Your address is in ADMIN_SOURCES, or you are on the VPN of the hosting provider. Without this, the admin web UIs and the Incus API refuse you.
For a deploy from a workstation, you are a member of the fediversity-admins group in the lldap directory of the hosting provider.
Your Nix has the features nix-command and flakes enabled. The nix commands in this guide need them; Prerequisites tells how to enable them.

Receiving alerting notifications

Deployments use the ADMIN_EMAIL environment variable to set email addresses (;-delimited) to send alerting emails, in core/setups/ from Grafana. A deployment with the telemetry node needs it: without it, the evaluation stops. An evaluation in the tree with no ADMIN_EMAIL (core/eval.nix and the CI builds of the nodes) gives the alertEmails argument of core/default.nix in its place.

Check a change without a deploy

You can check a change at five levels. Only the last one needs access to the hosting provider.

Build the NixOS systems of the nodes. This needs no infrastructure and changes nothing:
```
nix build ./core#nixosConfigurations.operator-mastodon.config.system.build.toplevel
```
Each node has a nixosConfigurations.<group>-<machine> attribute. The operator-* nodes use the sample operator configuration of core/setups/groups/operator.nix.
Deploy into a VM test. The tests deploy the applications with OpenTofu, and then check them:
- checks.<system>.apps-tf deploys to NixOS VMs.
- checks.<system>.apps-tf-incus deploys to Incus containers in a VM.
- checks.<system>.apps-api deploys through the API, as the panel does.
These tests need much memory. CI runs them on each pull request.
Read the scripts of a deploy. Each deploy verb is a shell script. Build it, and do not run it, to see what it would run:
```
cat "$(nix build --impure --no-link --print-out-paths .#hosting-provider-tf-incus-all)"/bin/*
```
The script calls other scripts by their store paths. You can read these with cat too. The first script only adds the dry run of the next step, and then runs the deploy script.
Evaluate a deploy, and do not run it. With DRY_RUN=1, a deploy evaluates the NixOS system of each node that it deploys, prints the derivation of each system and the command that it would run, and stops:
```
DRY_RUN=1 nix run --impure .#hosting-provider-tf-incus-all
```
It connects to no host and runs no tofu. It evaluates each node from the same configuration.nix as the deploy, with the same environment, such as DEPLOYMENT. Thus it can find errors that only your configuration causes, which step 1, with the sample configuration, does not find. It fails if a node does not evaluate. All the deploy attributes of the flake accept DRY_RUN=1. A verb that deploys no nodes, such as -plan, only prints its command.
Make a plan. The OpenTofu plan verb shows what a deploy would create, change and destroy, but changes nothing:
```
nix run --impure .#hosting-provider-tf-incus-all-plan
```
A plan reads the state and the coordinates of the hosting provider, so first prepare your shell as in Deploy from a workstation. To make a plan for an operator deployment, give its configuration as JSON in DEPLOYMENT, and use .#operator-tf-incus-all-plan. This is the configuration that the panel sends.

Deploy from CI

The CD workflows deploy each push to main that changes the files of a machine:

Workflow	What it deploys
`cd-hosting-provider.yaml`	All containers of the hosting provider, in one OpenTofu graph
`cd-hypervisor-incus.yaml`	The hypervisor that runs these containers, over SSH
`cd-dns-hosting-provider.yaml`	The DNS zone of the hosting provider

To run a different verb, start deploy-hosting-provider by hand in "Actions": select the verb (for example plan) and, if the verb needs them, the args. For example, plan with -destroy shows what a destroy would remove. Dispatching a verb other than apply lists the verbs.

The coordinates of the deploy are repository secrets and variables. notes/cd-credentials.md tells what each one is, and how to make it.

A CD run logs in to the OpenBao of the hosting provider with the OIDC token of the forge. OpenBao accepts only a run on main of this repository. A run on a different branch, or the run of a pull request, cannot get the deployer token, and the deploy refuses to start. A run that you start by hand on main gets the token, also when its ref input selects a different commit.

Deploy from a workstation

A deploy from a workstation uses your checkout, so it can deploy a branch that is not merged. It uses the same scripts as CI.

Enter the core dev shell, which has the tools:
```
nix develop ./core
```
Copy .forgejo/workflows/cd-env.example.sh to a location outside the worktree, and fill in the values. The template groups the variables by purpose, and has the same names as the repository secrets and variables. Get the values from the person who set up the hosting provider.
Load the values, and prepare the deploy:
```
. <your copy of cd-env.example.sh>
. ./.forgejo/workflows/cd-env.sh tf-incus
```
cd-env.sh refuses to continue if a required value is missing. With OPENBAO_ADDR set, it opens the SSO login of the hosting provider in your browser, and gets a deployer token that is valid for 30 minutes. The browser must be on the same machine, and port 8250 must be free.

Make a plan, and read it:

nix run --impure .#hosting-provider-tf-incus-all-plan

Apply:

nix run --impure .#hosting-provider-tf-incus-all-apply

Note --impure lets the deploy read the coordinates, so a plan proposes to create all containers again.

A full plan builds the whole fleet. To run it on a build host, forward the SSO callback from your workstation:

ssh -N -o ExitOnForwardFailure=yes -L 8250:127.0.0.1:8250 <build-host>

Then do steps 1 to 5 on the build host, and open the URL that bao shows in your browser. A deploy from a workstation tells more.

Be careful:

A deploy from a workstation deploys your checkout to the hosting provider that the coordinates name. If they name the production hosting provider, your branch replaces what CI deployed, until the next CD run.
Without the OPENBAO_* values, the deploy uses a different lane than CI, and changes every node. The next CD run changes them back.
Without CD_OIDC_ISSUER and CD_OIDC_REPOSITORY, the deploy removes the forge login from OpenBao, and CD fails after it. cd-env.sh shows a warning.
The filled-in file holds the repository secrets. Do not commit it, and do not load it automatically in each shell.

Test a feature branch

To deploy a branch or a pull request from CI, start the CD workflow by hand in "Actions", on main. Set ref to the name of the branch, or to refs/pull/<n>/head for pull request <n>. This also works for a pull request from a fork. Only a user with write access to the repository can run a workflow.

The run checks out the commit of ref, and runs its code with the secrets of the job. The workflow file stays that of main. verb: plan does a dry-run.

The run deploys to the same machines as main. Its deploy stays until the next deploy. A push to main deploys only when it changes the files of the machine. Thus, to restore main, start the workflow again with an empty ref. Runs of one workflow on main wait for each other, thus a run with ref does not overlap a deploy of main.

To test on machines that are not those of main, deploy from a workstation (see above) to a hosting provider that is for tests. Use coordinates of its own: a different state backend (TF_HTTP_ADDRESS), a different zone (DNS_DOMAIN), and a different Incus project or host. Then a test does not change the deploy of main.

To set up a hosting provider for tests, follow the bootstrap.

Test an operator branch

The operator code deploys the applications of a panel user, such as Mastodon. To test a change to that code, deploy it from the panel:

Push the branch to the main repository. The API accepts a source only from a repository that it trusts (trustedSourceUrls), and in that repository, only a branch. A pull request from a fork is not a branch, thus the API refuses it.
In the panel, open the form of a deployment, then open "Advanced".
In "Configuration source", type {"type": "git", "url": "https://git.fediversity.eu/fediversity/fediversity", "ref": "<branch>"}. The form then loads again, with the options of that branch.
Fill in the form, and deploy.

The ref floats: a deploy after a new push to the branch uses the new commit. To pin a commit, add "rev": "<commit id>". The branch must contain that commit.

To remove the test, click "Delete" on the deployment in the list of deployments. This destroys its resources.

To test a new application in the same way, first do the steps of Add an application.

Read the logs

A CD run: open the run in "Actions" on the forge.
A deploy from a workstation: the output of the nix run command.
A deploy that the panel started: the panel shows the end of the log. The full log is in Windmill, at https://jobs.<apex>/, in the fediversity workspace. The jobs of the f/fediversity/deploy and f/fediversity/destroy scripts are the deploys and the removals.
A different request to the panel or the API: only a deploy and a removal start a Windmill job. The services handle all other requests in their own process, for example the list of deployments, the forms and the build logs. These services write their requests and errors to the journal of their container: the unit fediversity-api on the node api, and the unit panel on the node panel.
Service in a container: ssh root@<incus> 'incus exec <node> -- journalctl -xeu <service>'
All containers in one place: each node sends its journal, at priority info and higher, to Loki. In Grafana at https://monitoring.<apex>/, open "Explore", select Loki, and find the records of a unit with a query such as {service_name="fediversity-api.service", service_instance_id="api"}.