Fediversity

Feature Specification: Resource garage

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code at 093e68b6f (core/resources/garage/default.nix, core/resources/garage/restic-s3.nix, core/resources/garage-forward-compat-test.nix, core/setups/wiring/garage-kv-stems.nix, core/setups/wiring/minted-garage-keys.nix, core/setups/wiring/supporting-requests.nix, core/setups/wiring/resource-routing.nix, core/setups/hosts-common.nix, core/components/operator/testing/lib.nix), pull requests #1184, #1238, #1246, #1266, #1361 and #1818, and issues #527, #1205 and #1206.

User Scenarios & Testing (mandatory)

User Story 1 - Give an application buckets and an S3 key on a shared garage (Priority: P1)

An operator application stores media in S3. Its component states ensureBuckets and ensureKeys. The policy prefixes every name with the tenant, so two deployments on one garage cluster do not collide. Under provisioning = "tf", the operator OpenTofu graph creates the buckets and the key on the shared garage node through its admin API. The graph then writes the S3 pair to OpenBao KV. The consumer node reads it back.

Why this priority: the Fediverse applications keep their media in S3.

Independent Test: deployment-tf-incus-operator runs the garage and OpenBao half of the operator graph. nix-unit-setups-garage-kv-stems checks the provisioning shape and the KV grants. nix-unit-setups-minted-garage-keys checks which keys have no minter, and nix-unit-setups-garage-key-minter checks that the deploy target then fails.

Acceptance Scenarios:

  1. Given tenant = "<deployment>" and a bucket media, When the deployment is evaluated, Then the physical bucket is <deployment>-media, and the node gets that name as contracts.s3.results.media.default.bucket.
  2. Given a key with kvName = "mastodon", When the graph mints the pair, Then it publishes vars/public/mastodon-s3/accessKey and vars/secret/mastodon-s3/secretKey. The stem carries no tenant prefix.
  3. Given the requests of a group, When provisioningOf runs, Then each bucket keeps only name and serveAsWebsite, and each key keeps name, kvName and the per-bucket read, write and owner grants.
  4. Given the same requests, When kvStemsOf runs, Then it returns one <kvName>-s3 stem per key without duplicates. The deployer token may write each stem, and the app node may read its own.
  5. Given provisioning = "tf" and a key with pair files that no generateFiles instance of the group mints, When the all or combined target of a tf-hosts, tf-incus-hosts or ssh-hosts deployment is evaluated, Then the evaluation fails and names the key. Only the tf-incus-operator-hosts graph mints pairs.

User Story 2 - Provision buckets and keys on the group's own garage node (Priority: P1)

A group with its own garage node provisions its own requests on that node. Examples are the buckets of the telemetry app and the cache-backup bucket of each valkey namespace. An OpenTofu reconciler (core/resources/garage/provision.nix) on the declarative-runtime library holds the buckets, the keys and the grants, with its state local to the node. The oneshot garage-provision-own applies it. The buckets are the option services.garage.runtime.buckets, typed from the provider schema of the declarative-runtime pairing, with one resource per bucket. The keys and the grants are each one for_each resource, because the config of a key depends on files that land at apply time. A key with no pair files has no other minter, so the garage node mints it with generateFiles.

Why this priority: the operator graph mints only the requests of the operator groups. Without the node-side route, a bucket of the group has no key.

Independent Test: nix-unit-setups-minted-garage-keys, integrations-garage-provision.

Acceptance Scenarios:

  1. Given an own key with s3AccessKeyFile = null and s3SecretKeyFile = null, When the garage node is evaluated, Then mintedGarageKeyInstances names <kvName>-s3, and the node generates accessKey (GK plus 24 hex digits, mode 0444) and secretKey (64 hex digits, mode 0400).
  2. Given such a minted key, When withMintedGarageKeyFiles runs, Then the key reads its pair from /run/vars/public/<kvName>-s3/accessKey and /run/vars/secret/<kvName>-s3/secretKey, and garage-provision-own imports it.
  3. Given a key whose pair files are set, When withMintedGarageKeyFiles runs, Then the key stays the same.
  4. Given garage is not healthy, When garage-provision-own runs, Then it polls /health of the admin API on 127.0.0.1:3903 for up to 120 seconds, fails, and restarts after 30 seconds.
  5. Given a key whose pair files are not there yet, When garage-provision-own runs, Then it applies everything else and writes the key to /var/lib/garage-provision/pending. Once the files are there, the timer of garage-provision-late-keys applies the key within a minute. The unit exits 0 while garage is down.
  6. Given a bucket or a key that exists in garage but not in the state, When the reconciler runs, Then it adopts the object with an import block and keeps its id.
  7. Given a bucket, a key or a grant that leaves the config, When the reconciler runs, Then it deletes it. It empties a bucket before it deletes it.
  8. Given a key in the state whose pair files went missing, When the reconciler runs, Then the key stays, with the id that the state holds.
  9. Given a state with one for_each resource for the buckets, from before each bucket had its own resource, When the reconciler runs, Then its moved blocks move each bucket to its own resource, and it changes no bucket.

User Story 3 - Run a co-located garage for a single-app test (Priority: P2)

A single-app check runs garage on the app node. It sets tenant = "", a local domain and provisioning = "in-nix". The app mints its own S3 pair through generateFiles. The garage-layout-init oneshot sets the layout. The ensure-garage oneshot then creates the buckets, applies the CORS rules and imports the key.

Why this priority: the application checks run without a hosting provider.

Independent Test: contracts-s3-minio covers the s3 contract. The single-app test-*-service checks use this path through core/components/operator/testing/lib.nix.

Acceptance Scenarios:

  1. Given provisioning = "in-nix" and a key with pair files, When the consumer node is evaluated, Then it wants contracts.generateFiles.want.<kvName>-s3 with accessKey (public) and secretKey (secret).
  2. Given a bucket with corsRules.enable = true, When ensure-garage runs, Then it sets the rules with bucket-cors.sh over the S3 API on loopback, with a key that it mints over the admin API and deletes again. A bucket with corsRules.enable = false loses its rules.

User Story 4 - Read the old website option (Priority: P3)

A stored configuration uses the old option website. The bucket option renames it to serveAsWebsite, so the old shape still evaluates.

Why this priority: stored configurations must keep their meaning.

Independent Test: nix-unit-resources-garage-forward-compat.

Acceptance Scenarios:

  1. Given a bucket with website = true, When it is evaluated, Then serveAsWebsite is true, and the result equals the new shape.
  2. Given a bucket with neither option, When it is evaluated, Then serveAsWebsite is false.

User Story 5 - Back up the media buckets of an application (Priority: P2)

An operator enables fediversity.mediaBackup.instances.<app> on the node of Mastodon, Pixelfed or PeerTube. Each day, media-backup-<app>.service copies the media buckets of the application into the prefix media/ of its <app>-backups bucket. rclone crypt encrypts the name and the content of each object. A replaced or deleted object moves to the history of that run. The media-backup-<app> command restores the buckets, lists the history and erases one object from the backup.

Three measures remove from the backup the media that the application or an erasure request deletes:

  • The history expires after retentionDays. Its default for an application is 30 days, the retention of all the backups of the application (backupRetention.application), so that the expiry removes the objects that the application deletes after the same period as the database backups.
  • For an erasure request (GDPR Art. 17), media-backup-<app> erase removes the object from the backup and from all of its history at once.
  • After a restore of the media without the database, the operator runs the cleanup of orphan media of the application, for example tootctl media remove-orphans of Mastodon, so that the objects of deleted records do not come back.

Why this priority: the buckets hold the uploads of the end users, and nothing else holds a copy.

Independent Test: integrations-media-backup.

Acceptance Scenarios:

  1. Given an enabled instance, When the timer starts the service, Then the target holds one encrypted object for each source object that excludes does not match, and a second run with no change transfers nothing.
  2. Given a credential that arrives empty, When the service starts, Then it fails and names the credential.
  3. Given a deleted or overwritten source object, or a deleted bucket that the provisioning made again, When media-backup-<app> restore runs, Then the object of the last backup is back with its metadata.
  4. Given a stamp from media-backup-<app> stamps, When restore --at <stamp> runs, Then each object is the version of that backup, and an object made later stays.
  5. Given an erasure request, When media-backup-<app> erase <bucket> <key> runs, Then no restore can put back that object.
  6. Given a history older than retentionDays, When a backup runs, Then it deletes that history.

Edge Cases

  • corsRules apply under each provisioning mode: on the co-located test path, in the operator graph and on the node reconciler (provisionFor). The v2 admin API has no verb for them, so each sets them over S3 with a temporary key (bucket-cors.sh). No key is shared between deployments or kept after the run.
  • An ensureAccess entry that sets none of read, write and owner stops the evaluation. The error names the key and the bucket.
  • garage-layout-init assigns the single-node layout only when the live layout version is 0. Later runs change nothing. The co-located path uses it too, and ensure-garage starts after it, so a run of ensure-garage makes no new layout version.
  • The S3 API has no TLS (api.url is http://<api.domain>:3900). The website endpoint is https://.
  • The wildcard names *.<api.domain> and *.<web.rootDomain> need a DNS-01 certificate.
  • A destroy of an operator deployment empties and deletes all of its buckets, the <app>-backups buckets too. No restore is possible after it, and an export for a migration must finish before it. A grace period for the backups (#2097) is not implemented.

Requirements (mandatory)

Functional Requirements

  • FR-001: The request MUST accept enable, ensureBuckets and ensureKeys. It MUST hold only data.
  • FR-002: A bucket MUST accept serveAsWebsite, corsRules (enable, allowedHeaders, allowedMethods, allowedOrigins), and MUST accept website as a renamed form of serveAsWebsite.
  • FR-003: A key MUST accept s3AccessKeyFile, s3SecretKeyFile (both nullable), kvName (default the key name) and ensureAccess.<bucket> with read, write and owner.
  • FR-004: The policy MUST accept enable, domain, tenant (default empty), provisioning (in-nix or tf, default tf), rpc.port (3901), api.domain (s3.<domain>), api.port (3900), web.rootDomain (web.<domain>) and web.internalPort (3902).
  • FR-005: The resource MUST prefix bucket, key and ensureAccess names as <tenant>-<name> when tenant is not empty, and MUST NOT prefix kvName.
  • FR-006: For each bucket of each request, the consumer side MUST set the want and the resolved instance of the s3 contract under the logical name, with the request bucket (the logical name) and the result bucket (the prefixed bucket name), endpoint (api.url), port (api.port), region (garage), apiDomain, webDomain and webUrl. A component reads its buckets from contracts.s3.results.<bucket>.default in a static module that sets nothing when its bucket has no result.
  • FR-007: Under provisioning = "in-nix", the consumer MUST want a generateFiles instance <kvName>-s3 for each key with pair files.
  • FR-008: Every key pair MUST publish to the KV stem <kvName>-s3, with accessKey public and secretKey secret, whoever mints it.
  • FR-009: provisioningOf MUST drop the key file paths, MUST keep serveAsWebsite, and MUST give corsRules without enable when it is enabled, else null.
  • FR-010: kvStemsOf MUST return the stems of all keys of the requests, without duplicates and without regard to the minter.
  • FR-011: The garage node MUST mint a pair for each own key whose two pair files are null, and MUST leave a key with set files alone.
  • FR-012: The garage node MUST run garage_2 with replication_factor = 1, and MUST read its admin token from the garage-admin vars instance (openssl rand -base64 32) through LoadCredential.
  • FR-013: The garage node MUST open the ports 80, 443, api.port, rpc.port and the admin port 3903.
  • FR-014: bucketEndpointVhosts MUST proxy *.<web.rootDomain> to port 3902 and *.<api.domain> to port 3900 on the local host.
  • FR-015: cacheBackupRequestFor MUST give one request per namespace, with a bucket and a key named <namespace>-cache-backups that has read, write and owner.
  • FR-016: The resource MUST request certificates for api.domain, web.rootDomain and their wildcards.
  • FR-017: ensure-garage MUST fail when a step fails. It MUST import a key only when garage does not have it, and MUST fail when a key file stays missing or empty.
  • FR-018: ensure-garage MUST allow the website of a bucket with serveAsWebsite = true, and MUST deny it for a bucket with serveAsWebsite = false.
  • FR-019: apply MUST fail the evaluation when an ensureAccess entry sets none of read, write and owner.
  • FR-020: The <app>-backups key of Mastodon, Pixelfed and PeerTube MUST have read on each media bucket of its application.
  • FR-021: Each of these applications MUST want a generateFiles instance <app>-media-backup with the secret files password and salt, also when the media backup is off.
  • FR-022: fediversity.mediaBackup.instances.<app> MUST be off by default. The resource MUST have no default for retentionDays, and mkMediaBackup MUST set its default to backupRetention.application.days. When it is off, the node MUST want no credential for it.
  • FR-023: The service MUST load only the <app>-backups key and the encryption keys. Only the transient unit media-backup-run-<app> of the command MUST load the <app> key, which may write the source buckets.
  • FR-024: Each component that requests <app>-backups MUST want a generateFiles instance <app>-backups-restic with the secret file password, and each valkey node MUST want <namespace>-cache-backups-restic for each namespace when its backup is on. restic-s3.nix MUST give this file to the restic job as the credential RESTIC_PASSWORD. Only the backups of openbao and terraform-backend MUST use a fixed password.
  • FR-025: When a repository has a key for the password ReallySafePasswordTrustMe39, the restic job MUST add a key for its own password and remove the key of the old password before it runs restic.
  • FR-026: Under provisioning = "tf", the all and combined deploy targets of a deployment that does not use tf-incus-operator-hosts MUST fail the evaluation when a key with pair files has no generateFiles instance in the group.
  • FR-027: All the backups of Mastodon, Pixelfed and PeerTube, the valkey backups, and the database backups of lldap, Authelia and the panel MUST keep their data for the same period, backupRetention.application of core/components/operator/app-resources.nix (30 days). restic MUST prune them with --keep-within, so that no snapshot stays that is older than this period before the newest snapshot.

Key Entities

  • Tenant: the prefix of all physical names on a shared cluster. The setup uses the deployment namespace, or the group when there is none.
  • KV stem: <kvName>-s3, the OpenBao vars instance of one key pair.
  • s3 contract result: the plain data of one requested bucket on the consumer node, in contracts.s3.results.<bucket>.default (nix/contracts/lib/contracts/definitions/s3.nix).
  • Provisioning shape: the JSON-safe form of the requests that the operator graph reads (provisioningOf).
  • Minted key: an own key with no pair files, which the group's garage node generates.
  • Media backup: media-backup.nix and media-backup.sh, which keep current/<bucket> and history/<stamp>/<bucket> below the target prefix, encrypted.
  • restic helper: restic-s3.nix, which gives a restic job the pair of a stem as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, with AWS_DEFAULT_REGION=garage, and the repository password as RESTIC_PASSWORD.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: nix-unit-resources-garage-consumer, nix-unit-resources-garage-forward-compat, nix-unit-resources-plain-requests, nix-unit-setups-garage-kv-stems, nix-unit-setups-minted-garage-keys, nix-unit-setups-operator-config, nix-unit-setups-valkey-config, nix-unit-setups-backup-retention and panel-api-frontend pass.
  • SC-002: deployment-tf-incus-operator, contracts-s3-minio and integrations-media-backup pass.
  • SC-003: No bucket that a provisioner creates is left without a key, and every published stem has a write grant for its publisher and a read grant for its consumer.

Assumptions

  • Garage has no entry in the provided table or in providerModules of setups and wiring. The provisioning policy decides the route.
  • The node that holds the buckets is the node whose component provides objectStore. The garage component declares it.
  • The key pairs travel through OpenBao KV with generateFiles.
  • The cache-backup keys feed the backups of the valkey component.
  • The admin port is not a setting. The garage resource, garage-webui and the operator TF effect read it from one file (core/resources/garage/admin-port.nix). The operator TF effect passes it as garage_admin_port, because its graph cannot read the settings of a node in another group.

Known Deviations

  • The media backup does not encrypt the metadata of an object, for example its content type. rclone crypt encrypts only the name and the content.
  • After a restore of the media, the next backup uploads all of the objects again. The backup compares the modification times of the server (--use-server-modtime --update), and the restored objects are newer than their copies.
  • A restore needs the bucket. The provisioning must make a deleted bucket again before media-backup-<app> restore runs.
  • FR-025 keeps the old password ReallySafePasswordTrustMe39 in the Nix store of each node with a generated password, also after each repository has changed its key. The migration does nothing when the repository has no key for the old password.