diff --git a/content/en/docs/next/applications/backup-and-recovery.md b/content/en/docs/next/applications/backup-and-recovery.md new file mode 100644 index 00000000..60bd571a --- /dev/null +++ b/content/en/docs/next/applications/backup-and-recovery.md @@ -0,0 +1,326 @@ +--- +title: "Application Backup and Recovery" +linkTitle: "Backup and Recovery" +description: "Back up and restore managed databases (Postgres, MariaDB, ClickHouse, FoundationDB) with BackupJob, Plan, and RestoreJob." +weight: 4 +--- + +This guide covers backing up and restoring **Cozystack-managed databases** — Postgres, MariaDB, ClickHouse, and FoundationDB — as a tenant user: running one-off and scheduled backups, checking status, and restoring from a backup either in place or into a separate target instance. + +{{% alert color="warning" %}} +**These backups are data-only.** Each strategy snapshots the database contents through the operator's native mechanism (CloudNativePG barman, mariadb-operator dumps, Altinity `clickhouse-backup`, FoundationDB `backup_agent`). They do **not** capture the `apps.cozystack.io/*` CR, its `HelmRelease`, chart values, or operator-managed Secrets. + +To restore you must either: +- keep the source application alive and restore in place (each driver re-bootstraps data into the existing operator-managed cluster), **or** +- pre-provision an empty target application of the same Kind, then restore into it. + +For backups that include the application's Helm release, CRs, and PVC snapshots (used for VMInstance / VMDisk), see [Backup and Recovery (VMs)]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}). +{{% /alert %}} + +## Prerequisites + +- A `BackupClass` exists in the cluster for the application Kind you want to back up. Run `kubectl get backupclasses` to confirm; if none is present, ask your administrator to follow the [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) guide. +- S3-compatible storage. Either provision an in-cluster `Bucket` (shown below) or use external S3 coordinates supplied by your administrator. +- `kubectl` and kubeconfig for the management cluster. + +## List available BackupClasses + +`BackupClass` resources are cluster-scoped and tell you which application Kinds can be backed up and which driver handles each: + +```bash +kubectl get backupclasses +``` + +Example output: + +``` +NAME AGE +postgres-data-backup 14m +mariadb-data-backup 14m +clickhouse-data-backup 14m +foundationdb-data-backup 14m +velero 1d +``` + +Use the `BackupClass` name when creating a `BackupJob` or `Plan`. The examples below assume `tenant-user` for the tenant namespace; substitute your own. + +## Provision the storage Bucket + +If your administrator has not pre-configured external S3, provision an in-cluster `Bucket` in the tenant namespace: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Bucket +metadata: + name: db-backups + namespace: tenant-user +spec: + users: + backup: + readonly: false +``` + +```bash +kubectl apply -f bucket.yaml +kubectl -n tenant-user wait hr/bucket-db-backups --for=condition=ready --timeout=300s +``` + +The `Bucket` controller materialises a `bucket--backup` Secret in the namespace carrying a `BucketInfo` JSON blob. The S3 endpoint, bucket name, and access keys come from there. + +## Create per-application backup credentials + +Each driver expects per-application credential Secrets in the application namespace — the strategy templates reference them by name. The snippets below assume a single shell session: first read the bucket credentials, then run only the per-driver block for the application Kind you are setting up. + +### Read the bucket credentials + +Run this once per shell session. Every per-driver block below reuses `$ACCESS_KEY`, `$SECRET_KEY`, and `/tmp/bucket.json`: + +```bash +kubectl -n tenant-user get secret bucket-db-backups-backup \ + -o jsonpath='{.data.BucketInfo}' | base64 -d > /tmp/bucket.json +ACCESS_KEY=$(jq -r .spec.secretS3.accessKeyID /tmp/bucket.json) +SECRET_KEY=$(jq -r .spec.secretS3.accessSecretKey /tmp/bucket.json) +``` + +If you start a new shell, re-run that snippet before continuing. + +### Postgres + +Project the credentials in the keys CNPG's barman client expects: + +```bash +kubectl -n tenant-user create secret generic my-postgres-cnpg-backup-creds \ + --from-literal=ACCESS_KEY_ID="$ACCESS_KEY" \ + --from-literal=ACCESS_SECRET_KEY="$SECRET_KEY" +``` + +When the S3 endpoint uses a self-signed certificate (the SeaweedFS default), also create a CA Secret: + +```bash +kubectl -n tenant-user create secret generic my-postgres-cnpg-backup-ca \ + --from-file=ca.crt=/path/to/ca.crt +``` + +### MariaDB + +```bash +kubectl -n tenant-user create secret generic my-mariadb-mariadb-backup-creds \ + --from-literal=AWS_ACCESS_KEY_ID="$ACCESS_KEY" \ + --from-literal=AWS_SECRET_ACCESS_KEY="$SECRET_KEY" +``` + +For self-signed endpoints, add `my-mariadb-mariadb-backup-ca` carrying `ca.crt` the same way. + +### ClickHouse + +ClickHouse backups read S3 credentials from the chart-emitted `-backup-s3` Secret directly. Set `backup.enabled: true` on the ClickHouse application and fill in `backup.*` with the bucket coordinates — no extra Secret is needed for the BackupClass flow. See the [ClickHouse application reference]({{% ref "/docs/next/applications/clickhouse" %}}) for the `backup.*` values. + +### FoundationDB + +FoundationDB's `backup_agent` requires a `blob_credentials.json` payload in a specific shape. This block reads the bucket endpoint from `/tmp/bucket.json` (created by [Read the bucket credentials](#read-the-bucket-credentials) above) and reuses `$ACCESS_KEY` / `$SECRET_KEY` from the same step: + +```bash +ENDPOINT_FULL=$(jq -r .spec.secretS3.endpoint /tmp/bucket.json) +ENDPOINT_HOSTPORT=${ENDPOINT_FULL#http://} +ENDPOINT_HOSTPORT=${ENDPOINT_HOSTPORT#https://} +ACCOUNT_NAME="${ACCESS_KEY}@${ENDPOINT_HOSTPORT}" + +jq -nc \ + --arg account "$ACCOUNT_NAME" \ + --arg key "$ACCESS_KEY" \ + --arg secret "$SECRET_KEY" \ + '{accounts: {($account): {api_key: $key, secret: $secret}}}' \ + > /tmp/blob_credentials.json + +kubectl -n tenant-user create secret generic my-fdb-fdb-backup-creds \ + --from-file=blob_credentials.json=/tmp/blob_credentials.json +``` + +Your administrator must also patch the FoundationDB `BackupClass` parameters with the resolved `accountName`, `bucket`, `region`, and `secureConnection` values before the first backup runs. Otherwise the first `BackupJob` fails fast with a validation error (`accountName is required`) — this is the intentional fail-loud behaviour for a half-configured tenant. + +## Run a backup + +### One-off backup + +Use a `BackupJob` for an ad-hoc backup (for example, before a risky change): + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: BackupJob +metadata: + name: my-postgres-adhoc + namespace: tenant-user +spec: + applicationRef: + apiGroup: apps.cozystack.io + kind: Postgres + name: my-postgres + backupClassName: postgres-data-backup +``` + +```bash +kubectl apply -f backupjob.yaml +kubectl -n tenant-user get backupjobs +kubectl -n tenant-user describe backupjob my-postgres-adhoc +``` + +When the `BackupJob` reaches `phase: Succeeded`, the driver creates a `Backup` object with the same name. That name is what you reference when restoring. + +Replace `Postgres` / `postgres-data-backup` with `MariaDB` / `mariadb-data-backup`, `ClickHouse` / `clickhouse-data-backup`, or `FoundationDB` / `foundationdb-data-backup` for the other drivers. + +### Scheduled backup + +Use a `Plan` for cron-driven recurring backups: + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: Plan +metadata: + name: my-postgres-daily + namespace: tenant-user +spec: + applicationRef: + apiGroup: apps.cozystack.io + kind: Postgres + name: my-postgres + backupClassName: postgres-data-backup + schedule: + type: cron + cron: "0 */6 * * *" # every 6 hours +``` + +Each scheduled run creates a `BackupJob` (and, on success, a `Backup`) named after the `Plan` with a timestamp suffix. + +```bash +kubectl apply -f plan.yaml +kubectl -n tenant-user get plans +kubectl -n tenant-user get backupjobs -l backups.cozystack.io/plan=my-postgres-daily +``` + +## Check backup status + +List `BackupJob` and `Backup` resources in the namespace: + +```bash +kubectl -n tenant-user get backupjobs +kubectl -n tenant-user get backups +``` + +Inspect a failed run: + +```bash +kubectl -n tenant-user get backupjob my-postgres-adhoc -o jsonpath='{.status.message}' +kubectl -n tenant-user describe backupjob my-postgres-adhoc +``` + +For driver-side detail, inspect the operator-native CR each driver materialises (one of `cnpg.io/Backup`, `k8s.mariadb.com/Backup`, `apps.foundationdb.org/FoundationDBBackup`, or the ClickHouse strategy `Pod`). + +## Restore in place + +An **in-place restore** replays the backup into the **same** application. Use this to roll back accidental deletion or corruption on a live database you intend to keep using under the same name. + +{{% alert color="warning" %}} +In-place restore is **destructive**. Each driver wipes or replaces existing data on the source application; any writes since the backup point are lost. If you cannot afford to lose recent writes, use [Restore to a copy](#restore-to-a-copy) instead. +{{% /alert %}} + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: RestoreJob +metadata: + name: my-postgres-restore-inplace + namespace: tenant-user +spec: + backupRef: + name: my-postgres-adhoc + # targetApplicationRef omitted: driver restores into Backup.spec.applicationRef. + # options: + # recoveryTime: "2026-05-01T12:00:00Z" # Postgres only; RFC3339 PITR +``` + +```bash +kubectl apply -f restorejob.yaml +kubectl -n tenant-user get restorejobs +kubectl -n tenant-user describe restorejob my-postgres-restore-inplace +``` + +### Per-driver caveats + +- **Postgres (CNPG)** — the driver deletes the live `cnpg.io/Cluster` and its PVCs, then re-bootstraps from the Barman archive. Connections drop for the duration. `spec.options.recoveryTime` (RFC3339) is supported for point-in-time recovery; omit it to restore to the latest WAL. +- **MariaDB** — the operator replays the logical dump into the live `MariaDB` via `mariadb-import`. Pre-existing tables will collide; pre-truncate the relevant schemas if your dump does not include `DROP TABLE`. +- **ClickHouse** — the Altinity strategy does **not** pass `clickhouse-backup --rm`. You are responsible for dropping conflicting tables on the source before submitting the `RestoreJob`; otherwise the operation fails with a duplicate-table error. +- **FoundationDB** — the operator pauses the FoundationDB cluster, clears the keyspace, and replays the backup via `fdbrestore`. Any data written after the snapshot is lost. Only one `FoundationDBBackup` directory may exist per cluster at a time — the driver stops any prior backup before starting a new one. + +## Restore to a copy + +A **to-copy restore** replays the backup into a **different**, freshly-provisioned application of the same Kind. Use this for disaster-recovery drills, side-by-side validation, branch databases, or migrating to a new version of the upstream operator. + +First, provision an empty target application with the same Kind. For example, an empty `Postgres`: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Postgres +metadata: + name: my-postgres-restored + namespace: tenant-user +spec: + # ...same shape as the source, no bootstrap data required... +``` + +Wait for the target to become Ready, then submit a `RestoreJob` that points at it: + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: RestoreJob +metadata: + name: my-postgres-restore-to-copy + namespace: tenant-user +spec: + backupRef: + name: my-postgres-adhoc + targetApplicationRef: + apiGroup: apps.cozystack.io + kind: Postgres + name: my-postgres-restored +``` + +The source application stays untouched. Cross-namespace restores are **not** supported — `targetApplicationRef` is a local reference; the target must live in the same namespace as the `RestoreJob`. + +## Limitations and lifecycle + +- **Data-only scope.** Application CRs, HelmReleases, chart values, and operator-managed Secrets (e.g. `cnpg.io` superuser secret, `clickhouse-installation` users) are not captured. Pre-provision the target application before a to-copy restore. +- **Archive retention is driver-owned.** Deleting a Cozystack `Backup` CR removes the artefact reference but leaves the actual S3 object intact. Each driver enforces its own retention: + - CNPG: `retentionPolicy` on the strategy (`30d` default in the admin example). + - MariaDB: configure `cleanupStrategy` on the operator-side `Backup` CR or rotate at the bucket level. + - ClickHouse: governed by the in-pod sidecar's retention configuration. Tenants who need to purge an archive call `DELETE /backup//remote` on the sidecar. + - FoundationDB: each `BackupJob` owns a discrete blob-store directory; clean up at the bucket level. +- **One running backup per FoundationDB cluster.** The driver enforces this by stopping any prior `FoundationDBBackup` on the same cluster before starting a new one. +- **ClickHouse depends on the in-chart sidecar.** The Altinity strategy is a thin HTTP client; the backup itself runs inside each `chi-*` Pod via `clickhouse-backup`. Disabling `backup.enabled` on the application also disables the BackupClass flow. + +## Troubleshooting + +If a `BackupJob` or `RestoreJob` ends in `phase: Failed`, check the message field: + +```bash +kubectl -n tenant-user get backupjob my-postgres-adhoc -o jsonpath='{.status.message}' +kubectl -n tenant-user get restorejob my-postgres-restore-inplace -o jsonpath='{.status.message}' +``` + +Then look at the operator-native CR the driver created: + +```bash +# Postgres +kubectl -n tenant-user get backups.cnpg.io +# MariaDB +kubectl -n tenant-user get backups.k8s.mariadb.com,restores.k8s.mariadb.com +# ClickHouse +kubectl -n tenant-user logs -l backups.cozystack.io/owned-by.BackupJobName=my-clickhouse-adhoc +# FoundationDB +kubectl -n tenant-user get foundationdbbackups.apps.foundationdb.org,foundationdbrestores.apps.foundationdb.org \ + -l backups.cozystack.io/owned-by.BackupJobName=my-fdb-adhoc +``` + +## See also + +- [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) — how administrators define strategies and `BackupClass` resources. +- [Backup and Recovery (VMs)]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}) — the parallel guide for VMInstance / VMDisk backups (HelmRelease + CRs + PVC snapshots). +- [Velero Backup Configuration]({{% ref "/docs/next/operations/services/velero-backup-configuration" %}}) — administrator setup for the Velero-driven VM backups. diff --git a/content/en/docs/next/applications/clickhouse.md b/content/en/docs/next/applications/clickhouse.md index bb998eed..df315859 100644 --- a/content/en/docs/next/applications/clickhouse.md +++ b/content/en/docs/next/applications/clickhouse.md @@ -20,6 +20,10 @@ It is used for online analytical processing (OLAP). ### How to restore backup from S3 +{{% alert color="warning" %}} +**Backups: prefer the `BackupClass` flow.** `backup.enabled` and the S3 fields (`s3Region`, `s3Bucket`, `endpoint`, `s3PathOverride`, `s3AccessKey`/`s3SecretKey` or `s3CredentialsSecret`) are still required — they materialise the in-pod `clickhouse-backup` sidecar that the Altinity backup strategy talks to. However, `backup.schedule`, `backup.cleanupStrategy`, and `backup.resticPassword` (which drive the legacy chart-managed CronJob doing dump + restic, and the matching restic restore flow documented below) are **superseded** by the Cozystack backups framework: define a `BackupClass` + `Altinity` strategy once, then drive scheduled backups via `Plan` and restores via `RestoreJob`. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup). +{{% /alert %}} + 1. Find the snapshot: ```bash diff --git a/content/en/docs/next/applications/foundationdb.md b/content/en/docs/next/applications/foundationdb.md index 71e94323..8804b2d6 100644 --- a/content/en/docs/next/applications/foundationdb.md +++ b/content/en/docs/next/applications/foundationdb.md @@ -66,6 +66,11 @@ resources: ### Backup (Optional) +{{% alert color="warning" %}} +**The chart-level `backup.*` values documented below are deprecated.** The in-chart `FoundationDBBackup` wiring is superseded by the Cozystack backups framework: define a `BackupClass` + `FoundationDB` strategy once, then drive backups via `BackupJob` / `Plan` and restores via `RestoreJob`. Existing tenants with `backup.enabled=true` continue to render the legacy `FoundationDBBackup` CR unchanged. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup). +{{% /alert %}} + + ```yaml backup: enabled: true diff --git a/content/en/docs/next/applications/mariadb.md b/content/en/docs/next/applications/mariadb.md index 284e8f26..d7926735 100644 --- a/content/en/docs/next/applications/mariadb.md +++ b/content/en/docs/next/applications/mariadb.md @@ -110,6 +110,9 @@ more details: ### Backup parameters +{{% alert color="warning" %}} +**The chart-level `backup.*` values documented below are deprecated.** The legacy `mariadb-dump` + `restic` flow is superseded by the Cozystack backups framework: define a `BackupClass` + `MariaDB` strategy once, then drive backups via `BackupJob` / `Plan` and restores via `RestoreJob`. Existing tenants with `backup.enabled=true` continue to render the legacy resources unchanged. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup). +{{% /alert %}} | Name | Description | Type | Value | | ------------------------ | ----------------------------------------------- | -------- | ------------------------------------------------------ | diff --git a/content/en/docs/next/applications/postgres.md b/content/en/docs/next/applications/postgres.md index cf282bbc..f5d9a008 100644 --- a/content/en/docs/next/applications/postgres.md +++ b/content/en/docs/next/applications/postgres.md @@ -27,6 +27,10 @@ This managed service is controlled by the CloudNativePG operator, ensuring effic ## Operations +{{% alert color="warning" %}} +**Backups: prefer the `BackupClass` flow.** The chart-level `backup.*` values documented below still configure the Barman object store and S3 credentials that backups read from, but the chart-emitted `ScheduledBackup` and the `bootstrap`-based recovery flow have been **superseded** by the Cozystack backups framework: define a `BackupClass` + `CNPG` strategy once, then drive scheduled backups via `Plan` and restores via `RestoreJob`. See [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) (tenant guide) and [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}) (admin setup). +{{% /alert %}} + ### How to enable backups To back up a PostgreSQL application, an external S3-compatible storage is required. diff --git a/content/en/docs/next/operations/services/managed-app-backup-configuration.md b/content/en/docs/next/operations/services/managed-app-backup-configuration.md new file mode 100644 index 00000000..a6d86373 --- /dev/null +++ b/content/en/docs/next/operations/services/managed-app-backup-configuration.md @@ -0,0 +1,343 @@ +--- +title: "Managed Application Backup Configuration" +linkTitle: "Managed Application Backup Configuration" +description: "Configure strategies and BackupClasses for logical data backups of managed databases (Postgres, MariaDB, ClickHouse, FoundationDB)." +weight: 31 +--- + +This guide is for **cluster administrators** who configure backup strategies for Cozystack-managed database applications: Postgres, MariaDB, ClickHouse, and FoundationDB. Once strategies and `BackupClass` resources are in place, tenants run backups and restores by creating [BackupJob, Plan, and RestoreJob]({{% ref "/docs/next/applications/backup-and-recovery" %}}) resources with no further admin action. + +{{% alert color="info" %}} +This page covers **data-only** backups driven by each operator's native backup mechanism (CloudNativePG barman, mariadb-operator dumps, Altinity `clickhouse-backup`, FoundationDB `backup_agent`). The `apps.cozystack.io/*` CR, its `HelmRelease`, chart values, and operator-managed Secrets are **not** captured by these strategies. + +For backups that bundle Helm release + CRs + PVC snapshots (used by VMInstance / VMDisk), see [Velero Backup Configuration]({{% ref "/docs/next/operations/services/velero-backup-configuration" %}}). +{{% /alert %}} + +## Prerequisites + +- Administrator access to the Cozystack (management) cluster. +- The `backup-controller` and `backupstrategy-controller` components are installed and running. +- S3-compatible storage reachable from the management cluster — either the in-cluster SeaweedFS provisioned via the `Bucket` application, or any external S3 endpoint. +- The corresponding upstream operator is deployed for each application Kind you want to back up: CloudNativePG, mariadb-operator, ClickHouse operator, or fdb-kubernetes-operator. These ship with Cozystack by default. + +## How a managed-application strategy works + +The flow on every `BackupJob`: + +1. A tenant creates a `BackupJob` (or a `Plan` that materialises one on a cron) that references a `BackupClass` and an `apps.cozystack.io/` application. +2. The core backup controller resolves the `BackupClass` and matches the application Kind to a driver-specific `strategy.backups.cozystack.io/` strategy. +3. The driver renders its strategy template against the live application object (`.Application`) and the BackupClass parameters (`.Parameters`), then creates the operator-native backup CR (`Backup` for mariadb, `FoundationDBBackup` for FoundationDB, an HTTP call against the in-pod sidecar for ClickHouse, a barman-driven snapshot in `cnpg.io` for Postgres). +4. On success the driver creates a Cozystack `Backup` artefact in the same namespace; `RestoreJob` resources reference that artefact later. + +`BackupClass` is **cluster-scoped**: a single instance covers every tenant namespace. + +## Per-driver setup + +The strategies below are written for the in-cluster SeaweedFS `Bucket` application. If you use external S3 storage, drop the `endpointCA` / TLS sections and point the endpoint at your provider. + +### Postgres (CNPG strategy) + +The CNPG driver delegates to CloudNativePG's native barman backup. Each `BackupJob` is a barman snapshot streamed to S3; `RestoreJob` recreates the `cnpg.io/Cluster` from the archive. + +Create the strategy: + +```yaml +apiVersion: strategy.backups.cozystack.io/v1alpha1 +kind: CNPG +metadata: + name: postgres-data-cnpg-strategy +spec: + template: + serverName: "{{ .Application.metadata.name }}" + barmanObjectStore: + destinationPath: "s3://REPLACE_WITH_COSI_BUCKET_NAME/{{ .Application.metadata.name }}/" + endpointURL: "https://REPLACE_WITH_S3_ENDPOINT" + retentionPolicy: "30d" + endpointCA: + secretRef: + name: "{{ .Application.metadata.name }}-cnpg-backup-ca" + key: "ca.crt" + s3Credentials: + secretRef: + name: "{{ .Application.metadata.name }}-cnpg-backup-creds" + data: + compression: gzip + wal: + compression: gzip +``` + +Bind the application Kind: + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: BackupClass +metadata: + name: postgres-data-backup +spec: + strategies: + - application: + apiGroup: apps.cozystack.io + kind: Postgres + strategyRef: + apiGroup: strategy.backups.cozystack.io + kind: CNPG + name: postgres-data-cnpg-strategy +``` + +Per-application Secrets the tenant must provision in the application namespace: + +| Secret | Keys | Purpose | +|---|---|---| +| `-cnpg-backup-creds` | `ACCESS_KEY_ID`, `ACCESS_SECRET_KEY` | S3 credentials consumed by barman | +| `-cnpg-backup-ca` *(only for self-signed endpoints)* | `ca.crt` | CA bundle the barman client trusts | + +Drop the `endpointCA` block in the strategy when your S3 endpoint has a publicly-trusted certificate. + +### MariaDB + +The MariaDB driver delegates to [mariadb-operator](https://github.com/mariadb-operator/mariadb-operator). Backups materialise as `k8s.mariadb.com/v1alpha1 Backup` CRs (logical `mariadb-dump`); restores materialise as `Restore` CRs that `mariadb-import` the dump back into the live database. + +Create the strategy: + +```yaml +apiVersion: strategy.backups.cozystack.io/v1alpha1 +kind: MariaDB +metadata: + name: mariadb-data-strategy +spec: + template: + storage: + s3: + bucket: "REPLACE_WITH_COSI_BUCKET_NAME" + endpoint: "REPLACE_WITH_S3_ENDPOINT" + prefix: "{{ .Application.metadata.name }}/" + accessKeyIdSecretKeyRef: + name: "{{ .Application.metadata.name }}-mariadb-backup-creds" + key: "AWS_ACCESS_KEY_ID" + secretAccessKeySecretKeyRef: + name: "{{ .Application.metadata.name }}-mariadb-backup-creds" + key: "AWS_SECRET_ACCESS_KEY" + tls: + enabled: true + caSecretKeyRef: + name: "{{ .Application.metadata.name }}-mariadb-backup-ca" + key: "ca.crt" + compression: gzip +``` + +The `endpoint` is **path-style without scheme** (e.g. `seaweedfs-s3..svc:8333` for the default in-cluster SeaweedFS — substitute the namespace where SeaweedFS is deployed in your environment). Drop the `tls` block entirely when the endpoint serves a publicly-trusted certificate. + +Bind the application Kind: + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: BackupClass +metadata: + name: mariadb-data-backup +spec: + strategies: + - application: + apiGroup: apps.cozystack.io + kind: MariaDB + strategyRef: + apiGroup: strategy.backups.cozystack.io + kind: MariaDB + name: mariadb-data-strategy +``` + +Per-application Secrets the tenant must provision in the application namespace: + +| Secret | Keys | Purpose | +|---|---|---| +| `-mariadb-backup-creds` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | S3 credentials consumed by mariadb-operator | +| `-mariadb-backup-ca` *(only for self-signed endpoints)* | `ca.crt` | CA bundle for TLS verification | + +{{% alert color="info" %}} +The chart-level `backup.*` block in `apps.cozystack.io/MariaDB` (the legacy `mariadb-dump` + `restic` path) is **deprecated** in favour of this BackupClass flow. Existing tenants with `backup.enabled=true` continue to render the legacy resources unchanged. +{{% /alert %}} + +### ClickHouse (Altinity strategy) + +The Altinity driver does **not** template a backup CR. It renders a small `PodTemplateSpec` that runs `curl + jq` against the in-pod [`clickhouse-backup`](https://github.com/Altinity/clickhouse-backup) HTTP API (port 7171) provided by a sidecar inside every `chi-*` Pod. + +{{% alert color="warning" %}} +The Altinity strategy **requires** `backup.enabled=true` on every ClickHouse application instance — that flag is what materialises the in-pod sidecar and the `clickhouse--backup-api-auth` Secret the strategy authenticates with. Unlike MariaDB/FoundationDB, ClickHouse's chart-level `backup.*` block is **not** deprecated; the BackupClass flow piggybacks on the same sidecar. +{{% /alert %}} + +Create the strategy. The `template` is a `PodTemplateSpec` driving the sidecar; for the full reference template (with the shell script that POSTs `create_remote` / `restore_remote` and polls the action log) see [`examples/backups/clickhouse/01-create-strategy.sh`](https://github.com/cozystack/cozystack/blob/main/examples/backups/clickhouse/01-create-strategy.sh) in the cozystack repo. + +```yaml +apiVersion: strategy.backups.cozystack.io/v1alpha1 +kind: Altinity +metadata: + name: clickhouse-data-altinity-strategy +spec: + template: + spec: + restartPolicy: Never + containers: + - name: ch-backup-client + image: alpine:3.19 + env: + - name: API_USERNAME + valueFrom: + secretKeyRef: + name: clickhouse-{{ .Release.Name }}-backup-api-auth + key: username + - name: API_PASSWORD + valueFrom: + secretKeyRef: + name: clickhouse-{{ .Release.Name }}-backup-api-auth + key: password + command: ["/bin/sh", "-c"] + args: + # See examples/backups/clickhouse/01-create-strategy.sh for the + # full script: branches on .Mode (backup|restore) and either + # POSTs /backup/create_remote or /backup/restore_remote/, + # then polls /backup/actions for terminal status. + - | + # ... (truncated; see linked example) +``` + +Bind the application Kind. No parameters are required — the strategy template addresses the sidecar by deterministic Pod DNS and reads S3 credentials from the chart-emitted `-backup-s3` Secret directly. + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: BackupClass +metadata: + name: clickhouse-data-backup +spec: + strategies: + - application: + apiGroup: apps.cozystack.io + kind: ClickHouse + strategyRef: + apiGroup: strategy.backups.cozystack.io + kind: Altinity + name: clickhouse-data-altinity-strategy +``` + +### FoundationDB + +The FoundationDB driver delegates to the [fdb-kubernetes-operator](https://github.com/FoundationDB/fdb-kubernetes-operator). Each `BackupJob` materialises a `FoundationDBBackup` CR (a continuous `backup_agent` Deployment that streams to a blob store); each `RestoreJob` materialises a `FoundationDBRestore` CR (a one-shot `fdbrestore` against the destination cluster). + +{{% alert color="warning" %}} +The operator allows **one running backup directory per cluster**. The driver stops any prior `FoundationDBBackup` on the same FoundationDB before starting a new one, and each Cozystack `BackupJob` owns a discrete blob-store directory keyed by its name. +{{% /alert %}} + +Create the strategy: + +```yaml +apiVersion: strategy.backups.cozystack.io/v1alpha1 +kind: FoundationDB +metadata: + name: foundationdb-data-strategy +spec: + template: + blobStoreConfiguration: + accountName: "{{ .Parameters.accountName }}" + bucket: "{{ .Parameters.bucket }}" + # BackupName left empty: driver fills with the BackupJob name so each + # Cozystack BackupJob owns a discrete S3 directory. + urlParameters: + - 'secure_connection={{ .Parameters.secureConnection | default "0" }}' + - 'region={{ .Parameters.region | default "us-east-1" }}' + snapshotPeriodSeconds: 3600 + customParameters: + - "--blob_credentials=/var/fdb-blob-credentials/blob_credentials.json" + backupDeploymentPodTemplateSpec: + spec: + containers: + - name: foundationdb + volumeMounts: + - name: blob-credentials + mountPath: /var/fdb-blob-credentials + readOnly: true + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 200m + memory: 256Mi + securityContext: + runAsUser: 0 + volumes: + - name: blob-credentials + secret: + secretName: "{{ .Application.metadata.name }}-fdb-backup-creds" + items: + - key: blob_credentials.json + path: blob_credentials.json +``` + +Bind the application Kind. Parameters carry blob-store routing; fill them once the tenant has provisioned a `Bucket` and you have its endpoint and bucket name: + +```yaml +apiVersion: backups.cozystack.io/v1alpha1 +kind: BackupClass +metadata: + name: foundationdb-data-backup +spec: + strategies: + - application: + apiGroup: apps.cozystack.io + kind: FoundationDB + strategyRef: + apiGroup: strategy.backups.cozystack.io + kind: FoundationDB + name: foundationdb-data-strategy + parameters: + accountName: "@:" + bucket: "" + region: "us-east-1" + secureConnection: "0" # "1" for https endpoints +``` + +Per-application Secrets the tenant must provision in the application namespace: + +| Secret | Keys | Purpose | +|---|---|---| +| `-fdb-backup-creds` | `blob_credentials.json` | JSON in the fdb-operator's expected shape (below) | + +`blob_credentials.json` must follow this exact shape so `backup_agent` can resolve `accountName`: + +```json +{ + "accounts": { + "@:": { + "api_key": "", + "secret": "" + } + } +} +``` + +{{% alert color="info" %}} +The chart-level `backup.*` block in `apps.cozystack.io/FoundationDB` is **deprecated** in favour of this BackupClass flow. Existing tenants with `backup.enabled=true` continue to render the legacy `FoundationDBBackup` CR unchanged. +{{% /alert %}} + +## Apply and verify + +Apply the strategy and `BackupClass` manifests: + +```bash +kubectl apply -f .yaml +kubectl apply -f .yaml +``` + +List the resources: + +```bash +kubectl get cnpgs.strategy.backups.cozystack.io +kubectl get mariadbs.strategy.backups.cozystack.io +kubectl get altinities.strategy.backups.cozystack.io +kubectl get foundationdbs.strategy.backups.cozystack.io +kubectl get backupclasses +``` + +Each strategy should report no error conditions; each `BackupClass` should list the strategy entries you defined. + +## Handing off to tenants + +Tenants run backups and restores against the `BackupClass` names you created above using `BackupJob`, `Plan`, and `RestoreJob` resources. Walk them through the [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}) guide; they do not need admin permissions to operate against an existing `BackupClass`. diff --git a/content/en/docs/next/operations/services/velero-backup-configuration.md b/content/en/docs/next/operations/services/velero-backup-configuration.md index 81a48193..6b59fc5d 100644 --- a/content/en/docs/next/operations/services/velero-backup-configuration.md +++ b/content/en/docs/next/operations/services/velero-backup-configuration.md @@ -7,6 +7,10 @@ weight: 30 This guide is for **cluster administrators** who configure the backup infrastructure in Cozystack: S3 storage, Velero locations, backup **strategies**, and **BackupClasses**. Tenant users then use existing BackupClasses to create [BackupJobs and Plans]({{% ref "/docs/next/virtualization/backup-and-recovery" %}}). +{{% alert color="info" %}} +This page covers **Velero-driven** backups that bundle the application HelmRelease, CRs, and PVC snapshots — the model used for VMInstance / VMDisk. For data-only backups of managed databases (Postgres, MariaDB, ClickHouse, FoundationDB) driven by each operator's native mechanism, see [Managed Application Backup Configuration]({{% ref "/docs/next/operations/services/managed-app-backup-configuration" %}}). +{{% /alert %}} + ## Prerequisites - Administrator access to the Cozystack (management) cluster. diff --git a/content/en/docs/next/virtualization/backup-and-recovery.md b/content/en/docs/next/virtualization/backup-and-recovery.md index bde9b5b0..3f43c300 100644 --- a/content/en/docs/next/virtualization/backup-and-recovery.md +++ b/content/en/docs/next/virtualization/backup-and-recovery.md @@ -14,6 +14,10 @@ This guide covers backing up and restoring **VMInstance** and **VMDisk** resourc Cozystack uses [Velero](https://velero.io/docs/v1.17/) under the hood for backup storage and volume snapshots. +{{% alert color="info" %}} +This guide covers VM backups (HelmRelease + CRs + PVC snapshots bundled by Velero). For data-only backups of managed databases (Postgres, MariaDB, ClickHouse, FoundationDB), see [Application Backup and Recovery]({{% ref "/docs/next/applications/backup-and-recovery" %}}). +{{% /alert %}} + ## Prerequisites - The Velero add-on is enabled for your cluster (by an administrator).