k3s Backup and Restore

Karios can back up a k3s cluster and restore it, either into the same cluster or into a different one. A backup holds the cluster’s Kubernetes objects and, unless you turn it off, the data inside its persistent volumes. Backups are stored in a UDP Backup repository on the platform’s own object storage (Karios RGW), so they outlive the cluster: a cluster can be deleted and still be restored later.

Backup uses Velero, the open-source Kubernetes backup tool. When you enable backup on a cluster, Karios installs Velero into the cluster and connects it to the repository for you. There is nothing to install or configure inside the cluster yourself.

UI path: Kubernetes -> k3s -> View clusters -> <cluster> -> Backup

See also

How It Works

  1. Enable backup on a cluster and choose a repository. Karios installs the backup controller and, unless you turn it off, a node agent on every node, which copies the contents of persistent volumes. The cluster stays with that repository from then on.

  2. Take backups, on demand with Back up now or automatically with a schedule.

  3. Restore a backup with the Restore from a backup wizard: the whole backup into a different, empty cluster, or chosen namespaces into any cluster that uses the same repository, including the cluster the backup came from.

Several clusters can share one repository; each cluster’s backups are kept in its own folder of the bucket. Karios issues and manages the storage keys, so you never see or enter them.

Prerequisites and Access

Permissions

To do this

Your role needs

See a cluster’s Backup tab: its backups, schedule and restores

k3s View

Enable backup on a cluster

k3s Manage and Backup Manage, plus Backup View to list the repositories

Back up now, set or remove a schedule, delete a backup, disable backup, restore

k3s Manage

Manage does not include View: a role that manages backups also needs k3s View. Without it, the Backup tab shows Not authorized. Enabling backup needs Backup Manage on top of k3s Manage because it gives the cluster access to a backup repository.

Repository

Create the repository in UDP Backup before you enable backup. It must be:

  • An S3 repository. NFS and SMB repositories cannot hold cluster backups.

  • On Karios RGW, the platform’s own object storage: choose Karios RGW in the Connect Backup Storage dialog. A repository on other S3 storage, such as AWS or MinIO, is refused.

  • Not a bucket that already holds another tenant’s cluster backups. Such a bucket is refused.

A restore from one cluster into another needs both clusters on the same repository. If you may ever restore across clusters, enable backup on all of them with the same repository.

Cluster

The cluster must be Running. Its nodes write backups straight to object storage, so they must be able to reach the Karios RGW endpoint (port 7480) from the cluster’s network.

Quick Start

Back up a cluster

  1. Once: create an S3 repository on Karios RGW in UDP Backup.

  2. Open the cluster, select the Backup tab and click Enable backup. Choose the repository in Backup repository and click Enable backup (details).

  3. When the Backup card shows Controller Ready, click Back up now, then Back up now in the panel (details). To take backups automatically, click Set a schedule (details).

Restore a backup

  1. Make sure the cluster you restore into has backup enabled on the same repository. To restore the whole backup, it must also be a different, empty cluster.

  2. Click the restore icon on the backup’s row on the Backup tab, or Restore on the k3s clusters page.

  3. From Restore, first choose the source cluster and its backup. Then choose The whole backup or Choose namespaces, choose the target cluster, check the Review step and click Restore (details).

Enabling Backup on a Cluster

Enable backup once per cluster, before its first backup. A cluster must also have backup enabled before anything can be restored into it.

Steps

  1. Open the cluster and select the Backup tab. While backup is off, the Backup card reads Backup is not enabled on this cluster.

  2. Click Enable backup. The Enable backup panel opens.

  3. In Backup repository, choose the repository. The list shows your S3 repositories as <repository name> - <bucket>. If backup was enabled on this cluster before, the panel shows that repository and its bucket instead, and they cannot be changed.

  4. Leave Include volume contents on. It installs the node agent, so backups capture the data inside persistent volumes, not just the Kubernetes objects.

  5. Click Enable backup. Installing runs in the background and usually finishes within a minute. You can leave the page; the progress is shown at the top of the Backup tab.

Warning

The cluster stays with the repository you pick. Every later enable of this cluster uses the same repository, and moving the cluster to another repository needs an administrator. Pick the repository you intend to keep.

Important

With Include volume contents off, backups hold Kubernetes objects only, and restoring them brings back empty volumes. To change this later you have to disable backup and enable it again.

Expected outcome

The Backup card reads Velero is installed in this cluster. and shows these rows:

Row

Healthy value

What it means

Controller

Ready

The Velero server inside the cluster. It takes the backups and runs the restores; nothing can be backed up while it is down.

Node agent

Ready (2/2), one per node

Copies the contents of persistent volumes. Not installed means backups hold Kubernetes objects only.

Storage

Available

Velero can reach the repository’s object storage.

Backups

A number

How many backups of this cluster exist.

Repository

The repository name

Where this cluster’s backups are written. It cannot be changed afterwards.

Volume encryption

Repository password

The password that encrypts volume data. Volume data can only be read back through the repository that wrote it.

The Schedule card appears next to it, with the backups table and the Restores table below. Nothing has been copied yet: take a backup or set a schedule.

Taking a Backup (Back up now)

Take a backup before a risky change, before deleting a cluster, or whenever you want a restore point.

  1. On the Backup tab, click Back up now in the header of the backups table.

  2. Fill in the Back up now panel:

    Field

    Required

    Notes

    Name

    No

    Leave blank for an automatic name, such as karios-1762444800. Lowercase letters, digits, dots and hyphens; up to 63 characters; not already used by another backup of this cluster.

    Namespaces

    No

    Nothing selected backs up the whole cluster. Select namespaces to limit the backup to them.

    Keep for

    No

    How many days to keep the backup: 30 unless you change it, 1 or more. After that it is deleted automatically.

    Include volume contents

    No (on by default)

    Captures the data inside persistent volumes. Not offered when backup was enabled without the node agent.

  3. Click Back up now. Backup started confirms it, and the backup appears in the table as Queued, then Running, and finally Completed.

When it has finished, the backup’s Volume data column reads Included (volume contents captured) or Objects only.

Note

Velero copies a volume only while a running pod mounts it. A backup can therefore finish with warnings such as Skip pod volume ... (pod is not running). The ones for k3s’s own finished helm-install pods in kube-system are expected and harmless. If the warning names one of your workloads, start its pod and back up again.

Scheduling Backups

A cluster has one schedule. It takes backups automatically and keeps each one for a set number of days.

  1. In the Schedule card on the Backup tab, click Set a schedule (Edit schedule once one exists).

  2. Fill in the form:

    Field

    Notes

    Schedule

    A new schedule starts on Daily at 02:00. Click a preset (Daily at 02:00, Every 6 hours, Weekly, Sunday 03:00, Monthly, 1st at 04:00), or choose Daily, Weekly or Monthly and set the Time (plus the Day for weekly or the Day of month for monthly), or choose Custom and enter a five-field Cron expression: minute, hour, day of month, month, day of week. Times are on the cluster’s clock, which is UTC; below the field the form shows Next run: ... (your time) in your local time. A monthly day that a month does not have is skipped that month: the 31st skips February and every 30-day month.

    Keep each backup for

    Days each scheduled backup is kept before it is deleted. Default 30. Does not affect backups taken with Back up now.

    Namespaces

    Nothing selected backs up every namespace, which is the safe default. A namespace you leave out cannot be restored later.

    Include volume contents

    On by default. Copies the data inside persistent volumes.

    Active

    On by default. Turn it off to keep the schedule but stop taking backups.

  3. Click Save schedule. Backup schedule saved confirms it.

The Schedule card then shows Active (or Paused), the schedule in words, for example Every day at 02:00 UTC, and the Next run (in your local time), Keep for, Namespaces and Volume data. Backups taken by the schedule are marked Scheduled - <schedule name> in the backups table; the others are marked Manual.

To stop automatic backups, click Remove schedule. Backups already taken are kept.

Note

If the card says This schedule is saved but is not active in the cluster., the cluster could not apply it, for example because it was unreachable. Save the schedule again to retry.

Viewing and Deleting Backups

The backups table on the Backup tab has two views:

  • This cluster’s backups - the cluster’s own backups, with Name, Status, Namespaces, Volume data, Metadata (the size of the backup’s Kubernetes objects, not of its volume data), Completed and Expires.

  • All backups - every backup in this cluster’s repository, from every cluster that uses it, including clusters that have been deleted (marked Deleted), with Name, Cluster, Status, Volume data and Completed.

Each row has a restore icon (Restore from this backup), active once the backup has finished as Completed or Completed with errors, and a delete icon (Delete backup), which is not offered for the backups of a deleted cluster.

Status

Meaning

Queued

Accepted and waiting to start.

Running

Copying objects and volume data.

Finishing / Finishing, with errors

Finishing up.

Completed

Finished. Can be restored.

Completed with errors

Finished, but some items failed. Can still be restored; check its errors first.

Failed / Failed validation

The backup did not finish and cannot be restored.

Deleting

Being removed from object storage.

Warning

Delete backup permanently removes the backup from object storage, including its volume data. It cannot be restored afterwards. The backup stays listed as Deleting for about 20 seconds while it is removed.

What Is in a Backup

Included

Not included

Every Kubernetes object in the backed-up namespaces: deployments, stateful sets, services, ingresses, config maps, secrets, volume claims, service accounts, roles and custom resources

The velero namespace (every cluster gets its own when backup is enabled)

The namespaces themselves, and cluster-wide objects: all of them for a whole-cluster backup; for a namespace backup, the ones tied to its objects, such as the persistent volumes behind its claims

Container images: the target cluster must be able to pull them at restore time

Volume contents, when volume data is included and a running pod mounts the volume

Files a container wrote outside a volume, and the nodes’ own disks

kube-system is captured in a whole-cluster backup, but no restore writes it back: every cluster runs its own copy.

Important

Volume data is encrypted with the repository’s password before it is stored. The Kubernetes objects, including Secrets, are stored in the bucket compressed but not encrypted. Anyone with access to the bucket can read them, so protect the bucket accordingly.

Restoring

What you can restore

What you restore

Where it can go

What comes back

The whole backup (disaster recovery)

A different, empty cluster that has backup enabled on the same repository

Everything in the backup except kube-system and velero

Chosen namespaces

Any cluster that has backup enabled on the same repository, including the cluster the backup came from

The namespaces you choose, each landing as shown below

For each chosen namespace, the target offers the choices that fit it:

Choice

Offered when

What happens

Restore under its own name

The namespace does not exist on the target

It comes back as it was.

Copy beside the live one

The namespace already exists on the target

It is restored under a new name, next to the original. Nothing is deleted. The new name is suggested as <namespace>-restored and must be free on the target.

Replace the live one

The namespace already exists on the target (not offered for default)

Karios takes a safety backup of the live namespace, deletes it, then restores it from the chosen backup. You must type the target cluster’s name to confirm.

A restore runs one choice at a time. If your namespaces need different choices, restore them in separate runs.

Warning

Replace the live one deletes the live namespace, including its volumes. The safety backup can only capture volumes that a running pod mounts. If any volume cannot be captured, the restore is refused, unless you select Continue even if some volumes cannot be captured, which destroys that data for good.

Note

A copy starts cut off from the network: its pods can reach each other and cluster DNS, and nothing else. This stops a copy of an application from connecting to the live application’s services and writing to its data. When you want the copy to reach other services, delete the network policy Karios put in it:

kubectl -n <copy-namespace> delete networkpolicy karios-restore-copy-isolation

Where to start

All of these open the same Restore from a backup wizard:

  • A backup of this cluster: on the cluster’s Backup tab, in This cluster’s backups, click the restore icon on the backup’s row. The wizard opens at the Namespaces step.

  • A backup of another cluster in the same repository, including a deleted cluster: on the Backup tab, switch to All backups and click the restore icon on the backup’s row.

  • Any cluster’s backup, including a deleted cluster: click Restore on the k3s clusters page, next to + k3s cluster. The wizard starts at the Source step.

  • From UDP Backup: open UDP Backup -> Backups -> By cluster and click the restore icon on the backup’s row.

Restore steps

The wizard has five steps: Source, Backup, Namespaces, Target and Review. When you start from a backup’s row, it skips the first two.

  1. Source - choose the cluster whose backup you want to restore. Clusters that have been deleted are listed too, marked Deleted; their backups are still in the repository.

  2. Backup - choose the backup. Backups are listed newest first, each with when it was taken, its status, whether it was Scheduled or Manual, and how much volume data it holds.

  3. Namespaces - under What do you want back?, choose:

    • Choose namespaces - then select the namespaces you want from the Workloads and Default groups, or

    • The whole backup - disaster recovery into a different, empty cluster.

    This step also shows what the backup holds: any errors or warnings recorded when it was taken, its volumes, and the full list of its resources.

  4. Target - choose the cluster to restore into.

    • For the whole backup, the list shows the clusters that are Running and have backup enabled. The cluster the backup came from is shown as Not available. A cluster that does not have backup enabled is not listed at all: enable backup on it first, with the same repository, then open the wizard again. Choose a cluster that uses the same repository as the backup; the Review step refuses any other.

    • For namespaces, every cluster is listed as Available or Not available, with the reason. When you pick one, How each namespace lands shows the choices for each namespace. For Copy beside the live one, enter the new name in Restore as.

  5. Review - check the summary and the Checks list. Karios runs these checks against the target each time you open this step, and Restore stays disabled until every check passes. If a check fails, fix what it names, then click Back and Continue to run the checks again. If the checks could not run at all, the step shows the error with a Run the checks again button. For Replace the live one, also type the target cluster’s name to confirm.

  6. Click Restore. The panel shows Restore started. and then the restore’s progress. The restore continues in the background; you can close the panel.

Note

The first time a cluster restores another cluster’s backup, it needs up to about a minute to read that cluster’s backups. The Review step says so and runs the checks by itself once the cluster has finished.

Namespace groups

In the Namespaces step, the backup’s namespaces are shown in three groups:

Group

Contains

Can you choose it?

Workloads

Every namespace you created. Restoring one brings back everything in it: deployments, services, configuration, secrets and volume data.

Yes

Default

The default namespace. It only matters if you deployed something into it. Every cluster already has one, so it restores as a copy under a new name.

Yes

System

kube-system, kube-public, kube-node-lease and velero. These belong to the cluster itself, and every cluster creates its own.

No (Not restorable)

Each namespace row shows its object count, how many of those objects are yours (3 yours, leaving out what Kubernetes creates by itself, such as events, ReplicaSets and each namespace’s default service account), and its volumes and their size. Expand a row to see your objects by kind.

Following a restore

A restore belongs to the cluster it lands in. It is listed in the Restores table on the target cluster’s Backup tab, named <backup>-restore-<number>, with its State, Items, Errors and Finished time. When you start a restore from another cluster’s tab, a banner there links to the target.

A restore that ends Completed with 0 in Errors is clean: your workloads start on the target, with their volumes filled from the backup. Completed with errors means some items could not be restored; the Errors column shows how many.

Restore checks

The Review step lists every check by name, as passed, failed, not_applicable or not_checked, with the reason when there is one. Only failed blocks the restore, and a failed check names what to fix.

Check

What it confirms

target_backup_ready, node_agent

Backup is enabled and running on the target, with the node agent when volume data is restored.

no_restore_in_progress, no_backup_in_progress

Nothing else is restoring into, or backing up, the target.

target_empty

For a whole-backup restore, the target runs no workloads of its own.

target_webhooks

No broken admission webhook on the target would reject the restored objects.

source_attached

The target can read the source cluster’s backups.

backup_present, backup_usable, backup_not_deleting

The backup exists, finished, and is not being deleted.

backup_format, kubernetes_version

The target can read what the backup contains.

namespaces_in_backup, copy_namespaces_free

The chosen namespaces are in the backup, and a copy’s new names are free.

replaced_volumes_backed_up

For a replace, the safety backup can capture every volume of the namespace.

archive_has_volume_data, volume_repository_present, volume_password_matches

The volume data is in the repository and the target can decrypt it.

archive_resources_servable

The target serves every kind of object in the backup.

target_capacity

The target has room for the restored volumes.

images_present

The container images are on the target’s nodes or can be pulled.

storage_classes_present, access_modes_supported

The volumes’ storage classes and access modes exist on the target.

node_ports_free, node_selectors_matchable

Fixed node ports are free, and pods pinned to particular nodes can be scheduled.

Restoring a Whole Cluster into Another Cluster

Use this to recover a cluster that was lost or deleted, or to rebuild it somewhere else. Deleting a cluster does not delete its backups.

A whole cluster can only be restored into a different, empty cluster that has backup enabled on the same repository as the backup.

  1. Create a new k3s cluster (see k3s on Karios), or pick an existing cluster that runs no workloads of its own.

  2. On the new cluster’s Backup tab, enable backup with the same repository the original cluster used. Wait until the Backup card shows the Controller Ready.

  3. Open the wizard: click Restore on the k3s clusters page, or click the restore icon on the backup’s row in All backups on the new cluster’s Backup tab.

  4. Choose the original cluster and its backup, then The whole backup.

  5. In Target, choose the new cluster. If it is not listed, backup is not enabled on it yet (step 2).

  6. Review the checks and click Restore. Follow the restore on the new cluster’s Backup tab.

Disabling Backup

Disable backup on the Backup card removes the backup controller from the cluster. The cluster stops taking scheduled backups, and nothing can be restored into it until backup is enabled again.

  • By default, the cluster’s existing backups stay in the repository and can still be restored later.

  • Select Also delete this cluster’s backups to remove them from object storage as well; the button then reads Disable and delete backups. This cannot be undone: nobody can restore this cluster afterwards, including an administrator.

Troubleshooting

When an action is refused, the panel shows the reason; when enabling backup is refused, it also shows what to do next. The table lists what you may see.

What you see

Cause

What to do

The cluster you want is not in the Target list for The whole backup

The list shows only clusters that are Running and have backup enabled.

Enable backup on that cluster with the same repository as the backup, wait until its Controller is Ready, then open the wizard again.

A cluster is Not available as a target

The reason is shown under the cluster: for example, backup is not enabled on it, it uses a different repository, or (for a whole backup) it is the cluster the backup came from.

Fix what the reason names, or choose another cluster.

Choosing a repository needs permission to view UDP Backup repositories.

Your role lacks Backup View.

Ask an administrator to add Backup View.

No S3 repository is registered. Create one in UDP Backup first - NFS and SMB repositories cannot hold cluster backups.

No S3 repository exists in UDP Backup.

Create one on Karios RGW in UDP Backup.

Enabling is refused because the repository is not on this platform’s object storage

The repository’s bucket is on other S3 storage, such as AWS or MinIO.

Use a repository on Karios RGW.

Enabling is refused because the bucket already holds another tenant’s cluster backups

Another tenant enabled cluster backup on that bucket first.

Use a bucket of your own.

Enabling is refused: This cluster's backups already live in another repository and cannot be moved.

The cluster was enabled with a different repository before.

Enable it with its own repository. Moving a cluster needs an administrator.

Enabling is refused: Start the cluster before enabling backup.

The cluster is not Running.

Start the cluster, then enable backup.

Enabling is refused: UDP is not responding. Try again shortly.

The UDP Backup service did not answer.

Try again shortly. If it persists, check the UDP Backup plugin in K-Store.

Backup storage is unavailable.

The cluster’s nodes cannot reach Karios RGW.

Allow the cluster’s network to reach the Karios RGW endpoint on port 7480.

Backup is enabled but Velero is not running in this cluster.

The backup controller in the cluster is down.

Disable backup and enable it again.

This cluster was enabled without the node agent, so its backups do not contain volume contents.

Backup was enabled with Include volume contents off.

Disable backup and enable it again with Include volume contents on.

Back up now refuses the name

Another backup of this cluster already has that name.

Choose a different name, or leave it blank.

Back up now is refused because the repository is nearly full

The repository is close to its capacity limit. The message says how full it is.

Delete old cluster backups or shorten how long they are kept, or ask for the repository’s limit to be raised.

Warnings Skip pod volume ... (pod is not running) on the backup

The pod that mounts the volume was not running, so its data was not copied.

Expected for k3s’s own helm-install pods. For your own workloads, start the pod and back up again.

... is reading ...'s backups for the first time on the Review step

The target is reading the source cluster’s backups for the first time.

Wait; the checks run by themselves when it finishes. If an error appears instead, click Run the checks again.

target_empty failed

The target already runs workloads of its own.

Use an empty cluster for a whole-backup restore, or choose namespaces instead.

replaced_volumes_backed_up failed

A volume in the namespace being replaced is not mounted by a running pod.

Start the pods that use it and retry, or knowingly select Continue even if some volumes cannot be captured.

Quick Reference

Setting

Value

Repository type

S3 on Karios RGW only (no NFS, SMB or custom S3 endpoint)

Repositories per cluster

One, fixed at the first enable

Schedules per cluster

One

Default retention

30 days, for Back up now and scheduled backups

Schedule clock

UTC (the form shows the next run in your local time)

Backup name

Optional; lowercase letters, digits, dots and hyphens; up to 63 characters

Volume data

On by default; only volumes mounted by running pods

Never restored

kube-system and velero

Whole-backup restore target

A different, empty cluster with backup enabled on the same repository

Restore between clusters

Both clusters on the same repository


→ Next: Karios DFS