k3s Backup and Restore
Karios can back up a k3s cluster and restore it, either into the same cluster or into a different one. A backup holds the cluster’s Kubernetes objects and, unless you turn it off, the data inside its persistent volumes. Backups are stored in a UDP Backup repository on the platform’s own object storage (Karios RGW), so they outlive the cluster: a cluster can be deleted and still be restored later.
Backup uses Velero, the open-source Kubernetes backup tool. When you enable backup on a cluster, Karios installs Velero into the cluster and connects it to the repository for you. There is nothing to install or configure inside the cluster yourself.
UI path: Kubernetes -> k3s -> View clusters -> <cluster> -> Backup
See also
k3s on Karios - creating and managing k3s clusters
UDP Backup - creating the repository that backups are written to
Kubernetes on Karios - roles and permissions
How It Works
Enable backup on a cluster and choose a repository. Karios installs the backup controller and, unless you turn it off, a node agent on every node, which copies the contents of persistent volumes. The cluster stays with that repository from then on.
Take backups, on demand with Back up now or automatically with a schedule.
Restore a backup with the Restore from a backup wizard: the whole backup into a different, empty cluster, or chosen namespaces into any cluster that uses the same repository, including the cluster the backup came from.
Several clusters can share one repository; each cluster’s backups are kept in its own folder of the bucket. Karios issues and manages the storage keys, so you never see or enter them.
Prerequisites and Access
Permissions
To do this |
Your role needs |
|---|---|
See a cluster’s Backup tab: its backups, schedule and restores |
k3s View |
Enable backup on a cluster |
k3s Manage and Backup Manage, plus Backup View to list the repositories |
Back up now, set or remove a schedule, delete a backup, disable backup, restore |
k3s Manage |
Manage does not include View: a role that manages backups also needs k3s View.
Without it, the Backup tab shows Not authorized. Enabling backup needs Backup
Manage on top of k3s Manage because it gives the cluster access to a backup
repository.
Repository
Create the repository in UDP Backup before you enable backup. It must be:
An S3 repository. NFS and SMB repositories cannot hold cluster backups.
On Karios RGW, the platform’s own object storage: choose Karios RGW in the Connect Backup Storage dialog. A repository on other S3 storage, such as AWS or MinIO, is refused.
Not a bucket that already holds another tenant’s cluster backups. Such a bucket is refused.
A restore from one cluster into another needs both clusters on the same repository. If you may ever restore across clusters, enable backup on all of them with the same repository.
Cluster
The cluster must be Running. Its nodes write backups straight to object storage, so
they must be able to reach the Karios RGW endpoint (port 7480) from the cluster’s
network.
Quick Start
Back up a cluster
Once: create an S3 repository on Karios RGW in UDP Backup.
Open the cluster, select the Backup tab and click Enable backup. Choose the repository in Backup repository and click Enable backup (details).
When the Backup card shows Controller
Ready, click Back up now, then Back up now in the panel (details). To take backups automatically, click Set a schedule (details).
Restore a backup
Make sure the cluster you restore into has backup enabled on the same repository. To restore the whole backup, it must also be a different, empty cluster.
Click the restore icon on the backup’s row on the Backup tab, or Restore on the k3s clusters page.
From Restore, first choose the source cluster and its backup. Then choose The whole backup or Choose namespaces, choose the target cluster, check the Review step and click Restore (details).
Enabling Backup on a Cluster
Enable backup once per cluster, before its first backup. A cluster must also have backup enabled before anything can be restored into it.
Steps
Open the cluster and select the Backup tab. While backup is off, the Backup card reads
Backup is not enabled on this cluster.Click Enable backup. The Enable backup panel opens.
In Backup repository, choose the repository. The list shows your S3 repositories as
<repository name> - <bucket>. If backup was enabled on this cluster before, the panel shows that repository and its bucket instead, and they cannot be changed.Leave Include volume contents on. It installs the node agent, so backups capture the data inside persistent volumes, not just the Kubernetes objects.
Click Enable backup. Installing runs in the background and usually finishes within a minute. You can leave the page; the progress is shown at the top of the Backup tab.
Warning
The cluster stays with the repository you pick. Every later enable of this cluster uses the same repository, and moving the cluster to another repository needs an administrator. Pick the repository you intend to keep.
Important
With Include volume contents off, backups hold Kubernetes objects only, and restoring them brings back empty volumes. To change this later you have to disable backup and enable it again.
Expected outcome
The Backup card reads Velero is installed in this cluster. and shows these rows:
Row |
Healthy value |
What it means |
|---|---|---|
Controller |
|
The Velero server inside the cluster. It takes the backups and runs the restores; nothing can be backed up while it is down. |
Node agent |
|
Copies the contents of persistent volumes. |
Storage |
|
Velero can reach the repository’s object storage. |
Backups |
A number |
How many backups of this cluster exist. |
Repository |
The repository name |
Where this cluster’s backups are written. It cannot be changed afterwards. |
Volume encryption |
|
The password that encrypts volume data. Volume data can only be read back through the repository that wrote it. |
The Schedule card appears next to it, with the backups table and the Restores table below. Nothing has been copied yet: take a backup or set a schedule.
Taking a Backup (Back up now)
Take a backup before a risky change, before deleting a cluster, or whenever you want a restore point.
On the Backup tab, click Back up now in the header of the backups table.
Fill in the Back up now panel:
Field
Required
Notes
Name
No
Leave blank for an automatic name, such as
karios-1762444800. Lowercase letters, digits, dots and hyphens; up to 63 characters; not already used by another backup of this cluster.Namespaces
No
Nothing selected backs up the whole cluster. Select namespaces to limit the backup to them.
Keep for
No
How many days to keep the backup: 30 unless you change it, 1 or more. After that it is deleted automatically.
Include volume contents
No (on by default)
Captures the data inside persistent volumes. Not offered when backup was enabled without the node agent.
Click Back up now.
Backup startedconfirms it, and the backup appears in the table asQueued, thenRunning, and finallyCompleted.
When it has finished, the backup’s Volume data column reads Included (volume
contents captured) or Objects only.
Note
Velero copies a volume only while a running pod mounts it. A backup can therefore
finish with warnings such as Skip pod volume ... (pod is not running). The ones for
k3s’s own finished helm-install pods in kube-system are expected and harmless.
If the warning names one of your workloads, start its pod and back up again.
Scheduling Backups
A cluster has one schedule. It takes backups automatically and keeps each one for a set number of days.
In the Schedule card on the Backup tab, click Set a schedule (Edit schedule once one exists).
Fill in the form:
Field
Notes
Schedule
A new schedule starts on
Daily at 02:00. Click a preset (Daily at 02:00,Every 6 hours,Weekly, Sunday 03:00,Monthly, 1st at 04:00), or choose Daily, Weekly or Monthly and set the Time (plus the Day for weekly or the Day of month for monthly), or choose Custom and enter a five-field Cron expression: minute, hour, day of month, month, day of week. Times are on the cluster’s clock, which is UTC; below the field the form showsNext run: ... (your time)in your local time. A monthly day that a month does not have is skipped that month: the 31st skips February and every 30-day month.Keep each backup for
Days each scheduled backup is kept before it is deleted. Default 30. Does not affect backups taken with Back up now.
Namespaces
Nothing selected backs up every namespace, which is the safe default. A namespace you leave out cannot be restored later.
Include volume contents
On by default. Copies the data inside persistent volumes.
Active
On by default. Turn it off to keep the schedule but stop taking backups.
Click Save schedule.
Backup schedule savedconfirms it.
The Schedule card then shows Active (or Paused), the schedule in words, for
example Every day at 02:00 UTC, and the Next run (in your local time), Keep
for, Namespaces and Volume data. Backups taken by the schedule are marked
Scheduled - <schedule name> in the backups table; the others are marked Manual.
To stop automatic backups, click Remove schedule. Backups already taken are kept.
Note
If the card says This schedule is saved but is not active in the cluster., the
cluster could not apply it, for example because it was unreachable. Save the schedule
again to retry.
Viewing and Deleting Backups
The backups table on the Backup tab has two views:
This cluster’s backups - the cluster’s own backups, with Name, Status, Namespaces, Volume data, Metadata (the size of the backup’s Kubernetes objects, not of its volume data), Completed and Expires.
All backups - every backup in this cluster’s repository, from every cluster that uses it, including clusters that have been deleted (marked
Deleted), with Name, Cluster, Status, Volume data and Completed.
Each row has a restore icon (Restore from this backup), active once the backup
has finished as Completed or Completed with errors, and a delete icon
(Delete backup), which is not offered for the backups of a deleted cluster.
Status |
Meaning |
|---|---|
|
Accepted and waiting to start. |
|
Copying objects and volume data. |
|
Finishing up. |
|
Finished. Can be restored. |
|
Finished, but some items failed. Can still be restored; check its errors first. |
|
The backup did not finish and cannot be restored. |
|
Being removed from object storage. |
Warning
Delete backup permanently removes the backup from object storage, including its
volume data. It cannot be restored afterwards. The backup stays listed as Deleting
for about 20 seconds while it is removed.
What Is in a Backup
Included |
Not included |
|---|---|
Every Kubernetes object in the backed-up namespaces: deployments, stateful sets, services, ingresses, config maps, secrets, volume claims, service accounts, roles and custom resources |
The |
The namespaces themselves, and cluster-wide objects: all of them for a whole-cluster backup; for a namespace backup, the ones tied to its objects, such as the persistent volumes behind its claims |
Container images: the target cluster must be able to pull them at restore time |
Volume contents, when volume data is included and a running pod mounts the volume |
Files a container wrote outside a volume, and the nodes’ own disks |
kube-system is captured in a whole-cluster backup, but no restore writes it back:
every cluster runs its own copy.
Important
Volume data is encrypted with the repository’s password before it is stored. The Kubernetes objects, including Secrets, are stored in the bucket compressed but not encrypted. Anyone with access to the bucket can read them, so protect the bucket accordingly.
Restoring
What you can restore
What you restore |
Where it can go |
What comes back |
|---|---|---|
The whole backup (disaster recovery) |
A different, empty cluster that has backup enabled on the same repository |
Everything in the backup except |
Chosen namespaces |
Any cluster that has backup enabled on the same repository, including the cluster the backup came from |
The namespaces you choose, each landing as shown below |
For each chosen namespace, the target offers the choices that fit it:
Choice |
Offered when |
What happens |
|---|---|---|
Restore under its own name |
The namespace does not exist on the target |
It comes back as it was. |
Copy beside the live one |
The namespace already exists on the target |
It is restored under a new name, next to the original. Nothing is deleted. The new name is suggested as |
Replace the live one |
The namespace already exists on the target (not offered for |
Karios takes a safety backup of the live namespace, deletes it, then restores it from the chosen backup. You must type the target cluster’s name to confirm. |
A restore runs one choice at a time. If your namespaces need different choices, restore them in separate runs.
Warning
Replace the live one deletes the live namespace, including its volumes. The safety backup can only capture volumes that a running pod mounts. If any volume cannot be captured, the restore is refused, unless you select Continue even if some volumes cannot be captured, which destroys that data for good.
Note
A copy starts cut off from the network: its pods can reach each other and cluster DNS, and nothing else. This stops a copy of an application from connecting to the live application’s services and writing to its data. When you want the copy to reach other services, delete the network policy Karios put in it:
kubectl -n <copy-namespace> delete networkpolicy karios-restore-copy-isolation
Where to start
All of these open the same Restore from a backup wizard:
A backup of this cluster: on the cluster’s Backup tab, in This cluster’s backups, click the restore icon on the backup’s row. The wizard opens at the Namespaces step.
A backup of another cluster in the same repository, including a deleted cluster: on the Backup tab, switch to All backups and click the restore icon on the backup’s row.
Any cluster’s backup, including a deleted cluster: click Restore on the k3s clusters page, next to + k3s cluster. The wizard starts at the Source step.
From UDP Backup: open UDP Backup -> Backups -> By cluster and click the restore icon on the backup’s row.
Restore steps
The wizard has five steps: Source, Backup, Namespaces, Target and Review. When you start from a backup’s row, it skips the first two.
Source - choose the cluster whose backup you want to restore. Clusters that have been deleted are listed too, marked
Deleted; their backups are still in the repository.Backup - choose the backup. Backups are listed newest first, each with when it was taken, its status, whether it was
ScheduledorManual, and how much volume data it holds.Namespaces - under What do you want back?, choose:
Choose namespaces - then select the namespaces you want from the Workloads and Default groups, or
The whole backup - disaster recovery into a different, empty cluster.
This step also shows what the backup holds: any errors or warnings recorded when it was taken, its volumes, and the full list of its resources.
Target - choose the cluster to restore into.
For the whole backup, the list shows the clusters that are
Runningand have backup enabled. The cluster the backup came from is shown asNot available. A cluster that does not have backup enabled is not listed at all: enable backup on it first, with the same repository, then open the wizard again. Choose a cluster that uses the same repository as the backup; the Review step refuses any other.For namespaces, every cluster is listed as
AvailableorNot available, with the reason. When you pick one, How each namespace lands shows the choices for each namespace. For Copy beside the live one, enter the new name in Restore as.
Review - check the summary and the Checks list. Karios runs these checks against the target each time you open this step, and Restore stays disabled until every check passes. If a check fails, fix what it names, then click Back and Continue to run the checks again. If the checks could not run at all, the step shows the error with a Run the checks again button. For Replace the live one, also type the target cluster’s name to confirm.
Click Restore. The panel shows
Restore started.and then the restore’s progress. The restore continues in the background; you can close the panel.
Note
The first time a cluster restores another cluster’s backup, it needs up to about a minute to read that cluster’s backups. The Review step says so and runs the checks by itself once the cluster has finished.
Namespace groups
In the Namespaces step, the backup’s namespaces are shown in three groups:
Group |
Contains |
Can you choose it? |
|---|---|---|
Workloads |
Every namespace you created. Restoring one brings back everything in it: deployments, services, configuration, secrets and volume data. |
Yes |
Default |
The |
Yes |
System |
|
No ( |
Each namespace row shows its object count, how many of those objects are yours (3
yours, leaving out what Kubernetes creates by itself, such as events, ReplicaSets and
each namespace’s default service account), and its volumes and their size. Expand a row
to see your objects by kind.
Following a restore
A restore belongs to the cluster it lands in. It is listed in the Restores table on the
target cluster’s Backup tab, named <backup>-restore-<number>, with its
State, Items, Errors and Finished time. When you start a restore from
another cluster’s tab, a banner there links to the target.
A restore that ends Completed with 0 in Errors is clean: your workloads start
on the target, with their volumes filled from the backup. Completed with errors means
some items could not be restored; the Errors column shows how many.
Restore checks
The Review step lists every check by name, as passed, failed,
not_applicable or not_checked, with the reason when there is one. Only failed
blocks the restore, and a failed check names what to fix.
Check |
What it confirms |
|---|---|
|
Backup is enabled and running on the target, with the node agent when volume data is restored. |
|
Nothing else is restoring into, or backing up, the target. |
|
For a whole-backup restore, the target runs no workloads of its own. |
|
No broken admission webhook on the target would reject the restored objects. |
|
The target can read the source cluster’s backups. |
|
The backup exists, finished, and is not being deleted. |
|
The target can read what the backup contains. |
|
The chosen namespaces are in the backup, and a copy’s new names are free. |
|
For a replace, the safety backup can capture every volume of the namespace. |
|
The volume data is in the repository and the target can decrypt it. |
|
The target serves every kind of object in the backup. |
|
The target has room for the restored volumes. |
|
The container images are on the target’s nodes or can be pulled. |
|
The volumes’ storage classes and access modes exist on the target. |
|
Fixed node ports are free, and pods pinned to particular nodes can be scheduled. |
Restoring a Whole Cluster into Another Cluster
Use this to recover a cluster that was lost or deleted, or to rebuild it somewhere else. Deleting a cluster does not delete its backups.
A whole cluster can only be restored into a different, empty cluster that has backup enabled on the same repository as the backup.
Create a new k3s cluster (see k3s on Karios), or pick an existing cluster that runs no workloads of its own.
On the new cluster’s Backup tab, enable backup with the same repository the original cluster used. Wait until the Backup card shows the Controller
Ready.Open the wizard: click Restore on the k3s clusters page, or click the restore icon on the backup’s row in All backups on the new cluster’s Backup tab.
Choose the original cluster and its backup, then The whole backup.
In Target, choose the new cluster. If it is not listed, backup is not enabled on it yet (step 2).
Review the checks and click Restore. Follow the restore on the new cluster’s Backup tab.
Disabling Backup
Disable backup on the Backup card removes the backup controller from the cluster. The cluster stops taking scheduled backups, and nothing can be restored into it until backup is enabled again.
By default, the cluster’s existing backups stay in the repository and can still be restored later.
Select Also delete this cluster’s backups to remove them from object storage as well; the button then reads Disable and delete backups. This cannot be undone: nobody can restore this cluster afterwards, including an administrator.
Troubleshooting
When an action is refused, the panel shows the reason; when enabling backup is refused, it also shows what to do next. The table lists what you may see.
What you see |
Cause |
What to do |
|---|---|---|
The cluster you want is not in the Target list for The whole backup |
The list shows only clusters that are |
Enable backup on that cluster with the same repository as the backup, wait until its Controller is |
A cluster is |
The reason is shown under the cluster: for example, backup is not enabled on it, it uses a different repository, or (for a whole backup) it is the cluster the backup came from. |
Fix what the reason names, or choose another cluster. |
|
Your role lacks Backup View. |
Ask an administrator to add Backup View. |
|
No S3 repository exists in UDP Backup. |
Create one on Karios RGW in UDP Backup. |
Enabling is refused because the repository is not on this platform’s object storage |
The repository’s bucket is on other S3 storage, such as AWS or MinIO. |
Use a repository on Karios RGW. |
Enabling is refused because the bucket already holds another tenant’s cluster backups |
Another tenant enabled cluster backup on that bucket first. |
Use a bucket of your own. |
Enabling is refused: |
The cluster was enabled with a different repository before. |
Enable it with its own repository. Moving a cluster needs an administrator. |
Enabling is refused: |
The cluster is not |
Start the cluster, then enable backup. |
Enabling is refused: |
The UDP Backup service did not answer. |
Try again shortly. If it persists, check the UDP Backup plugin in K-Store. |
|
The cluster’s nodes cannot reach Karios RGW. |
Allow the cluster’s network to reach the Karios RGW endpoint on port |
|
The backup controller in the cluster is down. |
Disable backup and enable it again. |
|
Backup was enabled with Include volume contents off. |
Disable backup and enable it again with Include volume contents on. |
Back up now refuses the name |
Another backup of this cluster already has that name. |
Choose a different name, or leave it blank. |
Back up now is refused because the repository is nearly full |
The repository is close to its capacity limit. The message says how full it is. |
Delete old cluster backups or shorten how long they are kept, or ask for the repository’s limit to be raised. |
Warnings |
The pod that mounts the volume was not running, so its data was not copied. |
Expected for k3s’s own |
|
The target is reading the source cluster’s backups for the first time. |
Wait; the checks run by themselves when it finishes. If an error appears instead, click Run the checks again. |
|
The target already runs workloads of its own. |
Use an empty cluster for a whole-backup restore, or choose namespaces instead. |
|
A volume in the namespace being replaced is not mounted by a running pod. |
Start the pods that use it and retry, or knowingly select Continue even if some volumes cannot be captured. |
Quick Reference
Setting |
Value |
|---|---|
Repository type |
S3 on Karios RGW only (no NFS, SMB or custom S3 endpoint) |
Repositories per cluster |
One, fixed at the first enable |
Schedules per cluster |
One |
Default retention |
30 days, for Back up now and scheduled backups |
Schedule clock |
UTC (the form shows the next run in your local time) |
Backup name |
Optional; lowercase letters, digits, dots and hyphens; up to 63 characters |
Volume data |
On by default; only volumes mounted by running pods |
Never restored |
|
Whole-backup restore target |
A different, empty cluster with backup enabled on the same repository |
Restore between clusters |
Both clusters on the same repository |
→ Next: Karios DFS