LINSTOR Storage
UI path: Control Center -> Storage -> Instance Storage (protocol linstor)
Where This Fits in Karios
This page is a sub-page of Storage. It covers one kind of Instance Storage pool: a pool that uses a LINSTOR cluster you already run. The common pool workflows (list, details, tags, maintenance) are in Instance Storage. Block disks, templates and snapshots that live on a LINSTOR pool work through the same screens as on any other pool: Block Storage, VM Templates and Snapshots. Shared terms are in the Glossary.
What LINSTOR Storage Is
LINSTOR is open-source software that manages replicated block volumes. It places each volume on two or more storage nodes and uses DRBD, a replication layer in the Linux kernel, to keep the copies identical. A LINSTOR cluster has one controller (the management service and its REST API) and several satellites, the services on the nodes that hold or use data.
Karios supports LINSTOR as a customer-owned storage backend:
You own and operate the LINSTOR controller and the storage nodes that hold the data. Karios does not create, upgrade, repair or monitor that cluster beyond the checks described below.
Every Karios host in the pool’s scope joins your cluster as a diskless client. It stores no data. When a VM disk is attached, Karios asks your controller to make the volume available on that host, and the host reads and writes the volume over the network to your storage nodes. When the disk is detached, that attachment is removed again.
Karios installs and keeps up to date the DRBD 9 kernel module and the LINSTOR satellite on its own hosts. Do not install either yourself.
Important
Only KVM hosts can use a LINSTOR pool.
Purpose
Use this page to:
Check that your LINSTOR cluster is ready for Karios before you add the pool.
Add the pool and read its health.
Rotate the controller API token.
Know what works, what does not, and what to expect when something fails.
Quick Start (Minimum Path)
Prepare the cluster: see Before You Start Checklist.
Add the pool: see Add the Pool.
If your controller uses a private certificate authority, trust it: see Controller With a Private Certificate Authority.
Open the pool and confirm
HealthreadsHealthy: see Read the Pool Health.Create one disk on the pool and attach it to a VM: see Block Storage.
Before You Start Checklist
Do these on your own LINSTOR cluster and network. Karios opens the matching ports on its own hosts and checks the rest, but it cannot change your side.
Requirement |
Detail |
|---|---|
Controller version |
The controller must run exactly the same LINSTOR version as the satellite Karios installs on its hosts (LINSTOR refuses a satellite of another version). Karios shows both versions in the health card and reports a host whose satellite differs. This release was tested only with a 1.35.3 controller, DRBD 9.3.4 on the hosts and satellite 1.35.3. No other controller version has been tested, so do not assume a range. |
HTTPS on port 3371 |
The controller must answer HTTPS on |
Certificate |
The controller certificate is checked by default. Use a certificate that a public authority signed, or follow Controller With a Private Certificate Authority. You can skip the check for a lab, but then the API token travels to an unverified endpoint. |
API token |
Create an API token on the controller. Karios uses it to register nodes, create and delete volumes and snapshots, and read the cluster state, so a read-only token is not enough. The token is stored encrypted, is never shown again and is never returned by the API. |
Resource group |
Create a resource group for Karios. Karios carves every volume from it, so
it must have free space behind it. Use a place count of 2 or more for
redundancy (tested with 2). A place count of 1 keeps a single copy.
One resource group backs one Karios pool. Two pools on the same
controller and resource group would share the copies of each template, and
one pool could delete a copy the other is still writing. Karios refuses to
create the second pool; give each pool its own resource group. The check
compares the controller address as typed, so |
Storage pool type |
Snapshots work only when the resource group’s backing pools are
|
Tiebreaker |
With two copies, add a diskless tiebreaker so the cluster keeps quorum
when one storage node is lost. Include |
Block size |
Set |
Network ports |
TCP |
DRBD port range |
Set the controller property |
Node names |
A LINSTOR node name must equal the Karios host name exactly (letters,
digits, |
Host network |
DRBD traffic and the host firewall rules use the host’s storage network. Your controller must reach each Karios host on its storage network address. The health check reports a host that the controller reaches only on its management address. |
Secure Boot |
Turn Secure Boot off on every Karios host. See Limits. |
Permissions |
Adding the pool, and every LINSTOR route in API Routes, needs
a root administrator (the root account, with the admin role or the |
Offerings |
A storage tag on the pool does not keep untagged offerings or system VMs off it. Decide what may land on the pool and tag the other offerings, see What a Storage Tag Does and Does Not Do. |
Add the Pool
Steps:
Open
Control Center -> Storage -> Instance Storage.Click
+ Add Storage Pool.Set
Protocoltolinstor. The option is always listed. It can be chosen only when the platform’s provider list includes LINSTOR; otherwise its tooltip says why.Fill in the fields:
Field
Required
Description
Storage NameYes
Unique name for the pool.
ScopeYes
The form opens on
CLUSTER. Prefer it. AZONEpool does not report a host that cannot reach your cluster, so it can readUpwith no working host.
ZoneandClusterYes
Where the pool is available.
Clusteris required forCLUSTERscope.
ProviderRead-only
Shows the LINSTOR provider. Karios hosts join your cluster as diskless clients.
Controller addressYes
Host name or IP address of your controller, optionally
:port. Type the host only, with no scheme, path or credentials. The port defaults to3371.
Resource groupYes
The resource group Karios carves volumes from: 2 to 48 letters, digits, hyphens or underscores, starting with a letter or underscore. It must already exist on the controller, and no other Karios pool may use it on the same controller. It cannot be changed after the pool is created.
API tokenYes
A token your controller accepts.
Skip certificate verificationNo
Off by default. Tick it only for a lab, or for the first step of Controller With a Private Certificate Authority.
Storage tagsYes
At least one, so that offerings which carry the tag can be sent to the pool on purpose. The tag does not keep untagged offerings or system VMs off the pool: see What a Storage Tag Does and Does Not Do.
Click
Add Storage Pool.
Karios checks the controller before it creates anything. It refuses the pool, and says why, when the controller refuses the token, has no resource group of that name, presents a certificate it cannot verify, or when another Karios pool already uses that controller address, as typed, and resource group (the message names that pool; create the new pool on another resource group, or use the existing pool). If the pool list cannot be read the create goes on with a warning. Pools that were created on one resource group before this rule are not changed and are not flagged: give the second pool its own resource group and move its volumes.
Then Karios prepares every host in the pool’s scope (see
What Karios Does Automatically). This can take up to a few minutes. If a
host cannot be made ready, the pool is not created and the message names each
host and the reason. The token stays in the form: fix the cause and click
Retry. A host that is down or in maintenance is skipped and named.
Expected Outcome:
The pool appears in Instance Storage and reaches
Up.Opening the pool shows the
LINSTOR clustercard withHealthHealthy.
If this fails: see Troubleshooting.
What a Storage Tag Does and Does Not Do
The platform chooses a pool for a new disk by comparing the storage tags of the disk offering with the tags of the pools in scope. The rule is the same for every kind of pool, LINSTOR included:
An offering that has tags reaches only a pool that carries every one of them.
An offering that has no tags reaches every pool in its scope that has no tag rule, whether the pool is tagged or not. The disk of a system VM is placed the same way.
So the storage tag on a LINSTOR pool lets you aim tagged offerings at it. It
does not keep untagged offerings, such as the stock Small Instance, or
system VMs off it. Karios does not register a LINSTOR pool with a tag rule (a
tag rule is the one thing in the platform’s placement code that an untagged
offering does not match, and it needs the platform setting
js.interpretation.enabled, which is off by default), so no setting of the
pool changes this.
What backs this: the placement code of the platform version Karios ships
(Apache CloudStack 4.23.0.0: the pool lookup with an empty tag list returns
every pool that is up in scope, minus the pools that have a tag rule), and one
observation on a three-host test cluster, where an untagged compute offering
put a root disk on a tagged Ceph pool. The same code serves every pool type; it
was not repeated on a LINSTOR pool. The setting
match.storage.pool.tags.with.disk.offering only governs changing a disk’s
offering later, not the first placement.
To keep other workloads off a LINSTOR pool, work in this order. Tagging an offering before a pool carries its tag makes every deploy from that offering fail with “Unable to find suitable primary storage”.
Give the pools that the other workloads should keep using (for example your Ceph pool) a storage tag of their own.
Then give every compute offering and disk offering that can reach the LINSTOR pool’s scope, the stock untagged ones such as
Small Instanceincluded, that tag. An offering whose tag the LINSTOR pool does not carry cannot land on it. How an existing VM behaves after its offering was tagged (for example at its next start) was not tested; try it on a test VM first.If no system VM may land on the pool either, do the same for the system offerings named by the platform settings
consoleproxy.service.offering,secstorage.service.offeringandrouter.service.offering: a pool that carries their tag must beUpin every zone before you tag them, because otherwise the system VMs cannot be recreated andCreate VMstays disabled without a healthy secondary storage VM. The platform’supdateServiceOfferingcall takesstoragetags; a tag on a system offering was not tried on the test cluster.Prefer cluster scope: a cluster-scoped pool serves only the VMs of its cluster, a zone-scoped one every cluster of the zone.
Karios still requires at least one tag on a LINSTOR pool, because without a tag no offering can be aimed at it.
What Karios Does Automatically
When you add the pool, and for each host that joins later, Karios prepares the host. For every host in scope that is up and enabled:
Step |
What happens |
|---|---|
Register the node |
If your controller does not hold a node with the host’s name, Karios registers it as a satellite node, at the host’s storage network address when it has one. An existing node is never replaced. Karios does not remove these registrations when you delete the pool: remove them on your controller if the hosts should no longer be part of LINSTOR. |
Open the host firewall |
Karios declares the rules the host needs: TCP |
Start the satellite |
The satellite is bound to the address your controller holds for the node and started. An existing bind file is never overwritten. |
Wait for the node |
Karios waits until your controller shows the node online and the authentication file your controller writes is on the host with a token the controller accepts. |
Trust the certificate |
For a pool that checks the certificate, the host gets the authority
certificate you sent (see Controller With a Private Certificate Authority).
When you delete the pool, Karios takes that certificate off the hosts and
management nodes again, from the system trust, from the agent’s own
keystore and, with the last pool that needs it, the agent’s start-up hook
that keeps the keystore in step. Each host reports what became of the
hook, and a host that kept it is named in the Karios log. A hook that is
left behind with nothing staged removes itself at the next agent start.
A host that was down or unreachable when the pool was deleted keeps the
certificate until you run |
Hosts that join later
A host that joins a cluster (or, for a zone pool, a zone) that has a LINSTOR
pool gets the same preparation. This happens when a host is onboarded, when
it is added through Add Host and when it joins through the agent’s own pull
join (a node installed from the Karios ISO, a pre-installed agent, or an
onboard that Karios runs again after a restart); for a pull join it runs once
the onboard has verified the host. The same step also provisions a dedicated
storage policy that was waiting for the cluster’s first host. For a pool that checks the certificate the
host also receives the authority certificate, so a scale-out does not silently
break certificate checking.
The certificate is given only when Karios holds exactly one valid, unexpired certificate for the pool that still verifies the controller. If not, nothing is sent, a warning in the Karios log names the pool and host, and the pool health shows that host as not trusting the certificate.
After staging, Karios reads what the host’s agent trusts and, if it does not trust yet, sends the certificate once more.
A host that is not yet
Upis waited for up to 5 minutes. A host still not ready after that is named in the log, never reported as prepared.If your controller cannot be reached when the host joins, or the satellites are not online yet, or Karios holds no certificate for the pool yet and a host in scope does not trust the controller (none staged on this management node and none recorded; when every host already trusts the controller there is nothing to wait for), Karios keeps a record of the join in its database and tries again, first after 30 seconds and then after longer pauses of up to 15 minutes, until it works: a join that waits for a certificate finishes by itself once you send it. The record survives a restart of Karios. A run keeps its record claimed by renewing it every minute, and a claim lapses 3 minutes after the last renewal, so a Karios that stopped in the middle of a join is picked up again within about 3 minutes of the next retry tick (every 15 seconds), by whichever management node is active. A run that hangs lets go of its record after 30 minutes. Karios stops after 7 days and says so in the log. What needs you (a certificate that no longer verifies the controller, one that cannot be used, a host that needs a change of setup) is not retried. Only the active management node runs the retries. Because the certificate is also recorded in the database, and Karios’s own client trusts the recorded certificates as well as the staged files, a management node without a staged file of its own completes the whole join: it registers the node, opens the firewall, starts the satellite and hands the certificate to the hosts. The engine on that node still reads the staged file (see the note under Controller With a Private Certificate Authority). The clocks of the management nodes are assumed to agree to within a few seconds.
Several hosts that join one cluster together are prepared one after another, and so are joins of different clusters that share a zone pool.
Any other way a host appears in the platform does not trigger this (a host added straight in the engine rather than through Karios). Such a host reads as not trusting the certificate. Send the authority certificate again with the trust route in API Routes, and run the satellite route if the host is not prepared.
Note
Hosts that join are prepared in the background and never block onboarding. This behavior, the retry included, is covered by automated tests. It has not yet been observed on a cluster that adds a second host to a certificate-checking pool, by any of the join paths.
Read the Pool Health
Open the pool. The LINSTOR cluster card shows a live check of your
controller and of every host in the pool’s scope. Check again repeats it.
A result younger than ten seconds is reused.
Row |
Meaning |
|---|---|
|
|
|
When the live check ran. |
|
The version your controller reports. |
|
The controller’s |
|
Each problem the check found, under the word |
|
|
|
See Capacity. |
What each Health value means
Health |
Meaning |
|---|---|
|
The controller answers, accepts the token, and every host in scope passes every check. A skipped certificate check is advice only and does not lower the health. |
|
Some hosts have a problem while others work, or the pool has a problem of
its own (for example the DRBD port range is outside |
|
Karios cannot reach the controller, or every host in scope has a problem. |
|
Karios could not tell: the controller refused Karios’s token, or no host
in scope could be listed. This is not the same as |
Checks run for each host
Karios checks that the host can be reached, that its satellite service is running, that the DRBD 9 module is loaded, that your controller lists the node as online under the host’s name and on an address the firewall rules cover, that the satellite version equals the controller version, and, when the pool checks the certificate, that the host’s agent trusts the controller.
Checks on your storage nodes
The copies of your volumes live on your storage nodes, which are not Karios
hosts. Karios reads your controller’s own view of them: every node that has a
diskful storage pool your resource group may place on (the pools and nodes the
group names, or every diskful pool when it names none), except the Karios hosts,
which are checked above. A storage node that the controller does not list as
online turns the pool Degraded and is named in the Controller problem
with its state and how many of the nodes that can hold volumes are online. It then
says what that means for the copies, from the group’s place count: with a place
count of 1 the volumes on that node have no online copy until it is back; with
as many nodes down as the group keeps copies, or more, some volumes may have
no online copy; otherwise every volume still has at least the place count minus the
nodes down of its copies online. Karios does not know which volumes sit on which
node, so it states the worst case and never says that the other copies serve every
volume. When fewer nodes are online than the group keeps copies, it also says that
new volumes cannot be created until one is back. A node of another storage pool or resource group on a
shared controller is ignored. A diskless node that is not online (your tiebreaker, or another client of a shared
controller) holds no copy and only adds a line to the advice; if it is your
tiebreaker the pool has one voter less for its quorum. If the controller’s
storage pool list cannot be read, the advice says the storage nodes were not
checked; if the resource group cannot be read, every diskful pool of the
controller is judged and the advice says so. Measured on the three-host test cluster: with the satellite of one of
two storage nodes stopped, creating a disk was refused with “Not enough
available nodes” while the pool still read Healthy; this check closes that
gap, and the automated tests cover it, but it was not run on a cluster after the
change.
Important
A ZONE scope pool can read Up in Instance Storage while this check
finds no working host. Use the card, not the pool state, to judge LINSTOR pools.
Capacity
Usable capacity is the raw figure from your resource group divided by its
replica count. Raw capacity and Replicas are shown beside it. The
numbers in the Instance Storage list for a LINSTOR pool are the usable ones. If
Karios cannot read the replica count, the pool shows the raw figure and says
so.
Warning
The platform’s allocator still counts every copy: it allows allocation up to
its overprovisioning factor times the raw figure. With two copies and a
factor of 2 that is about four times the usable space. Watch the pool’s
Used figure, and keep the overprovisioning factor low for LINSTOR pools.
Karios warns when the pool is created that the platform allocates against the
raw figure, and the pool health names the factor.
Tune DRBD for Write Latency
With DRBD’s default settings a small synchronous write on a replicated volume spends most of its time on DRBD’s own metadata, not on the network or the data disk. DRBD records the area being written in its activity log and, when a peer is connected, updates its out-of-sync bitmap each time an activity-log slot is reused; each of those metadata writes is flushed to the disk.
Two settings change that. Set them on the resource group that backs the pool, so every volume Karios creates from it inherits them:
al-extents(recommended)A larger activity log is reused less often, so fewer metadata writes and flushes reach the disk. On a two-copy pool this roughly halves the latency of a single 4 KiB synchronous write and nearly doubles small random writes under a 70/30 read/write mix. It is safe on any disk; the cost is a longer resync of the recently written area after a crash:
linstor resource-group drbd-options --al-extents 6433 <resource-group>
disk-flushesandmd-flushes(only on suitable disks)Turning flushes off removes most of the remaining metadata cost, but it is only safe when every disk under the pool keeps written data across a power loss: a disk or controller with power-loss protection, or a disk with its volatile write cache turned off. On any other disk a power cut can lose writes DRBD already reported as done. Check every model in the pool before you use it:
linstor resource-group drbd-options --disk-flushes no --md-flushes no <resource-group>
Changes to a resource group apply to its existing volumes too. A setting made directly on one resource definition overrides the group, so remove such per-volume overrides if you want the group’s value to apply.
Large sequential writes and deep queues are usually limited by the replication network rather than by these settings: give the storage nodes a replication link at least as fast as the disks can write.
Choices for Snapshots, Flushes, Offerings and Disk Size
Snapshots stay copied to secondary storage
By default Karios copies every volume snapshot of a LINSTOR disk to secondary
storage, as it does for other pools. Keep it that way unless you have a reason
not to. The copy is what lets you restore a disk if the LINSTOR cluster itself is
lost. It is a full copy each time, so it is most of the time a snapshot takes
and it grows with the amount of data on the disk: a root snapshot with a backup
took about 116 seconds against 57 on Ceph RBD in the comparison under
Supported Operations Compared With Ceph RBD (that run does not record the
disk size, so read it as a comparison, not a promise). A snapshot that lives only on your
cluster is faster to take, but it is lost together with the pool. The copy is
controlled by the global setting lin.backup.snapshots (default true).
Turn flushes off only on protected disks
Turning off disk-flushes and md-flushes (see Tune DRBD for Write
Latency) brings a single small synchronous write down to about a third of its
latency. Do it only when every disk under the pool has power-loss protection or
has its volatile write cache turned off. On any other disk a power cut can lose
writes that were already reported as done. When in doubt, leave the flushes on
and use only al-extents.
The LINSTOR Standard offering and its IOPS cap
Coming in the release that ships the LINSTOR Standard offering. It is not part of earlier releases.
When you add a LINSTOR pool that has a storage tag, Karios also creates a disk
offering called LINSTOR Standard. It has a custom size, carries all of the
pool’s storage tags, and limits each disk to 10000 reads and 10000 writes per
second. The limit keeps one busy virtual machine from slowing the others that
share the same storage.
Two things to know:
There is one offering per set of storage tags. If an offering called
LINSTOR Standardalready exists with the same tags, Karios leaves it as it is. A LINSTOR pool with different tags gets its own offering, named after its tags, for exampleLINSTOR Standard (fast,ssd).It is created with no domain restriction, so every account can choose it and create disks on your LINSTOR cluster. Restrict its domain if the pool is not meant for every tenant.
The limit cannot be changed on an existing offering from the disk offering page.
To get a different limit, create a new disk offering with the pool’s storage tag
and the IOPS you want, then hide or disable LINSTOR Standard. A limit set too
low throttles your own virtual machines, so measure your workload before you
lower it.
Disks show exactly the requested size
Coming in the release that ships exact disk size. It is not part of earlier releases.
DRBD keeps its own metadata inside the volume, so today a LINSTOR disk is a little larger than the size you asked for. In the release that ships exact disk size, a newly created LINSTOR disk shows inside the guest exactly the size you asked for. The difference removed depends on the pool behind the volume: about 8 KiB on ZFS thin and about 3.5 MiB on LVM thin (measured on a 2 GiB disk). Existing disks keep their current size. The change applies only to volumes created after the release is installed.
Two things change with it:
Growing a LINSTOR disk no longer reaches a running guest. The grow succeeds on the pool, and the guest sees the new size after the virtual machine is stopped and started. On Ceph RBD the running guest sees it at once.
A disk with a GUID partition table (GPT) keeps a second copy of the table at the very end of the disk. If a guest sees a new disk whose end is not where that copy sits, for example a disk made from a template that was prepared on one of the older, slightly larger volumes, the guest warns that the backup GPT is not at the end of the disk, or tools refuse to change the partitions. Move the copy to the end of the disk from inside the guest:
sudo sgdisk -e /dev/vdb sudo partprobe /dev/vdb
Use the device name of the disk (
/dev/vda,/dev/vdband so on). Take a snapshot first. If a partition itself was grown into the extra space,sgdisk -ecannot save the tail of that partition. Shrinking the file system does not work for every case (XFS cannot be shrunk, and a mounted root cannot be shrunk online). The fix that works for every file system is to grow the disk in Karios by a small amount (for example 1 GiB), stop and start the virtual machine, then runsgdisk -eand, if you want the space, growpart andxfs_growfsorresize2fs.
Rotate the API Token
Create a new token on your controller. Keep the old one valid for now.
Open the pool and click
Rotate token. (It needs a root administrator.)Paste the new token and confirm.
Revoke the old token on your controller after the pool reads
Healthy.
Karios tries the new token against your controller first and changes nothing if the controller refuses it. Then it replaces the stored token. Each host’s own token file is written by your controller, so nothing is copied to the hosts and no restart is needed. A token that works for the controller but fails part way through the change is reported with the instruction to send the same request again.
The token is never displayed, returned by the API or written to logs.
Supported Operations Compared With Ceph RBD
Both pool types were run through the same steps on one test cluster. “Same” means both behaved alike.
Operation |
LINSTOR |
Compared with Ceph RBD |
|---|---|---|
Create a disk, attach, detach, re-attach |
Works |
Same. Data is intact after a detach and re-attach. |
Grow a disk online |
Works. In the release that ships exact disk size, a new disk’s running guest sees the new size after a VM stop and start. |
Same. Shrinking is refused on both. On RBD the running guest sees the new size at once (see Disks show exactly the requested size). |
Delete a disk |
Works. The disk is removed at once and cannot be recovered. A snapshot’s backup on secondary storage can restore it. |
Same for a disk with no snapshot. Different for a disk that has a snapshot: RBD keeps the deleted disk, so it can still be recovered until the platform’s storage cleanup runs (by default after 24 hours). |
Deploy a VM from a template, stop, start, reboot, destroy, recover, reinstall |
Works |
Same. |
Grow the root disk |
Works |
Same. |
Disk IOPS limits from the disk offering |
Works |
Same. |
Volume snapshot (backed up to secondary storage) |
Works. A full copy to secondary storage each time. |
Different. On RBD the snapshot stays on the pool and no copy appears on secondary storage in the time we waited. |
Create a disk from a snapshot |
Works |
Same. |
Revert a disk to a snapshot |
Works when the disk is detached. Refused while the disk is attached to an instance that is not stopped. |
Same. |
Template from a root snapshot |
Works |
Same. |
Template from a root disk |
Works, with a size rule: see Limits. |
Works. |
Whole-VM disk-only snapshot, revert, delete |
Works |
Not supported: the snapshot ends in |
Recurring snapshot policy |
Works |
Same. |
Download a disk (extract) and upload a disk from a URL |
Works |
Same. |
Host agent restart with VMs running |
Works. VMs and disk I/O continue. |
Same. |
HA restart of a VM after its process is killed |
Works |
Same. |
Account isolation and account resource limits |
Works |
Same. |
Disable the pool, or put it in maintenance |
Works. A disabled pool refuses new disks (4.0 seconds, “Unable to find suitable primary storage”). Maintenance stops the VMs that have disks on the pool, and no write follows the stop. Deleting a pool is refused until it is in maintenance and has no disks left on it. |
Same, on a throwaway RBD pool of the three-host test cluster. |
Encrypted disk offering |
Fails unless the Karios engine has an encryption master key. |
Also fails on this setup. |
VM live migration |
Works between two hosts (see Live Migration). |
Not part of this comparison. |
Moving a disk between pools while the VM runs |
Not tested. Moving a disk into a LINSTOR pool while the VM runs is known not to work on the engine version Karios ships when the target host holds no copy of the data, which is always the case for diskless clients. |
Not part of this comparison. |
Speed, measured on one test cluster with the same host and steps: creating and attaching a 1 GiB disk took about 6.7 seconds on LINSTOR and 3.3 seconds on RBD; deploying a VM from a cached template took about 52 and 33 seconds; a root snapshot with a backup to secondary storage took about 116 and 57 seconds. Expect LINSTOR to be slower than RBD for control operations because every volume operation also goes through your controller and every snapshot backup crosses the network through a diskless host.
Behavior During Failures
Host failure detection (HA)
Declaring a host dead makes Karios fence it and restart its VMs elsewhere, which cannot be undone if the host was in fact alive; noticing a dead host late only delays HA. So Karios reads a host as dead in one narrow case, and in every other case it cannot tell and keeps the host.
When Karios checks whether another host is alive, the checking host first asks
DRBD whether it has a connection to that host: a connection in state
Connected means the host is alive, and nothing else is asked. Every other
answer from DRBD (a connection that is not Connected, no connection to the
host, the two share no resource, drbdsetup fails, or its output cannot be
read) is no evidence of death, and one rule applies to all of them. The checking
host asks your controller for its list of nodes, once, waiting at most 5 seconds
to connect and 15 seconds for the answer, and judges from that one answer. The
host is read as dead only when all of these hold:
your controller answered and lists the node, and shows it
OFFLINE;not every node is
OFFLINE, and no node is still connecting (a controller that has just restarted shows every node offline or connecting until the satellites reconnect);the checking host is itself shown
ONLINEin the same answer (this also proves that the name Karios uses for the host is the node name your controller holds);your controller has not been seen unsettled during the last 5 minutes: unreachable, listing no node, every node offline, or nodes still connecting;
the second signal below holds. It is needed whatever DRBD says, because a satellite that is down on a host whose VMs are running reads exactly like a dead host to your controller, a DRBD connection can be stuck (
StandAlone,Connecting) or blocked on a host that is up, and fencing a host that is alive is the mistake that cannot be undone.
Both parts of the second signal must hold:
It holds over time. Your controller showed the node
OFFLINEin an earlier reading taken at least one settle window (5 minutes) before, and no reading since showed the node alive or your controller unsettled. The first reading only starts the clock; the agent keeps it in memory, so a later check by the same agent at least 5 minutes afterwards can read the host as dead. The agent knows only the readings it took itself: when the gap since the previous reading of that node is longer than 2 windows (10 minutes) the clock starts again, so a short drop and a new one 40 minutes later count as two first readings, not as 40 minutes offline. A node that came back and went down again within two readings of the agent is not noticed.It holds on the wire, from the checking host and not through your controller. A TCP connection from the checking host to every address your controller lists for the node gets no answer at all, neither a connection nor a refusal (a refusal comes from the host’s own kernel, so the host is up whatever its satellite does), while the same connection to another
ONLINEnode does get an answer. Silence is a timeout, or a “no route to host” that took at least as long as the kernel needs to give up resolving an address on the checking host’s own network (about 3 seconds). A “no route to host” that came back at once was sent by a router or a firewall that answered (an ICMP “administratively prohibited” reply, a reject), so it is not silence: the node is not read as dead on it. Each address is connected to once per check. Two things can still make a path that refuses look silent or a dead host look unreadable. Linux limits the rate of its own refusals, per source host: after the first one or two, a router or firewall of yours that rejects the traffic to the node answers only some of the connections and the others time out. The traffic that uses up that allowance is normally DRBD’s own attempts to reconnect to the same node, one for each resource it shares with it. In a test with simulated reconnects every 10 seconds, 17 of 30 probes read silent with 5 shared resources and 24 of 30 with 20, against none with 0 or 1. So a device that rejects the traffic to a live node protects against a false “dead” reading only on a quiet path; with many shared resources it mostly reads as silent, and the node is read as dead once the other conditions hold. Do not reject traffic between LINSTOR nodes either. And a connection that joins an address resolution that is already about to fail fails at once, like a refusal, so a dead host on the checking host’s own network sometimes reads as “cannot tell” on one check (and as dead on the next). The second half of the comparison shows that the path and the port are open from this host to a live node. That rules out a drop that hits this host’s traffic to every node alike, and no more than that: it does not show that the silence is the node’s. A firewall or router of your own that silently drops only the traffic between the checking host and the node, while the node is up, looks exactly like a dead node from here; if your controller also shows that nodeOFFLINEfor a whole window and DRBD does not show itConnected, it is read as dead. Karios cannot tell the two apart over the wire, so do not drop traffic between LINSTOR nodes selectively. The port is 7999, the top of the DRBD port range, which the firewall rules Karios writes open to every LINSTOR node (the satellite port is open to the controller only). With Karios’s own host firewall a live host answers on any port, because what the plane does not allow it refuses with a TCP reset, and a reset counts as an answer. The probe sends nothing and closes the connection at once.
What counts as an address and as the comparison node:
Loopback addresses, the any-local address and the checking host’s own addresses (the ones your controller lists for it and the ones on its network interfaces) are left out, for the node and for the comparison node: a connection to them would be answered by the checking host itself. If nothing is left of the node’s addresses, the node is not read as dead.
An address that is not an IP literal (a name), an IPv6 address with a zone, a link-local IPv6 address or a multicast address cannot be judged. A node with such an address is never read as dead, because that address may be the live path; a comparison node with one cannot qualify by answering on all its addresses. Names are never looked up.
A comparison node counts only if, for each address of the node that was probed, it answers where that address can be calibrated. When one of the checking host’s own network interfaces covers the address, the network is known (LINSTOR lists no prefix length, so it is the prefix of that interface, the longest if several do), and the comparison node must answer on an address in that network. An answer from another network shows that the checking host reaches a live node, not that it reaches the node’s network, so a comparison node whose only address is on another network does not qualify. When no interface covers the address, no network can be named for it (the node is behind a router), and the comparison node must answer on all its addresses, none of them one that cannot be judged, and at least one of them must itself be behind a router. A comparison node that answers only on the checking host’s own networks does not qualify: a fault on the uplink that also cut your controller off would look the same.
The check asks at most two comparison nodes, skipping any with no address it can use or that cannot qualify, so a check can take up to about nine seconds on top of the read of the node list.
A cluster with only two nodes has no other node to calibrate against, so in such
a cluster a host is never read as dead, whatever DRBD shows: the other node
cannot be told from a path that is down. The port
(cloudstack.linstor.ha.probe.port, default 7999) and the connect time
(cloudstack.linstor.ha.probe.timeout.ms, default 3000) and the least time a
“no route to host” must take to count as silence
(cloudstack.linstor.ha.probe.noroute.min.ms, default 1000) are system
properties of the agent, like cloudstack.linstor.ha.settle.ms; add -Dname=value to
JAVA_OPTS in /etc/default/cloudstack-agent on each host and restart the
agent. Keep the last one below the time your network takes to give up
resolving an address (about 3 seconds with the kernel’s defaults): above it, a
dead host on the checking host’s own network, whose connection ends with that
reply when the connect time is longer than the resolution takes, reads as
“cannot tell”. If you changed the DRBD port range so
that 7999 is outside it, or your firewall does not let nodes reach it, no node
answers, and no host is read as dead.
What the platform makes of it was read from the platform’s source, with its default settings, and was not measured:
With Host HA enabled, the activity check runs every 60 seconds (
kvm.ha.activity.check.interval) in rounds of 10 attempts (kvm.ha.activity.check.max.attempts) and fences only when at least 70 percent of a round read “no activity” (kvm.ha.activity.check.failure.ratio). Because the first reading only starts the clock, the first attempts of a round read alive, so the first round normally ends under 70 percent: the host is marked degraded for up to 5 minutes (kvm.ha.degraded.max.period), and the next round, in which every attempt reads dead, fences it. A host that really died is therefore fenced roughly one round plus the degraded period later than before, whatever its DRBD connection looked like, which is about 10 to 15 minutes with the defaults (less when the platform’s health checks, which ask every online neighbour, already started the clock). A check that takes longer thankvm.ha.activity.check.timeout(60 seconds) is counted as “no activity”, so keep that value above the longest check: about 35 seconds with the node list (up to 20), the probe of the node and two comparison nodes (up to 9) and, with a plain http or linstor address, the version probe of the LINSTOR client (up to 6).The activity check is sent to the first online host of the cluster in the platform’s own order, which is kept from one check to the next, so it is the same agent while that host stays up. Each agent keeps its own clock.
Without Host HA, when a host stops answering the management server, the platform investigates once (the neighbours are asked, and any that reads the host alive wins) and does not investigate that host again until its agent reconnects and drops again. With this second signal the first investigation reads any host as alive, because the reading has not held a window yet, so the host goes to
Disconnectedand laterAlertand its VMs are not restarted by the platform: you restart them (after making sure the host is really off). That is the price of not fencing a live host, and with this rule it applies to every host that goes silent, not only to one that shares no DRBD resource with its neighbour.
A host is never read as dead because your controller cannot be asked (refused,
no answer in time, token refused, a failing gateway), because it lists no node
(for example after it was reinstalled), because it does not know the node, or
because the host has no name registered with it. DRBD alone is never enough
either: a connection that is found but not Connected, a drbdsetup that
fails, or output that cannot be read only mean that your controller has to be
asked and the second signal has to hold. The name Karios uses for a host is the platform’s host name, which is
also the node name Karios registers; names are compared without regard to case,
and a host whose platform name differs from its node name is never read as dead
(its HA does not act on it) and is named in the agent log.
A host also checks itself every minute, and the management server’s check of a
host asks that host itself first, so a check of this host that the platform
starts reaches the same code. A failed check is logged by the agent
(Write heartbeat for pool ... failed), and the host restarts itself only
when the agent setting reboot.host.and.alert.management.on.heartbeat.timeout
is set to true. That setting is off by default in the 4.23.0.0 agent this
page was written for; an agent of another version can default to on, so look at
the value on your hosts.
With the setting on, a failing drbdsetup never restarts the host, and a
Primary resource with promotion score 0 restarts it only when your controller
also shows this host OFFLINE while at least one other node is ONLINE
(the conditions on the controller’s answer listed above apply; with every other
node OFFLINE or in an error state the controller may see nobody, and the host
is not restarted), and that reading has held one
settle window (5 minutes): it was taken at least 5 minutes after the first one,
and no reading in between showed this host healthy, drbdsetup failing, or
your controller showing this host ONLINE or unsettled. This is the same
held window as for another host; the wire part of the second signal does not
apply, because a host cannot test itself over the wire. A host that reads
itself dead also reaches the management server’s own check, whatever the
setting says: the platform asks the host itself first, then its neighbours, and
when no neighbour can answer (a cluster of one host, or every neighbour
disconnected) the host’s own answer stands. That is read from the platform’s
source and was not measured. The activity check that fences a host goes to its
neighbours only, so the host’s own answer alone does not fence it.
A host that has lost both its peers and your controller does not restart itself
on DRBD evidence alone; the management server’s check of that host from the other
hosts still applies.
The other side of these choices is that a host that really died is detected only
once your controller answers, shows it offline and has been settled for 5
minutes, and a second reading at least 5 minutes after the first agrees and the
wire is silent; the 5 minutes can be changed with the system property
cloudstack.linstor.ha.settle.ms of the agent (set it as described above). LINSTOR’s controller retries a
satellite connection about every 10 seconds, so the window is many retries long.
Each agent remembers the window for itself and starts it when that agent sees
your controller unsettled, so an agent that restarted together with your
controller has no memory of it. A satellite that your controller shows stuck in
a connecting state (UNKNOWN, NO_STLT_CONN, CONNECTED or
AUTHENTICATED) keeps Karios from reading any host as dead until it is fixed
or removed; the agent log names it.
Note
The host liveness answer is proven by unit tests and brute-force tests, and by one live observation on a three-host cluster (A lost host on a three-host cluster). The dead verdict, and with it the wire test, was not reached there. The tests cover every combination of controller state (answering, refusing, not answering, listing no node), node state (every connection status your controller can report, and not listed, for the node, the checking host and a third node), DRBD state (connected, not connected, no connection to the node, no resources, failing), the history of the offline reading, and the addresses and outcomes of the connection test (loopback, any-local and own addresses, names, IPv6, several networks, one or two comparison nodes), checked against a second implementation, and the probe over real sockets and the real network interfaces of the test machine (a refusal and an accepted connection). Silence and the fast and slow “no route to host” replies were measured separately on a test machine in a private network namespace (a router with a prohibit route, an unused address on the checking host’s own network, and a connection that joins an address resolution already failing), and in the unit tests they are simulated. The wire test has not been run on a cluster: on the three-host test cluster the first reading ended the check before the wire test. On the a test host a healthy diskless Primary with two up-to-date peers was measured to report promotion score 2, and the score stayed 2 during two minutes with the controller unreachable and through a restart of the controller service, so while the score is above 0 the host’s own check does not need the controller. Nothing was fenced or restarted in those runs (host HA was disabled there).
A lost host on a three-host cluster
The rule above was watched once on a live cluster: three Karios hosts running as virtual machines on one physical host, next to a LINSTOR cluster of two storage nodes and a tiebreaker. A host that ran an HA VM (root disk and data disk on the LINSTOR pool) was powered off through its virtual BMC. Host HA was disabled for it, because no out-of-band controller (OOBM) was reachable from the management nodes. Times are counted from the power-off:
After |
What happened |
|---|---|
5 seconds |
Your controller shows the node |
4 minutes 16 seconds |
The platform’s one investigation asks a neighbour. The neighbour’s agent logs
that it cannot tell whether the host is up, because the |
4 minutes 21 seconds |
The host goes to |
4 minutes 21 seconds to 21 minutes |
Nothing more. The platform runs no second investigation (asking for a
reconnect did not start one), so the dead verdict, which needs a second
reading at least 5 minutes after the first, was never reached. The VM stays
|
20 minutes 50 seconds |
An operator force-stops the VM (accepted in 2 seconds with the host dead) and starts it: it runs on another host after 6 seconds with both disks attached. |
30 minutes 45 seconds |
The host is powered on. Its satellite is online after 72 seconds, the host
reads |
What this shows:
The platform does not restart the VM by itself. Without Host HA and an OOBM that fences the host there is no automatic VM restart, and the LINSTOR liveness rule holds its verdict as designed: it will not call a host dead on a first reading. The VM stayed down for 21 minutes 16 seconds, until the operator acted. After making sure the host is really off, restart its VMs yourself.
Data was safe. After the restart all 20 checksummed files read back identical and the guest’s write ledger had no gap and no lost acknowledged write.
Not measured: the dead verdict (it needs Host HA with an OOBM that a management node can reach), and the time a real fence and restart would take. The platform marked the host
Alertafter 4 minutes 21 seconds here and after 12 minutes 31 seconds on a Ceph pool of the same hosts; the cause of the difference was not isolated, so do not read it as a LINSTOR speed-up.
Host agent restart
Restarting the host agent with VMs running keeps the VMs and their disks
running. The host was back to Up in about 30 seconds in the test. If the
pool checks the certificate, the agent re-imports the authority certificate at
every start, so a rebuilt agent keystore does not leave the host untrusting.
Host reboot
The DRBD module and the satellite are set to load at boot. This was run on a
one-node test VM with Karios’s DRBD layer applied (Debian 13, kernel 6.12.107,
the LINSTOR controller and satellite on the same VM, an LVM thin pool, Secure
Boot off). After a clean reboot the DRBD 9.3.4 module was loaded from Karios’s
copy, the satellite was active and Online in the controller, the volume was
UpToDate and its data matched its checksum. The step that enables the
satellite persisted: the unit stayed enabled and a second call changed nothing.
After a hard reset with the volume Primary, a mounted ext4 file system and a
writer running, the data was intact (checksums equal, journal replayed, fsck
clean) and the module and the satellite came back, but the volume came back only
about 10 minutes later, because the test pool needed a manual lvconvert
--repair first. That is a property of the pool and not of DRBD or LINSTOR.
On the three-host test cluster, a host that was powered off and on again had its
satellite Online after 72 seconds and read Up after 96 seconds (see A
lost host on a three-host cluster); the module was not checked separately
there.
Not tested: a reboot with Secure Boot on (the module is refused), a ZFS-backed pool, a host installed from the Karios image rather than by applying the layer, and a reboot of a host that has VMs running.
Live Migration
Live migration between two Karios hosts was run on the three-host test cluster with an HA VM whose root disk and data disk were both on the LINSTOR pool, a guest writer that acknowledged every durable write to the test host, and 20 checksummed files:
The migration took 14.5 seconds (10.5 seconds back).
DRBD, sampled every 0.7 seconds on both hosts, showed the destination attach both disks as
Secondary, then both hosts ``Primary`` on both disks for about 2.9 seconds, then the source demote and release its attachment.The largest write gap in the guest was 0.39 seconds and all 20 checksums matched.
The cluster ran as virtual machines on one physical host, so the timings bound the mechanism and are not customer figures. Moving a disk between pools while the VM runs is a different case and was not tested (see Supported Operations Compared With Ceph RBD).
Upgrading
Upgrade the hosts before the management servers. A management server that has the template size fix sends a new command when a template is made from a LINSTOR disk. A host that is not upgraded yet does not know that command, and the template creation fails with an unsupported command error until the host is upgraded. A new host under an old management server is harmless: it receives the plain command and does what it always did. Upgrade both within a few minutes and do not create a template from a disk in between. To go back, roll back the management servers first and then the hosts, so a new management server never meets an old host.
A template is reused from a LINSTOR pool only when the copy into the pool ran to
the end. The management server marks the template resource as complete after the
host reports that the copy succeeded. A template resource without that mark is not
trusted: the next deploy deletes it and copies the template again, which takes the
time of a first deploy (about 7 minutes for the 5 GiB test template on the nested
test cluster). A clone for a new VM is only cut from a template resource that has
the mark: an unmarked one is never cloned, because it can be a copy that is still
being written. A template that was cached before this version has no mark, and the
platform does not copy it again because it has recorded it as ready, so a deploy
from it fails with a message that names the resource. If you know the resource holds a
good copy, mark it on your cluster (linstor resource-definition set-property
cs-<template uuid> Aux/_cs-template-complete true); otherwise delete the template
from the pool in Karios and deploy again so that it is copied again. Do this after the
upgrade for every template that was already in the pool. The wait for the cluster to
remove such a resource is 30 seconds; change it with
cloudstack.linstor.template.discard.wait.ms in JAVA_OPTS of
/etc/default/cloudstack-management. Without the fix, a copy that was cut off by a
host crash left a resource that the next deploy took for the template; the guest then
started with a damaged root file system. A template that was taken over that way
before the upgrade cannot be told apart from a good one: delete it from the pool and
let it be copied again if you have had a host go down during a first deploy.
While one pool is writing a template, no other pool of the same controller and
resource group may touch the unfinished resource: a copy that finds it without the
mark fails and names the other pool instead of deleting it. Karios no longer lets two
pools share a resource group, so this only protects pools that were created that way
before.
A rolling swap of the Karios core binary under load was run once on the three-host test cluster, agents first and the management node last (the same order as above). It was a core swap only: no DRBD, satellite or LINSTOR plugin component was upgraded in it, and the cluster had no LINSTOR pool or layer at that point (they had been rolled back before the run), so its five HA guests ran on the cluster’s own Ceph storage. An agent’s swap cost about 12 seconds of that agent’s core API. The management node’s swap was a 14 second outage of the management API (the engine’s own service never failed), with the database schema migration inside it (140 to 160 tables). Guest I/O saw no error in any step. It bounds the mechanism on a virtual test cluster; it says nothing about a LINSTOR pool under upgrade.
Limits
Limit |
What to know |
|---|---|
Template from a LINSTOR root disk |
DRBD makes the device a few KiB larger than the volume, so a template cut
straight from a LINSTOR root disk would be too big to deploy with the same
disk size. Karios cuts the template back to the volume size, but only
when it can prove nothing is lost: a GPT or MBR partition table whose
partitions all end inside the target size, an ext2/3/4 or XFS filesystem
that fits, or a blank disk. A boot signature over four empty partition
entries is not a partition table: such a disk is judged like one with no
table, so a filesystem that declares more than the cut leaves is kept. A GPT must also stay valid in the shorter
image: its last usable sector has to lie inside the target size, because
the guest’s kernel drops a table whose last usable sector is past the end of
the disk and then sees no partitions. That holds when the pool pads the
device by at most 33 sectors (16.5 KiB: one sector for the backup header and
32 for the backup entries of a table with the usual 128 entries; a table
with another entry count moves the number); the 8 KiB of the ZFS-backed test
pool is fine, while a pool that pads more keeps the device size for a
partition grown to the end. The entries array of the primary table must also
end before the table’s first usable sector, so that the cut keeps it. A disk
is called blank only when its first MiB (sector 0 may hold the disk identifier
and the signature of an empty partition table, nothing else), the last MiB
before the cut and everything past the cut are all zero, so a whole-disk btrfs, reiserfs or ZFS
(whose superblock or labels sit in the first MiB) is never cut as blank. The backup copy of the
partition table lies past the cut and is left behind; the guest reads the
primary table, and |
Encryption |
An encrypted disk offering needs an encryption master key in the Karios engine. Without one the create fails. |
Secure Boot |
The DRBD module Karios installs is not signed, so a host with Secure Boot
on refuses it and the pool health reports |
Kernel and image changes |
A kernel update, or an image switch to one that does not carry the DRBD layer, removes the module until the layer is applied again. Karios’s loader then refuses to load an older built-in DRBD 8 and the health reports the module as not loaded. |
Live migration |
Works between hosts (Live Migration). Moving a disk between pools while the VM runs is not tested. |
Host failure |
The platform does not restart the VMs of a lost host by itself without Host HA and an out-of-band controller (OOBM) that fences it; see A lost host on a three-host cluster. |
Hypervisor |
KVM only. |
Deleted disks |
A deleted disk is removed at once and cannot be recovered. The backup of a snapshot on secondary storage can restore it. |
Snapshots |
Need |
Tested versions |
One controller version (1.35.3), see Before You Start Checklist. |
Certificate for a joining host |
Recorded in Karios’s database as well as staged as a file on each management node, so any management node can give it to a joining host; the engine on a node that has no staged file still reads the file; see Hosts that join later. |
Troubleshooting
Symptom |
Likely cause |
What to do |
|---|---|---|
The pool is refused: “These hosts are not ready” |
A host in scope is not online in your controller, its firewall blocks the controller, or it has a structural problem such as a name held with another host’s address. |
Read the host and reason in the message, fix it, then click |
The pool is refused: the controller “refused the API token” |
Wrong, expired or read-only token. |
Create a working token on the controller and send it again. |
The pool is refused: no resource group named … |
The group does not exist on that controller. |
Create it, or use an existing group’s name. |
The pool is refused: the pool … already uses the resource group … |
Another Karios pool uses the same controller and resource group. One resource group backs one pool. |
Create the new pool on another resource group, or use the existing pool. |
The pool is refused: the certificate is not trusted |
A private authority signed the controller certificate. |
|
|
Your controller cannot be reached, or every host has a problem. |
Check port |
|
The controller refused Karios’s token, or no host in scope was listed. |
|
A host row says the satellite is not running |
The host was not prepared, or the satellite stopped. |
Run the satellite route (see API Routes) for the pool. |
A host row says the DRBD module is not loaded |
Secure Boot is on, the kernel changed, or the DRBD layer is missing. |
Turn Secure Boot off and apply the current Karios image layer, then |
A host row says the controller has no node with that name, or lists it as not online |
The node name differs from the host name, or the satellite cannot reach the controller. |
Register the node under the exact host name, or run the satellite route. |
A host row says the satellite and controller versions differ |
Your controller runs another LINSTOR version than the host’s satellite. |
Run the controller at the satellite version shown on the host. |
|
|
Set it to a range inside |
|
A host never received the authority certificate (it joined another way, or the certificate was missing or stale at join), its keystore was rebuilt, or a management node’s staged certificate is stale (the advice line names it). |
Send the certificate again with the trust route. The agent restores it at its next start. |
Deploying a VM from a template fails: root disk size smaller than the template size |
The template was cut from a LINSTOR root disk and kept the device size. |
Use a root disk offering larger than the source disk. See Limits. |
Deploying a VM from a template fails and names a template resource that holds an unfinished copy and cannot be deleted |
An earlier copy of the template into the pool did not finish (a host went down during it) and the cluster has not been able to remove what it left, for example because that resource is still open on a host. |
Karios will not reuse a template that was not copied to the end. Wait for the cluster to remove the resource, or delete it on your cluster, then deploy again. |
Deploying a VM from a template fails: there is no finished copy of template resource … |
The template resource on the controller has no completion mark: it was cached before the mark existed, or its copy never finished. |
Delete the template from the pool and deploy again, or, if the resource is known to be good, set |
Copying a template fails: the template resource has no completion property and another pool … has not finished copying it |
Two pools of one controller and resource group (created before Karios refused that) are copying the same template. |
Wait for the other pool’s copy to finish and deploy again. If the other pool’s copy cannot finish, delete the template from that pool. Give each pool its own resource group. |
An attach, a detach or a VM start takes minutes, then fails naming the controller |
Your controller is unreachable, or accepts connections and does not answer, and the host is waiting for it. |
Restore the controller. See Behavior During Failures for how long each step can wait. |
An encrypted disk cannot be created |
The Karios engine has no encryption master key. |
Configure the master key, or use an unencrypted offering. |
A snapshot failed and a snapshot is left on your cluster |
Secondary storage was full. |
Free space on secondary storage and delete the leftover snapshot on your cluster. |
Revert is refused |
The disk is attached to an instance that is not stopped. |
Detach the disk or stop the instance, then revert. |
API Routes
All routes are POST under /api/v2/storage/pools/linstor/ and need
a root administrator (the root account, with the admin role or the CONTROL_CENTER_MANAGE permission). The pool health is the same: the
LINSTOR cluster card is not shown to other users.
Route |
Use |
|---|---|
|
|
|
|
|
|
|
|
|
|
A disk is made from a snapshot with POST /api/v2/storage/volumes/create-from-snapshot,
{"name": ..., "snapshot_id": ..., "vm_id": ...} (vm_id optional). It
works on any pool, LINSTOR included. The size and disk offering come from the
snapshot. A tenant can use only its own domain’s snapshots and attach only to a
VM it can see.
Done Criteria
The pool is
Upand theLINSTOR clustercard readsHealthy.Certificate verificationreadsVerified, or you have chosen to skip it for a lab.One disk was created, attached to a VM, written to, detached and deleted on the pool.
You know your controller’s version, tiebreaker and port range, and who rotates the token.