LINSTOR Storage

UI path: Control Center -> Storage -> Instance Storage (protocol linstor)

Where This Fits in Karios

This page is a sub-page of Storage. It covers one kind of Instance Storage pool: a pool that uses a LINSTOR cluster you already run. The common pool workflows (list, details, tags, maintenance) are in Instance Storage. Block disks, templates and snapshots that live on a LINSTOR pool work through the same screens as on any other pool: Block Storage, VM Templates and Snapshots. Shared terms are in the Glossary.

What LINSTOR Storage Is

LINSTOR is open-source software that manages replicated block volumes. It places each volume on two or more storage nodes and uses DRBD, a replication layer in the Linux kernel, to keep the copies identical. A LINSTOR cluster has one controller (the management service and its REST API) and several satellites, the services on the nodes that hold or use data.

Karios supports LINSTOR as a customer-owned storage backend:

  • You own and operate the LINSTOR controller and the storage nodes that hold the data. Karios does not create, upgrade, repair or monitor that cluster beyond the checks described below.

  • Every Karios host in the pool’s scope joins your cluster as a diskless client. It stores no data. When a VM disk is attached, Karios asks your controller to make the volume available on that host, and the host reads and writes the volume over the network to your storage nodes. When the disk is detached, that attachment is removed again.

  • Karios installs and keeps up to date the DRBD 9 kernel module and the LINSTOR satellite on its own hosts. Do not install either yourself.

Important

Only KVM hosts can use a LINSTOR pool.

Purpose

Use this page to:

  • Check that your LINSTOR cluster is ready for Karios before you add the pool.

  • Add the pool and read its health.

  • Rotate the controller API token.

  • Know what works, what does not, and what to expect when something fails.

Quick Start (Minimum Path)

  1. Prepare the cluster: see Before You Start Checklist.

  2. Add the pool: see Add the Pool.

  3. If your controller uses a private certificate authority, trust it: see Controller With a Private Certificate Authority.

  4. Open the pool and confirm Health reads Healthy: see Read the Pool Health.

  5. Create one disk on the pool and attach it to a VM: see Block Storage.

Before You Start Checklist

Do these on your own LINSTOR cluster and network. Karios opens the matching ports on its own hosts and checks the rest, but it cannot change your side.

Requirement

Detail

Controller version

The controller must run exactly the same LINSTOR version as the satellite Karios installs on its hosts (LINSTOR refuses a satellite of another version). Karios shows both versions in the health card and reports a host whose satellite differs. This release was tested only with a 1.35.3 controller, DRBD 9.3.4 on the hosts and satellite 1.35.3. No other controller version has been tested, so do not assume a range.

HTTPS on port 3371

The controller must answer HTTPS on 3371. Karios connects as linstor+ssl://<controller>:3371. Port 3371 must be reachable from the Karios management servers and from every Karios host in the pool’s scope: the hosts call the controller themselves when a disk is attached, detached or deleted.

Certificate

The controller certificate is checked by default. Use a certificate that a public authority signed, or follow Controller With a Private Certificate Authority. You can skip the check for a lab, but then the API token travels to an unverified endpoint.

API token

Create an API token on the controller. Karios uses it to register nodes, create and delete volumes and snapshots, and read the cluster state, so a read-only token is not enough. The token is stored encrypted, is never shown again and is never returned by the API.

Resource group

Create a resource group for Karios. Karios carves every volume from it, so it must have free space behind it. Use a place count of 2 or more for redundancy (tested with 2). A place count of 1 keeps a single copy. One resource group backs one Karios pool. Two pools on the same controller and resource group would share the copies of each template, and one pool could delete a copy the other is still writing. Karios refuses to create the second pool; give each pool its own resource group. The check compares the controller address as typed, so linstor1.example.com and its IP address count as two controllers: use the same address for every pool on one controller. If the pool list cannot be read, the create goes on with a warning.

Storage pool type

Snapshots work only when the resource group’s backing pools are LVM_THIN, ZFS or ZFS_THIN. A thick LVM or file-based backing pool cannot snapshot. Avoid a dash in the name of an LVM volume group: on the engine version Karios ships, a snapshot backup of such a volume fails.

Tiebreaker

With two copies, add a diskless tiebreaker so the cluster keeps quorum when one storage node is lost. Include DISKLESS in the resource group’s provider list and set the quorum to majority, so LINSTOR places a diskless voter on a third node. Restrict placement to your storage nodes (for example with a node property and ReplicasOnSame) so that a Karios host is never chosen as a replica or tiebreaker. Without a tiebreaker, losing one of two copies makes the volume return I/O errors.

Block size

Set Linstor/Drbd/auto-block-size to false and use a 512 byte block size. Left alone, LINSTOR picks 4096 and overrides the resource group.

Network ports

TCP 3366 and 3367 from your controller to the Karios hosts (the satellite). The controller reaches the satellite on 3366 in plain TCP, unencrypted, unlike the HTTPS on 3371: keep it on the storage network. The host firewall opens it to your controller only. TCP 7600 to 7999 in both directions between the Karios hosts and your storage nodes (DRBD).

DRBD port range

Set the controller property TcpPortAutoRange to a range inside 7600-7999. LINSTOR’s default is 7000-7999. A volume given a port below 7600 is blocked by the host firewall while everything else looks healthy, so Karios reports the pool Degraded when the range is outside.

Node names

A LINSTOR node name must equal the Karios host name exactly (letters, digits, ., _ and -, starting with a letter or digit). The pool page lists the names under Prepare your LINSTOR cluster. Karios registers a missing node itself when you add the pool (see What Karios Does Automatically); register hosts yourself only if you want to control the address, and use the exact name.

Host network

DRBD traffic and the host firewall rules use the host’s storage network. Your controller must reach each Karios host on its storage network address. The health check reports a host that the controller reaches only on its management address.

Secure Boot

Turn Secure Boot off on every Karios host. See Limits.

Permissions

Adding the pool, and every LINSTOR route in API Routes, needs a root administrator (the root account, with the admin role or the CONTROL_CENTER_MANAGE permission). These calls register hosts in your controller, start their satellites and change their firewall and trust as root, so a tenant’s storage rights are not enough. Anyone else is refused with a message that says so.

Offerings

A storage tag on the pool does not keep untagged offerings or system VMs off it. Decide what may land on the pool and tag the other offerings, see What a Storage Tag Does and Does Not Do.

Add the Pool

Steps:

  1. Open Control Center -> Storage -> Instance Storage.

  2. Click + Add Storage Pool.

  3. Set Protocol to linstor. The option is always listed. It can be chosen only when the platform’s provider list includes LINSTOR; otherwise its tooltip says why.

  4. Fill in the fields:

Field

Required

Description

Storage Name

Yes

Unique name for the pool.

Scope

Yes

The form opens on CLUSTER. Prefer it. A ZONE pool does not report a host that cannot reach your cluster, so it can read Up with no working host.

Zone and Cluster

Yes

Where the pool is available. Cluster is required for CLUSTER scope.

Provider

Read-only

Shows the LINSTOR provider. Karios hosts join your cluster as diskless clients.

Controller address

Yes

Host name or IP address of your controller, optionally :port. Type the host only, with no scheme, path or credentials. The port defaults to 3371.

Resource group

Yes

The resource group Karios carves volumes from: 2 to 48 letters, digits, hyphens or underscores, starting with a letter or underscore. It must already exist on the controller, and no other Karios pool may use it on the same controller. It cannot be changed after the pool is created.

API token

Yes

A token your controller accepts.

Skip certificate verification

No

Off by default. Tick it only for a lab, or for the first step of Controller With a Private Certificate Authority.

Storage tags

Yes

At least one, so that offerings which carry the tag can be sent to the pool on purpose. The tag does not keep untagged offerings or system VMs off the pool: see What a Storage Tag Does and Does Not Do.

  1. Click Add Storage Pool.

Karios checks the controller before it creates anything. It refuses the pool, and says why, when the controller refuses the token, has no resource group of that name, presents a certificate it cannot verify, or when another Karios pool already uses that controller address, as typed, and resource group (the message names that pool; create the new pool on another resource group, or use the existing pool). If the pool list cannot be read the create goes on with a warning. Pools that were created on one resource group before this rule are not changed and are not flagged: give the second pool its own resource group and move its volumes.

Then Karios prepares every host in the pool’s scope (see What Karios Does Automatically). This can take up to a few minutes. If a host cannot be made ready, the pool is not created and the message names each host and the reason. The token stays in the form: fix the cause and click Retry. A host that is down or in maintenance is skipped and named.

Expected Outcome:

  • The pool appears in Instance Storage and reaches Up.

  • Opening the pool shows the LINSTOR cluster card with Health Healthy.

If this fails: see Troubleshooting.

What a Storage Tag Does and Does Not Do

The platform chooses a pool for a new disk by comparing the storage tags of the disk offering with the tags of the pools in scope. The rule is the same for every kind of pool, LINSTOR included:

  • An offering that has tags reaches only a pool that carries every one of them.

  • An offering that has no tags reaches every pool in its scope that has no tag rule, whether the pool is tagged or not. The disk of a system VM is placed the same way.

So the storage tag on a LINSTOR pool lets you aim tagged offerings at it. It does not keep untagged offerings, such as the stock Small Instance, or system VMs off it. Karios does not register a LINSTOR pool with a tag rule (a tag rule is the one thing in the platform’s placement code that an untagged offering does not match, and it needs the platform setting js.interpretation.enabled, which is off by default), so no setting of the pool changes this.

What backs this: the placement code of the platform version Karios ships (Apache CloudStack 4.23.0.0: the pool lookup with an empty tag list returns every pool that is up in scope, minus the pools that have a tag rule), and one observation on a three-host test cluster, where an untagged compute offering put a root disk on a tagged Ceph pool. The same code serves every pool type; it was not repeated on a LINSTOR pool. The setting match.storage.pool.tags.with.disk.offering only governs changing a disk’s offering later, not the first placement.

To keep other workloads off a LINSTOR pool, work in this order. Tagging an offering before a pool carries its tag makes every deploy from that offering fail with “Unable to find suitable primary storage”.

  1. Give the pools that the other workloads should keep using (for example your Ceph pool) a storage tag of their own.

  2. Then give every compute offering and disk offering that can reach the LINSTOR pool’s scope, the stock untagged ones such as Small Instance included, that tag. An offering whose tag the LINSTOR pool does not carry cannot land on it. How an existing VM behaves after its offering was tagged (for example at its next start) was not tested; try it on a test VM first.

  3. If no system VM may land on the pool either, do the same for the system offerings named by the platform settings consoleproxy.service.offering, secstorage.service.offering and router.service.offering: a pool that carries their tag must be Up in every zone before you tag them, because otherwise the system VMs cannot be recreated and Create VM stays disabled without a healthy secondary storage VM. The platform’s updateServiceOffering call takes storagetags; a tag on a system offering was not tried on the test cluster.

  4. Prefer cluster scope: a cluster-scoped pool serves only the VMs of its cluster, a zone-scoped one every cluster of the zone.

Karios still requires at least one tag on a LINSTOR pool, because without a tag no offering can be aimed at it.

Controller With a Private Certificate Authority

Use this when your controller’s certificate is signed by an authority that Karios hosts do not already trust (a company authority, or a self-signed certificate). Karios checks the controller certificate by default, and the pool form has no field for the authority, so this is a three-step flow that uses the Karios API for steps 2 and 3.

  1. Add the pool with Skip certificate verification ticked (see Add the Pool).

  2. Send the authority’s certificate. It must be one PEM certificate that verifies the controller’s certificate (the authority that signed it, or the controller’s own certificate when it is self-signed). Karios proves that it verifies the controller before it stages anything, then installs it on every host in the pool’s scope and on the management nodes. No service restart is needed.

    Sign in first (POST /api/v2/users/login returns jwt_token) and put it in TOKEN. The examples use -k only for a Control Center that still has a self-signed certificate; drop it otherwise.

    curl -sk -X POST "https://<control-center>/api/v2/storage/pools/linstor/trust" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d "$(jq -n --arg id "<pool id>" --rawfile pem ca.pem '{pool_id:$id, ca_pem:$pem}')"
    

    The caller must be a root administrator (the root account, with the admin role or the CONTROL_CENTER_MANAGE permission).

  3. Switch the check on:

    curl -sk -X POST "https://<control-center>/api/v2/storage/pools/linstor/credentials" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"pool_id":"<pool id>","ssl_insecure":false}'
    

Open the pool. Certificate verification reads Verified and the pool health stays Healthy when every host trusts the controller.

Note

The certificate is staged as a file on every management node that was listed when you sent it, and Karios also records it in its database, so whichever management node runs a later join can give it to a new host. A management node added later has no staged file. Karios’s own client on that node verifies the controller through the recorded certificate, which is enough to finish a join there (register the node, open the firewall, start the satellite, hand the certificate to the hosts). The engine on that node reads the staged file, so until you send the certificate again after adding a management node, the pool health does not report Certificate verification as Verified and its advisories name the management node that answered the health request and lacks the file (a node that did not answer is not seen). The health decides this with a handshake that uses only that node’s staged files and the system roots, so the same holds for a node whose staged file is stale or is another authority’s, not only for one with no file. A request to switch a pool to verifying (the credentials route with ssl_insecure false) is checked with what that node’s files trust, so it is refused there until the certificate is staged.

What Karios Does Automatically

When you add the pool, and for each host that joins later, Karios prepares the host. For every host in scope that is up and enabled:

Step

What happens

Register the node

If your controller does not hold a node with the host’s name, Karios registers it as a satellite node, at the host’s storage network address when it has one. An existing node is never replaced. Karios does not remove these registrations when you delete the pool: remove them on your controller if the hosts should no longer be part of LINSTOR.

Open the host firewall

Karios declares the rules the host needs: TCP 3366 and 3367 from your controller only, and TCP 7600 to 7999 from your LINSTOR nodes only. The rules are withdrawn when the last pool that needs them is deleted.

Start the satellite

The satellite is bound to the address your controller holds for the node and started. An existing bind file is never overwritten.

Wait for the node

Karios waits until your controller shows the node online and the authentication file your controller writes is on the host with a token the controller accepts.

Trust the certificate

For a pool that checks the certificate, the host gets the authority certificate you sent (see Controller With a Private Certificate Authority). When you delete the pool, Karios takes that certificate off the hosts and management nodes again, from the system trust, from the agent’s own keystore and, with the last pool that needs it, the agent’s start-up hook that keeps the keystore in step. Each host reports what became of the hook, and a host that kept it is named in the Karios log. A hook that is left behind with nothing staged removes itself at the next agent start. A host that was down or unreachable when the pool was deleted keeps the certificate until you run /usr/local/sbin/karios-stage-linstor-trust.sh remove <pool id> on it.

Hosts that join later

A host that joins a cluster (or, for a zone pool, a zone) that has a LINSTOR pool gets the same preparation. This happens when a host is onboarded, when it is added through Add Host and when it joins through the agent’s own pull join (a node installed from the Karios ISO, a pre-installed agent, or an onboard that Karios runs again after a restart); for a pull join it runs once the onboard has verified the host. The same step also provisions a dedicated storage policy that was waiting for the cluster’s first host. For a pool that checks the certificate the host also receives the authority certificate, so a scale-out does not silently break certificate checking.

  • The certificate is given only when Karios holds exactly one valid, unexpired certificate for the pool that still verifies the controller. If not, nothing is sent, a warning in the Karios log names the pool and host, and the pool health shows that host as not trusting the certificate.

  • After staging, Karios reads what the host’s agent trusts and, if it does not trust yet, sends the certificate once more.

  • A host that is not yet Up is waited for up to 5 minutes. A host still not ready after that is named in the log, never reported as prepared.

  • If your controller cannot be reached when the host joins, or the satellites are not online yet, or Karios holds no certificate for the pool yet and a host in scope does not trust the controller (none staged on this management node and none recorded; when every host already trusts the controller there is nothing to wait for), Karios keeps a record of the join in its database and tries again, first after 30 seconds and then after longer pauses of up to 15 minutes, until it works: a join that waits for a certificate finishes by itself once you send it. The record survives a restart of Karios. A run keeps its record claimed by renewing it every minute, and a claim lapses 3 minutes after the last renewal, so a Karios that stopped in the middle of a join is picked up again within about 3 minutes of the next retry tick (every 15 seconds), by whichever management node is active. A run that hangs lets go of its record after 30 minutes. Karios stops after 7 days and says so in the log. What needs you (a certificate that no longer verifies the controller, one that cannot be used, a host that needs a change of setup) is not retried. Only the active management node runs the retries. Because the certificate is also recorded in the database, and Karios’s own client trusts the recorded certificates as well as the staged files, a management node without a staged file of its own completes the whole join: it registers the node, opens the firewall, starts the satellite and hands the certificate to the hosts. The engine on that node still reads the staged file (see the note under Controller With a Private Certificate Authority). The clocks of the management nodes are assumed to agree to within a few seconds.

  • Several hosts that join one cluster together are prepared one after another, and so are joins of different clusters that share a zone pool.

  • Any other way a host appears in the platform does not trigger this (a host added straight in the engine rather than through Karios). Such a host reads as not trusting the certificate. Send the authority certificate again with the trust route in API Routes, and run the satellite route if the host is not prepared.

Note

Hosts that join are prepared in the background and never block onboarding. This behavior, the retry included, is covered by automated tests. It has not yet been observed on a cluster that adds a second host to a certificate-checking pool, by any of the join paths.

Read the Pool Health

Open the pool. The LINSTOR cluster card shows a live check of your controller and of every host in the pool’s scope. Check again repeats it. A result younger than ten seconds is reused.

Row

Meaning

Health

Healthy, Degraded, Down or Unknown. See the next table.

Checked

When the live check ran.

Controller version

The version your controller reports.

DRBD port range

The controller’s TcpPortAutoRange.

Controller and host rows

Each problem the check found, under the word Controller or the host’s name.

Certificate verification

Verified: the pool checks the certificate, Karios reached the controller with the check on, and every host’s agent trusts it. Checking is on, but not every host or management node trusts the certificate: the pool is set to check, but Karios, a management node or at least one host does not trust the certificate yet (the problem line or the advice line says which). Not verified (skipped): the pool does not check the certificate.

Replicas, Usable capacity, Raw capacity

See Capacity.

What each Health value means

Health

Meaning

Healthy

The controller answers, accepts the token, and every host in scope passes every check. A skipped certificate check is advice only and does not lower the health.

Degraded

Some hosts have a problem while others work, or the pool has a problem of its own (for example the DRBD port range is outside 7600-7999, or a storage node that can hold the pool’s volumes is not online). Volumes on the working hosts keep working.

Down

Karios cannot reach the controller, or every host in scope has a problem.

Unknown

Karios could not tell: the controller refused Karios’s token, or no host in scope could be listed. This is not the same as Down.

Checks run for each host

Karios checks that the host can be reached, that its satellite service is running, that the DRBD 9 module is loaded, that your controller lists the node as online under the host’s name and on an address the firewall rules cover, that the satellite version equals the controller version, and, when the pool checks the certificate, that the host’s agent trusts the controller.

Checks on your storage nodes

The copies of your volumes live on your storage nodes, which are not Karios hosts. Karios reads your controller’s own view of them: every node that has a diskful storage pool your resource group may place on (the pools and nodes the group names, or every diskful pool when it names none), except the Karios hosts, which are checked above. A storage node that the controller does not list as online turns the pool Degraded and is named in the Controller problem with its state and how many of the nodes that can hold volumes are online. It then says what that means for the copies, from the group’s place count: with a place count of 1 the volumes on that node have no online copy until it is back; with as many nodes down as the group keeps copies, or more, some volumes may have no online copy; otherwise every volume still has at least the place count minus the nodes down of its copies online. Karios does not know which volumes sit on which node, so it states the worst case and never says that the other copies serve every volume. When fewer nodes are online than the group keeps copies, it also says that new volumes cannot be created until one is back. A node of another storage pool or resource group on a shared controller is ignored. A diskless node that is not online (your tiebreaker, or another client of a shared controller) holds no copy and only adds a line to the advice; if it is your tiebreaker the pool has one voter less for its quorum. If the controller’s storage pool list cannot be read, the advice says the storage nodes were not checked; if the resource group cannot be read, every diskful pool of the controller is judged and the advice says so. Measured on the three-host test cluster: with the satellite of one of two storage nodes stopped, creating a disk was refused with “Not enough available nodes” while the pool still read Healthy; this check closes that gap, and the automated tests cover it, but it was not run on a cluster after the change.

Important

A ZONE scope pool can read Up in Instance Storage while this check finds no working host. Use the card, not the pool state, to judge LINSTOR pools.

Capacity

Usable capacity is the raw figure from your resource group divided by its replica count. Raw capacity and Replicas are shown beside it. The numbers in the Instance Storage list for a LINSTOR pool are the usable ones. If Karios cannot read the replica count, the pool shows the raw figure and says so.

Warning

The platform’s allocator still counts every copy: it allows allocation up to its overprovisioning factor times the raw figure. With two copies and a factor of 2 that is about four times the usable space. Watch the pool’s Used figure, and keep the overprovisioning factor low for LINSTOR pools. Karios warns when the pool is created that the platform allocates against the raw figure, and the pool health names the factor.

Tune DRBD for Write Latency

With DRBD’s default settings a small synchronous write on a replicated volume spends most of its time on DRBD’s own metadata, not on the network or the data disk. DRBD records the area being written in its activity log and, when a peer is connected, updates its out-of-sync bitmap each time an activity-log slot is reused; each of those metadata writes is flushed to the disk.

Two settings change that. Set them on the resource group that backs the pool, so every volume Karios creates from it inherits them:

al-extents (recommended)

A larger activity log is reused less often, so fewer metadata writes and flushes reach the disk. On a two-copy pool this roughly halves the latency of a single 4 KiB synchronous write and nearly doubles small random writes under a 70/30 read/write mix. It is safe on any disk; the cost is a longer resync of the recently written area after a crash:

linstor resource-group drbd-options --al-extents 6433 <resource-group>
disk-flushes and md-flushes (only on suitable disks)

Turning flushes off removes most of the remaining metadata cost, but it is only safe when every disk under the pool keeps written data across a power loss: a disk or controller with power-loss protection, or a disk with its volatile write cache turned off. On any other disk a power cut can lose writes DRBD already reported as done. Check every model in the pool before you use it:

linstor resource-group drbd-options --disk-flushes no --md-flushes no <resource-group>

Changes to a resource group apply to its existing volumes too. A setting made directly on one resource definition overrides the group, so remove such per-volume overrides if you want the group’s value to apply.

Large sequential writes and deep queues are usually limited by the replication network rather than by these settings: give the storage nodes a replication link at least as fast as the disks can write.

Choices for Snapshots, Flushes, Offerings and Disk Size

Snapshots stay copied to secondary storage

By default Karios copies every volume snapshot of a LINSTOR disk to secondary storage, as it does for other pools. Keep it that way unless you have a reason not to. The copy is what lets you restore a disk if the LINSTOR cluster itself is lost. It is a full copy each time, so it is most of the time a snapshot takes and it grows with the amount of data on the disk: a root snapshot with a backup took about 116 seconds against 57 on Ceph RBD in the comparison under Supported Operations Compared With Ceph RBD (that run does not record the disk size, so read it as a comparison, not a promise). A snapshot that lives only on your cluster is faster to take, but it is lost together with the pool. The copy is controlled by the global setting lin.backup.snapshots (default true).

Turn flushes off only on protected disks

Turning off disk-flushes and md-flushes (see Tune DRBD for Write Latency) brings a single small synchronous write down to about a third of its latency. Do it only when every disk under the pool has power-loss protection or has its volatile write cache turned off. On any other disk a power cut can lose writes that were already reported as done. When in doubt, leave the flushes on and use only al-extents.

The LINSTOR Standard offering and its IOPS cap

Coming in the release that ships the LINSTOR Standard offering. It is not part of earlier releases.

When you add a LINSTOR pool that has a storage tag, Karios also creates a disk offering called LINSTOR Standard. It has a custom size, carries all of the pool’s storage tags, and limits each disk to 10000 reads and 10000 writes per second. The limit keeps one busy virtual machine from slowing the others that share the same storage.

Two things to know:

  • There is one offering per set of storage tags. If an offering called LINSTOR Standard already exists with the same tags, Karios leaves it as it is. A LINSTOR pool with different tags gets its own offering, named after its tags, for example LINSTOR Standard (fast,ssd).

  • It is created with no domain restriction, so every account can choose it and create disks on your LINSTOR cluster. Restrict its domain if the pool is not meant for every tenant.

The limit cannot be changed on an existing offering from the disk offering page. To get a different limit, create a new disk offering with the pool’s storage tag and the IOPS you want, then hide or disable LINSTOR Standard. A limit set too low throttles your own virtual machines, so measure your workload before you lower it.

Disks show exactly the requested size

Coming in the release that ships exact disk size. It is not part of earlier releases.

DRBD keeps its own metadata inside the volume, so today a LINSTOR disk is a little larger than the size you asked for. In the release that ships exact disk size, a newly created LINSTOR disk shows inside the guest exactly the size you asked for. The difference removed depends on the pool behind the volume: about 8 KiB on ZFS thin and about 3.5 MiB on LVM thin (measured on a 2 GiB disk). Existing disks keep their current size. The change applies only to volumes created after the release is installed.

Two things change with it:

  • Growing a LINSTOR disk no longer reaches a running guest. The grow succeeds on the pool, and the guest sees the new size after the virtual machine is stopped and started. On Ceph RBD the running guest sees it at once.

  • A disk with a GUID partition table (GPT) keeps a second copy of the table at the very end of the disk. If a guest sees a new disk whose end is not where that copy sits, for example a disk made from a template that was prepared on one of the older, slightly larger volumes, the guest warns that the backup GPT is not at the end of the disk, or tools refuse to change the partitions. Move the copy to the end of the disk from inside the guest:

    sudo sgdisk -e /dev/vdb
    sudo partprobe /dev/vdb
    

    Use the device name of the disk (/dev/vda, /dev/vdb and so on). Take a snapshot first. If a partition itself was grown into the extra space, sgdisk -e cannot save the tail of that partition. Shrinking the file system does not work for every case (XFS cannot be shrunk, and a mounted root cannot be shrunk online). The fix that works for every file system is to grow the disk in Karios by a small amount (for example 1 GiB), stop and start the virtual machine, then run sgdisk -e and, if you want the space, growpart and xfs_growfs or resize2fs.

Rotate the API Token

  1. Create a new token on your controller. Keep the old one valid for now.

  2. Open the pool and click Rotate token. (It needs a root administrator.)

  3. Paste the new token and confirm.

  4. Revoke the old token on your controller after the pool reads Healthy.

Karios tries the new token against your controller first and changes nothing if the controller refuses it. Then it replaces the stored token. Each host’s own token file is written by your controller, so nothing is copied to the hosts and no restart is needed. A token that works for the controller but fails part way through the change is reported with the instruction to send the same request again.

The token is never displayed, returned by the API or written to logs.

Supported Operations Compared With Ceph RBD

Both pool types were run through the same steps on one test cluster. “Same” means both behaved alike.

Operation

LINSTOR

Compared with Ceph RBD

Create a disk, attach, detach, re-attach

Works

Same. Data is intact after a detach and re-attach.

Grow a disk online

Works. In the release that ships exact disk size, a new disk’s running guest sees the new size after a VM stop and start.

Same. Shrinking is refused on both. On RBD the running guest sees the new size at once (see Disks show exactly the requested size).

Delete a disk

Works. The disk is removed at once and cannot be recovered. A snapshot’s backup on secondary storage can restore it.

Same for a disk with no snapshot. Different for a disk that has a snapshot: RBD keeps the deleted disk, so it can still be recovered until the platform’s storage cleanup runs (by default after 24 hours).

Deploy a VM from a template, stop, start, reboot, destroy, recover, reinstall

Works

Same.

Grow the root disk

Works

Same.

Disk IOPS limits from the disk offering

Works

Same.

Volume snapshot (backed up to secondary storage)

Works. A full copy to secondary storage each time.

Different. On RBD the snapshot stays on the pool and no copy appears on secondary storage in the time we waited.

Create a disk from a snapshot

Works

Same.

Revert a disk to a snapshot

Works when the disk is detached. Refused while the disk is attached to an instance that is not stopped.

Same.

Template from a root snapshot

Works

Same.

Template from a root disk

Works, with a size rule: see Limits.

Works.

Whole-VM disk-only snapshot, revert, delete

Works

Not supported: the snapshot ends in Error on RBD.

Recurring snapshot policy

Works

Same.

Download a disk (extract) and upload a disk from a URL

Works

Same.

Host agent restart with VMs running

Works. VMs and disk I/O continue.

Same.

HA restart of a VM after its process is killed

Works

Same.

Account isolation and account resource limits

Works

Same.

Disable the pool, or put it in maintenance

Works. A disabled pool refuses new disks (4.0 seconds, “Unable to find suitable primary storage”). Maintenance stops the VMs that have disks on the pool, and no write follows the stop. Deleting a pool is refused until it is in maintenance and has no disks left on it.

Same, on a throwaway RBD pool of the three-host test cluster.

Encrypted disk offering

Fails unless the Karios engine has an encryption master key.

Also fails on this setup.

VM live migration

Works between two hosts (see Live Migration).

Not part of this comparison.

Moving a disk between pools while the VM runs

Not tested. Moving a disk into a LINSTOR pool while the VM runs is known not to work on the engine version Karios ships when the target host holds no copy of the data, which is always the case for diskless clients.

Not part of this comparison.

Speed, measured on one test cluster with the same host and steps: creating and attaching a 1 GiB disk took about 6.7 seconds on LINSTOR and 3.3 seconds on RBD; deploying a VM from a cached template took about 52 and 33 seconds; a root snapshot with a backup to secondary storage took about 116 and 57 seconds. Expect LINSTOR to be slower than RBD for control operations because every volume operation also goes through your controller and every snapshot backup crosses the network through a diskless host.

Behavior During Failures

Your controller is unavailable

The data path does not depend on the controller. These results were measured by stopping the controller, which then refuses connections:

Operation

What happened

Running VMs and disk I/O

Unaffected. A write loop in a guest ran with no errors for the whole outage.

Pool health

Down, with the cause shown.

Create a disk, create and attach

Refused within a few seconds. A refused create-and-attach can leave an Allocated disk row behind; delete it from Block Storage.

Detach a disk

Completed in about 3 seconds. The VM no longer lists the disk. The host releases the disk without asking the controller.

Delete a disk after the controller returns

Works, and leaves no resource on the controller or on the host.

After a detach during an outage, the host’s diskless attachment stays registered on your controller until the disk is deleted or attached again.

On the three-host test cluster, with the controller down, stopping a VM worked (6.4 seconds: the detach does not need the controller) and starting it was refused after 4.3 seconds with the platform’s generic “Unable to orchestrate the start of VM instance” (the controller is named only in the management log). The start worked 6.4 seconds after the controller returned. The one-host test bed took many minutes to refuse the same start with the same plugin build. The two runs differ in when the controller was stopped. On the one-host bed the VM was stopped first, while the controller still answered, so the host let go of its disks completely. On the three-host cluster the VM was stopped during the outage: the host could not tell the controller, so the disks’ DRBD devices stayed up on the host, the start found them there and never entered the long lookup below, and was refused at once when it then asked the controller to make the disks available. This reading comes from the code and from the order of the two tests; the three-host run’s host log was not kept, so it was not confirmed from a log, and it was measured once. What it does not explain is why the platform’s start retries (start.retry is 10) did not move to another host, where there is no local device and the long lookup below would have run: the total of 4.3 seconds says they did not, and no management log was kept to show why. The wait below therefore applies to a VM whose disks are not up on the host that starts it (stopped before the outage, or started on another host); a VM stopped during the outage and started again on the same host was refused within seconds.

How long a call waits depends on how your controller fails. A controller that refuses connections is the quick case. One that does not answer at the network level (powered off, cut off by a firewall) costs the 5 second connection limit. The slow case is a controller that accepts the connection and then never answers (a hung process): each call then waits for its read limit. The limits are 5 seconds to connect, 15 seconds for a read of state, and 300 seconds for a call that changes something (create, delete, snapshot, resize, make a disk available), because your controller carries such a call on after the caller gives up, and a shorter limit would leave resources behind. These waits come from the limits. Automated tests check the create, delete, snapshot, resize, detach and statistics paths against real sockets with shortened limits; only the first table above was measured on a cluster, and so were the attach with no answer at the network level and two cases of the VM start row (a controller that refuses connections, and one that does not answer at the network level); the remaining figures are derived from the limits and the host code, not tested.

Operation

Controller refuses connections

No answer at the network level

Accepts the connection, never answers

Attach a disk

At once

About 5 seconds on top of the call (measured: 6.2, 6.2 and 10.2 seconds)

About 5 minutes

Detach a disk

A few seconds

About 10 seconds

About 30 seconds (two reads of 15 seconds)

Start a VM that has disks on the pool, on a cluster with one host (disks not up on the host)

About 10 minutes (measured: 10 minutes 14 seconds)

About 27 minutes (measured: 27 minutes 25 seconds)

Up to about 60 minutes

Create, delete, snapshot, resize (management server)

At once

About 5 seconds

Up to 5 minutes per call

Pool capacity and disk usage reads (management server)

At once

About 5 seconds

15 seconds

Most of these end with an error that names your controller, for example Linstor controller https://controller.example:3371 is unreachable (read timed out: it accepted the connection and did not answer): attaching a disk, creating or deleting one, resizing, the statistics reads, and the VM snapshot create and revert. Three do not, because their text comes from the platform and not from the LINSTOR plugin. Creating a snapshot of a disk answers Failed to create Snapshot due to an internal error creating Snapshot for volume ..., deleting a VM snapshot answers Failed to delete Instance Snapshot, and a VM start that fails answers Unable to orchestrate the start of VM instance .... For these, the sentence that names your controller is in the management server log (Failed to take snapshot: Linstor controller ... is unreachable; Linstor: Linstor controller ... is unreachable followed by Unable to delete Linstor resource ... for the VM snapshot delete; and the host’s answer for the start). A VM snapshot that could not be created or deleted while the controller was down is left in state Error; delete it again once the controller answers. Every call of the LINSTOR storage driver and of the VM snapshot strategy on the management server completes with an answer instead of a task that stays running. If the pool’s URL is a plain http:// or linstor:// address, the client first asks the controller for its version, waiting up to 3 seconds to connect and 3 to read, before the call itself; the figures above are for the usual linstor+ssl:// address, which has no such step.

Starting a VM is slow because the host first looks every disk of the VM up, and the Karios engine’s host code repeats a failed lookup 100 times, 3 seconds apart (about 5 minutes when the controller refuses connections). The platform then tries the start again: on the one-host test cluster a start with the controller refusing connections ended after 10 minutes 14 seconds, two attempts of 5 minutes 6 seconds each. With a controller that did not answer at the network level (its packets dropped) a start ended after 27 minutes 25 seconds, again two attempts, of about 13 minutes 40 seconds each. The figure for a controller that accepts the connection and never answers is twice the derived time of one attempt and was not measured. How many attempts a cluster with more hosts gets was not checked (the engine’s start.retry setting is 10, and its planner avoids a host that failed), so treat these figures as the one-host case, not as a bound. That loop is not part of the LINSTOR plugin and is not changed here. It does not affect a disk that is already up on the host, which is why a detach works during an outage. The error the platform returns for a failed start says only that it could not orchestrate the start; the message that names your controller is in the management server log and in the host’s answer. Deleting a disk is refused with the same message; retry it after the controller returns. Treat the pool as unavailable for new attachments and for starting VMs until the controller returns.

Host failure detection (HA)

Declaring a host dead makes Karios fence it and restart its VMs elsewhere, which cannot be undone if the host was in fact alive; noticing a dead host late only delays HA. So Karios reads a host as dead in one narrow case, and in every other case it cannot tell and keeps the host.

When Karios checks whether another host is alive, the checking host first asks DRBD whether it has a connection to that host: a connection in state Connected means the host is alive, and nothing else is asked. Every other answer from DRBD (a connection that is not Connected, no connection to the host, the two share no resource, drbdsetup fails, or its output cannot be read) is no evidence of death, and one rule applies to all of them. The checking host asks your controller for its list of nodes, once, waiting at most 5 seconds to connect and 15 seconds for the answer, and judges from that one answer. The host is read as dead only when all of these hold:

  • your controller answered and lists the node, and shows it OFFLINE;

  • not every node is OFFLINE, and no node is still connecting (a controller that has just restarted shows every node offline or connecting until the satellites reconnect);

  • the checking host is itself shown ONLINE in the same answer (this also proves that the name Karios uses for the host is the node name your controller holds);

  • your controller has not been seen unsettled during the last 5 minutes: unreachable, listing no node, every node offline, or nodes still connecting;

  • the second signal below holds. It is needed whatever DRBD says, because a satellite that is down on a host whose VMs are running reads exactly like a dead host to your controller, a DRBD connection can be stuck (StandAlone, Connecting) or blocked on a host that is up, and fencing a host that is alive is the mistake that cannot be undone.

Both parts of the second signal must hold:

  • It holds over time. Your controller showed the node OFFLINE in an earlier reading taken at least one settle window (5 minutes) before, and no reading since showed the node alive or your controller unsettled. The first reading only starts the clock; the agent keeps it in memory, so a later check by the same agent at least 5 minutes afterwards can read the host as dead. The agent knows only the readings it took itself: when the gap since the previous reading of that node is longer than 2 windows (10 minutes) the clock starts again, so a short drop and a new one 40 minutes later count as two first readings, not as 40 minutes offline. A node that came back and went down again within two readings of the agent is not noticed.

  • It holds on the wire, from the checking host and not through your controller. A TCP connection from the checking host to every address your controller lists for the node gets no answer at all, neither a connection nor a refusal (a refusal comes from the host’s own kernel, so the host is up whatever its satellite does), while the same connection to another ONLINE node does get an answer. Silence is a timeout, or a “no route to host” that took at least as long as the kernel needs to give up resolving an address on the checking host’s own network (about 3 seconds). A “no route to host” that came back at once was sent by a router or a firewall that answered (an ICMP “administratively prohibited” reply, a reject), so it is not silence: the node is not read as dead on it. Each address is connected to once per check. Two things can still make a path that refuses look silent or a dead host look unreadable. Linux limits the rate of its own refusals, per source host: after the first one or two, a router or firewall of yours that rejects the traffic to the node answers only some of the connections and the others time out. The traffic that uses up that allowance is normally DRBD’s own attempts to reconnect to the same node, one for each resource it shares with it. In a test with simulated reconnects every 10 seconds, 17 of 30 probes read silent with 5 shared resources and 24 of 30 with 20, against none with 0 or 1. So a device that rejects the traffic to a live node protects against a false “dead” reading only on a quiet path; with many shared resources it mostly reads as silent, and the node is read as dead once the other conditions hold. Do not reject traffic between LINSTOR nodes either. And a connection that joins an address resolution that is already about to fail fails at once, like a refusal, so a dead host on the checking host’s own network sometimes reads as “cannot tell” on one check (and as dead on the next). The second half of the comparison shows that the path and the port are open from this host to a live node. That rules out a drop that hits this host’s traffic to every node alike, and no more than that: it does not show that the silence is the node’s. A firewall or router of your own that silently drops only the traffic between the checking host and the node, while the node is up, looks exactly like a dead node from here; if your controller also shows that node OFFLINE for a whole window and DRBD does not show it Connected, it is read as dead. Karios cannot tell the two apart over the wire, so do not drop traffic between LINSTOR nodes selectively. The port is 7999, the top of the DRBD port range, which the firewall rules Karios writes open to every LINSTOR node (the satellite port is open to the controller only). With Karios’s own host firewall a live host answers on any port, because what the plane does not allow it refuses with a TCP reset, and a reset counts as an answer. The probe sends nothing and closes the connection at once.

What counts as an address and as the comparison node:

  • Loopback addresses, the any-local address and the checking host’s own addresses (the ones your controller lists for it and the ones on its network interfaces) are left out, for the node and for the comparison node: a connection to them would be answered by the checking host itself. If nothing is left of the node’s addresses, the node is not read as dead.

  • An address that is not an IP literal (a name), an IPv6 address with a zone, a link-local IPv6 address or a multicast address cannot be judged. A node with such an address is never read as dead, because that address may be the live path; a comparison node with one cannot qualify by answering on all its addresses. Names are never looked up.

  • A comparison node counts only if, for each address of the node that was probed, it answers where that address can be calibrated. When one of the checking host’s own network interfaces covers the address, the network is known (LINSTOR lists no prefix length, so it is the prefix of that interface, the longest if several do), and the comparison node must answer on an address in that network. An answer from another network shows that the checking host reaches a live node, not that it reaches the node’s network, so a comparison node whose only address is on another network does not qualify. When no interface covers the address, no network can be named for it (the node is behind a router), and the comparison node must answer on all its addresses, none of them one that cannot be judged, and at least one of them must itself be behind a router. A comparison node that answers only on the checking host’s own networks does not qualify: a fault on the uplink that also cut your controller off would look the same.

  • The check asks at most two comparison nodes, skipping any with no address it can use or that cannot qualify, so a check can take up to about nine seconds on top of the read of the node list.

A cluster with only two nodes has no other node to calibrate against, so in such a cluster a host is never read as dead, whatever DRBD shows: the other node cannot be told from a path that is down. The port (cloudstack.linstor.ha.probe.port, default 7999) and the connect time (cloudstack.linstor.ha.probe.timeout.ms, default 3000) and the least time a “no route to host” must take to count as silence (cloudstack.linstor.ha.probe.noroute.min.ms, default 1000) are system properties of the agent, like cloudstack.linstor.ha.settle.ms; add -Dname=value to JAVA_OPTS in /etc/default/cloudstack-agent on each host and restart the agent. Keep the last one below the time your network takes to give up resolving an address (about 3 seconds with the kernel’s defaults): above it, a dead host on the checking host’s own network, whose connection ends with that reply when the connect time is longer than the resolution takes, reads as “cannot tell”. If you changed the DRBD port range so that 7999 is outside it, or your firewall does not let nodes reach it, no node answers, and no host is read as dead.

What the platform makes of it was read from the platform’s source, with its default settings, and was not measured:

  • With Host HA enabled, the activity check runs every 60 seconds (kvm.ha.activity.check.interval) in rounds of 10 attempts (kvm.ha.activity.check.max.attempts) and fences only when at least 70 percent of a round read “no activity” (kvm.ha.activity.check.failure.ratio). Because the first reading only starts the clock, the first attempts of a round read alive, so the first round normally ends under 70 percent: the host is marked degraded for up to 5 minutes (kvm.ha.degraded.max.period), and the next round, in which every attempt reads dead, fences it. A host that really died is therefore fenced roughly one round plus the degraded period later than before, whatever its DRBD connection looked like, which is about 10 to 15 minutes with the defaults (less when the platform’s health checks, which ask every online neighbour, already started the clock). A check that takes longer than kvm.ha.activity.check.timeout (60 seconds) is counted as “no activity”, so keep that value above the longest check: about 35 seconds with the node list (up to 20), the probe of the node and two comparison nodes (up to 9) and, with a plain http or linstor address, the version probe of the LINSTOR client (up to 6).

  • The activity check is sent to the first online host of the cluster in the platform’s own order, which is kept from one check to the next, so it is the same agent while that host stays up. Each agent keeps its own clock.

  • Without Host HA, when a host stops answering the management server, the platform investigates once (the neighbours are asked, and any that reads the host alive wins) and does not investigate that host again until its agent reconnects and drops again. With this second signal the first investigation reads any host as alive, because the reading has not held a window yet, so the host goes to Disconnected and later Alert and its VMs are not restarted by the platform: you restart them (after making sure the host is really off). That is the price of not fencing a live host, and with this rule it applies to every host that goes silent, not only to one that shares no DRBD resource with its neighbour.

A host is never read as dead because your controller cannot be asked (refused, no answer in time, token refused, a failing gateway), because it lists no node (for example after it was reinstalled), because it does not know the node, or because the host has no name registered with it. DRBD alone is never enough either: a connection that is found but not Connected, a drbdsetup that fails, or output that cannot be read only mean that your controller has to be asked and the second signal has to hold. The name Karios uses for a host is the platform’s host name, which is also the node name Karios registers; names are compared without regard to case, and a host whose platform name differs from its node name is never read as dead (its HA does not act on it) and is named in the agent log.

A host also checks itself every minute, and the management server’s check of a host asks that host itself first, so a check of this host that the platform starts reaches the same code. A failed check is logged by the agent (Write heartbeat for pool ... failed), and the host restarts itself only when the agent setting reboot.host.and.alert.management.on.heartbeat.timeout is set to true. That setting is off by default in the 4.23.0.0 agent this page was written for; an agent of another version can default to on, so look at the value on your hosts. With the setting on, a failing drbdsetup never restarts the host, and a Primary resource with promotion score 0 restarts it only when your controller also shows this host OFFLINE while at least one other node is ONLINE (the conditions on the controller’s answer listed above apply; with every other node OFFLINE or in an error state the controller may see nobody, and the host is not restarted), and that reading has held one settle window (5 minutes): it was taken at least 5 minutes after the first one, and no reading in between showed this host healthy, drbdsetup failing, or your controller showing this host ONLINE or unsettled. This is the same held window as for another host; the wire part of the second signal does not apply, because a host cannot test itself over the wire. A host that reads itself dead also reaches the management server’s own check, whatever the setting says: the platform asks the host itself first, then its neighbours, and when no neighbour can answer (a cluster of one host, or every neighbour disconnected) the host’s own answer stands. That is read from the platform’s source and was not measured. The activity check that fences a host goes to its neighbours only, so the host’s own answer alone does not fence it. A host that has lost both its peers and your controller does not restart itself on DRBD evidence alone; the management server’s check of that host from the other hosts still applies.

The other side of these choices is that a host that really died is detected only once your controller answers, shows it offline and has been settled for 5 minutes, and a second reading at least 5 minutes after the first agrees and the wire is silent; the 5 minutes can be changed with the system property cloudstack.linstor.ha.settle.ms of the agent (set it as described above). LINSTOR’s controller retries a satellite connection about every 10 seconds, so the window is many retries long. Each agent remembers the window for itself and starts it when that agent sees your controller unsettled, so an agent that restarted together with your controller has no memory of it. A satellite that your controller shows stuck in a connecting state (UNKNOWN, NO_STLT_CONN, CONNECTED or AUTHENTICATED) keeps Karios from reading any host as dead until it is fixed or removed; the agent log names it.

Note

The host liveness answer is proven by unit tests and brute-force tests, and by one live observation on a three-host cluster (A lost host on a three-host cluster). The dead verdict, and with it the wire test, was not reached there. The tests cover every combination of controller state (answering, refusing, not answering, listing no node), node state (every connection status your controller can report, and not listed, for the node, the checking host and a third node), DRBD state (connected, not connected, no connection to the node, no resources, failing), the history of the offline reading, and the addresses and outcomes of the connection test (loopback, any-local and own addresses, names, IPv6, several networks, one or two comparison nodes), checked against a second implementation, and the probe over real sockets and the real network interfaces of the test machine (a refusal and an accepted connection). Silence and the fast and slow “no route to host” replies were measured separately on a test machine in a private network namespace (a router with a prohibit route, an unused address on the checking host’s own network, and a connection that joins an address resolution already failing), and in the unit tests they are simulated. The wire test has not been run on a cluster: on the three-host test cluster the first reading ended the check before the wire test. On the a test host a healthy diskless Primary with two up-to-date peers was measured to report promotion score 2, and the score stayed 2 during two minutes with the controller unreachable and through a restart of the controller service, so while the score is above 0 the host’s own check does not need the controller. Nothing was fenced or restarted in those runs (host HA was disabled there).

A lost host on a three-host cluster

The rule above was watched once on a live cluster: three Karios hosts running as virtual machines on one physical host, next to a LINSTOR cluster of two storage nodes and a tiebreaker. A host that ran an HA VM (root disk and data disk on the LINSTOR pool) was powered off through its virtual BMC. Host HA was disabled for it, because no out-of-band controller (OOBM) was reachable from the management nodes. Times are counted from the power-off:

After

What happened

5 seconds

Your controller shows the node OFFLINE, and DRBD on both storage nodes shows the connection to it as Connecting.

4 minutes 16 seconds

The platform’s one investigation asks a neighbour. The neighbour’s agent logs that it cannot tell whether the host is up, because the OFFLINE reading has not yet held a window over two readings, and so does not believe it. The answer is “alive”.

4 minutes 21 seconds

The host goes to Disconnected and then Alert.

4 minutes 21 seconds to 21 minutes

Nothing more. The platform runs no second investigation (asking for a reconnect did not start one), so the dead verdict, which needs a second reading at least 5 minutes after the first, was never reached. The VM stays Running on the dead host in the platform. Enabling Host HA reported success and the host’s HA state stayed Disabled; the likely cause is the missing OOBM, which was not confirmed.

20 minutes 50 seconds

An operator force-stops the VM (accepted in 2 seconds with the host dead) and starts it: it runs on another host after 6 seconds with both disks attached.

30 minutes 45 seconds

The host is powered on. Its satellite is online after 72 seconds, the host reads Up after 96 seconds, and its stale diskless entries for the disks are harmless.

What this shows:

  • The platform does not restart the VM by itself. Without Host HA and an OOBM that fences the host there is no automatic VM restart, and the LINSTOR liveness rule holds its verdict as designed: it will not call a host dead on a first reading. The VM stayed down for 21 minutes 16 seconds, until the operator acted. After making sure the host is really off, restart its VMs yourself.

  • Data was safe. After the restart all 20 checksummed files read back identical and the guest’s write ledger had no gap and no lost acknowledged write.

  • Not measured: the dead verdict (it needs Host HA with an OOBM that a management node can reach), and the time a real fence and restart would take. The platform marked the host Alert after 4 minutes 21 seconds here and after 12 minutes 31 seconds on a Ceph pool of the same hosts; the cause of the difference was not isolated, so do not read it as a LINSTOR speed-up.

One storage node or one replica is unavailable

With two or more copies and a tiebreaker (see Before You Start Checklist), volumes stay online. The test cut the replication traffic to one storage node for about 26 seconds while a guest wrote continuously: no write failed, and the volumes resynchronized about six seconds after the traffic returned.

A whole storage node was also lost on the three-host test cluster: one of the two storage nodes was destroyed while a guest wrote the ledger and a 300 MiB direct-I/O burst. Writes froze for 8.0 seconds, the time DRBD takes to notice, and then ran with no failed write on the remaining copy and the tiebreaker (two voters of three). When the node was started again both volumes went from Inconsistent to SyncTarget and were UpToDate about 66 seconds after the start (the resynchronization itself took about 33 seconds), with write gaps of at most 1.4 seconds meanwhile. DRBD’s online verify found nothing out of sync, and a direct read of both backing volumes gave identical checksums for the 2 GiB data volume and the 5 GiB root volume.

With a storage node’s satellite down for about 70 seconds, the pool health turns Degraded and names the node (see Read the Pool Health), and creating and attaching a disk was refused after 4.0 seconds with “Not enough available nodes”, because a place count of 2 had only one copy holder left. A failure with no tiebreaker was not tested. Without a tiebreaker and with two copies, a volume returns I/O errors when one copy is lost.

Until a resynchronization finishes, the node being resynchronized is not a usable copy. If the up-to-date node is lost in that time, the volume stops, and the blocks that had not been copied yet exist only on that node. Let a resynchronization finish before taking another storage node down.

Host agent restart

Restarting the host agent with VMs running keeps the VMs and their disks running. The host was back to Up in about 30 seconds in the test. If the pool checks the certificate, the agent re-imports the authority certificate at every start, so a rebuilt agent keystore does not leave the host untrusting.

Host reboot

The DRBD module and the satellite are set to load at boot. This was run on a one-node test VM with Karios’s DRBD layer applied (Debian 13, kernel 6.12.107, the LINSTOR controller and satellite on the same VM, an LVM thin pool, Secure Boot off). After a clean reboot the DRBD 9.3.4 module was loaded from Karios’s copy, the satellite was active and Online in the controller, the volume was UpToDate and its data matched its checksum. The step that enables the satellite persisted: the unit stayed enabled and a second call changed nothing.

After a hard reset with the volume Primary, a mounted ext4 file system and a writer running, the data was intact (checksums equal, journal replayed, fsck clean) and the module and the satellite came back, but the volume came back only about 10 minutes later, because the test pool needed a manual lvconvert --repair first. That is a property of the pool and not of DRBD or LINSTOR.

On the three-host test cluster, a host that was powered off and on again had its satellite Online after 72 seconds and read Up after 96 seconds (see A lost host on a three-host cluster); the module was not checked separately there.

Not tested: a reboot with Secure Boot on (the module is refused), a ZFS-backed pool, a host installed from the Karios image rather than by applying the layer, and a reboot of a host that has VMs running.

Live Migration

Live migration between two Karios hosts was run on the three-host test cluster with an HA VM whose root disk and data disk were both on the LINSTOR pool, a guest writer that acknowledged every durable write to the test host, and 20 checksummed files:

  • The migration took 14.5 seconds (10.5 seconds back).

  • DRBD, sampled every 0.7 seconds on both hosts, showed the destination attach both disks as Secondary, then both hosts ``Primary`` on both disks for about 2.9 seconds, then the source demote and release its attachment.

  • The largest write gap in the guest was 0.39 seconds and all 20 checksums matched.

The cluster ran as virtual machines on one physical host, so the timings bound the mechanism and are not customer figures. Moving a disk between pools while the VM runs is a different case and was not tested (see Supported Operations Compared With Ceph RBD).

Upgrading

Upgrade the hosts before the management servers. A management server that has the template size fix sends a new command when a template is made from a LINSTOR disk. A host that is not upgraded yet does not know that command, and the template creation fails with an unsupported command error until the host is upgraded. A new host under an old management server is harmless: it receives the plain command and does what it always did. Upgrade both within a few minutes and do not create a template from a disk in between. To go back, roll back the management servers first and then the hosts, so a new management server never meets an old host.

A template is reused from a LINSTOR pool only when the copy into the pool ran to the end. The management server marks the template resource as complete after the host reports that the copy succeeded. A template resource without that mark is not trusted: the next deploy deletes it and copies the template again, which takes the time of a first deploy (about 7 minutes for the 5 GiB test template on the nested test cluster). A clone for a new VM is only cut from a template resource that has the mark: an unmarked one is never cloned, because it can be a copy that is still being written. A template that was cached before this version has no mark, and the platform does not copy it again because it has recorded it as ready, so a deploy from it fails with a message that names the resource. If you know the resource holds a good copy, mark it on your cluster (linstor resource-definition set-property cs-<template uuid> Aux/_cs-template-complete true); otherwise delete the template from the pool in Karios and deploy again so that it is copied again. Do this after the upgrade for every template that was already in the pool. The wait for the cluster to remove such a resource is 30 seconds; change it with cloudstack.linstor.template.discard.wait.ms in JAVA_OPTS of /etc/default/cloudstack-management. Without the fix, a copy that was cut off by a host crash left a resource that the next deploy took for the template; the guest then started with a damaged root file system. A template that was taken over that way before the upgrade cannot be told apart from a good one: delete it from the pool and let it be copied again if you have had a host go down during a first deploy. While one pool is writing a template, no other pool of the same controller and resource group may touch the unfinished resource: a copy that finds it without the mark fails and names the other pool instead of deleting it. Karios no longer lets two pools share a resource group, so this only protects pools that were created that way before.

A rolling swap of the Karios core binary under load was run once on the three-host test cluster, agents first and the management node last (the same order as above). It was a core swap only: no DRBD, satellite or LINSTOR plugin component was upgraded in it, and the cluster had no LINSTOR pool or layer at that point (they had been rolled back before the run), so its five HA guests ran on the cluster’s own Ceph storage. An agent’s swap cost about 12 seconds of that agent’s core API. The management node’s swap was a 14 second outage of the management API (the engine’s own service never failed), with the database schema migration inside it (140 to 160 tables). Guest I/O saw no error in any step. It bounds the mechanism on a virtual test cluster; it says nothing about a LINSTOR pool under upgrade.

Limits

Limit

What to know

Template from a LINSTOR root disk

DRBD makes the device a few KiB larger than the volume, so a template cut straight from a LINSTOR root disk would be too big to deploy with the same disk size. Karios cuts the template back to the volume size, but only when it can prove nothing is lost: a GPT or MBR partition table whose partitions all end inside the target size, an ext2/3/4 or XFS filesystem that fits, or a blank disk. A boot signature over four empty partition entries is not a partition table: such a disk is judged like one with no table, so a filesystem that declares more than the cut leaves is kept. A GPT must also stay valid in the shorter image: its last usable sector has to lie inside the target size, because the guest’s kernel drops a table whose last usable sector is past the end of the disk and then sees no partitions. That holds when the pool pads the device by at most 33 sectors (16.5 KiB: one sector for the backup header and 32 for the backup entries of a table with the usual 128 entries; a table with another entry count moves the number); the 8 KiB of the ZFS-backed test pool is fine, while a pool that pads more keeps the device size for a partition grown to the end. The entries array of the primary table must also end before the table’s first usable sector, so that the cut keeps it. A disk is called blank only when its first MiB (sector 0 may hold the disk identifier and the signature of an empty partition table, nothing else), the last MiB before the cut and everything past the cut are all zero, so a whole-disk btrfs, reiserfs or ZFS (whose superblock or labels sit in the first MiB) is never cut as blank. The backup copy of the partition table lies past the cut and is left behind; the guest reads the primary table, and sgdisk -e or the grow step of cloud-init puts a backup back. For any other content, and for a partition or filesystem that runs to the very end of the device, the template keeps the device size and logs why. Deploy such a template with a root disk offering that is larger than the source disk (for example 6 GB for a 5 GB disk). The cloud image used in testing (a GPT disk whose root partition was grown to the end of the usable area, on a pool that pads 8 KiB) is cut correctly. A template made from a root snapshot is not affected. A disk download (extract) is still a few KiB larger than the volume. These sizes describe volumes that exist today; a volume created after exact disk size ships (see Disks show exactly the requested size) has no such difference, and the extract size of such a volume has not been checked.

Encryption

An encrypted disk offering needs an encryption master key in the Karios engine. Without one the create fails.

Secure Boot

The DRBD module Karios installs is not signed, so a host with Secure Boot on refuses it and the pool health reports the drbd module is not loaded. Keep Secure Boot off on every host that uses a LINSTOR pool.

Kernel and image changes

A kernel update, or an image switch to one that does not carry the DRBD layer, removes the module until the layer is applied again. Karios’s loader then refuses to load an older built-in DRBD 8 and the health reports the module as not loaded.

Live migration

Works between hosts (Live Migration). Moving a disk between pools while the VM runs is not tested.

Host failure

The platform does not restart the VMs of a lost host by itself without Host HA and an out-of-band controller (OOBM) that fences it; see A lost host on a three-host cluster.

Hypervisor

KVM only.

Deleted disks

A deleted disk is removed at once and cannot be recovered. The backup of a snapshot on secondary storage can restore it.

Snapshots

Need LVM_THIN, ZFS or ZFS_THIN backing pools. Each snapshot is copied to secondary storage and the primary copy is deleted, so secondary storage needs room. A snapshot that fails on a full secondary store can leave a snapshot behind on your cluster: remove it there.

Tested versions

One controller version (1.35.3), see Before You Start Checklist.

Certificate for a joining host

Recorded in Karios’s database as well as staged as a file on each management node, so any management node can give it to a joining host; the engine on a node that has no staged file still reads the file; see Hosts that join later.

Troubleshooting

Symptom

Likely cause

What to do

The pool is refused: “These hosts are not ready”

A host in scope is not online in your controller, its firewall blocks the controller, or it has a structural problem such as a name held with another host’s address.

Read the host and reason in the message, fix it, then click Retry. The token stays in the form.

The pool is refused: the controller “refused the API token”

Wrong, expired or read-only token.

Create a working token on the controller and send it again.

The pool is refused: no resource group named …

The group does not exist on that controller.

Create it, or use an existing group’s name.

The pool is refused: the pool … already uses the resource group …

Another Karios pool uses the same controller and resource group. One resource group backs one pool.

Create the new pool on another resource group, or use the existing pool.

The pool is refused: the certificate is not trusted

A private authority signed the controller certificate.

Follow Controller With a Private Certificate Authority.

Health is Down

Your controller cannot be reached, or every host has a problem.

Check port 3371 from the management servers and the hosts, then read the problem rows.

Health is Unknown

The controller refused Karios’s token, or no host in scope was listed.

Rotate token with a token the controller accepts. Check the pool’s scope.

A host row says the satellite is not running

The host was not prepared, or the satellite stopped.

Run the satellite route (see API Routes) for the pool.

A host row says the DRBD module is not loaded

Secure Boot is on, the kernel changed, or the DRBD layer is missing.

Turn Secure Boot off and apply the current Karios image layer, then Check again.

A host row says the controller has no node with that name, or lists it as not online

The node name differs from the host name, or the satellite cannot reach the controller.

Register the node under the exact host name, or run the satellite route.

A host row says the satellite and controller versions differ

Your controller runs another LINSTOR version than the host’s satellite.

Run the controller at the satellite version shown on the host.

DRBD port range is flagged

TcpPortAutoRange is outside 7600-7999.

Set it to a range inside 7600-7999.

Certificate verification reads “Checking is on, but not every host or management node trusts the certificate”

A host never received the authority certificate (it joined another way, or the certificate was missing or stale at join), its keystore was rebuilt, or a management node’s staged certificate is stale (the advice line names it).

Send the certificate again with the trust route. The agent restores it at its next start.

Deploying a VM from a template fails: root disk size smaller than the template size

The template was cut from a LINSTOR root disk and kept the device size.

Use a root disk offering larger than the source disk. See Limits.

Deploying a VM from a template fails and names a template resource that holds an unfinished copy and cannot be deleted

An earlier copy of the template into the pool did not finish (a host went down during it) and the cluster has not been able to remove what it left, for example because that resource is still open on a host.

Karios will not reuse a template that was not copied to the end. Wait for the cluster to remove the resource, or delete it on your cluster, then deploy again.

Deploying a VM from a template fails: there is no finished copy of template resource …

The template resource on the controller has no completion mark: it was cached before the mark existed, or its copy never finished.

Delete the template from the pool and deploy again, or, if the resource is known to be good, set Aux/_cs-template-complete to true on it. See Upgrading.

Copying a template fails: the template resource has no completion property and another pool … has not finished copying it

Two pools of one controller and resource group (created before Karios refused that) are copying the same template.

Wait for the other pool’s copy to finish and deploy again. If the other pool’s copy cannot finish, delete the template from that pool. Give each pool its own resource group.

An attach, a detach or a VM start takes minutes, then fails naming the controller

Your controller is unreachable, or accepts connections and does not answer, and the host is waiting for it.

Restore the controller. See Behavior During Failures for how long each step can wait.

An encrypted disk cannot be created

The Karios engine has no encryption master key.

Configure the master key, or use an unencrypted offering.

A snapshot failed and a snapshot is left on your cluster

Secondary storage was full.

Free space on secondary storage and delete the leftover snapshot on your cluster.

Revert is refused

The disk is attached to an instance that is not stopped.

Detach the disk or stop the instance, then revert.

API Routes

All routes are POST under /api/v2/storage/pools/linstor/ and need a root administrator (the root account, with the admin role or the CONTROL_CENTER_MANAGE permission). The pool health is the same: the LINSTOR cluster card is not shown to other users.

Route

Use

health

{"pool_id": ...}. The live check the card shows.

credentials

{"pool_id": ..., "api_token": ..., "ssl_insecure": ...}. Rotate the token and/or switch certificate checking. Send at least one of the two. ssl_insecure true switches the check off: from then on the API token is sent to an endpoint whose certificate nobody checked.

trust

{"pool_id": ..., "ca_pem": ...}. Stage the authority certificate.

satellite/enable

{"pool_id": ...}. Run the host preparation again for the pool’s scope, for instance after fixing a refused host.

firewall/sync

{"pool_id": ...}. Bring each host’s firewall rules in line.

A disk is made from a snapshot with POST /api/v2/storage/volumes/create-from-snapshot, {"name": ..., "snapshot_id": ..., "vm_id": ...} (vm_id optional). It works on any pool, LINSTOR included. The size and disk offering come from the snapshot. A tenant can use only its own domain’s snapshots and attach only to a VM it can see.

Done Criteria

  • The pool is Up and the LINSTOR cluster card reads Healthy.

  • Certificate verification reads Verified, or you have chosen to skip it for a lab.

  • One disk was created, attached to a VM, written to, detached and deleted on the pool.

  • You know your controller’s version, tiebreaker and port range, and who rotates the token.