I/O Isolation

When multiple workloads co-located on the same Kubernetes node share the underlying block storage, a heavy workload in one of them (a large COPY, an aggressive VACUUM, a runaway query, or any I/O-intensive non-StackGres tenant) can saturate the device and degrade every other pod on that node. This is the noisy-neighbor problem at the storage layer.

Co-locating multiple StackGres clusters on the same node is itself an anti-pattern that StackGres recommends against. The protection described here is mostly relevant when a node is genuinely a shared resource — StackGres co-existing with other I/O-active workloads, multi-tenant nodes, or environments where node affinity cannot be fully controlled.

This runbook walks a Kubernetes cluster administrator through the steps needed to apply hard per-pod IOPS and bandwidth caps on the nodes that host StackGres clusters. It relies on three pieces, all standard:

  • The container runtime’s native BlockIO classes mechanism (CRI-O and containerd both support it).
  • A pod annotation that selects a class.
  • A standard Kubernetes extended resource advertised on the node, used by the kube-scheduler to prevent overcommitment.

No hostPath mounts, no privileged init containers in the cluster pods, no custom cgroup writes from inside the workload.

StackGres 1.19+ provides a complementary mechanism that configures per-pod I/O caps directly in the SGCluster spec (spec.pods.persistentVolume.ioLimits), with no node-level preparation. It is simpler to enable, but requires host-cgroup access from the cluster pods, which SCC policies in many OpenShift environments forbid. This runbook covers the alternative path: more upfront node preparation, but using only standard, supported configuration surfaces. Both approaches coexist; choose whichever fits the constraints of your environment. See the I/O isolation overview for a side-by-side comparison, and the storage configuration page for the ioLimits reference.

The runbook covers configuring CRI-O and containerd directly (applicable to any Kubernetes distribution), then documents the OpenShift-specific path that delivers the same configuration through the Machine Config Operator.

WARNING: the initial setup requires restarting the container runtime on each database node, because enabling the BlockIO configuration changes the runtime’s startup parameters. What that costs depends on the path:

  • CRI-O/containerd direct: a systemctl restart crio (or containerd) does not stop running containers: workloads keep running while the daemon restarts. Draining the node first remains the conservative recommendation, but is precautionary rather than strictly required.
  • OpenShift (MCO): for existing nodes the Machine Config Operator delivers the change as a drain + node reboot, one node at a time; new nodes provisioned with the role label set from the start get the configuration baked in via Ignition at first boot and require no restart.

Plan the rollout accordingly: on MCO a pool of n nodes needs at least n+1 capacity during the initial enablement, unless you are willing to absorb the downtime.

What you get

Three pieces, working together:

  1. Per-pod caps via cgroup io.max. The container runtime writes the cap into the pod’s cgroup at container creation, the kernel’s blk-throttle subsystem enforces it. Two pods on the same device, each capped at N IOPS, never affect each other below the saturation point.
  2. Drive characterization. fio measures the device’s sustained peak, a conservative fraction of that becomes the node’s safe ceiling.
  3. Scheduler-enforced no-overcommit. The safe ceiling is advertised as an extended resource (local-lvm-nvme/iops); each StackGres pod requests its share; the kube-scheduler refuses to place a pod that would push the node over capacity. The same shape of FailedScheduling event you already get for cpu or memory.

Pieces 1 and 3 are independent layers. You can adopt 1 first (noisy-neighbor protection only) and add 3 later (bounded SLA across the node).

Prerequisites

  • A Kubernetes cluster (any distribution) where the nodes use the unified cgroup v2 hierarchy. If your nodes are pinned to cgroup v1, the BlockIO classes mechanism described here does not apply.
  • cluster-admin permissions for the kubectl session.
  • Container runtime on the database nodes is either CRI-O 1.20+ or containerd 1.7+. Older runtimes either lack BlockIO support entirely or expose it through a different and unstable API surface.
  • For non-OpenShift clusters: a mechanism for delivering files to /etc/crio/ or /etc/containerd/ on each database node (SSH + scp, Ansible/Salt/Puppet, golden node image, or a privileged DaemonSet).
  • A representative spare block device to run fio against during characterization, or a maintenance window during which an existing device can be benchmarked safely.

Overview of components

Layer Component Purpose
Node Node taint Reserve the nodes for the capped workloads: the taint repels pods that don’t tolerate it; the SGClusters placed here carry a matching toleration. (On OpenShift a node label additionally drives MachineConfigPool membership.)
Node /etc/crio/blockio.yaml or /etc/containerd/blockio.yaml Defines the class ladder (named classes mapping to io.max values).
Runtime CRI-O blockio_config_file option / containerd CRI plugin BlockIO option Tells the runtime to load the class ladder.
Pod Annotation blockio.resources.beta.kubernetes.io/pod: <class> Selects the class for a given pod. The runtime resolves it at container creation and merges throttle parameters into the OCI runtime spec; runc writes them as io.max entries when it creates the pod’s cgroup.
Cluster Extended resource local-lvm-nvme/iops Node-level capacity advertised on status.capacity; pods request a share; the scheduler refuses overcommit.

1. Reserve the database nodes with a taint

Reserve the nodes that host the capped devices so that only the workloads you deliberately place there can land on them. A taint repels every pod that does not carry a matching toleration, which keeps uncapped, IOPS-stealing workloads off the nodes – the protection that actually makes the budget hold (see Reserving the nodes for capped workloads for why a taint, and not a nodeSelector, is the control that matters).

Naming the taint (and the class ladder, extended resource, and config files) after the StorageClass that provisions the capped devices – local-lvm-nvme in this runbook – keeps the scheme generic: it reflects the device being throttled, not any single workload that happens to use it.

kubectl taint node nvme-db-01 node-role.kubernetes.io/local-lvm-nvme=:NoSchedule
kubectl taint node nvme-db-02 node-role.kubernetes.io/local-lvm-nvme=:NoSchedule
kubectl taint node nvme-db-03 node-role.kubernetes.io/local-lvm-nvme=:NoSchedule

Verify:

kubectl describe node nvme-db-01 | grep -i taints

Expected output:

Taints:             node-role.kubernetes.io/local-lvm-nvme:NoSchedule

The SGClusters you place on these nodes carry the matching toleration (see step 6). You do not need a nodeSelector: a pod that requests the local-lvm-nvme/iops extended resource (step 5) is already restricted by the scheduler to the nodes that advertise that resource – which are exactly these nodes – so the request pins placement on its own. If you adopt only the BlockIO layer and skip step 5, the request is absent: add a nodeSelector or nodeAffinity to the pod spec to pin it to the prepared nodes instead.

OpenShift. Apply the taint with oc adm taint nodes …, or declaratively through the node’s spec.taints in the MachineSet so new nodes come up already tainted. OpenShift additionally needs a node label on these nodes to drive MachineConfigPool membership – that label is applied in step 4.3 and is used only for config delivery, not for scheduling.

2. Characterize the drive with fio

You need a single integer per node hardware profile: the safe IOPS ceiling you will advertise as the extended resource and use as the upper bound of your BlockIO class ladder. The procedure is run once per hardware profile, not per node –identical hardware yields identical numbers.

Critical: prefill the SSD before measuring. Empty-SSD performance is 30–50% higher than steady-state because the FTL has not yet allocated its over-provisioning. Skipping this yields numbers customers will hit the wall on after a few weeks of production load.

Deploy a temporary privileged Pod on the target node that has fio installed and /dev from the host mounted (raw block device access is required so the test bypasses any filesystem). The Pod targets the specific node via nodeName:

kubectl create ns fio
cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: fio-characterize
  namespace: fio
spec:
  nodeName: nvme-db-01
  restartPolicy: Never
  containers:
    - name: fio
      image: debian:bookworm-slim
      command:
        - sh
        - -c
        - "apt-get update -qq && apt-get install -y -qq fio && sleep infinity"
      securityContext:
        privileged: true
      volumeMounts:
        - name: dev
          mountPath: /dev
  volumes:
    - name: dev
      hostPath:
        path: /dev
EOF

On OpenShift, the privileged Pod requires the privileged SCC to be bound to the namespace’s default service account: oc adm policy add-scc-to-user privileged -z default -n fio. You will also have to label the fio namespace with:

kubectl label namespace fio \
    pod-security.kubernetes.io/enforce=privileged \
    pod-security.kubernetes.io/warn=privileged \
    pod-security.kubernetes.io/audit=privileged \
    --overwrite

Alternatively, run the fio commands directly on the node via oc debug node/nvme-db-01.

Wait for the Pod to be Running. --for=condition=Ready flips as soon as the container starts –not when the in-container apt-get install finishes– so we follow it with a poll for the fio binary actually being on PATH:

kubectl wait -n fio --for=condition=Ready pod/fio-characterize --timeout=120s

for i in $(seq 1 60); do
  kubectl exec -n fio fio-characterize -- which fio >/dev/null 2>&1 && break
  sleep 5
done
kubectl exec -n fio fio-characterize -- fio --version

Prefill (run once per drive –destroys the data on the device!):

kubectl exec -n fio -it fio-characterize -- \
  fio --name=prefill --filename=/dev/nvme0n1 \
      --ioengine=libaio --direct=1 --rw=write --bs=1M \
      --iodepth=16 --numjobs=1 \
      --size=100% --loops=2

Test 1 — random 4k read IOPS:

kubectl exec -n fio -it fio-characterize -- \
  fio --name=randread-4k --filename=/dev/nvme0n1 \
      --ioengine=libaio --direct=1 --rw=randread --bs=4k \
      --iodepth=64 --numjobs=4 \
      --time_based --runtime=300s --ramp_time=60s \
      --group_reporting

Test 2 — random 4k write IOPS:

kubectl exec -n fio -it fio-characterize -- \
  fio --name=randwrite-4k --filename=/dev/nvme0n1 \
      --ioengine=libaio --direct=1 --rw=randwrite --bs=4k \
      --iodepth=64 --numjobs=4 \
      --time_based --runtime=300s --ramp_time=60s \
      --group_reporting

Test 3 — sequential 1M read bandwidth:

kubectl exec -n fio -it fio-characterize -- \
  fio --name=read-1m --filename=/dev/nvme0n1 \
      --ioengine=libaio --direct=1 --rw=read --bs=1M \
      --iodepth=16 --numjobs=2 \
      --time_based --runtime=180s --ramp_time=30s \
      --group_reporting

Test 4 — sequential 1M write bandwidth:

kubectl exec -n fio -it fio-characterize -- \
  fio --name=write-1m --filename=/dev/nvme0n1 \
      --ioengine=libaio --direct=1 --rw=write --bs=1M \
      --iodepth=16 --numjobs=2 \
      --time_based --runtime=180s --ramp_time=30s \
      --group_reporting

Why 4k and not Postgres' 8k page size? The kernel’s blk-throttle subsystem counts BIOs (block I/O operations), not bytes –and a single Postgres 8k page operation is not necessarily one BIO (it can be split, merged, or aggregated with adjacent I/O depending on alignment, readahead, and the I/O scheduler). 4k is the standard block-layer atomic unit and matches the IOPS numbers vendors quote on NVMe spec sheets, making characterization comparable across hardware. The cap you derive is in BIOs/sec regardless of each BIO’s size, so 4k is the right unit. If you want a Postgres-relevant cross-check, repeat tests 1 and 2 at --bs=8k; you typically observe ~70-80% of the 4k IOPS number (each operation moves more bytes, fewer fit in the device queue), but the cap derived from 4k is the conservative reference.

Each test ends with a summary block. The relevant lines look like:

randread-4k: (groupid=0, jobs=4): err= 0: pid=...
  read: IOPS=212k, BW=826MiB/s (866MB/s)(242GiB/300004msec)
  ...

Record IOPS= from tests 1 and 2, BW= from tests 3 and 4.

Derive the safe ceilings:

local-lvm-nvme/iops          = floor(0.75 × min(test1_iops, test2_iops))
local-lvm-nvme/io-bandwidth  = floor(0.75 × min(test3_bw,   test4_bw))   # if used

The min() is conservative –it ensures the cap holds for the worst-case read/write mix. The 0.75 factor leaves headroom for system I/O, filesystem overhead, and run-to-run variance.

Clean up the characterization Pod:

kubectl delete ns fio

Carry the numbers above into step 3 (to pick class values that fit under the ceiling) and step 5 (to advertise the ceiling itself).

A note on fio vs Postgres-level tools. fio is the right tool for drive characterization –it bypasses Postgres, the page cache, and the filesystem to measure what the kernel and the device can actually sustain in BIOs/sec. That’s exactly the layer at which the cap is enforced. pgbench (or replay of your production traffic) has a different role: once a cap is set, use it to validate that the tier you picked fits your specific workload’s tps target. The kernel cap is in BIOs/sec; the resulting Postgres tps depends on your workload’s mix (read/write ratio, transaction size, WAL volume, cache hit ratio), and only an end-to-end Postgres benchmark can answer “is this cap tight enough that my application is unhappy?”. The two tools answer different questions at different stages of capacity planning.

3. Author the BlockIO class ladder

Write a blockio.yaml file that defines the named classes you want to offer. The format is shared between CRI-O and containerd. Class names and tier values are entirely up to you –StackGres imposes no specific naming convention. Stay below the safe ceiling derived in step 2.

Tier-design guidance. Three to five classes is the practical range. Fewer leaves too little flexibility for workloads of different intensity; more is over-engineering and creates choice fatigue when placing clusters. The values should span an order of magnitude or so (e.g. 1k / 5k / 20k / 50k IOPS), with the highest tier well below the node’s safe ceiling so multiple high-tier pods can coexist on the same node without exceeding the budget enforced by the extended resource in step 5.

Target the underlying physical disks, not the dm devices. When the PV stack uses LVM-based CSI (TopoLVM, LVMS, OpenEBS LVM-LocalPV), every PVC gets its own /dev/dm-N mapping that the Postgres process writes to. It is tempting to glob /dev/dm-* in the class so each LV is throttled at its own dm. Don’t. That path is fragile (see Limitations for why). Instead, glob the underlying physical block devices (/dev/sd[b-z], /dev/nvme[1-9]n[0-9]). The kernel propagates the originating cgroup tag through the device-mapper layer, so blk-throttle on the physical disk caps I/O submitted by the workload via its LV. This was verified end-to-end: with a class globbing /dev/sd[b-z] and a 5000 IOPS cap, fio inside a Postgres pod whose data PV is an LVM-LocalPV LV measured 5006 IOPS read / 5007 IOPS write (within 0.1% of the cap). The physical-disk approach is stable across PVC churn, restarts, and node lifecycle events; the per-dm approach is not.

Example ladder (adjust device paths and values for your environment):

# blockio.yaml
Classes:

  local-lvm-nvme-io-1k:
    - Devices: ["/dev/sd[b-z]", "/dev/nvme[1-9]n[0-9]"]
      ThrottleReadIOPS: 1k
      ThrottleWriteIOPS: 1k
      ThrottleReadBps: 50M
      ThrottleWriteBps: 50M

  local-lvm-nvme-io-5k:
    - Devices: ["/dev/sd[b-z]", "/dev/nvme[1-9]n[0-9]"]
      ThrottleReadIOPS: 5k
      ThrottleWriteIOPS: 5k
      ThrottleReadBps: 200M
      ThrottleWriteBps: 200M

  local-lvm-nvme-io-20k:
    - Devices: ["/dev/sd[b-z]", "/dev/nvme[1-9]n[0-9]"]
      ThrottleReadIOPS: 20k
      ThrottleWriteIOPS: 20k
      ThrottleReadBps: 800M
      ThrottleWriteBps: 800M

The globs above match whole SATA/SCSI disks and whole NVMe namespaces respectively. Adjust to whatever names the database nodes' kernels actually assign to the physical disks hosting the LVM VG. The glob is expanded by the runtime at startup against the real devices present on each node; entries that don’t match anything are silently dropped –but for physical disks this isn’t a problem because their device names are stable from boot.

Write the globs so they can only match whole disks, never partitions. A trailing * (e.g. /dev/sd[a-z]* or /dev/nvme[0-9]n[0-9]*) also matches partition nodes (/dev/sda1, /dev/nvme0n1p1, …). The runtime stores their major:minor in the class as-is, and the kernel rejects partition devices in io.max with ENODEV, at which point the runtime fails the container create, and every pod carrying a BlockIO annotation on that node is stuck in CreateContainerError with write 'rbps': No such device. This bites on any node whose matched disks are partitioned, which the OS disk always is. The single-character forms above ([b-z], n[0-9] with no trailing *) can only match whole disks. [b-z] rather than [a-z] also keeps the OS disk (sda) out of the class: throttling it would cap the pods' overlayfs and log I/O along with the database volumes. nvme[1-9]n[0-9] rather than nvme[0-9]n[0-9] does the same for NVMe: the OS disk is nvme0n1 on most cloud instance types (all AWS Nitro ones, for example), so starting the range at 1 leaves it uncapped. Confirm the numbering on your own hardware with lsblk -dno NAME,SIZE,TYPE before settling on the glob – if the OS disk is not nvme0n1 there, adjust the range accordingly.

Save this file locally; the next step places it on the database nodes via the path appropriate to your runtime/platform.

4. Install and enable the BlockIO configuration

This step has three alternative paths. Pick exactly one per node hardware profile, based on the container runtime in use and whether the cluster is managed by the OpenShift Machine Config Operator:

  • 4.1 –CRI-O nodes on any Kubernetes (manual configuration)
  • 4.2 –containerd nodes on any Kubernetes (manual configuration)
  • 4.3 –OpenShift (configuration delivered through MCO)

How you deliver the configuration files to each node in 4.1 / 4.2 is a node-management problem outside the scope of this runbook –typical choices are SSH + scp, a configuration-management tool (Ansible/Salt/Puppet), bakes into a golden node image, or a privileged DaemonSet that writes the files and restarts the runtime. The commands below are written as if executed directly on each node; adapt them to your delivery mechanism.

4.1 CRI-O direct configuration

Applies to clusters running CRI-O as the container runtime, where you have direct write access to /etc/crio/ on each node.

Step 1. Place the class ladder on each database node:

# on each db node:
install -m 0644 blockio.yaml /etc/crio/blockio.yaml

Step 2. Add a CRI-O drop-in configuration file that points the runtime at the class ladder:

# on each db node:
mkdir -p /etc/crio/crio.conf.d
cat > /etc/crio/crio.conf.d/99-local-lvm-nvme-blockio.conf << 'EOF'
[crio.runtime]
blockio_config_file = "/etc/crio/blockio.yaml"
blockio_reload = true
EOF

Why blockio_reload = true. With it, CRI-O re-reads blockio.yaml and re-expands its device globs against the devices present at that moment. That’s a plain systemctl reload crio (SIGHUP), with zero disruption to running containers. Day-2 class-ladder edits then need no drain and no restart: edit the file, systemctl reload crio, done. Note the semantics: containers created after the reload get the new values; existing containers keep the values they were created with until they are recreated (e.g. via an SGDbOps restart). This is the observed behavior on CRI-O 1.31.

Step 3. Restart CRI-O so the configuration takes effect (this initial restart is still needed. The drop-in itself is only read at startup). Draining the node first is the conservative choice, though a CRI-O daemon restart does not stop running containers:

# on the control plane:
kubectl drain nvme-db-01 --ignore-daemonsets --delete-emptydir-data

# on the node:
systemctl restart crio

# on the control plane:
kubectl uncordon nvme-db-01

Step 4. Verify the runtime picked up the configuration:

# on the node:
crio status config | grep blockio_config_file

Expected output:

blockio_config_file = "/etc/crio/blockio.yaml"

Inspect the CRI-O journal for the class load message:

journalctl -u crio --since "5 min ago" | grep -i blockio

Expected output: lines acknowledging the config file path and a final Blockio config successfully loaded message.

Check for device wildcard does not match warnings. If you see lines like:

[ blockio ] WARN: device wildcard "/dev/dm-[0-9]*" does not match any device nodes
[ blockio ] WARN: no matches on any of Devices: [/dev/dm-[0-9]*], parameters ignored

the class loaded with empty parameters for that wildcard, and any annotated pod that needs a matching device will get no throttle. This is the symptom of the one-shot expansion described in step 3 and discussed in Limitations and caveats. See that section for the recommended remediation.

Skip to step 5.

4.2 containerd direct configuration

Applies to clusters running containerd as the container runtime, where you have direct write access to /etc/containerd/ on each node.

⚠ Verify the exact configuration field on your containerd version. Containerd’s BlockIO support landed in the 1.7 series and the configuration plumbing has shifted between minor releases. Run containerd config dump on a target node and confirm the path of the BlockIO option in your installed version before applying the snippet below. The example uses the v2 plugin path for the CRI plugin; on some distributions or older builds you may need to use a cri_blockio_config_file key or place the file in a runtime-specific subsection.

Step 1. Place the class ladder on each database node:

# on each db node:
install -m 0644 blockio.yaml /etc/containerd/blockio.yaml

Step 2. Edit /etc/containerd/config.toml to point containerd at the class ladder. Add (or merge into the existing CRI plugin section):

version = 2

[plugins."io.containerd.grpc.v1.cri"]
  enable_blockio = true

[plugins."io.containerd.grpc.v1.cri".containerd]
  blockio_config_file = "/etc/containerd/blockio.yaml"

Step 3. Restart containerd. Drain the node first to avoid in-flight pod disruption:

# on the control plane:
kubectl drain nvme-db-01 --ignore-daemonsets --delete-emptydir-data

# on the node:
systemctl restart containerd

# on the control plane:
kubectl uncordon nvme-db-01

Step 4. Verify containerd loaded the configuration:

# on the node:
containerd config dump | grep -A1 blockio

Expected output (subset, exact key names depend on the version):

    enable_blockio = true
    blockio_config_file = "/etc/containerd/blockio.yaml"

Inspect the containerd journal for parse errors and for device wildcard ... does not match warnings –the same one-shot-glob-expansion caveat applies to containerd as to CRI-O. See Limitations and caveats:

journalctl -u containerd --since "5 min ago" | grep -iE 'blockio|wildcard'

Skip to step 5.

4.3 OpenShift: configure via Machine Config Operator

Applies to OpenShift clusters, where node configuration is managed declaratively through MachineConfigPool, MachineConfig, and ContainerRuntimeConfig. Direct edits under /etc/crio/ are not supported and would be reverted by MCO.

4.3.1 Create the MachineConfigPool

First label the nodes that will join the pool. Unlike the scheduling taint from step 1, this label is what MCO matches to select pool members – a taint cannot drive MachineConfigPool membership:

oc label node nvme-db-01 node-role.kubernetes.io/local-lvm-nvme=
oc label node nvme-db-02 node-role.kubernetes.io/local-lvm-nvme=
oc label node nvme-db-03 node-role.kubernetes.io/local-lvm-nvme=

A node can only be in one pool. A custom pool takes precedence over worker, so labelled nodes leave worker and join local-lvm-nvme. The other workers stay untouched.

cat << 'EOF' | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: local-lvm-nvme
  labels:
    pools.operator.machineconfiguration.openshift.io/local-lvm-nvme: ""
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values: [worker, local-lvm-nvme]
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/local-lvm-nvme: ""
  maxUnavailable: 1
EOF

Wait for the pool to reconcile. This triggers a rolling drain + reboot of existing nodes, one at a time, since they switch pools –make sure the pool has n+1 capacity if you cannot tolerate disruption:

oc get mcp local-lvm-nvme -w

Expected output once stable:

NAME            CONFIG                                           UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT
local-lvm-nvme    rendered-local-lvm-nvme-<hash>                     True      False      False      3              3                   3                     0

Tip: for new nodes provisioned with node-role.kubernetes.io/local-lvm-nvme= set from the start, the BlockIO configuration is baked in at first boot via Ignition. No drain, no rolling restart. This is the recommended provisioning path for greenfield deployments.

4.3.2 Deliver the BlockIO config via MachineConfig

Base64-encode the blockio.yaml authored in step 3 and embed it in a MachineConfig:

BLOCKIO_B64=$(base64 -w0 blockio.yaml)

cat << EOF | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 50-local-lvm-nvme-blockio-config
  labels:
    machineconfiguration.openshift.io/role: local-lvm-nvme
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/crio/blockio.yaml
          mode: 0644
          contents:
            source: data:text/plain;base64,${BLOCKIO_B64}
EOF

Verify the file landed on a node:

oc debug node/nvme-db-01 -- chroot /host cat /etc/crio/blockio.yaml

Expected output: the YAML you authored in step 3, verbatim.

4.3.3 Enable BlockIO in CRI-O

CRI-O has to be told to load /etc/crio/blockio.yaml. Ship a CRI-O drop-in configuration file via a second MachineConfig –it’s just another Ignition file, so MCO delivers it to every node in the pool the same way it delivered /etc/crio/blockio.yaml in step 4.3.2:

CRIO_CONF_B64=$(printf '[crio.runtime]\nblockio_config_file = "/etc/crio/blockio.yaml"\nblockio_reload = true\n' | base64 -w0)

cat << EOF | oc apply -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 51-local-lvm-nvme-blockio-crio-config
  labels:
    machineconfiguration.openshift.io/role: local-lvm-nvme
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/crio/crio.conf.d/99-local-lvm-nvme-blockio.conf
          mode: 0644
          contents:
            source: data:text/plain;base64,${CRIO_CONF_B64}
EOF

blockio_reload = true matters here too. It is the same option described in step 4.1, and it is not only a day-2 convenience on this path: CRI-O expands the class’s Devices: globs once, when it loads blockio.yaml, so a disk that is not present at that moment holds no throttle parameters. That is the normal case for anything but a disk soldered to the node: a cloud-attached data volume only becomes a block device on the node once a pod claiming it is scheduled there, which is necessarily later than the runtime’s startup. Without this option the class silently loads with empty parameters for that device and the annotation does nothing. With it, a plain systemctl reload crio (SIGHUP, no container disruption, no drain, no reboot) re-expands the globs against the devices present now:

oc debug node/nvme-db-01 -- chroot /host systemctl reload crio

Run it once after the first data volume is attached, and again whenever a node gains a disk the class should cover. It is also how a blockio.yaml edit is picked up without waiting for the MCO rollout, though MCO still has to deliver the edited file first.

MCO restarts CRI-O whenever its config files change, so applying this triggers a rolling drain + reboot of the MCP –one node at a time. Watch the rollout:

oc get mcp local-lvm-nvme -w

Wait until UPDATED=True and UPDATING=False again.

Verify CRI-O loaded the configuration on a node:

oc debug node/nvme-db-01 -- chroot /host crio status config | grep blockio

Expected output:

blockio_config_file = "/etc/crio/blockio.yaml"
blockio_reload = true

If the blockio drop-in was loaded recently, the CRI-O journal will also show the load message (this fires only on drop-in reloads, not on every restart –an empty result here doesn’t necessarily mean failure):

oc debug node/nvme-db-01 -- chroot /host journalctl -u crio --since "10 min ago" | grep -iE 'blockio|wildcard'

Look out for WARN: device wildcard ... does not match any device nodes entries –those mean the class loaded with empty parameters and the annotation will have no effect. See Limitations and caveats for the cause and the recommended workaround.

Continue to step 5.

5. Advertise IOPS capacity as an extended resource

This step is independent of the BlockIO class wiring above. Skip it if you only want noisy-neighbor protection without scheduler-enforced no-overcommit; come back to it later when you want the full bounded SLA.

Requesting this resource also constrains scheduling: the kube-scheduler only places a pod on nodes whose allocatable advertises enough local-lvm-nvme/iops. Because only the prepared nodes advertise it, the request doubles as the placement constraint – which is why the SGCluster in step 6 needs no nodeSelector, only a toleration for the step 1 taint.

Patch each node’s status capacity with the safe ceiling derived in step 2. The path uses JSON Pointer escaping –~1 is the literal escape for / and is easy to miss:

kubectl patch node nvme-db-01 \
  --subresource=status \
  --type=json \
  -p '[{"op": "add",
        "path": "/status/capacity/local-lvm-nvme~1iops",
        "value": "150000"}]'

Repeat per node, substituting the appropriate value if hardware profiles differ.

Verify. kubectl describe renders resource quantities in the friendly form (e.g. 150k for 150000); use kubectl get -o jsonpath if you need the exact integer:

kubectl describe node nvme-db-01 | sed -n '/^Capacity:/,/^System Info:/p' | grep local-lvm-nvme/iops

Expected output (two identical lines –Capacity and Allocatable):

  local-lvm-nvme/iops:  150k
  local-lvm-nvme/iops:  150k

For the exact integer:

kubectl get node nvme-db-01 -o jsonpath='{.status.capacity.local-lvm-nvme/iops}{"\n"}'

Expected output: 150000.

Persistence caveat. status.capacity is rewritten by the kubelet periodically. Kubernetes 1.20+ preserves unknown extended resources across status updates, but kubelet restarts, cluster upgrades, and node replacements are edge cases where the patched value can be lost. Treat the patch as part of node provisioning and document it in your runbooks; if a node loses the value, re-run the patch.

6. Validate end-to-end with an SGCluster

Create a test SGCluster that exercises the full path: the BlockIO class annotation is propagated to its pods, the local-lvm-nvme/iops resource request is set on the patroni container, and a custom fio sidecar mounts the same data PV so I/O performance can be measured against the actual storage path Postgres will use.

About the storage class. The example below uses local-lvm-nvme –a placeholder name. Substitute the StorageClass name your provisioner exposes (e.g. topolvm-default for TopoLVM, lvms-vg1 for LVMS, the name you gave to your OpenEBS LVM-LocalPV StorageClass). The choice of kind of provisioner is what matters, not the name: the BlockIO cap has to apply to the device the workload actually writes to.

  • Per-PVC LVM devices on top of a stable physical disk. LVM-based local CSI drivers expose each PVC as a /dev/dm-N mapping over a physical disk (the one that hosts the LVM VG). The cap from step 3’s class targets the physical disk; the kernel propagates the workload’s cgroup tag through the dm layer, so the throttle binds correctly regardless of which dm minor the PVC happens to land on. See Why target physical disks and not dm devices for the underlying explanation.
  • Local storage. I/O stays on the node, so a kernel-block-layer cap is meaningful.
  • Cloud block volumes that the node sees as a real block device. What decides whether the cap works is not whether the storage is local, but whether the I/O passes through a block device on the node that blk-throttle can act on. On instance types that expose their volumes over NVMe – AWS Nitro instances, where an EBS volume shows up as /dev/nvme1n1 and up – it does, and a class globbing those namespaces caps the volume exactly as it caps a local disk. Two things change compared to a local disk, both consequences of the volume being attached on demand:
    1. The device does not exist when CRI-O loads blockio.yaml, so the class needs blockio_reload = true and a systemctl reload crio after the volume is attached (see step 4.3.3). Without it the class holds no parameters for that device.
    2. Each PVC is its own device rather than an LV over a shared disk, so the glob has to cover the namespaces the volumes land on, and a node that gains a namespace outside the already-expanded set needs another reload.
  • Storage the node reaches over the network only (Ceph RBD, NFS and other network filesystems, iSCSI-backed volumes) is the case where a per-device cgroup throttle is at the wrong layer: the bottleneck is the network, not a block device the kernel is throttling.

Equivalent alternatives that provide the same shape of local LVM-backed PV:

  • OpenEBS LVM-LocalPV — vanilla Kubernetes, same TopoLVM-style semantics.
  • Red Hat LVM Storage Operator (LVMS) –recommended on OpenShift. Available from OperatorHub, it is TopoLVM repackaged with proper SCC handling and an LVMCluster CR that drives VG setup. Installing the upstream TopoLVM Helm chart on OpenShift requires manual SCC bindings and (depending on chart version) has container-image packaging issues that LVMS avoids. The provisioner StorageClass LVMS creates is typically named lvms-<device-class> (for example lvms-vg1).
  • Any other CSI driver that creates one block device per PVC and lives on the node –Ondat, the Local Storage Operator paired with LocalVolume raw block PVs, etc.

Substitute the storageClass field below for whichever provisioner you use; the rest of this section is independent of the storage choice.

If you skipped step 5 (extended resource layer), remove the pods.resources block from the SGCluster spec below –otherwise the cluster pod will stay Pending because no node advertises the requested resource. Without that request the pod is no longer pinned to the prepared nodes either, so add a nodeSelector (or nodeAffinity) to the scheduling block alongside the toleration to keep it on the BlockIO-configured nodes.

The clusterPods key under spec.metadata.annotations is a StackGres convention. Its entries are propagated as annotations on each cluster pod, which is how the BlockIO class selection reaches the container runtime.

Adjust the storageClass and Postgres version to values that exist in your environment (the size-xs SGInstanceProfile is created inline below; substitute an existing profile if you prefer):

kubectl create ns io-isolation
cat << 'EOF' | kubectl apply -f -
apiVersion: stackgres.io/v1
kind: SGInstanceProfile
metadata:
  name: size-xs
  namespace: io-isolation
spec:
  cpu: "500m"
  memory: "1Gi"
---
apiVersion: stackgres.io/v1
kind: SGCluster
metadata:
  name: io-isolation-test
  namespace: io-isolation
spec:
  instances: 1
  postgres:
    version: '16'
  sgInstanceProfile: size-xs
  metadata:
    annotations:
      clusterPods:
        blockio.resources.beta.kubernetes.io/pod: local-lvm-nvme-io-5k
  pods:
    persistentVolume:
      storageClass: local-lvm-nvme
      size: 10Gi
    scheduling:
      tolerations:
        - key: node-role.kubernetes.io/local-lvm-nvme
          operator: Exists
          effect: NoSchedule
    resources:
      containers:
        patroni:
          requests:
            local-lvm-nvme/iops: "5000"
          limits:
            local-lvm-nvme/iops: "5000"
    customContainers:
      - name: fio
        image: openeuler/fio
        command:
          - sh
          - -c
          - "sleep infinity"
        volumeMounts:
          - name: io-isolation-test-data
            mountPath: /var/lib/postgresql
EOF

Note: the custom container declared as fio appears in the pod as custom-fio –StackGres prepends custom- to custom container names.

Changing the class on a running cluster requires an explicit restart. And the pod annotation is not evidence the throttle is active. StackGres StatefulSets use updateStrategy: OnDelete: patching spec.metadata.annotations.clusterPods on an existing SGCluster updates the pod template and patches the annotation onto the live pod, but does not recreate the pod; the cluster is flagged PendingRestart=True (PodRequiresRestart) and left running. Since the container runtime resolves BlockIO classes only at container creation, the live pod then shows the new annotation while still running with the previous (or no) throttle. Apply the change with an SGDbOps restart or a controlled pod delete, then re-verify io.max (section 6.1): the cgroup file, not the annotation, is the source of truth.

Wait for the cluster pod to be running:

kubectl wait -n io-isolation --for=condition=Ready pod/io-isolation-test-0 --timeout=300s

If the pod stays Pending, inspect the events –Insufficient local-lvm-nvme/iops means more capacity needs to be advertised (step 5) or the request needs to be lowered.

6.1 Verify the cgroup io.max

Get the pod UID and the node it was scheduled onto:

POD_UID=$(kubectl get pod -n io-isolation io-isolation-test-0 -o jsonpath='{.metadata.uid}')
NODE=$(kubectl get pod -n io-isolation io-isolation-test-0 -o jsonpath='{.spec.nodeName}')
echo "POD_UID=${POD_UID} NODE=${NODE}"

Open a shell on ${NODE} (via SSH, kubectl debug node/${NODE}, or oc debug node/${NODE}) and locate the pod’s io.max files:

# on the node:
find /sys/fs/cgroup -name 'io.max' -path "*pod${POD_UID//-/_}*" 2>/dev/null

(With the systemd cgroup driver, the path embeds the pod UID with dashes replaced by underscores. With the cgroupfs driver, dashes are preserved –adjust the substitution if the search returns nothing.)

The find returns one cgroup per process structure in the pod. The path where the throttle entries actually land depends on the cgroup-driver and OCI-runtime combination on your nodes:

  • With cgroup_manager = "systemd" and crun as the runtime (the modern CRI-O default –used by OpenShift 4.x), the throttle is written by crun into an inner cgroup the runtime itself creates: kubepods-…-pod<UID>.slice/crio-<container-ID>.scope/container/io.max. The outer crio-<container-ID>.scope/io.max stays empty in this case.
  • With older configurations (cgroup_manager = "systemd" + runc, or cgroup_manager = "cgroupfs"), the throttle lands at kubepods-…-pod<UID>.slice/crio-<container-ID>.scope/io.max directly.
  • kubepods-…-pod<UID>.slice/io.max is the pod-level cgroup and is always empty in this setup –the throttle is per-container, not per-pod.
  • If monitor_cgroup = "pod" is configured (a non-default mode), you may additionally see crio-conmon-<container-ID>.scope/io.max siblings –CRI-O’s container-monitor sidecars. These are not workload containers and are expected to be empty. With the modern default monitor_cgroup = "system.slice", these scopes are not under the pod slice at all and you can ignore this category.

To find the file that actually holds the throttle entries regardless of which layout your node uses, look for the io.max files that contain a riops= line:

# on the node:
find /sys/fs/cgroup -name 'io.max' -path "*pod${POD_UID//-/_}*" 2>/dev/null \
  | xargs -r grep -l riops=

To map the <container-ID> values back to container names –so you know which scope corresponds to the patroni container– run:

kubectl -n io-isolation get pod io-isolation-test-0 -o json | \
  jq -r '.status.containerStatuses[] | "\(.name)\t\(.containerID | sub("cri-o://"; ""))"'

Expected output:

cluster-controller             3c6a5eb9c9c2b027eee9692c6a89d864b6c0c20b55c678546ec3062f82b1f51f
custom-fio                     35ae858a4d3a73f3a09a0a7419e9eeb85fd180a7ae7ea1755b7b38e011cc86c1
patroni                        0576fcfca34ff9c195d9b9715da5fbcbff150a6bfa99c38f1863b774d28ee47a
pgbouncer                      7e5d9e6e169b1923fe0964efd0098afb65c958ca7d8460458bc9354f4c66c684
postgres-util                  1f64dbb394e447959ca6b3eaec94bfceead578ce978326bdacbafd76c5da954a
prometheus-postgres-exporter   a2144dcf8a1e28109f37dd7055340e81c4254d5389cdb0b0951dd1486da4776f

Dump io.max for the patroni container’s scope, substituting the container ID from the mapping above. On modern CRI-O + crun the file lives one level deeper, under container/:

# on the node:
PATRONI_ID=<patroni container ID from the mapping above>
BASE=/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod${POD_UID//-/_}.slice/crio-${PATRONI_ID}.scope

# try the inner path first (crun creates an inner cgroup):
cat ${BASE}/container/io.max 2>/dev/null || cat ${BASE}/io.max

Expected output (exact device numbers depend on your storage stack; one line per device matched by the class’s Devices: glob):

259:0 rbps=209715200 wbps=209715200 riops=5000 wiops=5000

With the recommended physical-disk glob from step 3, major:minor corresponds to a physical disk on the node, not to the LV’s /dev/dm-N. That is deliberate: the kernel propagates the cgroup tag through the dm layer, so the throttle on the physical disk caps I/O the workload submits via its LV. Common majors:

Major Device type
7 loop devices (lab setups, e.g. file-backed LVM)
8 SCSI / SATA (/dev/sd*)
252, 253 device-mapper (only if the class globs /dev/dm-* directly)
259 NVMe (/dev/nvme*)

If the major:minor matches one of the physical disks underlying your LVM VG, the throttle is attached to the right device.

This confirms:

  • The annotation was recognized by the container runtime on the node.
  • The throttle parameters from the local-lvm-nvme-io-5k class were resolved.
  • The kernel wrote the matching io.max entries into the pod’s cgroup.

If the find … | xargs -r grep -l riops= above returned no files, or the file it returned contains max for every field, the annotation was silently ignored. The most common cause is the one-shot glob expansion described in step 3 –see Limitations and caveats for diagnosis and remediation. If that’s not it, see Troubleshooting below.

6.2 Load-test against the data PV

Run fio from inside the custom-fio custom container, against a file on the same data PV that Patroni uses. The cgroup applies at the pod level so the cap that throttles Postgres also throttles fio here –the measured IOPS therefore reflect exactly the cap Postgres will see at runtime.

Once the pod is Ready, the openeuler/fio image used in the custom-fio container has fio already on $PATH –no startup install step is needed.

Run a random 4k read test (1 GB file, 60 seconds, direct I/O bypasses the page cache):

kubectl exec -n io-isolation -it io-isolation-test-0 -c custom-fio -- \
  fio --name=cap-check-read \
      --filename=/var/lib/postgresql/fio-test.dat \
      --size=1G \
      --ioengine=libaio --direct=1 --rw=randread --bs=4k \
      --iodepth=64 --numjobs=4 --runtime=60s --time_based \
      --group_reporting

Expected output (relevant line):

  read: IOPS=5012, BW=19.6MiB/s (20.5MB/s)(1176MiB/60003msec)

The reported IOPS= should land within a few percent of the local-lvm-nvme-io-5k cap (5000), not the device’s native peak. If the number is much higher, the throttle is not applied –re-check section 6.1 and the troubleshooting section.

Optionally run the write side to confirm symmetric throttling:

kubectl exec -n io-isolation -it io-isolation-test-0 -c custom-fio -- \
  fio --name=cap-check-write \
      --filename=/var/lib/postgresql/fio-test.dat \
      --size=1G \
      --ioengine=libaio --direct=1 --rw=randwrite --bs=4k \
      --iodepth=64 --numjobs=4 --runtime=60s --time_based \
      --group_reporting

6.3 Clean up the test cluster

kubectl delete ns io-isolation

Operational considerations

What triggers a node reboot vs. what doesn’t

Once the runbook is applied, day-2 operations fall into two distinct buckets. Knowing which is which matters for change-management and capacity planning.

Change Node reboot? Impact
Edit blockio.yaml (add/remove/modify class definitions) Depends on the path CRI-O direct with blockio_reload = true (step 4.1): no restart at all. systemctl reload crio re-reads the ladder with zero container disruption; containers created afterwards get the new values, existing ones keep theirs until recreated. OpenShift/MCO: drain + reboot one node at a time; the MCP needs n+1 capacity to avoid downtime.
Add/remove blockio_config_file in the runtime config Yes Same as above –runtime config change.
Change which class an SGCluster’s pods use (annotation only) No No node touch, but the pods must be restarted explicitly (SGDbOps restart or a controlled pod delete): StackGres flags the cluster PendingRestart and does not restart it automatically, and until the restart the live pod shows the new annotation with the old throttle still in force (see the warning in step 6).
Add/remove the local-lvm-nvme/iops request/limit on a cluster No Pod-level change; no node touch.
Patch status.capacity["local-lvm-nvme/iops"] on a node No Online operation on the node object.
Scale a cluster up (new pod, new PVC) No With the recommended physical-disk classes (step 3) new PVCs are covered automatically. The throttle lives on the physical disk, not on the per-PVC dm device. Only the static-provisioning dm variant needs its pre-created LV slots. Where each PVC is its own attached device instead of an LV over a shared disk (cloud block volumes), a systemctl reload crio is needed once the new volume is attached, unless its device was already in the expanded set.
Provision a brand-new database node The BlockIO config is baked in via Ignition at first boot –no rolling restart, no drain. This is the recommended provisioning path.

The practical implication: most day-2 changes (tier reassignments, scaling, capacity adjustments) are online and pod-level. Reboots are confined to the relatively rare event of changing the class ladder itself –which typically only happens when adding a new tier or retuning values after a hardware change.

Behavior under contention from uncapped workloads

The cap is a hard upper bound but only a soft lower bound:

  • Hard upper bound: a capped pod never exceeds its configured IOPS/bandwidth limit, regardless of what else runs on the node. This is enforced in the kernel by blk-throttle and is independent of any other I/O activity.
  • Soft lower bound: a capped pod achieves its limit only when the underlying device is not saturated by other I/O. If uncapped pods –or workloads running outside the BlockIO class system entirely –saturate the drive, capped pods can be starved well below their cap. The cap reserves nothing; it only ceilings.

To turn the cap into a guaranteed share rather than just an upper bound, the sum of all concurrent I/O on the device must be ≤ the drive’s characterized capacity. The extended resource layer enforces this for pods that request local-lvm-nvme/iops. Pods that don’t request it (DaemonSets, system workloads, non-StackGres application pods) bypass the budget –the scheduler doesn’t know about their I/O.

Why you cannot enforce “must request the extended resource” at the node level

The budget enforced by the extended resource layer constrains only the pods that actually request local-lvm-nvme/iops. A pod that omits the request is invisible to the scheduler’s I/O accounting: it is placed on the node regardless of remaining budget, and – carrying no BlockIO annotation – runs with no cgroup io.max cap at all. One such pod (a COPY-heavy job, a backup tool, an unrelated tenant) can saturate the device and starve the capped pods well below their configured cap. This is the soft lower bound described above: the cap ceilings, it never reserves.

It is tempting to look for a node-level admission rule of the form “refuse any pod that does not request local-lvm-nvme/iops”. Kubernetes has no such rule. The closest primitive, a LimitRange with a defaultRequest (to inject the request) or a min (to require it), operates only at the namespace level – it applies to pods created in that one namespace and does nothing for pods created in other namespaces, for DaemonSets, or for any other workload that happens to land on the node. There is no built-in object that says “on these nodes, admit only pods that request this resource”. The request is therefore a convention you must enforce, not a constraint the platform can guarantee per node.

Reserving the nodes for capped workloads

Since the request cannot be forced, the practical control is to keep the workloads that would steal IOPS off the nodes entirely, using the taint applied in step 1. Any pod that does not carry the matching toleration – uncapped application pods, batch jobs, other tenants – is refused with the usual node(s) had untolerated taint event. The only remaining co-tenants are the system DaemonSets (CNI, kube-proxy, log and metrics agents) that tolerate all taints by design; these do negligible disk I/O and do not threaten the budget.

A nodeSelector is not the right tool here, and is not needed. A toleration only permits a pod onto a tainted node; it does not attract it there. Placement is handled instead by the local-lvm-nvme/iops request itself: the scheduler only fits a pod onto nodes whose allocatable advertises the requested resource, and only the prepared nodes do, so the request pins the pod to them with no selector. (A nodeSelector or nodeAffinity is only required in the BlockIO-only configuration that skips step 5, where no such request exists to do the pinning.)

The taint does not by itself force the pods that remain to request local-lvm-nvme/iops – a StackGres pod with the toleration but without the request would still bypass the budget. It removes the much larger risk: arbitrary, unmanaged workloads landing on the node and saturating the device. Combine it with the convention that every SGCluster placed here sets both the BlockIO annotation and the local-lvm-nvme/iops request. If non-StackGres workloads genuinely must coexist on these nodes, they have to carry BlockIO annotations and request local-lvm-nvme/iops on the same terms; otherwise the guarantee dissolves into “best effort against an unbounded set of contenders.”

Limitations and caveats

Why target physical disks and not dm devices

The container runtime (CRI-O or containerd) resolves each class’s Devices globs once, when it loads blockio.yaml at service startup. Each glob is matched against the block devices that exist on the node at that moment, and the result is stored as a fixed list of (major:minor → throttle) entries. Devices that don’t exist at startup are not in the class; devices that appear later are not retroactively added.

Mitigation on CRI-O: with blockio_reload = true (step 4.1), systemctl reload crio re-reads blockio.yaml and re-expands the globs against the devices present at that moment, with no disruption to running containers, so a reload after new devices appear refreshes the classes without a restart. Containers created before the reload keep their original entries until recreated.

That property is benign for physical block devices (/dev/sda, /dev/nvme0n1, …) because their device names are stable from boot and don’t come or go in normal operation. It is hostile to device-mapper devices created on demand by LVM CSI drivers (/dev/dm-N), which appear when a PVC is bound and disappear when the corresponding LV is removed.

There are two failure modes if you glob /dev/dm-* against a dynamic LVM-CSI stack:

  1. Cold-start blind spot. On a fresh node where no PVCs have been provisioned yet, /dev/dm-* matches nothing. Every class loads with empty parameters. The very first PVC’s pod gets the annotation, the runtime resolves the class to an empty list, no throttle is applied, and nothing emits an event –it just doesn’t work. The CRI-O journal shows WARN: device wildcard "..." does not match any device nodes at startup; see step 4.1 step 4 for the explicit check.

  2. Stale-entry container-create failure. If /dev/dm-* matches at startup but the matching dm devices later disappear (e.g. the LV is removed or its activating Pod terminates), the class still contains their (major:minor) entries. The next container creation will try to write throttle parameters for those stale entries; the kernel returns ENODEV and crun fails the container create with write 'rbps': No such device. The Pod can get stuck in Init:CreateContainerError indefinitely. Verified end-to-end during runbook validation.

Both failure modes vanish if you glob the underlying physical disks (/dev/sd*, /dev/nvme*n*) as recommended in step 3. The kernel propagates the BIO’s cgroup tag through dm’s clone/split path, so a throttle on the physical device caps I/O the workload submitted via its LV. Each pod’s per-cgroup limit is enforced independently –multiple capped pods on the same node, sharing the same physical disk, each hold their own cap (validated: two pods, 5000 IOPS cap each, fio concurrently, both observed 5003 IOPS).

This is why step 3 recommends physical-disk globs only. If you have an unusual stack where the workload writes to a dm device that is itself the underlying device (no further block layer beneath), see Static provisioning as an alternative below for a dm-stable approach. The in-cluster approach available in StackGres 1.19+ also sidesteps the class mechanism entirely: it writes the cgroup io.max entries directly per pod and per device, with no class-load timing dependency.

conmon and cgroup-path variations

The exact cgroup path where the throttle ends up depends on CRI-O’s cgroup_manager and monitor_cgroup settings (see step 6.1 for the resolution recipe). The recommended diagnostic is to find all io.max files under the pod’s slice and grep -l riops= to locate the one with throttle entries, rather than to assume a fixed path.

Static provisioning as an alternative

The physical-disk-glob approach from step 3 is the recommended default for LVM-based CSI stacks. Static provisioning is an alternative worth considering if your team prefers strict capacity planning over dynamic expansion, or if you have a stack where physical-disk throttling isn’t appropriate (e.g. dm-multipath where the workload device is the multipath dm and the underlying paths are several sd devices that you don’t want to throttle individually).

Pre-create a fixed number of LVs at known names by hand (or by an Ansible / MachineConfig step at node-prep time):

# on each db node, at provisioning time:
for i in $(seq 1 16); do
  lvcreate -L 500G -n local-lvm-nvme-data-${i} myvg
done

Expose each LV as a static PersistentVolume (one per LV) with volumeMode: Block or volumeMode: Filesystem as appropriate, and a nodeAffinity that pins it to the node it lives on. StackGres PVCs then bind to these pre-created PVs (the matching is by capacity and StorageClass; use a dedicated StorageClass with no provisioner, i.e. provisioner: kubernetes.io/no-provisioner).

Each LV’s /dev/dm-N is fixed at LV creation time and doesn’t move unless the LV is removed. The BlockIO class can then safely glob /dev/dm-* and bind deterministically. No restart-on-growth concern, but you lose the dynamic-expansion property of CSI provisioning.

This is appropriate for environments where the workload’s storage layout is known up-front and stable, and where the cost of pre-sizing storage slots is acceptable. For most StackGres deployments with dynamic provisioning, the physical-disk approach in step 3 is simpler.

ZFS-backed PVs

ZFS does not correctly attribute buffered writes to the originating cgroup, so write throttling is ineffective on ZFS-backed PVs. Read throttling still works. This is a kernel/filesystem property, not a runtime configuration error. Use a non-ZFS filesystem under the PV if write throttling is required.

Troubleshooting

Annotation silently ignored –io.max shows no throttle. Most likely the class loaded but its Devices: glob matched nothing. Check the CRI-O journal for WARN: device wildcard ... does not match any device nodes (see Why target physical disks and not dm devices for the underlying explanation and the recommended remedy: glob the physical disks, not the dm devices).

Pod stuck in Init:CreateContainerError with write 'rbps': No such device. A Devices: entry in the class references a major:minor the kernel cannot throttle, and container creation fails for every annotated pod on the node. Two causes:

  1. The glob matched a partition node. A trailing * (e.g. /dev/sd[a-z]*) also matches /dev/sda1-style partitions, and io.max rejects partitions with ENODEV. Since the OS disk is always partitioned, this reproduces on essentially every node. Use whole-disk globs with no trailing * (see the callout in step 3), then restart the runtime.
  2. A matched device has since been removed (stale entry). This happens mainly when the glob covers dm devices (LV present at runtime startup, removed later); the physical-disk globs avoid it. See Why target physical disks and not dm devices.

Annotation has no effect after changing the class on a running cluster. The pods were not restarted: StackGres leaves the cluster PendingRestart on annotation changes and the runtime only resolves classes at container creation, even though the live pod already shows the new annotation. Restart via SGDbOps restart (see the warning in step 6) and re-verify io.max.

If that’s not it, the container runtime may not have loaded the BlockIO config file. Causes to check, in order:

  1. The configuration key is wrong for your runtime/version (see the verification notes in step 4.1, step 4.2, and step 4.3.3).
  2. The runtime was not restarted after the configuration change –for CRI-O direct: systemctl status crio; for containerd direct: systemctl status containerd; for OpenShift: oc get mcp local-lvm-nvme should show UPDATED=True.
  3. The class name in the pod annotation does not match any class defined in blockio.yaml. The runtime ignores unknown class names without emitting an event.
  4. The YAML is malformed and the runtime failed to parse it at startup — journalctl -u crio or journalctl -u containerd on the node will show the parse error.

Pod stays Pending with Insufficient local-lvm-nvme/iops. The scheduler refuses to overcommit. Either reduce the pod’s request, free up capacity by deleting/draining another pod, or expand the node’s advertised capacity (only if your characterization supports it).

kubectl describe node no longer shows local-lvm-nvme/iops. The kubelet dropped the value during a status refresh. Re-run the patch from step 5.

For ZFS-backed PVs, see the ZFS caveat in Limitations.

Rollback

The rollback procedure mirrors the configuration path you took in step 4. Take the corresponding subsection below.

Rollback for CRI-O direct configuration (step 4.1)

On each database node:

rm /etc/crio/crio.conf.d/99-local-lvm-nvme-blockio.conf
rm /etc/crio/blockio.yaml
systemctl restart crio

Drain each node beforehand to avoid disruption to running pods.

Rollback for containerd direct configuration (step 4.2)

On each database node, remove the enable_blockio and blockio_config_file entries from /etc/containerd/config.toml, then:

rm /etc/containerd/blockio.yaml
systemctl restart containerd

Drain each node beforehand to avoid disruption to running pods.

Rollback for OpenShift (step 4.3)

Delete the two MachineConfigs created in step 4.3.2 and step 4.3.3. MCO rolls the deletion out the same way it rolled out the creation (drain + reboot, one node at a time):

oc delete machineconfig 51-local-lvm-nvme-blockio-crio-config
oc delete machineconfig 50-local-lvm-nvme-blockio-config

To dismantle the pool entirely and return nodes to the worker pool:

oc label node nvme-db-01 node-role.kubernetes.io/local-lvm-nvme-
oc label node nvme-db-02 node-role.kubernetes.io/local-lvm-nvme-
oc label node nvme-db-03 node-role.kubernetes.io/local-lvm-nvme-
oc delete mcp local-lvm-nvme

Rollback for the extended resource (step 5)

Independent of which configuration path you used:

kubectl patch node nvme-db-01 \
  --subresource=status \
  --type=json \
  -p '[{"op": "remove",
        "path": "/status/capacity/local-lvm-nvme~1iops"}]'

Rollback for the node taint (step 1)

Remove the taint from each node (note the trailing -); on OpenShift, if you tainted declaratively through the MachineSet, remove it there instead:

kubectl taint node nvme-db-01 node-role.kubernetes.io/local-lvm-nvme:NoSchedule-
kubectl taint node nvme-db-02 node-role.kubernetes.io/local-lvm-nvme:NoSchedule-
kubectl taint node nvme-db-03 node-role.kubernetes.io/local-lvm-nvme:NoSchedule-

Each rollback step is independent –for example, you can leave the BlockIO classes loaded and only remove the extended resource if you want to keep noisy-neighbor protection but stop enforcing scheduler-level no-overcommit.