Why systemctl daemon-reload drops GPUs from running Kubernetes pods

A systemctl daemon-reload on a Kubernetes GPU node can detach the GPU from containers that are already running. The pod stays Running, nothing restarts, and no component reports a fault. The workload simply loses the ability to open the device. Almost every package that ships a systemd unit runs a reload on install, so agent rollouts, apt-daily-upgrade and snapd refreshes all trigger it.

I hit this with HAMi chart 2.10.0 alongside the NVIDIA GPU Operator. The mechanism applies to any device plugin configured the same way. This covers the symptoms, why systemd does it, and the two Helm values that stop it.

Everything looks healthy from outside

From the host and the control plane, nothing is wrong:

• Pods are Running with zero restarts.
• nvidia-smi on the node works, and the device plugin reports all GPUs healthy.
• No Xid errors in the kernel log, and no OOM kill (memory.events shows oom_kill 0).
• containerd and the kubelet have not restarted.

Inside the container you get Failed to initialize NVML: Unknown Error. With HAMi, HAMi-core also logs cuInit failed:100.

The distinguishing detail is that /dev/nvidia* is still present inside the container, but open() on /dev/nvidiactl fails with EPERM. You can confirm this from the host by entering the container's namespaces with nsenter. A device node that exists but cannot be opened points to the cgroup device controller, not the driver or the hardware.

Three conditions, one silent failure

All three need to hold:

• containerd runs with SystemdCgroup = true, the recommended setup for kubeadm clusters.
• The device plugin uses deviceListStrategy: envvar.
• The device plugin does not pass device specs (PASS_DEVICE_SPECS=false). In the HAMi chart this is the default, devicePlugin.passDeviceSpecsEnabled: false.

Nodes using the NVIDIA GPU Operator's device plugin with CDI (Container Device Interface) and PASS_DEVICE_SPECS=true are not affected. In a mixed cluster, the same reload will break pods on one node type and leave the other untouched, which is a useful diagnostic.

With the envvar strategy, the device plugin tells the container runtime which GPU to attach through the NVIDIA_VISIBLE_DEVICES environment variable. The NVIDIA runtime hook then injects the device nodes and modifies the container's cgroup device rules itself. The GPU never appears in the OCI spec, so runc never learns about it, and neither does systemd.

With SystemdCgroup = true, systemd owns each container's cgroup. On a reload, systemd rebuilds the device filter (a BPF program on cgroup v2) for every unit from the device list it holds. That list does not include the GPU, so the rules the hook added are dropped. The device nodes remain in the container's mount namespace, but the kernel now denies access to them.

NVIDIA has had a pinned notice on this family of failure since 2023, and documents it in the Container Toolkit troubleshooting guide. Both say the affected container has to be deleted and recreated.

Two Helm values, and the trap with a containerised driver

In the HAMi Helm values:

devicePlugin:
  passDeviceSpecsEnabled: true
  nvidiaDriverRoot: /run/nvidia/driver   # only with a containerised driver

passDeviceSpecsEnabled: true renders PASS_DEVICE_SPECS=true on the device plugin DaemonSet. The plugin then returns the device nodes and their permissions to the kubelet at allocation time. They land in the OCI spec, runc passes them to systemd as part of the unit's device policy, and systemd includes them every time it rebuilds the filter.

Render the chart with helm template before rolling and confirm the environment variable appears on the DaemonSet. Chart value names and the variables they produce do not always match across versions.

On nodes where the GPU Operator runs the driver in a container, enabling device specs on its own breaks every new GPU pod:

CreateContainerError: failed to generate spec: lstat /dev/nvidiactl: no such file or directory

With a containerised driver there is no /dev/nvidia* on the host. The device nodes live under /run/nvidia/driver/dev. The plugin defaults nvidiaDriverRoot and nvidiaDevRoot to /, so the device specs it passes point at host paths that do not exist.

Setting devicePlugin.nvidiaDriverRoot: /run/nvidia/driver makes the chart set NVIDIA_DRIVER_ROOT and mount the driver root at /driver-root. The plugin derives the device root from the driver root when NVIDIA_DEV_ROOT is unset. If the driver is installed on the host from packages, the device nodes are under /dev and this value is not needed. Check where the device nodes are before enabling device specs. Pods that are already running are not affected by the misconfiguration, only pods created while it is in place.

What rolling the plugin does and does not fix

Rolling the device plugin DaemonSet does not restart workloads. The plugin re-registers and running pods keep their devices. A container keeps the device state it was created with, so only pods created after the roll are protected. Pods that have already lost the GPU do not recover. They must be deleted and recreated. Pods created before the roll that still have a working GPU remain exposed until they are recreated.

That last point is the one that bites during a planned host package rollout. You can land the Helm change, watch the DaemonSet come up clean, and still have a canary window where half the GPU estate is safe and half is not. That is why a node without GPUs is useless as a canary for this class of failure.

How to prove it before the next apt run

Test it by triggering the failure on purpose:

1. Start a fresh GPU pod on the node after the device plugin has rolled.
2. Run sudo systemctl daemon-reload on the host.
3. Inside the pod, run nvidia-smi -L and confirm that open("/dev/nvidiactl", O_RDWR) succeeds.
4. Inspect the container's OCI spec and confirm it lists the NVIDIA devices.

A pod created before the roll should still fail step 3 after a reload, which confirms the test is exercising the right path. Node-level checks do not detect this failure. Node readiness, service timestamps, host nvidia-smi and device plugin health all pass while containers are broken. Only a probe that runs inside a GPU container catches it.

Before the next host-level package rollout, check the device plugin DaemonSet for PASS_DEVICE_SPECS or confirm the node is on CDI, check containerd for SystemdCgroup = true, find where /dev/nvidia* lives on the node, and run the reload test on one node per GPU node type. Include a GPU node in the canary group. Add an in-container GPU check to platform health monitoring.

If your cluster still uses the envvar strategy with device specs off, the next unattended package refresh is a reload away from a silent outage. Fix the values first. If you have hit the same failure on a different device plugin and solved it another way, I would like to hear how.