Handing one node's GPUs to HAMi without touching the live models

The box has four Blackwell RTX PRO 6000s in it. Two are Server Edition at 600 W, two are Max-Q Workstation Edition at 300 W. Kubernetes advertised four identical nvidia.com/gpu and handed out whichever card it felt like.

For a single notebook that is fine. For tensor parallelism across two cards it is not. A mixed pair paces every collective at the slower card, so a TP=2 deployment that lands on one SE and one Max-Q quietly runs at Max-Q speed forever. That was the blocker on a research AI platform we run for a New Zealand university: three GPU nodes, two of them carrying eight H200s each and serving a live model to researchers, one RTX node for development and benchmarking.

This post is about the scheduling handover on that RTX node. It is not about GPU slicing, which we deliberately left switched off.

The fix we tried first, and why it failed

The obvious move is to make the two classes look different to Kubernetes. NVIDIA's device plugin takes a config file, and that config has a resources field that looks like it will let you advertise nvidia.com/gpu-se and nvidia.com/gpu-maxq off the same node.

It does not. We tested it against the operator in late August and the field was ignored outright. The plugin kept registering four generic GPUs and no amount of config coaxing changed that. So renaming was falsified, and we recorded the alternative as a commitment against the GPU operator package: wattage-class policy gets enforced by the scheduler, not by resource names.

That points at HAMi, which is Heterogeneous AI Computing Virtualization Middleware. It sits in front of the default scheduler as an extender, with its own mutating webhook and its own device plugin. HAMi was accepted as a CNCF Incubating project on 2 July 2026, having entered the Sandbox in August 2024, so it is no longer a curiosity you have to defend in a design review.

We went with chart v2.10.0, pinned the extender's kube-scheduler image to the cluster's own version, and put the scheduler on the management tier rather than a GPU node.

Class targeting by annotation, whole GPUs only

HAMi's class mechanism is a pod annotation, substring-matched against the card names that nvidia-smi -L reports:

metadata:
  annotations:
    nvidia.com/use-gputype: "Server Edition"

Before authoring anything I checked the strings on the node. One card family reports "Blackwell Server Edition", the other "Blackwell Max-Q Workstation Edition". Distinct substrings, no overlap, so the annotation is enough. UUID pinning would have worked too and would have broken the first time a card was replaced.

The second decision was to set deviceSplitCount: 1. Every physical GPU registers as exactly one nvidia.com/gpu, no memory or core slicing at all. HAMi's headline feature is sharing, and we turned it off.

That was on purpose. HAMi isolates VRAM in software through CUDA API interception, but it does not isolate compute the way MIG does. Sharing changes what a workload sees at runtime. Scheduling only changes where it lands. Doing both in one change means that when something misbehaves you cannot tell which half did it, so phase one keeps the blast radius on placement and sharing gets its own review later.

One node, three gates

The upstream constraint is blunt: HAMi's device plugin and the operator's cannot both run on a node, because both register nvidia.com/gpu. The original design disabled the operator's plugin cluster-wide and handed over all three nodes in one sitting. I threw that out. A cluster-wide values change to take over the nodes serving a live model, in order to fix a problem that only exists on a third node, is a bad trade.

The rewrite hands over one node with no GPU operator values change at all, behind three independent gates:

HAMi's plugin DaemonSet selects on gpu=on. No node carries that label when the app merges.
The operator's plugin leaves the RTX node on its own documented per-node skip label, nvidia.com/gpu.deploy.device-plugin=false. The ClusterPolicy is untouched, and preflight fails if anyone ever sets devicePlugin.enabled=false.
The webhook is opt-in by namespace, hami.io/webhook: enabled, rather than the chart's default of mutating everything.

That third one matters most. The chart default would have caught the inference namespace, and the alternative fix, a pod-level ignore label, is a pod template change, which means a Recreate roll of a whole-node model deployment. A public LLM outage to protect a node the change never goes near.

Honest accounting on the window: the webhook goes live the moment Argo syncs, before any node carries gpu=on. Between those two steps a new GPU pod in an opted-in namespace gets steered to a scheduler that knows about no devices and pends. So merge and label happen in one sitting, and you warn the workspace owners first.

The error that ate deploy day

Handover worked on the first attempt. The operator's plugin left the node, HAMi's registered four GPUs, capacity stayed conserved across the cluster, and both model deployments elsewhere stayed on the default scheduler. Verification came back 11 pass on platform and scope, and 3 fail on every GPU test pod, all dying at container create:

failed to inject CDI devices:
unresolvable CDI devices management.nvidia.com/gpu=GPU-<uuid>

Scheduling, bind and HAMi's own allocation were all correct. The failure was purely at OCI create. The operator's nvidia RuntimeClass handler runs the toolkit in CDI mode, and HAMi passes devices the old way, as NVIDIA_VISIBLE_DEVICES=GPU-<uuid>. CDI mode reads that as a CDI device name that no spec provides, so it refuses.

The fix is one values line. NVIDIA's operator also creates nvidia-cdi and nvidia-legacy runtime classes alongside the default nvidia one, and pinning runtimeClassName: nvidia-legacy puts device injection back on the path HAMi expects. The webhook then injects it into every GPU pod it schedules. Same sitting, 17 of 17 checks passing after the second run.

Nothing in the HAMi docs or the operator docs joins those two dots for you. If you are running the operator with CDI enabled, which is now the default, assume you will hit this.

What I would tell a peer before they start

Two things.

Adoption of a pre-existing allocation is not documented upstream, and I would not rely on it. A GPU pod that was placed by the old plugin holds a card that HAMi has no record of, which is how you get two workloads on one GPU. We wrote a step that re-creates those pods so their cards are allocated through HAMi, and verification proves every GPU pod on the node carries HAMi's allocation annotation afterwards. In the end the workspace owners stopped their sessions beforehand, so it was a no-op with zero restarts, which is the outcome you want.

Second, keep the workloads portable. Nothing we deploy sets schedulerName in a manifest. Pods carry annotations and the webhook does the steering. KAI Scheduler is coming in on the training and batch side once tenancy exists, and workloads with a hardcoded scheduler name would have to be rewritten to compose the two. It costs nothing now and saves a migration later, which is most of what platform work on a research build actually consists of.

Phase two is the H200s, and it is a scheduled serving window rather than a label flip, because re-creating those pods means dropping the model. I am in no hurry.

If you are staring at a mixed-class GPU node and wondering whether your scheduler can tell the cards apart, the answer is almost certainly no until you tell it how.