Kubernetes GPU Operator Deployment and Driver Lifecycle

The NVIDIA GPU Operator is the Kubernetes add-on that installs and reconciles the driver, container toolkit, device plugin, DCGM, and optional MIG manager so GPU nodes present a consistent device API to pods. It is an operations product. Treating it as a one-line Helm install is how clusters end up with mixed driver versions and surprise node drains.

Platform teams adopt it when they are tired of baking drivers into AMIs by hand and then discovering that a toolkit mismatch broke the runtime. They regret it when an automatic driver upgrade reboots a node that was holding a production replica.

This article covers what the operator owns, how to deploy it without fighting preinstalled drivers, and how to run driver lifecycle as a planned change.

What the Operator Owns

The operator is a set of cluster-scoped controllers plus DaemonSets. Typical pieces:

• Driver: kernel module matching the node OS and GPU SKU, unless you tell it to use a host driver.
• Container toolkit: the runtime hook that lets containerd or CRI-O expose GPUs to pods.
• Device plugin: the kubelet advertisement of nvidia.com/gpu and related resources.
• DCGM and exporter: telemetry for temperature, ECC, and utilization.
• MIG manager: optional partitioning on supported GPUs.

If any one of those is installed outside the operator, you must declare that fact. Two drivers or two device plugins on the same node is not a high-availability design. It is a race.

Deploy Against the Node You Actually Have

Read the operator version against the Kubernetes version and the container runtime. Toolkit installs that assume Docker on a containerd cluster waste a day. Validate with a small DaemonSet that requests one GPU and prints nvidia-smi before you move production charts.

On dedicated hosts, time-sync, IOMMU, and the base kernel matter as much as Helm values. The operator cannot compensate for a kernel that the vendor driver does not support. Put that check in the image pipeline, not in a Slack thread after the first CrashLoop.

Driver Lifecycle Is a Drain Plan

A driver change is a node reboot or an equivalent disruption for most enterprise stacks. Schedule it like a kernel upgrade. Cordon, drain, upgrade, wait for the device plugin to re-advertise, then uncordon. Keep a surge replica or a second pool so QPS does not fall on the floor during the drain.

Turn off unattended driver upgrades on serving pools. Development pools can be more aggressive. The same cluster can use two policies if you label the node groups. The failure mode is a single ClusterPolicy that treats a research box and a production replica as equals.

MIG changes are also disruptive. Recreating MIG profiles evicts work on that GPU. Treat profile edits as capacity changes, and document which workloads are allowed to land on MIG slices versus full GPUs.

Observability and Multi-Team Use

DCGM is how you see throttling, ECC, and XID before users report a hung job. Export it to the same system that already pages on disk and network. A GPU that is thermally limited looks like a slow model if you only watch application latency.

Scheduling still needs a policy above the device plugin. The plugin advertises GPUs. It does not stop one namespace from claiming the last eight H100s. The OnePlus Platform, OneSource Cloud's AI orchestration platform, sits on that cluster to assign quotas, workspaces, and job queues so Kubernetes GPU resources are not a first-come-first-served pile.

Keep training and serving in separate node groups when their drain windows differ. A weekly driver rollout that is fine for batch training is not fine for a latency SLO. Private AI infrastructure makes those groups real machines you can cordon. Shared cloud node pools can vanish or reshape underneath the operator if the SKU is withdrawn.

Failure Patterns Worth a Runbook

The common breaks are toolkit-runtime mismatch, a driver that does not match the GPU firmware, a device plugin that is Ready while nvidia-smi is not, and a leftover host install after you switched to the operator. The first debug step is "what owns the driver on this node," not another Helm upgrade.

Managed AI infrastructure should include the operator version, the last drain, and the owner of ClusterPolicy. If nobody owns those three facts, the next CVE will be applied by whoever is on call and least afraid of a reboot.

FAQ

Do we need the GPU Operator if we already install drivers in the node image?

You may still want it for the toolkit, device plugin, DCGM, and MIG manager. Tell it to use the host driver so it does not install a second one. Two driver sources on one node is the usual outage.

Will a GPU Operator upgrade interrupt inference?

It will if the upgrade changes the driver or restarts the toolkit on a node that still has pods. Plan a drain with surplus capacity, or pin serving nodes to a version and upgrade them in a window.

Can the GPU Operator configure MIG?

Yes, through the MIG manager on supported GPUs. Profile changes evict work on that device. Apply them as a scheduled capacity change, not as a live experiment on a serving node.

How do we run GPU Operator on a multi-team cluster?

Install one operator, then separate node groups and quotas. The operator makes devices visible. It does not implement fairness. Add namespace quotas or an orchestration layer so one team cannot consume the pool.

What should we watch after install?

Device plugin health, nvidia-smi on a canary pod, DCGM temperature and ECC, and the count of advertised GPUs versus physical GPUs. A mismatch is an incident even if pods still schedule on the remaining devices.

Summary

The GPU Operator is the lifecycle manager for drivers, toolkit, device plugin, and DCGM on Kubernetes. Deploy it against the node image you really have, disable surprise driver upgrades on serving pools, and treat every driver or MIG change as a drain. Pair advertised GPUs with quotas so multi-team clusters stay predictable.

If you want that operator lifecycle on dedicated U.S. GPU nodes instead of a shared node pool, OneSource Cloud can provide the hosts and the operations window. Request an architecture review to map ClusterPolicy to serving and training pools.