GPU Passthrough Stack

Proxmox PCIe → Talos → GPU Operator → Pod

Hardware Inventory physical

Node GPU VRAM Compute Architecture Driver
k8s-cp-1 none — — — —
k8s-worker-1 RTX 3090 24 GB 8.6 Ampere 580.178.04
k8s-worker-2 RTX 4070 Ti Super 16 GB 8.9 Ada Lovelace 580.178.04

Both GPUs are PCIe-passthrough from the Proxmox host (192.168.20.10) to their respective VMs. The host has no NVIDIA userspace tools — nvidia-smi runs from inside the K8s nodes or pods.

Full Stack how GPU reaches a pod

Layer 1 Proxmox — PCIe Passthrough

Each GPU is assigned to a VM via PCIe passthrough in the Proxmox VM config. The VM gets exclusive access to the physical device. IOMMU must be enabled in BIOS and Proxmox kernel args (intel_iommu=on or amd_iommu=on).

Proxmox host cannot use the GPU at all — no driver, no nvidia-smi. The device is fully owned by the VM.

↓
Layer 2 Talos Linux — NVIDIA Extensions

Talos ships two system extensions that handle the driver and container toolkit at the OS level, compiled into the Talos image:

extensions:
  - nonfree-kmod-nvidia-lts    # kernel module (580.178.04)
  - nvidia-container-toolkit-lts  # injects driver libs into containers

Because Talos is immutable, you cannot apt install the NVIDIA driver. The extensions are baked into the Talos machine image and loaded at boot. The toolkit extension provides the nvidia container runtime hook.

↓
Layer 3 GPU Operator — Cluster-level Management

The NVIDIA GPU Operator (v24.9.2) runs several components. On Talos, the driver and toolkit pods are disabled — the Talos extensions handle those. The operator still manages:

driver:        enabled: false   # Talos extension handles this
toolkit:       enabled: false   # Talos extension handles this
devicePlugin:  enabled: true    # advertises nvidia.com/gpu to kubelet
gfd:           enabled: true    # labels nodes with GPU metadata
dcgmExporter:  enabled: true    # Prometheus GPU metrics
validator:     enabled: true    # confirms GPU is functional

The device plugin is the critical piece: it tells kubelet that nvidia.com/gpu: 1 is an allocatable resource on the node. Without it, the scheduler doesn't know GPUs exist.

The GPU Feature Discovery (GFD) daemonset labels each node with detailed GPU metadata used for scheduling decisions:

nvidia.com/gpu.product=NVIDIA-GeForce-RTX-3090
nvidia.com/gpu.memory=24576
nvidia.com/gpu.compute.major=8
nvidia.com/gpu.family=ampere
↓
Layer 4 RuntimeClass — Driver Injection

A RuntimeClass named nvidia tells containerd to use the nvidia runtime hook when starting the pod. This hook injects the host's driver libraries (libcuda.so, libnvidia-ml.so, nvidia-smi) into the container at startup.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

Without runtimeClassName: nvidia, the container toolkit does not inject host driver libs. The container falls back to any bundled cuda-compat stub — which may be a different driver version, causing CUDA failure 35: driver version insufficient.

↓
Layer 5 Pod Spec — Requesting the GPU

A pod claims a GPU with three things in its spec:

spec:
  runtimeClassName: nvidia          # triggers driver injection
  containers:
    - image: ghcr.io/.../ml:v3.0.0-cuda  # CUDA-enabled image
      resources:
        limits:
          nvidia.com/gpu: 1         # claims 1 GPU from device plugin

The nvidia.com/gpu: 1 limit does double duty: it requests the GPU resource and acts as an implicit node selector — the pod can only land on a node where the device plugin advertises a free GPU.

GPUs are exclusive — each nvidia.com/gpu: 1 claim takes the entire device. Two pods requesting 1 GPU each need two physical GPUs (or two nodes with one GPU each).

This Cluster immich ML example

The Immich machine learning pod runs face detection, CLIP search, and OCR on whichever worker has a free GPU:

# gitops/values/immich-values.yaml
machine-learning:
  controllers:
    main:
      strategy: Recreate        # RWO PVC can't multi-attach
      pod:
        runtimeClassName: nvidia
      containers:
        main:
          image:
            tag: "v3.0.0-cuda"
          resources:
            limits:
              nvidia.com/gpu: 1

Why strategy: Recreate?

The ML pod uses a ReadWriteOnce PVC for its model cache. With the default RollingUpdate strategy, Kubernetes tries to start the new pod before killing the old one. The PVC can't attach to two nodes simultaneously, creating a deadlock. Recreate kills old, then starts new.

Targeting a specific GPU

If you need to pin a workload to a specific card (e.g. RTX 3090 for its 24 GB VRAM), use a nodeSelector with GFD labels:

spec:
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-GeForce-RTX-3090

Verification Commands debug

Check GPU resources per node
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
Check GPU metadata labels
kubectl get node k8s-worker-1 -o json | \
  jq '.metadata.labels | with_entries(select(.key | startswith("nvidia")))'
nvidia-smi from inside a GPU pod
kubectl -n immich exec deploy/immich-machine-learning -- nvidia-smi
Watch GPU utilization live
watch -n 1 'kubectl -n immich exec deploy/immich-machine-learning -- nvidia-smi'
Verify CUDA providers in ONNX Runtime
kubectl -n immich exec deploy/immich-machine-learning -- \
  python3 -c "import onnxruntime as ort; print(ort.get_available_providers())"
# Expect: ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
Check if driver libs were injected (RuntimeClass working)
kubectl -n immich exec deploy/immich-machine-learning -- \
  ls /usr/local/nvidia/lib64/libcuda.so* 2>/dev/null && echo "injected" || echo "missing"

Gotchas learned the hard way

trap CUDA driver version insufficient (failure 35)

If you see CUDA failure 35, the container is using a bundled cuda-compat stub instead of the host driver. Fix: add runtimeClassName: nvidia to the pod spec. The Talos container toolkit only injects host libs when this runtime is specified.

trap No nvidia-smi on Proxmox host

The Proxmox host has no NVIDIA userspace — GPUs are fully passed through to VMs. Run nvidia-smi from inside a GPU pod, not on the Proxmox shell.

trap RWO PVC deadlock on rolling update

A ReadWriteOnce volume can only attach to one node. If the new pod schedules to a different node than the old one, the volume can't attach and the rollout deadlocks. Use strategy: Recreate for any deployment with RWO PVCs.

trap GPU Operator driver/toolkit disabled on Talos

Talos uses system extensions for the NVIDIA driver and container toolkit. The GPU Operator's driver and toolkit fields must be enabled: false in the ClusterPolicy. The operator still manages the device plugin, GFD, and validators.