Intro

AI capex 2026 is $401B, while average GPU utilization across 23k+ Kubernetes clusters sits near 5%. That gap is the ugly truth no one’s talking about.
In my recent blog vLLM DeepSeek deployment, we sharded a 643GB whale across 16 H100s because it was too big to fit on one node. Today we’re facing the opposite problem: one GPU, more than one model, in Kubernetes with no way to split it. Take two vLLM pods with small models that could both fit on one H100. Kubernetes won’t let them share: nvidia.com/gpu:1 is the smallest unit it understands, so each grabs a whole card and barely touches it. On an 8-GPU node, the ninth model sits Pending while eight cards run half empty.

So how do you put more than one model on that card without wrecking the others? That’s what we’ll show in this post.

You’ve Been Lied to About GPU Sharing

A CPU swaps threads thousands of times a second, so everyone gets a fair slice. A GPU can’t afford to: a running kernel’s state spans every SM (registers, shared memory, caches) and runs to tens of megabytes, so pausing it mid-run is virtually impossible. That’s why a GPU’s unit of work holds the card until it’s done, and everything else waits.

Kubernetes makes it worse. It caps CPU and memory with cgroups, but there is no GPU cgroup, so it hands out whole cards or nothing. And a vLLM process takes what it’s given: the HBM for weights and KV cache, all the compute, until it dies. One card, one tenant. Every tool that “shares” GPUs fakes it. Some enforce the split, some just hope you behave.

GPU Sharing: What People Try, Why It Fails

How a Pod Gets a GPU

Before you can share a GPU, here’s how a pod gets one. Three layers decide it, and only the top one is Kubernetes:

The isolation you actually lean on is NVIDIA_VISIBLE_DEVICES: NVIDIA’s toolkit hides every card except the one you’re assigned, which is why nvidia-smi in a healthy pod shows one GPU (i.e gpt-oss-20b on an H100 taking 75 of 81 GB):

root@vllm-gpu-stack-gpt-oss-20b:/vllm-workspace$ nvidia-smi     # output trimmed
+-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:5D:00.0 Off |                    0 |
| N/A   29C    P0            114W /  700W |   75079MiB /  81559MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+
|    0   N/A  N/A            272      C   VLLM::EngineCore                       75070MiB |
+-----------------------------------------------------------------------------------------+

root@vllm-gpu-stack-gpt-oss-20b:/vllm-workspace$ nvidia-smi     # output trimmed
+-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:5D:00.0 Off |                    0 |
| N/A   29C    P0            114W /  700W |   75079MiB /  81559MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+
|    0   N/A  N/A            272      C   VLLM::EngineCore                       75070MiB |
+-----------------------------------------------------------------------------------------+

⌛Time-Slicing Doesn’t Slice Time

Where the lie starts. In CPU land, time-slicing means the kernel forcibly pauses a process at fixed intervals so everyone gets a fair turn. GPU time-slicing does none of that. It just tells the device plugin to advertise one card as N GPUs so Kubernetes packs N pods onto it. One pod launches a five-second kernel, the other three wait five seconds. That’s all.

You don’t even need the device plugin for that. My vLLM Production Stack hack pulled the same stunt by hand:
drop the GPU request so K8s never locks the card, and let vLLM manage VRAM itself.

# vllm-production-stack Helm values
modelSpec:
  - name: "tinyllama-gpu"
    replicaCount: 2
    # requestGPU: 1     # REMOVED: bypasses the K8s device lock
    nodeSelectorTerms:
      - matchExpressions:
          - key: workload-type
            operator: "In"
            values: ["gpu"]

# vllm-production-stack Helm values
modelSpec:
  - name: "tinyllama-gpu"
    replicaCount: 2
    # requestGPU: 1     # REMOVED: bypasses the K8s device lock
    nodeSelectorTerms:
      - matchExpressions:
          - key: workload-type
            operator: "In"
            values: ["gpu"]

Two replicas, one GPU, zero isolation. It works right up until one replica’s context grows and the other one dies. That is the pain the next three tools were built to solve, each at a different layer of the stack (scheduling, software, physical).

The Field At A Glance

The gist of it: three levels of enforcement, orchestrate only, enforce in software, enforce in silicon. The choice is yours.

I. KAI Scheduler: The Same Lie With Lipstick

KAI, NVIDIA’s open-source Kubernetes scheduler for AI workloads, is no better than time-slicing. It schedules the sharing neatly, it’s the core scheduling engine of NVIDIA’s Run:ai, but at runtime nobody is refereeing. K8s only sees whole cards, so KAI fakes fractions: a reservation pod in the kai-resource-reservation namespace claims the card with a normal nvidia.com/gpu request, kube-scheduler sees the card as taken, and KAI decides which pods share it.

Look at the gpu-memory: 10GB annotation, it only decides where the pod lands. Nothing enforces it after. The pod hits the real driver, still sees all 80GB, and vLLM at 0.9 takes 72GB. The two neighbours that booked 10GB each get bumped with an OOM. No eviction, no throttling, nothing steps in.

Installing KAI is its own chart, not the GPU Operator:

helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  -n kai-scheduler --create-namespace \
  --set "global.gpuSharing=true"      # sharing is off by default

helm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \
  -n kai-scheduler --create-namespace \
  --set "global.gpuSharing=true"      # sharing is off by default

Routing a pod to KAI looks like this, and notice what is missing:

apiVersion: v1
kind: Pod
metadata:
  name: gemma-4-e4b
  labels:
    kai.scheduler/queue: default-queue   # which queue this bills against
  annotations:
    gpu-memory: "10000"         # the booking, in MiB
    # gpu-fraction: "0.5"       # or book a share of the card instead
spec:
  schedulerName: kai-scheduler  # send it to KAI, not kube-scheduler
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "google/gemma-4-E4B-it"]

apiVersion: v1
kind: Pod
metadata:
  name: gemma-4-e4b
  labels:
    kai.scheduler/queue: default-queue   # which queue this bills against
  annotations:
    gpu-memory: "10000"         # the booking, in MiB
    # gpu-fraction: "0.5"       # or book a share of the card instead
spec:
  schedulerName: kai-scheduler  # send it to KAI, not kube-scheduler
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "google/gemma-4-E4B-it"]

The only way to use resources.limits: nvidia.com/gpu is for a whole GPU. For a slice, metadata.annotations is required, because Kubernetes has no way to express a fraction. And the pod never schedules without the queue label.

Enforcement is optional and not native: a fence can be added via a HAMi-core, which is the subject of the next section.

II. HAMi: The Software Fence

HAMi is GPU virtualization done in software: if the silicon will not enforce the split, a library will. It is a CNCF Incubating project, and the only one here that is not NVIDIA-only: its device matrix spans AMD, Ascend NPUs, Cambricon MLUs and a handful of other accelerators. Where MIG needs data-center silicon, HAMi runs on any CUDA GPU you already own.

How HAMi Catches Every CUDA Call

HAMi ships a library, libvgpu.so, and preloads it into your container with LD_PRELOAD. Every CUDA call from your app passes through it on the way to the driver. Every allocation gets checked against the container’s limit. Under the limit, the call goes through. Beyond it, your app gets a CUDA OOM and the driver never hears about it.

The limits themselves are plain environment variables, which is the whole enforcement layer in two lines:

# what HAMi injects into your K8s container
CUDA_DEVICE_MEMORY_LIMIT_0=3000m    # hard memory ceiling, one var per GPU (_0, _1, ...)
CUDA_DEVICE_SM_LIMIT=50             # percentage of SMs, set once for the container

# what HAMi injects into your K8s container
CUDA_DEVICE_MEMORY_LIMIT_0=3000m    # hard memory ceiling, one var per GPU (_0, _1, ...)
CUDA_DEVICE_SM_LIMIT=50             # percentage of SMs, set once for the container

The number in your pod spec becomes that env var, and HAMi-core checks it on every cudaMalloc your app makes.

Slicing a GPU in Kubernetes with HAMi

No profiles, no carving, no drain. A slice is resource limits on the pod, any size you like:

apiVersion: v1
kind: Pod
metadata:
  name: qwen3-5-9b
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "Qwen/Qwen3.5-9B"]
      resources:
        limits:
          nvidia.com/gpu: 1                  # physical GPUs this pod needs
          nvidia.com/gpumem: 12000           # hard memory ceiling, in MiB (i.e 12G)
          nvidia.com/gpucores: 30            # 30% of the card's SMs
          # nvidia.com/gpumem-percentage: 50 # or ask by share instead of MiB

apiVersion: v1
kind: Pod
metadata:
  name: qwen3-5-9b
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "Qwen/Qwen3.5-9B"]
      resources:
        limits:
          nvidia.com/gpu: 1                  # physical GPUs this pod needs
          nvidia.com/gpumem: 12000           # hard memory ceiling, in MiB (i.e 12G)
          nvidia.com/gpucores: 30            # 30% of the card's SMs
          # nvidia.com/gpumem-percentage: 50 # or ask by share instead of MiB

These are real resources the scheduler counts, not annotations like KAI Scheduler. And 12G is a size MIG can’t cut, no fixed shapes, so gpt-oss-20b and Qwen3.5-9B can share one H100 with the headroom each needs. They’re extended resources, so unlike CPU there’s no overcommit: you can’t request 4GB and cap 12GB, request and limit are the same.

Each physical card advertises 10 vGPUs by default (deviceSplitCount), so one card shows up as 10 schedulable devices. Both halves of that are checkable:

# the node side: one physical H100, ten schedulable vGPUs
kubectl get node gpu-node-1 -o jsonpath='{.status.allocatable}' | grep -o 'nvidia.com/gpu":"[0-9]*'
nvidia.com/gpu":"10

# the container side: nvidia-smi reports the slice, not the card (output trimmed)
kubectl exec -it qwen3-5-9b -- nvidia-smi
[HAMi-core Msg]: Initializing ...
|   0  NVIDIA H100 80GB HBM3    |    64MiB / 12000MiB   |

# the node side: one physical H100, ten schedulable vGPUs
kubectl get node gpu-node-1 -o jsonpath='{.status.allocatable}' | grep -o 'nvidia.com/gpu":"[0-9]*'
nvidia.com/gpu":"10

# the container side: nvidia-smi reports the slice, not the card (output trimmed)
kubectl exec -it qwen3-5-9b -- nvidia-smi
[HAMi-core Msg]: Initializing ...
|   0  NVIDIA H100 80GB HBM3    |    64MiB / 12000MiB   |

The Fence Costs a Little Latency

HAMi isolation used to cost ~10% in latency. It got much better lately, v2.9 numbers on an A100 confirm the drop trend:

What About AMD?

HAMi carries an AMD backend (amd.com/gpu), sized today around the MI300X. Scheduling and allocation work, with LD_AUDIT in place of LD_PRELOAD: memory and CU limits will ride a bitmask enforced by the ROCm runtime and the KFD driver. See below:

apiVersion: v1
kind: Pod
metadata:
  name: amd-gpu-pod
spec:
  containers:
    - name: rocm
      image: ubuntu:22.04
      command: ["sleep", "infinity"]
      resources:
        limits:
          amd.com/gpu: 1            # scheduled and allocated today
          amd.com/gpu-memory: 3000  # in MB, and note the name: gpu-memory, not gpumem
                                    # not enforced yet, scheduling only

apiVersion: v1
kind: Pod
metadata:
  name: amd-gpu-pod
spec:
  containers:
    - name: rocm
      image: ubuntu:22.04
      command: ["sleep", "infinity"]
      resources:
        limits:
          amd.com/gpu: 1            # scheduled and allocated today
          amd.com/gpu-memory: 3000  # in MB, and note the name: gpu-memory, not gpumem
                                    # not enforced yet, scheduling only

HAMi-core is now supported by NVIDIA’s KAI Scheduler to enforce memory: read the announcement.

III. NVIDIA MIG: The Silicon Wall

Multi-Instance GPU (MIG) is NVIDIA’s answer in hardware. It carves the GPU into as many as 7 hardware partitions (drawn from 7 compute slices and 8 memory chunks), each with its own compute, memory, bandwidth, and fault domain. Partitioning happens in silicon, so a crash in one slice cannot reach another, and each slice shows up as an independent CUDA device. Strongest isolation on the list, the only one that survives a hostile tenant, and the only one that strands silicon you already paid for.

MIG runs on NVIDIA data-center chips only (A/H100, …, B200, GB200, etc), not the 4090/5090 or the L40S/L4 (AMD equivalent is MxGPU). Beware that enabling MIG wants an idle card, so changing topology means draining the whole node.

MIG Profiles for Dummies

MIG profile names look like serial numbers, but let’s simplify it. An H100 keeps two separate budgets. Compute is cut into 7 slices, memory into 8 chunks of 10GB. A profile name is just the receipt, 3g.40gb means 3 of the 7 compute slices plus 40 of the 80GB. Two withdrawals, one from each budget, and whichever budget runs dry first decides how many instances fit on the card.

Same rule, real models. B200 tells the same story with bigger memory units, which is why its slices read 23, 45, 90:

Notice in the table: on the H100, two small models each take their own 1g slice of one card, while on the B200 Gemma rides a single 3g.90gb MIG, 3 compute slices + 90GB covering weights and KV cache. And carve carefully: two 3g.40gb slices use six of seven slots, roughly 14% of an H100 idle (about $718/mo of paid silicon you can’t schedule).

MIG Carving in Kubernetes

Under the hood a slice is two objects: a GPU Instance (the memory) and a Compute Instance (its compute slots). Let’s see how the carving works in Kubernetes:

# flip MIG on (Hopper+: driver toggle, no reset)
nvidia-smi -i 0 -mig 1              # -i 0: target GPU 0 | -mig 1: enable MIG

# carve two half-card instances on GPU 0, compute included (-C)
nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C

# GPU indexes count up from 0: carve the next card, with its own layout
nvidia-smi mig -i 1 -cgi 4g.40gb,2g.20gb,1g.10gb -C

# flip MIG on (Hopper+: driver toggle, no reset)
nvidia-smi -i 0 -mig 1              # -i 0: target GPU 0 | -mig 1: enable MIG

# carve two half-card instances on GPU 0, compute included (-C)
nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C

# GPU indexes count up from 0: carve the next card, with its own layout
nvidia-smi mig -i 1 -cgi 4g.40gb,2g.20gb,1g.10gb -C

Slices then surface as their own resource names, and a pod claims one like any other resource:

apiVersion: v1
kind: Pod
metadata:
  name: gpt-oss-20b
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "openai/gpt-oss-20b"]
      resources:
        limits:
          nvidia.com/mig-2g.20gb: 1   # one hardware-isolated slice

apiVersion: v1
kind: Pod
metadata:
  name: gpt-oss-20b
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "openai/gpt-oss-20b"]
      resources:
        limits:
          nvidia.com/mig-2g.20gb: 1   # one hardware-isolated slice

Note the pod asks for a shape, not an address: if a node has six 2g.20gb slices, there is no resource syntax to say “give me the 2g.20gb on GPU 3″ (inside the pod, nvidia-smi tells you which one you got). With mig.strategy: mixed, a custom config can carve different GPUs on the same node differently (heterogeneous shapes), and you can still route pods to a specific node’s MIG config through a nodeSelector.

Automating It: GPU Operator’s MIG Manager

You don’t have to nvidia-smi every node. The GPU Operator ships a MIG Manager that reconciles each node to a declared layout which lives in a ConfigMap: a node label picks which one applies, and MIG Manager does the carving. All of it fits in Helm values and manifests, so the whole thing is GitOps-friendly:

# install the operator with profile-named resources and your own config
helm upgrade -i gpu-operator nvidia/gpu-operator -n gpu-operator \
  --set mig.strategy=mixed \
  --set migManager.config.name=custom-mig-config

# install the operator with profile-named resources and your own config
helm upgrade -i gpu-operator nvidia/gpu-operator -n gpu-operator \
  --set mig.strategy=mixed \
  --set migManager.config.name=custom-mig-config

Let’s configure our previous example declaratively:

# custom-mig-config: the layouts. This one carves our H100 example fleet
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-disabled:
        - devices: all          # applies to every GPU on the node
          mig-enabled: false

      h100-fleet:
        - devices: all          # or a list like [0,1] to target specific GPUs
          mig-enabled: true
          mig-devices:
            "1g.10gb": 5     # Gemma 4 E4B + Qwen3.5-9B take two, three spare
            "2g.20gb": 1     # gpt-oss-20b

# custom-mig-config: the layouts. This one carves our H100 example fleet
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-disabled:
        - devices: all          # applies to every GPU on the node
          mig-enabled: false

      h100-fleet:
        - devices: all          # or a list like [0,1] to target specific GPUs
          mig-enabled: true
          mig-devices:
            "1g.10gb": 5     # Gemma 4 E4B + Qwen3.5-9B take two, three spare
            "2g.20gb": 1     # gpt-oss-20b

# point a node at a layout: MIG Manager carves it for you
kubectl label node gpu-node-1 nvidia.com/mig.config=h100-fleet --overwrite

# the operator also ships built-in uniform layouts, named all-:
# all-3g.40gb = every GPU on the node carved identically into 3g.40gb slices (2 per H100)
kubectl label node gpu-node-2 nvidia.com/mig.config=all-3g.40gb --overwrite

# point a node at a layout: MIG Manager carves it for you
kubectl label node gpu-node-1 nvidia.com/mig.config=h100-fleet --overwrite

# the operator also ships built-in uniform layouts, named all-:
# all-3g.40gb = every GPU on the node carved identically into 3g.40gb slices (2 per H100)
kubectl label node gpu-node-2 nvidia.com/mig.config=all-3g.40gb --overwrite

The MIG Manager gotcha comes from Saiyam Pathak’s Kubesimplify deep-dive linked above, worth the full read.

💡 HAMi x MIG: Best of Both Worlds

Yes, it’s possible to have both. Flip a node into MIG mode and HAMi’s device plugin creates the MIG slices itself through NVML, no drain needed, so a single pool can hold software-fenced nodes and hardware-partitioned ones. The pod spec barely changes, and no MIG profile name is needed (HAMi picks the profile for you):

apiVersion: v1
kind: Pod
metadata:
  name: gpt-oss-20b
  annotations:
    nvidia.com/vgpu-mode: "mig"        # satisfy this with real MIG instances
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "openai/gpt-oss-20b"]
      resources:
        limits:
          nvidia.com/gpu: 2            # HAMi's resources, no profile name anywhere
          nvidia.com/gpumem: 8000      # at least 8GB per instance

apiVersion: v1
kind: Pod
metadata:
  name: gpt-oss-20b
  annotations:
    nvidia.com/vgpu-mode: "mig"        # satisfy this with real MIG instances
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:latest
      args: ["--model", "openai/gpt-oss-20b"]
      resources:
        limits:
          nvidia.com/gpu: 2            # HAMi's resources, no profile name anywhere
          nvidia.com/gpumem: 8000      # at least 8GB per instance

Head To Head

When To Pick Which

What’s Coming: Dynamic Resource Allocation (DRA)

Dynamic Resource Allocation hit GA in Kubernetes 1.34, and it is the native device API the field is converging on. KAI, HAMi, and MIG are not replaced by it, they move onto it: typed ResourceClaims instead of opaque nvidia.com/gpumem strings. Typed means the claims are fields both the scheduler and the vendor driver understand, memory and cores, instead of abstract integers:

# Today: opaque extended resource
resources:
  limits:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 4000        # just a number

# DRA: typed request, scheduler does the accounting
kind: ResourceClaim
spec:
  devices:
    requests:
      - name: gpu-0
        exactly:
          capacity:
            requests:
              cores: 30            # % of SMs
              memory: "4Gi"

Final Thoughts

In retrospect, this is one question: how much do you trust the workloads sharing the card? Trust them fully and KAI buys utilization for almost nothing. Trust them halfway and HAMi holds the line on any GPU you own. If you don’t trust them at all, only MIG’s silicon wall will do, stranded capacity included.

You will probably run more than one, each at its own layer (scheduling, software, silicon), so hoping this gave you enough breadth in GPU sharing to pick the one your workloads actually need.

What’s Next

Slicing a GPU is half the battle. The other half is scaling the pods on those slices, and KEDA does not do it the way it scales CPU. That’s our next article. Stay tuned!

Run AI Your Way — In Your Cloud

Want full control over your AI backend? The CloudThrill VLLM Private Inference POC is still open — but not forever.

📢 Secure your spot (only a few left), 𝗔𝗽𝗽𝗹𝘆 𝗻𝗼𝘄!

---

Run AI assistants, RAG, or internal models on an AI backend 𝗽𝗿𝗶𝘃𝗮𝘁𝗲𝗹𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗰𝗹𝗼𝘂𝗱 –
✅ No external APIs
✅ No vendor lock-in
✅ Total data control

Claim YOur FREE VLLM POC

𝗬𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮. 𝗬𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹𝘀. 𝗬𝗼𝘂𝗿 𝗿𝘂𝗹𝗲𝘀…

🙋🏻‍♀️If you like this content please subscribe to our blog newsletter ❤️.