Intro

A few months back I deployed the vLLM Production Stack on Nebius Managed K8s with Terraform. This time it’s NVIDIA Dynamo on the same foundation, one terraform apply from an empty cloud project to a live OpenAI-compatible endpoint. Dynamo is a very different beast from vllm-production-stack: a Helm chart installs an operator, and each model is a Kubernetes CRD the operator turns into a Frontend and workers. That’s what unlocks KV-cache-aware routing, disaggregated prefill/decode, and multi-model serving behind one endpoint.

Most Dynamo examples online assume you already have a cluster with the operator running; this one starts from scratch. By the end you get two models on one L40S, Let’s Encrypt TLS, and Grafana dashboards live, all measured on a real cluster, troubleshooting included.

💡You can find our code in the CloudThrill repo ➡️ nvidia-dynamo-stack-terraform.

📂 Project Structure

./nebius/
├── main.tf                           # MK8s cluster + CPU/GPU node groups
├── network.tf                        # VPC, subnets, IP pools (Cilium overlay)
├── provider.tf                       # Nebius + Helm + kubectl providers
├── variables.tf                      # all input variables (30+)
├── output.tf                         # HTTPS endpoint + stack summary
├── cluster-tools.tf                  # cert-manager, NGINX Ingress, kube-prometheus
├── data_sources.tf                   # ingress IP + Grafana lookups
├── dynamo-stack.tf                   # platform chart + DGD + PVCs + ingress + dashboards
├── env-vars.template                 # copy -> env-vars (export TF_VAR_*)
├── terraform.tfvars.template         # copy -> terraform.tfvars (holds dynamo_models)
├── assets                            # architecture diagrams (SVG)
├── config
│   ├── dynamo
│   │   ├── model-deployment.tpl      # PVC-per-model + DGD (Frontend + N workers)
│   │   └── frontend-ingress.tpl      # nginx + Let's Encrypt ingress -> Frontend svc
│   ├── helm
│   │   └── kube-prome-stack.yaml     # monitor-selector relaxation for Dynamo
│   ├── manifests
│   │   ├── letsencrypt-issuer.yaml   # Let's Encrypt ClusterIssuer (HTTP-01)
│   │   └── dynamo-operator-tls-cert.yaml   # real cert for operator-mode ingress
│   ├── kubeconfig.tpl                # local kubeconfig template
│   ├── grafana-dynamo-dashboard.yaml      # shipped serving dashboard (ConfigMap)
│   └── grafana-operator-dashboard.yaml    # shipped operator dashboard (ConfigMap)
└── README.md                              # <- you are here

./nebius/
├── main.tf                           # MK8s cluster + CPU/GPU node groups
├── network.tf                        # VPC, subnets, IP pools (Cilium overlay)
├── provider.tf                       # Nebius + Helm + kubectl providers
├── variables.tf                      # all input variables (30+)
├── output.tf                         # HTTPS endpoint + stack summary
├── cluster-tools.tf                  # cert-manager, NGINX Ingress, kube-prometheus
├── data_sources.tf                   # ingress IP + Grafana lookups
├── dynamo-stack.tf                   # platform chart + DGD + PVCs + ingress + dashboards
├── env-vars.template                 # copy -> env-vars (export TF_VAR_*)
├── terraform.tfvars.template         # copy -> terraform.tfvars (holds dynamo_models)
├── assets                            # architecture diagrams (SVG)
├── config
│   ├── dynamo
│   │   ├── model-deployment.tpl      # PVC-per-model + DGD (Frontend + N workers)
│   │   └── frontend-ingress.tpl      # nginx + Let's Encrypt ingress -> Frontend svc
│   ├── helm
│   │   └── kube-prome-stack.yaml     # monitor-selector relaxation for Dynamo
│   ├── manifests
│   │   ├── letsencrypt-issuer.yaml   # Let's Encrypt ClusterIssuer (HTTP-01)
│   │   └── dynamo-operator-tls-cert.yaml   # real cert for operator-mode ingress
│   ├── kubeconfig.tpl                # local kubeconfig template
│   ├── grafana-dynamo-dashboard.yaml      # shipped serving dashboard (ConfigMap)
│   └── grafana-operator-dashboard.yaml    # shipped operator dashboard (ConfigMap)
└── README.md                              # <- you are here

🧰Prerequisites

Before you begin, ensure you have the following:

Tool	Version tested	Notes
Terraform	≥ 1.5.7	tested on 1.5.7 (use `-json` on 1.5.x, see Troubleshooting)
nebius CLI	0.12.109	profile / federation auth
kubectl	≥ 1.33	within ±1 of the control plane (cluster runs 1.34)
helm	≥ 3.14	used by `helm_release`
kubernetes provider	2.38.0	core K8s resources
kubectl provider	1.19.0	DGD + ingress manifests
local provider	2.5.3	local-exec, files
nebius provider	0.5.124	Nebius AI Cloud resources

Follow the below steps to Install the tools (expend)👇🏼

# Install tools
sudo apt-get install jq
curl -sSL https://storage.eu-north1.nebius.cloud/cli/install.sh | bash
###### Auto completion
nebius completion bash > ~/.nebius/completion.bash.inc
echo 'if [ -f ~/.nebius/completion.bash.inc ]; then source ~/.nebius/completion.bash.inc; fi' >> ~/.bashrc
source ~/.bashrc

# Install tools
sudo apt-get install jq
curl -sSL https://storage.eu-north1.nebius.cloud/cli/install.sh | bash

###### Auto completion
nebius completion bash > ~/.nebius/completion.bash.inc
echo 'if [ -f ~/.nebius/completion.bash.inc ]; then source ~/.nebius/completion.bash.inc; fi' >> ~/.bashrc
source ~/.bashrc

Configure Nebius CLI profile

$ nebius profile create
profile name: my-profile
Set api endpoint: api.nebius.cloud
Set federation endpoint: auth.nebius.com
# Opens browser for authentication
✔ Profile "my-profile" configured and activated

$ nebius profile create
profile name: my-profile
Set api endpoint: api.nebius.cloud
Set federation endpoint: auth.nebius.com

# Opens browser for authentication
✔ Profile "my-profile" configured and activated

What’s in the stack?📦

This Terraform stack delivers a production-ready vLLM serving environment on Nebius AI Cloud supporting GPU inference with operational best practices embedded in Nebius Managed Kubernetes.

It’s designed for real-world production workloads with:
✅ GPU-first architecture: Purpose-built for AI/ML with L40S, H100, H200, and B200 GPUs
✅ Pre-baked GPU drivers: No manual driver installation or GPU operator needed
✅ VPC-Cilium networking: eBPF-based networking with Hubble observability
✅ Lightning-fast deployment: Complete stack in ~21 minutes
✅ Secure endpoints: HTTPS-only model serving with NGINX Ingress + Nebius Load Balancer + Let’s Encrypt

🏛 Dynamo Architecture Overview

The Helm chart installs the Dynamo operator. Every model you serve is a Kubernetes object the operator reconciles into a running graph, so you add models without re-running Helm.

Two layers, two planes

Platform (operator, CRDs, NATS) installs once into dynamo-system. Model is a DynamoGraphDeployment applied into dynamo; the operator turns it into a Frontend (the AI API gateway that routes requests to models) plus one worker per model. Two kinds of traffic. Requests reach the workers over TCP. What the components track about each other (who’s up, what’s cached, how busy) rides a separate event plane: ZMQ by default, or NATS, the optional shared message bus that KV-aware routing, disaggregation, and the Planner need.

Serving modes: Aggregated vs Disaggregated

Dynamo core architecture, aggregated vs disaggregated

Aggregated keeps prefill and decode in one engine on a single GPU: lowest cost, no cross-GPU transfer. Disaggregated splits them across GPUs and moves the KV over NIXL: lower TTFT, more GPUs, a Planner. Under both, the KV Block Manager tiers KV across GPU/CPU/SSD/remote.

Topology chosen for this build One aggregated worker per model on a shared L40S.

The ingress dilemma: Multi-model exposure

Option A vs Option B multi-model exposure

One ingress endpoint, two shapes:

A gives each model its own Frontend and routes by path.
B shares one Frontend and routes by the model field, like a vLLM router.

Ingress topology chosen: Option B The default here: every entry in dynamo_models becomes a worker behind one shared Frontend, one host, one cert. Option A stays a documented fallback, not wired as a variable.

Learn more about Dynamo

Planner internals, KVBM tiers, and the disaggregation path live in NVIDIA’s design docs.

Read: NVIDIA Dynamo, Overall Architecture

Deployment layers – The stack provisions infrastructure in logical layers that adapt based on your hardware choice:

Layer	Component	Time
Infrastructure	VPC + Subnet + Managed K8s (MK8S)	~4m 10s
GPU Nodes	Auto-scaling L40S / H100 / H200 / B200	~2m 01s
Add-ons	cert-manager, NGINX Ingress, kube-prometheus-stack	~11m 18s
Dynamo Stack	platform chart + DGD apply	~2m
Pod convergence	runtime image (9.8 GB) + model downloads	~18m
Total	End-to-end, both models serving	~35–40 min

Why ~40 minutes Pod convergence is the long pole, not Terraform. The bulk is the first pull of the 9.8 GB runtime image, NVIDIA’s own baked vLLM image that can’t be swapped for a stock OSS build, plus the model-weight downloads.

1. 🛜Networking Foundation

The stack creates a production-grade network topology:

Single /16 private IP pool (10.20.0.0/16) shared for nodes + pods
Additional /16 service-CIDR pool (10.96.0.0/16) carved from the same parent pool
One private subnet per AZ (derived from the pools), no public subnets, no NAT Gateway
Native VPC-Cilium CNI (overlay), VXLAN/Geneve encapsulation, eBPF datapath, Hubble observability
NGINX Ingress Controller exposed via Nebius Load Balancer

2. ☸️MK8S Cluster

Managed control plane on v1.34, with CPU and GPU managed node groups. GPU nodes ship pre-baked NVIDIA drivers, so no separate GPU operator.

Pool	Instance	Purpose
`cpu-pool`	cpu-d3 (8 vCPU / 32 GiB)	Core Kubernetes workload
`gpu-pool`	gpu-l40s-d (8 vCPU / 64 GiB + 1 × L40S)	GPU inference workload

3. 📦Essential Add-ons

Core Nebius MK8s add-ons can be installed from the catalog, and GPU drivers are already baked in the gpu nodes.

Category	Add-on
CNI	Nebius VPC-Cilium (overlay, eBPF, Hubble)
Storage	Compute-CSI (block, RWO)
Ingress / LB	NGINX Ingress Controller (Nebius LB)
Observability	kube-prometheus-stack (relaxed monitor selectors)
Security	cert-manager (Let’s Encrypt HTTP-01)
GPU	pre-baked NVIDIA drivers (`cuda13.0` preset, no GPU operator)

4. 🐉 NVIDIA Dynamo Stack

The heart of the deployment, production-ready multi-model serving:

✅ Models: Qwen3-8B + TinyLlama-1.1B on one L40S (default, fully customizable via dynamo_models)
✅ Two layers: dynamo-platform chart (the operator) + a DynamoGraphDeployment serving the models
✅ Routing: one shared Frontend (×2) routes by model name to N vLLM workers
✅ Secrets: Hugging Face token stored as a Kubernetes Secret
✅ Storage: one persistent model-cache PVC per worker at /opt/models
✅ Monitoring: Prometheus scrapes Dynamo’s PodMonitor/ServiceMonitor; shipped Grafana dashboards
✅ HTTPS endpoint: automatic TLS with Let’s Encrypt certificates
✅ Chart source: NVIDIA packaged .tgz over HTTPS (dynamo-platform 1.3.0)

🖥️ Nebius GPU Instance Types Available

Available GPU instances (T4 · L4 · V100 · A10G · A100)

Platform	GPU	vCPUs	RAM (GiB)	Region	Use-case
Frontier Training
`gpu-b200-sxm`	8 × B200 NVL72	160	1792	us-central1	Frontier training
Large-scale Training
`gpu-h200-sxm`	8 × H200 NVLink	128	1600	eu-n/w/us	Large-scale training
`gpu-h100-sxm`	1-8 × H100 NVLink	16-128	200-1600	eu-north1	High-perf training
Cost-effective Inference
`gpu-l40s-a`	1 × L40S PCIe (Intel)	8-40	32-160	eu-north1	Cost-effective inference
`gpu-l40s-d`	1 × L40S PCIe (AMD)	16-192	96-1152	eu-north1	Cost-effective inference

Note: Check the full list of Nebius GPU instance offerings in our last blog

Getting started

The deployment automatically provisions only the required infrastructure based on your hardware selection.

Phase	Component	Action	Condition
1. Infra	IP pool + subnet	Single private pool (`10.20.0.0/16`) + service-CIDR (`10.96.0.0/16`)	Always
	MK8S cluster	Managed control plane + CPU node group	Always
	GPU node group	Auto-scaling L40S / H100 / H200 / B200	Always
2. Add-ons	Ingress + TLS	NGINX controller + cert-manager (Let’s Encrypt)	Always
3. Observability	Prometheus + Grafana	kube-prometheus-stack + Dynamo & operator dashboards	Always
4. Dynamo Stack	HF token secret	Create `hf-token-secret` in the model namespace	`enable_dynamo = true`
	dynamo-platform	Operator + NATS + CRDs (+ PodMonitors) in `dynamo-system`	`enable_dynamo = true`
	DGD + PVCs	Shared Frontend (×2) + one worker & model-cache PVC per model	`enable_dynamo = true`
5. Exposure	HTTPS endpoint	`https://dynamo-api.<ip>.nip.io/v1` (hand-rolled \| operator \| none)	`enable_dynamo = true`

dynamo_ingress_mode — changes how the endpoint is fronted (operator | hand-rolled | none).
dynamo_gpu_exclusive — changes model placement (share one card, or one GPU/worker).

🔵 Deployment Steps

NVIDIA Dynamo on Nebius MK8s Deployment Overview

1️⃣Clone the repository

The NVIDIA Dynamo Nebius MK8s deployment build is located under nvidia-dynamo-stack-terraform/nebius directory:

Navigate to the production-stack-terraform directory and terraform Nebius tutorial folder

git clone https://github.com/CloudThrill/nvidia-dynamo-stack-terraform
cd nvidia-dynamo-stack-terraform/nebius/

git clone https://github.com/CloudThrill/nvidia-dynamo-stack-terraform
cd nvidia-dynamo-stack-terraform/nebius/

2️⃣ Set Up Environment Variables

Two ways to configure:

cp env-vars.template 
env-vars
vim env-vars  
# Set HF token and customize deployment options
source env-vars

cp env-vars.template 
env-vars
vim env-vars  
# Set HF token and customize deployment options
source env-vars

Usage examples

Option 1: environment variables (export TF_VAR_*)

# Copy and customize
cp env-vars.template env-vars
vi env-vars
################################################################################
# Nebius project credentials and region
################################################################################
export TF_VAR_neb_project_id=""                    # (required) your Nebius project ID
export TF_VAR_neb_profile="my_nebius_profile"      # (required) your Nebius CLI profile
################################################################################
# Cluster
################################################################################
export TF_VAR_cluster_name="vllm-neb-gpu"
export TF_VAR_k8s_version="1.34"
################################################################################
# 🐉 Dynamo inference configuration
################################################################################
export TF_VAR_enable_dynamo="true"
export TF_VAR_hf_token=""                          # only needed for gated weights
export TF_VAR_dynamo_deployment_name="vllm-agg"    # DGD name; Frontend svc = <name>-frontend
export TF_VAR_dynamo_ingress_mode="operator"       # operator | hand-rolled | none
export TF_VAR_dynamo_gpu_exclusive="false"         # false = share the card; true = one GPU/worker
export TF_VAR_dynamo_kv_routing="false"            # KV-cache-aware routing (needs >=2 replicas)
################################################################################
# ⚙️ GPU / node-group settings
################################################################################
export TF_VAR_gpu_node_min="0"
export TF_VAR_gpu_node_max="3"
export TF_VAR_gpu_platform="gpu-l40s-a"
# .snip
source env-vars

# Copy and customize
cp env-vars.template env-vars
vi env-vars
################################################################################
# Nebius project credentials and region
################################################################################
export TF_VAR_neb_project_id=""                    # (required) your Nebius project ID
export TF_VAR_neb_profile="my_nebius_profile"      # (required) your Nebius CLI profile
################################################################################
# Cluster
################################################################################
export TF_VAR_cluster_name="vllm-neb-gpu"
export TF_VAR_k8s_version="1.34"
################################################################################
# 🐉 Dynamo inference configuration
################################################################################
export TF_VAR_enable_dynamo="true"
export TF_VAR_hf_token=""                          # only needed for gated weights
export TF_VAR_dynamo_deployment_name="vllm-agg"    # DGD name; Frontend svc = <name>-frontend
export TF_VAR_dynamo_ingress_mode="operator"       # operator | hand-rolled | none
export TF_VAR_dynamo_gpu_exclusive="false"         # false = share the card; true = one GPU/worker
export TF_VAR_dynamo_kv_routing="false"            # KV-cache-aware routing (needs >=2 replicas)
################################################################################
# ⚙️ GPU / node-group settings
################################################################################
export TF_VAR_gpu_node_min="0"
export TF_VAR_gpu_node_max="3"
export TF_VAR_gpu_platform="gpu-l40s-a"
# .snip
source env-vars

Option 2: Terraform variables (required for the dynamo_models list) or use variables.tf

# Copy and customize
cp terraform.tfvars.template terraform.tfvars
vim terraform.tfvars

dynamo_models = [
  { name = "qwen3",     model = "Qwen/Qwen3-8B",                      gpus = 1, tp = 1, mem = 0.6  },
  { name = "tinyllama", model = "TinyLlama/TinyLlama-1.1B-Chat-v1.0", gpus = 1, tp = 1, mem = 0.15, replicas = 1 },
]

# Copy and customize
cp terraform.tfvars.template terraform.tfvars
vim terraform.tfvars

dynamo_models = [
  { name = "qwen3",     model = "Qwen/Qwen3-8B",                      gpus = 1, tp = 1, mem = 0.6  },
  { name = "tinyllama", model = "TinyLlama/TinyLlama-1.1B-Chat-v1.0", gpus = 1, tp = 1, mem = 0.15, replicas = 1 },
]

Why models go in tfvars dynamo_models is a list of objects, so it has no clean TF_VAR_ form and must live in terraform.tfvars. Everything else works as an export.

Load the Variables into Your Shell Before running Terraform, source the env-vars file:

$ source env-vars

$ source env-vars

3️⃣ Run Terraform deployment:

You can now safely run Terraform plan & apply. You will deploy the 100 resources in total, including local kubeconfig.

terraform init
terraform plan
terraform apply

terraform init
terraform plan
terraform apply

After apply, the stack prints a summary output with everything you need to reach it:

$ terraform output -raw dynamo_stack_summary
🐉 NEBIUS × NVIDIA DYNAMO STACK 🐉
-----------------------------------------------------------
PROJECT ID        : project-xxxxxxx
CLUSTER           : vllm-neb-gpu (mk8scluster-e00r0cv9asvmbv7xr1)
K8s ENDPOINT      : pending
VPC / SUBNET      : vllm-neb-gpu-network / vpcsubnet-e00hx71v88et9ssfpa
NETWORK CIDR      : 10.20.0.0/16, 10.96.0.0/16
LOADBALANCER IP   : 89.169.127.214
🖥️  NODEPOOL INFRASTRUCTURE
-----------------------------------------------------------
CPU POOL          : vllm-neb-gpu-cpu
└─ Platform       : cpu-d3 (8vcpu-32gb)
GPU POOL          : vllm-neb-gpu-gpu
└─ Platform       : gpu-l40s-d (1gpu-16vcpu-96gb)
└─ Scaling        : [1 Min, 2 Max]
🐉 DYNAMO SERVING
-----------------------------------------------------------
STATUS            : ENABLED
PLATFORM CHART    : dynamo-platform 1.3.0  (NATS on)
MODELS            : Qwen/Qwen3-8B, TinyLlama/TinyLlama-1.1B-Chat-v1.0
DGD / NAMESPACE   : vllm-agg / dynamo
FRONTEND          : 2 replicas (shared, routes by model name)
WORKERS / GPUs    : 2 workers · 1 GPU(s)
INGRESS MODE      : operator
KV ROUTING        : DISABLED
🌐 ACCESS ENDPOINTS
-----------------------------------------------------------
DYNAMO API        : https://vllm-agg.59a97fd6.nip.io/v1
GRAFANA DASHBOARD : https://grafana.59a97fd6.nip.io
🛠️  QUICK START
-----------------------------------------------------------
1. Kubeconfig     : nebius mk8s cluster get-credentials mk8scluster-e00r0cv9asvmbv7xr1 --external
2. List models    : curl -sk https://vllm-agg.59a97fd6.nip.io/v1/models | jq
3. Chat tests     :
   [qwen3]     curl -sk -X POST "https://vllm-agg.59a97fd6.nip.io/v1/chat/completions" \
               -H "Content-Type: application/json" \
               -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hi in one line."}]}'
   [tinyllama] curl -sk -X POST "https://vllm-agg.59a97fd6.nip.io/v1/chat/completions" \
               -H "Content-Type: application/json" \
               -d '{"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", "messages": [{"role": "user", "content": "Say hi in one line."}]}'
Built with ❤️ by @Cloudthrill

$ terraform output -raw dynamo_stack_summary
🐉 NEBIUS × NVIDIA DYNAMO STACK 🐉
-----------------------------------------------------------
PROJECT ID        : project-xxxxxxx
CLUSTER           : vllm-neb-gpu (mk8scluster-e00r0cv9asvmbv7xr1)
K8s ENDPOINT      : pending
VPC / SUBNET      : vllm-neb-gpu-network / vpcsubnet-e00hx71v88et9ssfpa
NETWORK CIDR      : 10.20.0.0/16, 10.96.0.0/16
LOADBALANCER IP   : 89.169.127.214
🖥️  NODEPOOL INFRASTRUCTURE
-----------------------------------------------------------
CPU POOL          : vllm-neb-gpu-cpu
└─ Platform       : cpu-d3 (8vcpu-32gb)
GPU POOL          : vllm-neb-gpu-gpu
└─ Platform       : gpu-l40s-d (1gpu-16vcpu-96gb)
└─ Scaling        : [1 Min, 2 Max]
🐉 DYNAMO SERVING
-----------------------------------------------------------
STATUS            : ENABLED
PLATFORM CHART    : dynamo-platform 1.3.0  (NATS on)
MODELS            : Qwen/Qwen3-8B, TinyLlama/TinyLlama-1.1B-Chat-v1.0
DGD / NAMESPACE   : vllm-agg / dynamo
FRONTEND          : 2 replicas (shared, routes by model name)
WORKERS / GPUs    : 2 workers · 1 GPU(s)
INGRESS MODE      : operator
KV ROUTING        : DISABLED
🌐 ACCESS ENDPOINTS
-----------------------------------------------------------
DYNAMO API        : https://vllm-agg.59a97fd6.nip.io/v1
GRAFANA DASHBOARD : https://grafana.59a97fd6.nip.io
🛠️  QUICK START
-----------------------------------------------------------
1. Kubeconfig     : nebius mk8s cluster get-credentials mk8scluster-e00r0cv9asvmbv7xr1 --external
2. List models    : curl -sk https://vllm-agg.59a97fd6.nip.io/v1/models | jq
3. Chat tests     :
   [qwen3]     curl -sk -X POST "https://vllm-agg.59a97fd6.nip.io/v1/chat/completions" \
               -H "Content-Type: application/json" \
               -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hi in one line."}]}'
   [tinyllama] curl -sk -X POST "https://vllm-agg.59a97fd6.nip.io/v1/chat/completions" \
               -H "Content-Type: application/json" \
               -d '{"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", "messages": [{"role": "user", "content": "Say hi in one line."}]}'
Built with ❤️ by @Cloudthrill

4️⃣ Observability (Grafana Login)

You can access Grafana dashboards using grafana_url output or port forwarding .(i.e http://localhost:3000)

# Get Grafana HTTPS URL (already printed by Terraform) i.e https://grafana.xxxxx.nip.io
terraform output -raw grafana_url 
# Or port forward
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n kube-prometheus-stack

# Get Grafana HTTPS URL (already printed by Terraform) i.e https://grafana.xxxxx.nip.io
terraform output -raw grafana_url 
# Or port forward
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n kube-prometheus-stack

Run the below command to fetch the password

kubectl get secret -n kube-prometheus-stack kube-prometheus-stack-grafana -o jsonpath={.data.admin-password} | base64 -d

kubectl get secret -n kube-prometheus-stack kube-prometheus-stack-grafana -o jsonpath={.data.admin-password} | base64 -d

Serving dashboard: per-model RPS, E2E latency, TTFT/ITL

Operator dashboard: reconciliation, webhooks, inventory

Username: admin
Password : through kubectl command above

Monitor-selector gotcha By default kube-prometheus only scrapes monitors carrying its own release label, so Dynamo’s own PodMonitor and ServiceMonitor are ignored and the dashboards render empty. The stack relaxes those selectors in kube-prome-stack.yaml, so both dashboards import and populate with no manual setup.

For Benchmarking vLLM Production Stack Performance check the multi-round QA tutorial

5️⃣ Destroying the Infrastructure 🚧

To delete everything just run the below (Note: sometimes you need to run it twice as the loadbalancer gets tough to die)

terraform destroy -auto-approve

Plan: 0 to add, 0 to change, 20 to destroy.

Changes to Outputs:
  - dynamo_api_url       = "https://vllm-agg.59a97fd6.nip.io/v1" -> null
  - dynamo_stack_summary = <<-EOT ... EOT -> null

terraform destroy -auto-approve

Plan: 0 to add, 0 to change, 20 to destroy.

Changes to Outputs:
  - dynamo_api_url       = "https://vllm-agg.59a97fd6.nip.io/v1" -> null
  - dynamo_stack_summary = <<-EOT ... EOT -> null

🛠️Configuration knobs

This stack provides extensive customization options to tailor your deployment:

Variable	Default	What it does
`neb_project_id`	— (required)	Nebius project ID for the deployment
`gpu_platform`	`gpu-l40s-a`	GPU instance type
`gpu_node_min` / `_max`	0 / 3	GPU node autoscaling bounds
`enable_dynamo`	`true`	Deploy the Dynamo stack
`hf_token`	«secret»	Hugging Face token, only for gated weights
`dynamo_models`	qwen3 + tinyllama	List of `{name, model, gpus, tp, mem, replicas}`, one worker per entry
`dynamo_frontend_replicas`	2	Shared Frontend replicas (HA)
`dynamo_gpu_exclusive`	`false`	`true` = one GPU per worker; `false` = share the card
`dynamo_ingress_mode`	`operator`	operator \| hand-rolled \| none
`dynamo_kv_routing`	`false`	KV-cache-aware routing (needs ≥2 replicas of a model)
`dynamo_deployment_name`	`vllm-agg`	DGD name; Frontend svc = `<name>-frontend`
`dynamo_release_version`	1.3.0	`dynamo-platform` chart version
`dynamo_model_cache_size`	50Gi	Model-cache PVC size per worker
`grafana_admin_password`	«secret»	Admin password for the observability stack
`letsencrypt_email`	— (none)	Email for Let’s Encrypt certificates

📓 This is a subset. The full 30+ options live in env-vars.template and terraform.tfvars.template.

🧪 Quick Test

1️⃣ In ingress mode the endpoint is already public, so grab it straight from the Terraform output:

export dynamo_api_url=$(terraform output -raw dynamo_api_url)

export dynamo_api_url=$(terraform output -raw dynamo_api_url)

2️⃣ List models: one endpoint, both models answering.

curl -sk ${dynamo_api_url}/models | jq '.data[].id'
"TinyLlama/TinyLlama-1.1B-Chat-v1.0"
"Qwen/Qwen3-8B"

curl -sk ${dynamo_api_url}/models | jq '.data[].id'
"TinyLlama/TinyLlama-1.1B-Chat-v1.0"
"Qwen/Qwen3-8B"

3️⃣ Chat with a model (swap the "model" field for tinyllama):

curl -sk ${dynamo_api_url}/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [{ "role": "user", "content": "What does Toronto look like in summer?" }]
}' | jq -r .choices[].message.content
# Warm and green: 25-30°C, open patios, waterfront festivals, and beaches along Lake Ontario.

curl -sk ${dynamo_api_url}/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [{ "role": "user", "content": "What does Toronto look like in summer?" }]
}' | jq -r .choices[].message.content
# Warm and green: 25-30°C, open patios, waterfront festivals, and beaches along Lake Ontario.

Both models, one endpoint tinyllama answers the same way, just change the "model" field. The stack summary from step 3 prints a ready curl for each.

⚡ Bonus: KV-cache-aware routing

KV-Routing (2 replicas, one GPU) → 20/0

Flip on KV routing and give one model two replicas that share the card:

cat > terraform.tfvars <<'EOF'
enable_dynamo        = true
dynamo_kv_routing    = true
dynamo_gpu_exclusive = false
dynamo_models = [
  { name = "tinyllama", model = "TinyLlama/TinyLlama-1.1B-Chat-v1.0", gpus = 1, tp = 1, mem = 0.3, replicas = 2 },
]
EOF
terraform apply

Confirm the workers publish KV events:

kubectl logs -n dynamo -l nvidia.com/dynamo-component=tinyllama --tail=-1 | grep -m1 -o 'use_kv_events=\w*'
# use_kv_events=True

Fire 20 requests that share one system-prompt prefix, then count per replica:

API=$(terraform output -raw dynamo_api_url)
for i in $(seq 1 20); do curl -sk $API/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"TinyLlama/TinyLlama-1.1B-Chat-v1.0","messages":[{"role":"system","content":"Long shared system prompt used as a common cache prefix."},{"role":"user","content":"hi #'$i'"}]}' >/dev/null; done

for p in $(kubectl get pods -n dynamo -l nvidia.com/dynamo-component=tinyllama -o custom-columns=:metadata.name --no-headers); do
  echo "$p: $(kubectl exec -n dynamo $p -- curl -s localhost:9090/metrics | awk '/requests_total.*dynamo_endpoint=\"generate\"/ {print $NF}')"
done
# vllm-agg-tinyllama-...-tfcth: 20
# vllm-agg-tinyllama-...-ndf7v: 0

Result: 20 of 20 shared-prefix requests land on the cache-warm replica. 0 on the other.

🎯Troubleshooting:

The failures that actually broke, and the fix for each.

1. nip.io certificate rate limit on the Frontend ingress URL

429 … too many certificates (10000) already issued for “nip.io” in the last 168h0m0s, retry after <date>

Fix: swap the host in frontend-ingress.tpl from nip.io to sslip.io (or back), on both the tls.hosts and rules.host lines.

2. Second model CrashLoops on registration

a different model is already registered there

Fix: every dynamo.vllm worker defaults to component backend, so the second model’s registration is rejected. Each worker needs its own --endpoint, already wired in model-deployment.tpl.

3. GPU node group fails on the driver preset

no implementation found for the given drivers preset

Fix: the preset must match platform + k8s version. Check with nebius mk8s node-group get-compatibility-matrix --cluster-kubernetes-version 1.34 --platform gpu-l40s-a -> use cuda13.0.

4. Terraform 1.5.x panics rendering the plan

interface conversion: float64

Fix: old-renderer bug. Run terraform apply -json -auto-approve, or upgrade Terraform.

Check the full troubleshooting list in the repo README

Useful Nebius CLI Debugging Commands

# Check MK8s cluster status
nebius mk8s cluster list --parent-id <project-id>
nebius mk8s cluster get <cluster-id>
# List node groups
nebius mk8s node-group list --parent-id <cluster-id>
# Check GPU node group details
nebius mk8s node-group get <node-group-id> 
# View available GPU platforms
nebius compute platform list --parent-id <project-id>
# Get kubeconfig
nebius mk8s cluster get-credentials <cluster-id> --external  --kubeconfig <path>

# Check MK8s cluster status
nebius mk8s cluster list --parent-id <project-id>
nebius mk8s cluster get <cluster-id>

# List node groups
nebius mk8s node-group list --parent-id <cluster-id>

# Check GPU node group details
nebius mk8s node-group get <node-group-id> 

# View available GPU platforms
nebius compute platform list --parent-id <project-id>

# Get kubeconfig
nebius mk8s cluster get-credentials <cluster-id> --external  --kubeconfig <path>

Conclusion

Congratulations, you just went from an empty Nebius project to a full-fledged Dynamo production deployment with one terraform apply. That gave you the Dynamo operator, a DynamoGraphDeployment, and two models answering on a single L40S behind one OpenAI-compatible endpoint (ingress), with real Let’s Encrypt TLS and Grafana already scraping Dynamo’s own dashboards. KV-cache-aware routing is one flag away. Most Dynamo Kubernetes examples start with the operator already installed; this one took you there from scratch.

github.com/CloudThrill/nvidia-dynamo-stack-terraform

📚 Additional Resources

Run AI Your Way — In Your Cloud

Want full control over your AI backend? The CloudThrill VLLM Private Inference POC is still open — but not forever.

📢 Secure your spot (only a few left), 𝗔𝗽𝗽𝗹𝘆 𝗻𝗼𝘄!

Run AI assistants, RAG, or internal models on an AI backend 𝗽𝗿𝗶𝘃𝗮𝘁𝗲𝗹𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗰𝗹𝗼𝘂𝗱 –
✅ No external APIs
✅ No vendor lock-in
✅ Total data control

Claim YOur FREE VLLM POC

𝗬𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮. 𝗬𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹𝘀. 𝗬𝗼𝘂𝗿 𝗿𝘂𝗹𝗲𝘀…

🙋🏻‍♀️If you like this content please subscribe to our blog newsletter ❤️.

👋🏻Want to chat about your challenges?
We’d love to hear from you!

Get in touch

Latest Podcasts

NVIDIA Dynamo on Nebius Managed Kubernetes with Terraform

Intro

📂 Project Structure

🧰Prerequisites

What’s in the stack?📦

🏛 Dynamo Architecture Overview

Two layers, two planes

Serving modes: Aggregated vs Disaggregated

The ingress dilemma: Multi-model exposure

Learn more about Dynamo

1. 🛜Networking Foundation

2. ☸️MK8S Cluster

3. 📦Essential Add-ons

4. 🐉 NVIDIA Dynamo Stack

🖥️ Nebius GPU Instance Types Available

Getting started

🔵 Deployment Steps

1️⃣Clone the repository

2️⃣ Set Up Environment Variables

Usage examples

3️⃣ Run Terraform deployment:

4️⃣ Observability (Grafana Login)

5️⃣ Destroying the Infrastructure 🚧

🛠️Configuration knobs

🧪 Quick Test

⚡ Bonus: KV-cache-aware routing

🎯Troubleshooting:

Useful Nebius CLI Debugging Commands

Conclusion

📚 Additional Resources

Run AI Your Way — In Your Cloud

𝗬𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮. 𝗬𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹𝘀. 𝗬𝗼𝘂𝗿 𝗿𝘂𝗹𝗲𝘀…

👋🏻Want to chat about your challenges?
We’d love to hear from you!

Don't miss a Bit!

Join countless others!
Sign up and get awesome cloud content straight to your inbox. 🚀

Intro

📂 Project Structure

🧰Prerequisites

What’s in the stack?📦

🏛 Dynamo Architecture Overview

Two layers, two planes

Serving modes: Aggregated vs Disaggregated

The ingress dilemma: Multi-model exposure

Learn more about Dynamo

1. 🛜Networking Foundation

2. ☸️MK8S Cluster

3. 📦Essential Add-ons

4. 🐉 NVIDIA Dynamo Stack

🖥️ Nebius GPU Instance Types Available

Getting started

🔵 Deployment Steps

1️⃣Clone the repository

2️⃣ Set Up Environment Variables

Usage examples

3️⃣ Run Terraform deployment:

4️⃣ Observability (Grafana Login)

5️⃣ Destroying the Infrastructure 🚧

🛠️Configuration knobs

🧪 Quick Test

⚡ Bonus: KV-cache-aware routing

🎯Troubleshooting:

Useful Nebius CLI Debugging Commands

Conclusion

📚 Additional Resources

Run AI Your Way — In Your Cloud

𝗬𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮. 𝗬𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹𝘀. 𝗬𝗼𝘂𝗿 𝗿𝘂𝗹𝗲𝘀…

👋🏻Want to chat about your challenges? We’d love to hear from you!

Don't miss a Bit!

Join countless others! Sign up and get awesome cloud content straight to your inbox. 🚀

👋🏻Want to chat about your challenges?
We’d love to hear from you!

Join countless others!
Sign up and get awesome cloud content straight to your inbox. 🚀