Intro A few months back I deployed the vLLM Production Stack on Nebius Managed K8s with Terraform. This time itโs NVIDIA Dynamo on the same foundation, one terraform apply from an empty cloud project to a live OpenAI-compatible endpoint. Dynamo is a very different beast from vllm-production-stack: a Helm chart installs an operator, and each …
Read more “NVIDIA Dynamo on Nebius Managed Kubernetes with Terraform”
Intro AI capex 2026 is $401B, while average GPU utilization across 23k+ Kubernetes clusters sits near 5%. That gap is the ugly truth no one’s talking about.In my recent blog vLLM DeepSeek deployment, we sharded a 643GB whale across 16 H100s because it was too big to fit on one node. Today we’re facing the …
Intro The moment your app calls more than one model provider, the plumbing gets ugly. Different SDKs, different auth, no shared view of spend, and when one provider has a bad afternoon your bot just goes dark. The fix is an LLM gateway: one endpoint in front of every provider that handles routing, fallbacks, cost …
Read more “LLM Routing Solutions: Open Source vs Commercial”
Intro Conference demos always run out of clock, you show one or two outputs and the rest stay on the cutting-room floor. That’s what happened at my last Conf42 talk, How vLLM-Omni Unifies Multimodal Inference, so to make up for it I promised to share the whole code and demo videos in a proper blog. …
Read more “vLLM-Omni on Nebius H100: Serving Z-Image, Wan2.2, Qwen3-TTS & NVIDIA Cosmos 3”
Intro In the last CoreWeave post we ran vLLM on a single GPU box with mid-sized models (70-125B). DeepSeek-V3.2, on the other hand, doesn’t fit on one box (685B parameters, ~643GB even compressed). But how do you shard a whale this big across nodes and still serve it fast, at low latency, without it falling …
Read more “vLLM DeepSeek๐ Multinode Deployment on CoreWeave (KubeRay + Terraform)”