Intro Conference demos always run out of clock, you show one or two outputs and the rest stay on the cutting-room floor. That’s what happened at my last Conf42 talk, How vLLM-Omni Unifies Multimodal Inference, so to make up for it I promised to share the whole code and demo videos in a proper blog. …
Read more “vLLM-Omni on Nebius H100: Serving Z-Image, Wan2.2, Qwen3-TTS & NVIDIA Cosmos 3”
Intro In the last CoreWeave post we ran vLLM on a single GPU box with mid-sized models (70-125B). DeepSeek-V3.2, on the other hand, doesn’t fit on one box (685B parameters, ~643GB even compressed). But how do you shard a whale this big across nodes and still serve it fast, at low latency, without it falling …
Read more “vLLM DeepSeek๐ Multinode Deployment on CoreWeave (KubeRay + Terraform)”
Intro When scaling AI models like DeepSeek or Qwen on Amazon EKS, engineering teams obsess over GPU utilization while quietly bleeding money on storage bloat. Because standard EBS volumes force a 1:1 replica-to-disk ratio, scaling a single 70GB model to 20 pods doesn’t cost 70GB, it forces you to provision 1.4 Terabytes of redundant EBS …
Read more “vLLM on EKS: Cut LLM Storage Costs by 95% with S3 Mountpoint”
Intro If you’ve been following along, we’ve already covered vLLM-Omni and how diffusion models work. But here’s the dirty secret of diffusion models: they don’t run a single expensive computation, they run it many times per generation. 50 steps means 50 full forward passes through a multi-billion-parameter transformer. That’s a lot of GPU hours burned …
Intro The vLLM Production Stack is designed to run on any Kubernetes-based infrastructure. After covering AWS , Azure, Google Cloud and Nebius MK8s implementations, today we’re deploying vLLM production-stack on CoreWeave Kubernetes (CKS) with the same Terraform framework. CoreWeave is one of the hottest NeoCould built on the idea that GenAI workloads donโt need virtualization; they need direct access to …
Read more “vLLM Production Stack on CoreWeave CKS with Terraform๐ง๐ผโ๐”