Intro The moment your app calls more than one model provider, the plumbing gets ugly. Different SDKs, different auth, no shared view of spend, and when one provider has a bad afternoon your bot just goes dark. The fix is an LLM gateway: one endpoint in front of every provider that handles routing, fallbacks, cost …
Read more “LLM Routing Solutions: Open Source vs Commercial”
Intro Conference demos always run out of clock, you show one or two outputs and the rest stay on the cutting-room floor. That’s what happened at my last Conf42 talk, How vLLM-Omni Unifies Multimodal Inference, so to make up for it I promised to share the whole code and demo videos in a proper blog. …
Read more “vLLM-Omni on Nebius H100: Serving Z-Image, Wan2.2, Qwen3-TTS & NVIDIA Cosmos 3”
Intro When scaling AI models like DeepSeek or Qwen on Amazon EKS, engineering teams obsess over GPU utilization while quietly bleeding money on storage bloat. Because standard EBS volumes force a 1:1 replica-to-disk ratio, scaling a single 70GB model to 20 pods doesn’t cost 70GB, it forces you to provision 1.4 Terabytes of redundant EBS …
Read more “vLLM on EKS: Cut LLM Storage Costs by 95% with S3 Mountpoint”
Intro No invite? No quota? No problem. If youโve tried to create an account on CoreWeave, you already know the drill: thereโs No open self-registration, No free tier, and No โSign up with GitHubโโwithout an invite. That’s why I decided to write my first CoreWeave blog post. This post shows how to get started with …
Intro The vLLM Production Stack is designed to work across any cloud provider with Kubernetes. After covering AWS EKS, today we’re deploying vLLM production-stack on Azure AKS with the same Terraform approach. This guide shows you how to deploy the same production-ready LLM serving environment on Azure, with azure-specific optimizations. We’ll cover network architecture, certificate …
Read more “vLLM Production Stack on Azure AKS with Terraform๐ง๐ผโ๐”