Intro The moment your app calls more than one model provider, the plumbing gets ugly. Different SDKs, different auth, no shared view of spend, and when one provider has a bad afternoon your bot just goes dark. The fix is an LLM gateway: one endpoint in front of every provider that handles routing, fallbacks, cost …
Read more “LLM Routing Solutions: Open Source vs Commercial”
Intro When scaling AI models like DeepSeek or Qwen on Amazon EKS, engineering teams obsess over GPU utilization while quietly bleeding money on storage bloat. Because standard EBS volumes force a 1:1 replica-to-disk ratio, scaling a single 70GB model to 20 pods doesn’t cost 70GB, it forces you to provision 1.4 Terabytes of redundant EBS …
Read more “vLLM on EKS: Cut LLM Storage Costs by 95% with S3 Mountpoint”
Intro Deploying vLLM manually is fine for a lab, but running it in production means dealing with Kubernetes, autoscaling, GPU orchestration, and observability. Thatโs where the vLLM Production Stack comes in – a Terraform-based blueprint that delivers production-ready LLM serving with enterprise-grade foundations. In this post, we’ll deploy it on Amazon EKS, covering everything from …
Read more “vLLM Production Stack on Amazon EKS with Terraform๐ง๐ผโ๐”