Intro In the LLM routing post we wired a single gate in front of every model and promised a follow up for the part that actually keeps you out of trouble: guardrails. A guardrail is the thing that reads a prompt or a response and decides whether it gets through, gets masked, or gets dropped. …
Read more “LLM Prompt Guardrails Solutions: Open Source vs Commercial”
Intro The moment your app calls more than one model provider, the plumbing gets ugly. Different SDKs, different auth, no shared view of spend, and when one provider has a bad afternoon your bot just goes dark. The fix is an LLM gateway: one endpoint in front of every provider that handles routing, fallbacks, cost …
Read more “LLM Routing Solutions: Open Source vs Commercial”
Intro Conference demos always run out of clock, you show one or two outputs and the rest stay on the cutting-room floor. That’s what happened at my last Conf42 talk, How vLLM-Omni Unifies Multimodal Inference, so to make up for it I promised to share the whole code and demo videos in a proper blog. …
Read more “vLLM-Omni on Nebius H100: Serving Z-Image, Wan2.2, Qwen3-TTS & NVIDIA Cosmos 3”
Intro In the last CoreWeave post we ran vLLM on a single GPU box with mid-sized models (70-125B). DeepSeek-V3.2, on the other hand, doesn’t fit on one box (685B parameters, ~643GB even compressed). But how do you shard a whale this big across nodes and still serve it fast, at low latency, without it falling …
Read more “vLLM DeepSeek๐ Multinode Deployment on CoreWeave (KubeRay + Terraform)”
Intro When scaling AI models like DeepSeek or Qwen on Amazon EKS, engineering teams obsess over GPU utilization while quietly bleeding money on storage bloat. Because standard EBS volumes force a 1:1 replica-to-disk ratio, scaling a single 70GB model to 20 pods doesn’t cost 70GB, it forces you to provision 1.4 Terabytes of redundant EBS …
Read more “vLLM on EKS: Cut LLM Storage Costs by 95% with S3 Mountpoint”