vLLM for beginners: Key Features & Performance Optimization(PartII)

by CloudDudeAI, LLM, VllmPosted on Comments are Disabled

Intro In Part 1 of our vLLM for beginners Series, we covered the fundamentals—core concepts and terminology behind vLLM’s architecture. In Part 2, we go deeper into what makes vLLM excel at performance: features like PagedAttention, attention backends, prefill & decode management, and more. 💡This series is about building a strong foundation in vLLM—understanding how …

Read more “vLLM for beginners: Key Features & Performance Optimization(PartII)”

vLLM for beginners: The Fundamentals

by CloudDudeAI, LLM, VllmPosted on Comments are Disabled

Intro last year, I have dived deep into Ollama inference where I ended up building and speaking about Ollama Kubernetes deployments along with rich documentation in my ollama_lab repo and quantization article—This year’s Cloudtrhill focus is VLLM Inference which is a next level beast from a model serving standpoint. Exploring multiple inference options is time-intensive …

Read more “vLLM for beginners: The Fundamentals”

Ollama deployment on Civo K8s Cluster with terraform

by CloudDudeAI, kubernetes, LLM, Ollama, terraformPosted on Comments are Disabled

Intro Tired of sharing your IP & sensitive data to OpenAI ? What if you could run your own private AI chatbot powered by Local Inference & LLMs, with 100% data privacy—all inside a Kubernetes cluster?Today we’ll show you how to deploy an end-to-end LLM inference setup on a Civo Cloud Talos K8s cluster with …

Read more “Ollama deployment on Civo K8s Cluster with terraform”

kv_cache Explained: How It Enhances vLLM Inference

by CloudDudeAI, LLM, VllmPosted on Comments are Disabled

Intro Too often, machine learning concepts are explained like a mathematician talking to other mathematicians—leaving the rest of us scratching our heads. One of those is kv_cache, a key technique that makes large language models run faster and more efficient.This blog is my attempt to break it down simply, without drowning in dark math :). …

Read more “kv_cache Explained: How It Enhances vLLM Inference”

world of LLM

How to Quantize AI Models with Ollama CLI

by CloudDudeAI, LLMPosted on Comments are Disabled

Intro You’ve probably fired up ollama run some-cool-model tons of times, effortlessly pulling models from Ollama’s Repo or even directly from Hugging Face. But have you ever wondered how those CPU-friendly GGUF quantized models actually land on places like Hugging Face in the first place? What if I told you, you could contribute back with tools you might already be …

Read more “How to Quantize AI Models with Ollama CLI”