Intro

Any-to-any multimodal models combining text, images, video, and audio are advancing AI, but their complex architectures, mixing autoregressive LLMs and diffusion transformers, make efficient serving very difficult. Current systems like OpenAI’s ChatGPT (text) and Sora (video) run as separate engines, lacking unified any-to-any pipelines. vLLM-Omni solves just that with a fully disaggregated serving system designed for interconnected, multi-component models. 

I. What is vLLM-Omni?

vLLM-Omni is a purposeful extension of vLLM core that adds multimodal and non-autoregressive model support while preserving full compatibility with vLLM’s scheduling and KV cache optimizations. That means anything from Image generation text to speech even video generation can be done through this new serving engine flavor.

Design intent

The secret sauce is to serve both text generation that relies on Autoregressive(AR) models and Multi-Modal generation using Diffusion(Dit) model. All in a heterogenous pipeline (multimedia input and output).

Two In One Architecture: AR vs. DiT

How does it extend vLLM

• Intercepting the CLI --omni Switch
  • CLI checks for --omni flag; if absent, it calls vLLM’s main directly .
  • If present, it loads vLLM-Omni’s serve module and dispatches accordingly . If not the CLI behave like vLLM
• Runtime patching of vLLM symbols at import
  • Replaces vLLM classes with Omni equivalents at import
  • Ensures all vLLM internals use Omni-enhanced classes
• Inheriting key vLLM classes to add omni-specific capabilities.
  • Adds omni-specific configuration & connectors including an LLMEngine like vLLM 
  • AsyncOmniLLM extends vLLM’s AsyncLLM for async, per-stage execution 
  • OmniEngineArgs extends vLLM’s EngineArgs with omni fields like stage_id and model_stage
  • Schedulers and workers also inherit from vLLM bases, e.g., OmniARScheduler extends VLLMScheduler

II. Architecture Overview 🚀

• Modality Encoders: Thinker stage that uses separate encoders for each input modality(Vision/Audio Encoders)
• LLM AR (Autoregressive Language Model): Core Language model that generates text tokens sequentially
• Modality Generator: Uses different generators depending on the output modality (txt-to-img,TTS,img-to-vid…)
  

Omni-Modality Serving Architecture

In this Advanced dispatching and orchestration stack for multi-modal AI workloads. AutoRegressive(AR) Module handles all text based workload like vLLM while Diffusion takes care of other modalities.

Natively Disaggregated Serving

vLLM-Omni runs each model stage in its own process/device, connected via OmniConnector. This enables heterogeneous pipelines (AR + DiT) and dynamic resource allocation across stages

AR Vs Diffusion Module Design

Below is the core architecture of each AR and Difusion modules that makes Omni-modality work under vLLm-Omni.

Omni-Modality Pipeline Stages

Thinker/Talker/Code2Wav is defined in Qwen-Omni stage configs to produce audio output via waveform decoding.

Omni-Modal Request Workflow

Diffusion models

Diffusion models use a single diffusion stage and expose image/video outputs directly, without intermediate stages

🧠Model Support

vLLM-Omni supports 20+ popular omni and diffusion model architectures(growing rapidly) including Stable Diffusion 3.5

⚡Diffusion stage acceleration

• TeaCache: Caches transformer computations when consecutive timesteps are similar speeding-up by ~1.5x–2.0x
• Cache-DiT: speeds up diffusion transformers via DBCache (dual block), TaylorSeer (forecasting), & SCM
• Ulysses-SP: Splits seq dimensions & uses all-to-all communication so attention heads can be processed in parallel 
• Ring-Attention: Splits the sequence dimension and circulates KV blocks in a ring topology to accumulate attention & shard sequences
• Other: Parallelism, Quantization, Fused Ops, Timestep distillation

Interface design

• Offline Inference
  • The Omni class provides a Python interface for offline batched inference
• Online Inference
  • Similar to vLLM, vLLM-Omni also provides a FastAPI-based server for online serving. 

Vllm-Omni Endpoints

• All models expose all OpenAI-compatible endpoints such as /v1/chat/completions and /v1/images/generations.
• /v1/transcription and /v1/translation: Audio transcription/translation
• /v1/audio/speech: Text-to-speech (TTS) generation for supported models (e.g., Qwen3-TTS)
• /v1/messages: Anthropic-compatible messages (if supported_tasks include “generate”)

III. Online Inference Examples

1. Text-to-Image

# 1) Serve  
vllm serve Qwen/Qwen-Image --omni --port 8091  
  
# 2) Generate  
curl -s http://localhost:8091/v1/images/generations \  
  -H "Content-Type: application/json" \  
  -d '{  
    "prompt": "a cup of coffee on the table",  
    "model": "Qwen/Qwen-Image",  
    "n": 1,  
    "size": "1024x1024"  
  }' | jq -r '.data[0].b64_json' | base64 -d > coffee.png

# 1) Serve  
vllm serve Qwen/Qwen-Image --omni --port 8091  
  
# 2) Generate  
curl -s http://localhost:8091/v1/images/generations \  
  -H "Content-Type: application/json" \  
  -d '{  
    "prompt": "a cup of coffee on the table",  
    "model": "Qwen/Qwen-Image",  
    "n": 1,  
    "size": "1024x1024"  
  }' | jq -r '.data[0].b64_json' | base64 -d > coffee.png

2. Image-to-Image

# 1) Serve  
vllm serve Qwen/Qwen-Image-Edit --omni --port 8091  
  
# 2) Generate (edit)  
curl -s http://localhost:8091/v1/images/generations \  
  -H "Content-Type: application/json" \  
  -d '{  
    "prompt": "turn this cat to a dog",  
    "model": "Qwen/Qwen-Image-Edit",  
    "n": 1,  
    "size": "1024x1024",  
    "image": "data:image/png;base64,..."  
  }' | jq -r '.data[0].b64_json' | base64 -d > edited.png

# 1) Serve  
vllm serve Qwen/Qwen-Image-Edit --omni --port 8091  
  
# 2) Generate (edit)  
curl -s http://localhost:8091/v1/images/generations \  
  -H "Content-Type: application/json" \  
  -d '{  
    "prompt": "turn this cat to a dog",  
    "model": "Qwen/Qwen-Image-Edit",  
    "n": 1,  
    "size": "1024x1024",  
    "image": "data:image/png;base64,..."  
  }' | jq -r '.data[0].b64_json' | base64 -d > edited.png

3.Text-to-Speech (TTS)

# 1) Serve  
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091  
  
# 2) Generate speech  
curl -s http://localhost:8091/v1/audio/speech \  
  -H "Content-Type: application/json" \  
  -d '{  
    "model": "Qwen/Qwen2.5-Omni-7B",  
    "input": "Hello, this is a test.",  
    "voice": "Default"  
  }' --output speech.wav

# 1) Serve  
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091  
  
# 2) Generate speech  
curl -s http://localhost:8091/v1/audio/speech \  
  -H "Content-Type: application/json" \  
  -d '{  
    "model": "Qwen/Qwen2.5-Omni-7B",  
    "input": "Hello, this is a test.",  
    "voice": "Default"  
  }' --output speech.wav

4. OmniQwen3-Omni (Multimodal: video input→ text + audio output)

# 1) Serve  
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091  
  
# 2) Generate (video input → text + audio output)  
curl -s http://localhost:8091/v1/chat/completions \  
  -H "Content-Type: application/json" \  
  -d '{  
    "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",  
    "messages": [  
      {  
        "role": "system",  
        "content": [  
          {  
            "type": "text",  
            "text": "You are Qwen, a virtual Cyborg developed by the Qwen Team, 
            capable of perceiving auditory and visual inputs, as well as generating text and speech."  
          }  
        ]  
      },  
      {  
        "role": "user",  
        "content": [  
          {"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}},  
          {"type": "text", "text": "Describe the video briefly."}  
        ]  
      }  
    ],  
    "extra_body": {  
      "sampling_params_list": [  
        {"temperature": 0.4, "max_tokens": 2048},  
        {"temperature": 0.9, "max_tokens": 4096},  
        {"temperature": 0.0, "max_tokens": 65536}  
      ]  
    }  
  }' | jq -r '.choices[0].message.content' > response.json

# 1) Serve  
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091  
  
# 2) Generate (video input → text + audio output)  
curl -s http://localhost:8091/v1/chat/completions \  
  -H "Content-Type: application/json" \  
  -d '{  
    "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",  
    "messages": [  
      {  
        "role": "system",  
        "content": [  
          {  
            "type": "text",  
            "text": "You are Qwen, a virtual Cyborg developed by the Qwen Team, 
            capable of perceiving auditory and visual inputs, as well as generating text and speech."  
          }  
        ]  
      },  
      {  
        "role": "user",  
        "content": [  
          {"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}},  
          {"type": "text", "text": "Describe the video briefly."}  
        ]  
      }  
    ],  
    "extra_body": {  
      "sampling_params_list": [  
        {"temperature": 0.4, "max_tokens": 2048},  
        {"temperature": 0.9, "max_tokens": 4096},  
        {"temperature": 0.0, "max_tokens": 65536}  
      ]  
    }  
  }' | jq -r '.choices[0].message.content' > response.json

Coming Up Next

That wraps up our Beginners vLLm-Omni intro, where we explored the key features that make vLLM-omni fast, efficient, and production-ready multi-modal serving engine. Hopefully, this little breakdown made it a little easier to grasp as the project is still growing in restless pace.
In Part 2, we’ll show you all about Diffusion models from noise to pixel, covering the research papers from 2021 to 2024!

Stay tuned for Part 2⚡

Run AI Your Way — In Your Cloud

Want full control over your AI backend? The CloudThrill VLLM Private Inference POC is still open — but not forever.

📢 Secure your spot (only a few left), 𝗔𝗽𝗽𝗹𝘆 𝗻𝗼𝘄!

---

Run AI assistants, RAG, or internal models on an AI backend 𝗽𝗿𝗶𝘃𝗮𝘁𝗲𝗹𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗰𝗹𝗼𝘂𝗱 –
✅ No external APIs
✅ No vendor lock-in
✅ Total data control

Claim YOur FREE VLLM POC

𝗬𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮. 𝗬𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹𝘀. 𝗬𝗼𝘂𝗿 𝗿𝘂𝗹𝗲𝘀…

🙋🏻‍♀️If you like this content please subscribe to our blog newsletter ❤️.