vLLM-Omni on Nebius H100: Serving Z-Image, Wan2.2, Qwen3-TTS & NVIDIA Cosmos 3

by CloudDudein AI, LLM, MultiCloud, Multimodal, NeoCloudson Posted on

Intro

Conference demos always run out of clock, you show one or two outputs and the rest stay on the cutting-room floor. That’s what happened at my last Conf42 talk, How vLLM-Omni Unifies Multimodal Inference, so to make up for it I promised to share the whole code and demo videos in a proper blog. This is it.

So in this walkthrough, we’ll take you through serving four modalities on a single Nebius H100 node: provisioning the box, serving each model, hitting it with raw curl (or through vllm-WebUI), and driving the whole thing from one menu. We’ll go from an empty VM to Z-Image, Wan2.2 video, Qwen3-TTS speech, and Cosmos 3, a world model – with exact commands at each step. By the end, you’ll be able to reproduce the entire demo yourself.

▶Talk: vLLM-Omni Multimodal Inference ▤ Download The Slides (PDF)

What you’ll end up with

We picked Nebius as the GPU platform to run our lab with a H100 node, one serving stack, four modalities:

One engine, four modalities : swapped on a single H100

Modality Model Port Endpoint
Image Tongyi-MAI/Z-Image-Turbo 8091 /v1/images/generations
Image → Video Wan-AI/Wan2.2-TI2V-5B-Diffusers 8091 /v1/videos (async + poll)
Speech Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice 8091 /v1/audio/speech
World model nvidia/Cosmos3-Nano 8000 /v1/videos/sync

The first three share port 8091 and get swapped on the same GPU. Cosmos runs on its own image and port (8000).

Where things run

The Serve steps run on the Nebius H100 box. The Query steps run on your laptop, hitting localhost through the SSH tunnel; so the generated files land locally. Each heading is tagged so you always know which shell you’re in.

⌥ All the scripts in this walkthrough live in one repo — clone it and follow along:
git clone https://github.com/cloudthrill/vllm-omni-demo.git
📦 What’s in the repo

1. Provision a single-H100 box on Nebius · on your laptop

Authenticate first so no browser-login prompt fires mid-script:

nebius iam tenant list  
 # if this opens a browser login, do it now

Then bind tenant to a project (index 0 = your first project):

# 1. Tenant ID
export TENANT_ID=$(nebius iam tenant list --format json | jq -r '.items[0].metadata.id')

# 2. Project ID
export PROJECT_ID=$(nebius iam project list --parent-id "$TENANT_ID" --format json \
  | jq -r '.items[0].metadata.id')

# 3. Set the project ID
nebius config set parent-id "$PROJECT_ID"

# Confirm it stuck
nebius config get parent-id

Boot disk with CUDA drivers preinstalled:

export INF_VM_BOOT_DISK_ID=$(nebius compute disk create \
  --name inf-disk --size-gibibytes 200 --type network_ssd \
  --source-image-family-image-family ubuntu22.04-cuda12 \
  --block-size-bytes 4096 --format json | jq -r ".metadata.id")

Subnet + cloud-init user (injects your public key i.e id_rsa.pub): 

export SUBNET_ID=$(nebius vpc subnet list --format json | jq -r ".items[0].metadata.id")

export USER_DATA=$(jq -Rs '.' <<EOF
users:
  - name: user
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - $(cat ~/.ssh/id_rsa.pub)
EOF
)

Create the VM (gpu-h100-sxm, single GPU):

export INF_VM_ID=$(nebius compute instance create --format json - <<EOF | jq -r ".metadata.id"
{ "metadata": { "name": "vllm-omni-demo" },
  "spec": { "stopped": false, "cloud_init_user_data": $USER_DATA,
    "resources": { "platform": "gpu-h100-sxm", "preset": "1gpu-16vcpu-200gb" },
    "boot_disk": { "attach_mode": "READ_WRITE", "existing_disk": { "id": "$INF_VM_BOOT_DISK_ID" } },
    "network_interfaces": [ { "name": "nic0", "subnet_id": "$SUBNET_ID", "ip_address": {}, "public_ip_address": {} } ] } }
EOF
)

Grab the public IP:

export INF_IP=$(nebius compute instance get --id $INF_VM_ID --format json \
  | jq -r '.status.network_interfaces[0].public_ip_address.address | split("/")[0]')
echo "$INF_VM_ID  →  $INF_IP"
Tip: make the IP re-derivable. A fresh shell loses $INF_IP. Drop this in ~/box.env and source it any time: 
cat > ~/box.env <<'EOF'
export INF_VM_ID=$(nebius compute instance list --format json \
  | jq -r '.items[] | select(.metadata.name=="vllm-omni-demo") | .metadata.id')
export INF_IP=$(nebius compute instance get --id "$INF_VM_ID" --format json \
  | jq -r '.status.network_interfaces[0].public_ip_address.address | split("/")[0]')
echo "VM: $INF_VM_ID"; echo "IP: $INF_IP"
EOF

2. Connect, enable Docker, open the tunnel · laptop → box

SSH in and forward both serving ports back to your laptop in one connection. The forwards let you run curl and the playground UI locally while the models run on the box: i.e SSH key here id_rsa

ssh -i ~/.ssh/id_rsa \
  -L 8091:localhost:8091 -L 8000:localhost:8000 -L 8080:localhost:8080 \
  -o ServerAliveInterval=60 -o LogLevel=ERROR user@$INF_IP

On the box, enable Docker for your user and verify the GPU is visible inside a container:

sudo usermod -aG docker user
newgrp docker          # apply the group now, no relog
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01             Driver Version: 550.163.01     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:8D:00.0 Off |                    0 |
| N/A   29C    P0             70W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

If that prints the H100, the container toolkit is wired up correctly.

3. Install the vllm-playground · on your laptop

The vLLM Playground: a WEB-UI for poking at the models, runs on your laptop. Install uv, then the Playground:

curl -LsSf https://astral.sh/uv/install.sh | sh
echo '[ -f "$HOME/.local/bin/env" ] && . "$HOME/.local/bin/env"' >> ~/.bashrc
source ~/.bashrc
uv tool install vllm-playground

Run the vLLM-Playground locally and point it at the box’s /v1 (once a model is up) through the SSH tunnel:

vllm-playground --port 8080 &     # browse http://localhost:8080, add instance → http://localhost:8091/v1

4. Image — Z-Image-Turbo

▶ Z-Image · live through the vLLM Playground

Serve · on the box

docker run -d --rm --name omni --gpus all -p 8091:8091 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-omni:v0.22.0 vllm serve Tongyi-MAI/Z-Image-Turbo --omni --port 8091
  
# check the docker logs wait for the serving line, 
docker logs -f omni

Confirm it’s up:

# → Tongyi-MAI/Z-Image-Turbo
curl -s http://localhost:8091/v1/models | jq -r '.data[0].id'

Query (standalone) · on your laptop (via tunnel)

Z-Image is a turbo model: few steps, CFG 1. The response is base64; decode it to a PNG:

time curl -s http://localhost:8091/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model":"Tongyi-MAI/Z-Image-Turbo",
    "prompt":"a whale swimming through clouds above a city, surreal, golden hour",
    "negative_prompt":"blurry, low quality, deformed, distorted, washed out, artifacts",
    "seed":42,
    "num_inference_steps":6,
    "guidance_scale":1,
    "size":"1024x1024",
    "n":1
  }' \
  | jq -r '.data[0].b64_json' | base64 -d > whale.png && ls -lh whale.png
API primitive one call · base64 in the JSON
  1. 1
    POST /v1/images/generations

    OpenAI-images-compatible JSON. The image comes back as data[0].b64_json — pipe through base64 -d to get the file. Everything else (seed, num_inference_steps, guidance_scale, size) is a standard generation knob.

Swap off · on the box

# you can use --rm to auto-remove it, freeing the GPU and the name
docker stop omni

5. Image → Video — Wan2.2

▶ Wan2.2 · image → video

Serve · on the box

Wan benefits from Cache-DiT, which is a serve-time flag:

docker run -d --rm --name omni --gpus all -p 8091:8091 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-omni:v0.22.0 vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --omni --port 8091 --cache-backend cache_dit

# check the docker logs
docker logs -f omni

Query (standalone) · on your laptop (via tunnel)

Video is asynchronous: you POST a job, get an id, poll until completed, then download the content. The whole flow must stay in one shell so $vid survives:

vid=$(curl -s -X POST http://localhost:8091/v1/videos \
  -F 'model=Wan-AI/Wan2.2-TI2V-5B-Diffusers' \
  -F 'input_reference=@whale.png' \
  -F 'prompt=the whale glides slowly through drifting clouds, gentle camera push-in, cinematic, golden hour' \
  -F 'negative_prompt=blurry, distorted, flicker, warping, morphing, low quality' \
  -F 'seconds=5' -F 'num_inference_steps=50' -F 'guidance_scale=5.0' -F 'flow_shift=5.0' \
  -F 'enable_frame_interpolation=true' -F 'frame_interpolation_exp=2' -F 'seed=100' \
  | jq -r '.id'); echo "job: $vid"

# poll with an elapsed timer
start=$SECONDS
while :; do
  s=$(curl -s http://localhost:8091/v1/videos/$vid | jq -r '.status')
  e=$((SECONDS-start)); printf '\r[%02d:%02d] %s        ' "$((e/60))" "$((e%60))" "$s"
  [ "$s" = completed ] && { echo; break; }
  [ "$s" = failed ] && { echo FAILED; curl -s http://localhost:8091/v1/videos/$vid | jq .; break; }
  sleep 5
done

# download the finished clip
curl -s http://localhost:8091/v1/videos/$vid/content -o whale_i2v.mp4 && ls -lh whale_i2v.mp4
API primitive three calls, not one
  1. 1
    POST /v1/videos

    Multipart -F (the @ in input_reference=@file uploads the source image) → returns { "id": ... }

  2. 2
    GET /v1/videos/{id}

    Poll .status until completed / failed

  3. 3
    GET /v1/videos/{id}/content

    The MP4 bytes — save with -o

Note: use -F (not –form-string) for input_reference=@whale.png, the @ is what tells curl to upload the file. Text fields can be either.

Swap off · on the box

docker stop omni

6. Speech — Qwen3-TTS

▶ Qwen3-TTS · text → speech

Serve · on the box

TTS is small > cap the memory fraction and run eager:

docker run -d --rm --name omni --gpus all -p 8091:8091 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-omni:v0.22.0 vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
  --deploy-config vllm_omni/deploy/qwen3_tts.yaml --omni --port 8091 \
  --trust-remote-code --enforce-eager --gpu-memory-utilization 0.3

docker logs -f omni

Query (standalone) · on your laptop (via tunnel)

Synchronous > one call returns the audio directly (notice no need to specify the model in the payload):

time curl -s -X POST http://localhost:8091/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "input":"Sight. Sound. Voice. Motion. The new senses of artificial intelligence, with vLLM Omni.",
    "voice":"vivian",
    "language":"English",
    "max_new_tokens":4096
  }' \
  --output tts_demo.wav && ls -lh tts_demo.wav
API primitive one call · body is the audio
  1. 1
    POST /v1/audio/speech

    OpenAI-speech-compatible JSON. Unlike image/video, the response body is the audio file itself — no base64 unwrap, no polling. Just --output to a .wav. voice and language pick the speaker; max_new_tokens caps length (a short line never hits it).

Swap off · on the box

docker stop omni

7. World model — Cosmos 3

Cosmos has its own image (vllm/vllm-omni:cosmos3), its own port (8000), and it wants the whole GPU: stop the 8091 container first.

▶ Cosmos 3 · video + sound 🔊

⚠️Serve · on the box: Guardrail Gate

Cosmos pulls a gated guardrail model from HuggingFace at init. Unless you’ve been granted access to nvidia/Cosmos-1.0-Guardrail, that download 401s and the engine crashes on startup. Disable guardrails via a deploy-config YAML (see Troubleshooting for the why):

cat > ./no_guardrails.yaml <<'EOF'
async_chunk: false

stages:
  - stage_id: 0
    async_chunk: false
    max_num_seqs: 1
    enforce_eager: true
    trust_remote_code: true
    model_class_name: Cosmos3OmniDiffusersPipeline
    model_config:
      guardrails: false
      offload_guardrail_models: false
EOF
docker stop omni 2>/dev/null      # free the GPU first

docker run -d --rm --name cosmos --runtime nvidia --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface -v "$(pwd):/workspace" \
  vllm/vllm-omni:cosmos3 vllm serve nvidia/Cosmos3-Nano --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --deploy-config /workspace/no_guardrails.yaml \
  --allowed-local-media-path / --port 8000 --init-timeout 1800

# slow load — wait for "Application startup complete."
docker logs -f cosmos

Confirm the model endpoint:

curl -s http://localhost:8000/v1/models | jq -r '.data[0].id'   
# → nvidia/Cosmos3-Nano

Query (standalone) · on your laptop (via tunnel)

Cosmos is synchronous (/v1/videos/sync), one call blocks until the MP4 is written, no polling. Two paths:

Path A — full pipeline: text → image → video + sound. Generate the dashcam still, then animate it with audio:

# A1: text → image
curl -sS -X POST http://localhost:8000/v1/images/sync \
  -F "prompt=Generate a 16:9 image from a dashcam view of a formula 1 racing event" \
  -F "size=1280x720" -F "seed=0" \
  -o f1_gen.png

# A2: that image → video + sound
curl -sS -X POST http://localhost:8000/v1/videos/sync \
  -F "input_reference=@f1_gen.png" \
  -F "prompt=A high-speed racing event where a car navigates multiple winding turns" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" \
  -F "num_inference_steps=35" -F "guidance_scale=6.0" -F "flow_shift=10.0" -F "seed=0" \
  -F "generate_sound=true" \
  --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false}' \
  -o cosmos3_A.mp4

Path B — start from an existing frame: image → video. Skips text-to-image:
Image should exist.

curl -sS -X POST http://localhost:8000/v1/videos/sync \
  -F "input_reference=@f1_dashcam.png" \
  -F "prompt=A high-speed racing event where a car navigates multiple winding turns" \
  -F "size=1280x720" -F "num_frames=189" -F "fps=24" \
  -F "num_inference_steps=35" -F "guidance_scale=6.0" -F "flow_shift=10.0" -F "seed=0" \
  --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false}' \
  -o cosmos3_B.mp4
API primitive one sync call · body is the MP4
  1. 1
    POST /v1/videos/sync also /v1/images/sync

    Multipart like Wan, but synchronous — the response body is the MP4, written straight to -o, no job id and no poll. generate_sound=true muxes a stereo AAC track into the file. extra_params carries pipeline-level flags as a JSON blob — keep it on --form-string so curl passes it verbatim.

Swap off · on the box

docker stop cosmos

8. The all-in-one demo menu · on your laptop

Once each model’s serve + query is verified, drive the requests from one skippable menu. Servers are still started separately (they need the GPU and you swap them by hand); this just fires the right curl for whichever act you pick, prints the command first, and times it.

Assumes the right server is already running: Your image/video/speech on 8091, Cosmos on 8000. 
cat > ~/demo_menu.sh <<'EOF'
#!/bin/bash

cd ~/omni_demo 2>/dev/null

show() { printf '\033[36m$ %s\033[0m\n' "$*"; eval "$@"; }

image() {
  start=$SECONDS
  show "curl -s http://localhost:8091/v1/images/generations -H 'Content-Type: application/json' -d '{"model":"Tongyi-MAI/Z-Image-Turbo","prompt":"a whale swimming through clouds above a city, surreal, golden hour","negative_prompt":"blurry, low quality, deformed","seed":42,"n":1}' | jq -r '.data[0].b64_json' | base64 -d > whale.png"
  e=$((SECONDS-start)); printf '[%02d:%02d] done · ' "$((e/60))" "$((e%60))"; ls -lh whale.png
}

video() {
  printf '\033[36m$ POST /v1/videos  (input_reference=@whale.png) → poll → download\033[0m\n'
  vid=$(curl -s -X POST http://localhost:8091/v1/videos \
    -F 'model=Wan-AI/Wan2.2-TI2V-5B-Diffusers' -F 'input_reference=@whale.png' \
    -F 'prompt=the whale glides slowly through drifting clouds, gentle camera push-in, cinematic, golden hour' \
    -F 'seconds=5' -F 'num_inference_steps=50' -F 'seed=100' | jq -r '.id'); echo "job: $vid"
  start=$SECONDS
  while :; do
    s=$(curl -s http://localhost:8091/v1/videos/$vid | jq -r '.status')
    e=$((SECONDS-start)); printf '\r[%02d:%02d] %s        ' "$((e/60))" "$((e%60))" "$s"
    [ "$s" = completed ] && { echo; break; }
    [ "$s" = failed ] && { echo FAILED; return; }
    sleep 5
  done
  curl -s http://localhost:8091/v1/videos/$vid/content -o whale_i2v.mp4 && ls -lh whale_i2v.mp4
}

tts() {
  start=$SECONDS
  show "curl -s -X POST http://localhost:8091/v1/audio/speech -H 'Content-Type: application/json' -d '{"input":"Sight. Sound. Voice. Motion. The new senses of artificial intelligence, with vLLM Omni.","voice":"vivian","language":"English","max_new_tokens":4096}' --output tts_demo.wav"
  e=$((SECONDS-start)); printf '[%02d:%02d] done · ' "$((e/60))" "$((e%60))"; ls -lh tts_demo.wav
}

cosmos() {
  printf '\033[36m$ POST /v1/videos/sync  (input_reference=@f1_dashcam.png, generate_sound=true)\033[0m\n'
  start=$SECONDS
  curl -sS -X POST http://localhost:8000/v1/videos/sync -F "input_reference=@f1_dashcam.png" \
    -F "prompt=A high-speed racing event where a car navigates multiple winding turns" \
    -F "size=1280x720" -F "num_frames=189" -F "fps=24" -F "num_inference_steps=35" \
    -F "guidance_scale=6.0" -F "flow_shift=10.0" -F "seed=0" -F "generate_sound=true" \
    --form-string 'extra_params={"use_resolution_template":false,"use_duration_template":false}' \
    -o cosmos3_f1.mp4 &
  pid=$!
  while kill -0 $pid 2>/dev/null; do
    e=$((SECONDS-start)); printf '\r[%02d:%02d] generating…   ' "$((e/60))" "$((e%60))"; sleep 1
  done
  e=$((SECONDS-start)); printf '\r[%02d:%02d] done           \n' "$((e/60))" "$((e%60))"; ls -lh cosmos3_f1.mp4
}

PS3=$'\n>>> pick a step, or 5 to quit: '
while true; do
  echo
  select choice in "IMAGE (Z-Image)" "VIDEO (Wan2.2 i2v)" "SPEECH (Qwen3-TTS)" "COSMOS (F1 i2v+sound :8000)" "quit"; do
    case $REPLY in
      1) image; break ;;
      2) video; break ;;
      3) tts; break ;;
      4) cosmos; break ;;
      5) exit 0 ;;
      *) echo "pick 1-5"; break ;;
    esac
  done
done
EOF

chmod +x ~/demo_menu.sh

Run it:

~/demo_menu.sh
1) IMAGE  (Z-Image)
2) VIDEO  (Wan2.2 i2v)
3) SPEECH (Qwen3-TTS)
4) COSMOS (F1 i2v :8000)
5) quit

>>> pick a step, or 5 to quit:

Pick a number, it prints the curl in cyan then runs it with a timer, and returns to the menu so you can jump to any modality in any order. The sync calls (Cosmos) run in the background with a ticking spinner so a multi-minute generation doesn’t look frozen.

9. Troubleshooting

crash Cosmos: GatedRepoError: 401 on the guardrail repo

Cosmos loads a safety guardrail at init that downloads from a gated HF repo. No access → 401 → engine never starts. Three ways to disable:

  • Deploy-config (server-wide, no token) — what this guide uses. The guardrails: false YAML from §7. If you hit async_chunk=True in deploy, set async_chunk: false at both the top level and inside the stage.
  • Per-request — add guardrails:false to extra_params. Caveat: still needs the guardrail models present locally (skips running, not downloading) — won’t help without gate access.
  • --no-guardrails flag — a Cosmos Framework flag, not in vLLM-Omni. vllm serve … --no-guardrails errors with unrecognized arguments. Use the deploy-config.

If you do have gate access: accept the gate on HF and pass -e HF_TOKEN=… into the container.

docker Conflict. The container name "/omni" is already in use

A stopped-but-not-removed container is squatting the name. Force-remove:

docker rm -f omni

Put docker stop omni 2>/dev/null; docker rm -f omni 2>/dev/null at the top of every serve script so swaps never collide.

docker permission denied … /var/run/docker.sock

The docker group isn’t active in your current shell. newgrp docker activates it without a relog; if your shell predates the group change (e.g. an old tmux pane), open a fresh session or prefix with sudo.

slow Video generation is much slower than expected

Almost always cold start — the first generation pays for kernel compilation and an empty Cache-DiT. Warm with one throwaway run, then it’s fast. Confirm the GPU is working with nvidia-smi (util near 100% during a job). Cache-DiT is a serve-time flag (--cache-backend cache_dit), not a request param — check it’s on with docker ps --no-trunc.

loop $vid is empty / the poll loop never finishes

The video POST and the poll loop ran in separate shells, so $vid was lost. Keep submit + poll + download in one shell (one line joined with ;, or one function). The tell: the poll hits /v1/videos/ with no id appended.

noise SSH spam: channel N: connect failed: Connection refused

The tunnel is forwarding a port that isn’t listening yet (e.g. 8091 while only Cosmos on 8000 is up). Harmless. Silence with -o LogLevel=ERROR on the SSH command, or only forward the port you’re using.

Extracting the API primitive — one shape per modality

Model Method + path Sync? Where the output lives
Z-Image POST /v1/images/generations sync JSON → data[0].b64_json, base64 -d
Wan2.2 POST /v1/videos → GET /v1/videos/{id} → /content async · poll MP4 bytes from /content
Qwen3-TTS POST /v1/audio/speech sync response body is the audio
Cosmos 3 POST /v1/videos/sync (or /v1/images/sync) sync response body is the MP4

File generation pattern

JSON bodies for the OpenAI-compatible text-ish endpoints (images, speech), multipart -F wherever you upload a reference file (input_reference=@file), base64-decode only for the images endpoint, and poll only for async video (Wan). Cosmos collapses the video flow into one synchronous call.

10. Teardown · box, then laptop

Stop the meter. Containers first (on the box), then the VM and disk (from your laptop):

# on the box
docker rm -f $(docker ps -aq) 2>/dev/null
exit

# on the laptop — derive ids by name, delete instance then disk
export INF_VM_ID=$(nebius compute instance list --format json \
  | jq -r '.items[] | select(.metadata.name=="vllm-omni-demo") | .metadata.id')

export INF_DISK_ID=$(nebius compute instance get --id "$INF_VM_ID" --format json \
  | jq -r '.spec.boot_disk.existing_disk.id')

nebius compute instance delete --id "$INF_VM_ID"

nebius compute disk delete --id "$INF_DISK_ID"

nebius compute instance list && nebius compute disk list   # both empty = billing stopped

Order matters: delete the instance before the disk, and read the disk id before deleting the instance.

What’s next

That’s four modalities through one engine, end to end , exactly what didn’t fit in my vllm-omni talk at conf42.

📚 Go deeper

If you want to learn more about diffusion models and inference check the deep dives I created specifically aroun this topic:

Coming up next

An AI-gateway approach to inference traffic: routing through the vLLM Production Stack with LiteLLM, adding LLM Guard and token-based rate limiting, on Nebius. Stay tuned.

Slides and the full talk: 🎙️ How vLLM-Omni Unifies Multimodal Inference. Built on open-source vLLM-Omni.

Run AI Your Way — In Your Cloud

Want full control over your AI backend? The CloudThrill VLLM Private Inference POC is still open — but not forever.

📢 Secure your spot (only a few left), 𝗔𝗽𝗽𝗹𝘆 𝗻𝗼𝘄!

Run AI assistants, RAG, or internal models on an AI backend 𝗽𝗿𝗶𝘃𝗮𝘁𝗲𝗹𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗰𝗹𝗼𝘂𝗱 –
✅ No external APIs
✅ No vendor lock-in
✅ Total data control

Claim YOur FREE VLLM POC

𝗬𝗼𝘂𝗿 𝗶𝗻𝗳𝗿𝗮. 𝗬𝗼𝘂𝗿 𝗺𝗼𝗱𝗲𝗹𝘀. 𝗬𝗼𝘂𝗿 𝗿𝘂𝗹𝗲𝘀…

Share this…

Don't miss a Bit!

Join countless others!
Sign up and get awesome cloud content straight to your inbox. 🚀

Start your Cloud journey with us today .