vLLM

High-throughput LLM inference server for production workloads on CLORE.AI GPUs.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

Server Requirements

Parameter	Minimum	Recommended
RAM	16GB	32GB+
VRAM	16GB (7B)	24GB+
Network	500Mbps	1Gbps+
Startup Time	5-15 minutes	-

Important: vLLM requires significant RAM and VRAM. Servers with less than 16GB RAM will fail to run even 7B models.

Startup Time: The first launch downloads the model from HuggingFace (5-15 minutes depending on model size and network speed). HTTP 502 during this time is normal.

Why vLLM?

• Fastest throughput - PagedAttention for 24x higher throughput

• Production ready - OpenAI-compatible API out of the box

• Continuous batching - Efficient multi-user serving

• Streaming - Real-time token generation

• Multi-GPU - Tensor parallelism for large models

Quick Deploy on CLORE.AI

Docker Image:

vllm/vllm-openai:latest

Ports:

22/tcp
8000/http

Command:

vllm serve mistralai/Mistral-7B-Instruct-v0.2 --host 0.0.0.0 --port 8000

Verify It's Working

After deployment, find your http_pub URL in My Orders:

# Check health (may take 5-15 min on first run)
curl https://your-http-pub.clorecloud.net/health

# List models (only works after model loads)
curl https://your-http-pub.clorecloud.net/v1/models

If you get HTTP 502 for more than 15 minutes, check:

1. Server has 16GB+ RAM

2. Server has enough VRAM for the model

3. HuggingFace token is set for gated models

Accessing Your Service

When deployed on CLORE.AI, access vLLM via the http_pub URL:

# Chat completion
curl https://your-http-pub.clorecloud.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

All localhost:8000 examples below work when connected via SSH. For external access, replace with your https://your-http-pub.clorecloud.net/ URL.

Installation

Using Docker (Recommended)

docker run -d --gpus all \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --host 0.0.0.0

Using pip

pip install vllm

# Run server
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.2

Supported Models

Model	Parameters	VRAM Required	RAM Required
Mistral 7B	7B	14GB	16GB+
Llama 3.1 8B	8B	16GB	16GB+
Llama 3.1 70B	70B	140GB (or 2x80GB)	64GB+
Mixtral 8x7B	47B	90GB	32GB+
Qwen2.5 7B	7B	14GB	16GB+
Qwen2.5 72B	72B	145GB	64GB+
DeepSeek-V2	236B	Multi-GPU	128GB+
Phi-4	14B	28GB	32GB+
Gemma 2 9B	9B	18GB	16GB+
CodeLlama 34B	34B	68GB	32GB+

Server Options

Basic Server

vllm serve mistralai/Mistral-7B-Instruct-v0.2 \
    --host 0.0.0.0 \
    --port 8000

Production Server

vllm serve mistralai/Mistral-7B-Instruct-v0.2 \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --max-num-seqs 256 \
    --enable-prefix-caching

With Quantization (Lower VRAM)

# AWQ quantized model (uses less VRAM)
vllm serve TheBloke/Mistral-7B-Instruct-v0.2-AWQ \
    --host 0.0.0.0 \
    --quantization awq

API Usage

Chat Completions (OpenAI Compatible)

from openai import OpenAI

# For external access, use your http_pub URL:
client = OpenAI(
    base_url="https://your-http-pub.clorecloud.net/v1",
    api_key="not-needed"
)

# Or via SSH tunnel:
# client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[
        {"role": "user", "content": "Explain quantum computing"}
    ],
    max_tokens=500,
    temperature=0.7
)

print(response.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[{"role": "user", "content": "Write a poem"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

cURL

curl https://your-http-pub.clorecloud.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Text Completions

curl https://your-http-pub.clorecloud.net/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "prompt": "The capital of France is",
    "max_tokens": 50
  }'

Complete API Reference

vLLM provides OpenAI-compatible endpoints plus additional utility endpoints.

Standard Endpoints

Endpoint	Method	Description
/v1/models	GET	List available models
/v1/chat/completions	POST	Chat completion
/v1/completions	POST	Text completion
/health	GET	Health check (may return empty)

Additional Endpoints

Endpoint	Method	Description
/tokenize	POST	Tokenize text
/detokenize	POST	Convert tokens to text
/version	GET	Get vLLM version
/docs	GET	Swagger UI documentation
/metrics	GET	Prometheus metrics

Tokenize Text

Useful for counting tokens before sending requests:

curl https://your-http-pub.clorecloud.net/tokenize \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "prompt": "Hello world"
  }'

Response:

{"count": 2, "max_model_len": 32768, "tokens": [9707, 1879]}

Detokenize

Convert token IDs back to text:

curl https://your-http-pub.clorecloud.net/detokenize \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.2",
    "tokens": [9707, 1879]
  }'

Response:

{"prompt": "Hello world"}

Get Version

curl https://your-http-pub.clorecloud.net/version

Response:

{"version": "0.14.0"}

Swagger Documentation

Open in browser for interactive API documentation:

https://your-http-pub.clorecloud.net/docs

Prometheus Metrics

For monitoring:

curl https://your-http-pub.clorecloud.net/metrics

Reasoning Models: Some models like Qwen3 support reasoning mode and may include <think> tags in responses showing the model's reasoning process.

Benchmarks

Throughput (tokens/sec per user)

Model	RTX 3090	RTX 4090	A100 40GB	A100 80GB
Mistral 7B	100	170	210	230
Llama 3.1 8B	95	150	200	220
Llama 3.1 8B (AWQ)	130	190	260	280
Mixtral 8x7B	-	45	70	85
Llama 3.1 70B	-	-	25 (2x)	45 (2x)

Benchmarks updated January 2026.

Context Length vs VRAM

Model	4K ctx	8K ctx	16K ctx	32K ctx
8B FP16	18GB	22GB	30GB	46GB
8B AWQ	8GB	10GB	14GB	22GB
70B FP16	145GB	160GB	190GB	250GB
70B AWQ	42GB	50GB	66GB	98GB

Hugging Face Authentication

For gated models (Llama, etc.):

# Set token in command
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --env HUGGING_FACE_HUB_TOKEN=hf_xxxxx

Or set as environment variable:

export HUGGING_FACE_HUB_TOKEN=hf_xxxxx

GPU Requirements

Model	Min VRAM	Min RAM	Recommended
7-8B	16GB	16GB	24GB VRAM, 32GB RAM
13B	26GB	32GB	40GB VRAM
34B	70GB	32GB	80GB VRAM
70B	140GB	64GB	2x80GB

Cost Estimate

Typical CLORE.AI marketplace rates:

GPU	CLORE/day	Approx USD/hr	Best For
RTX 3090 24GB	~150	~$0.03	7-8B models
RTX 4090 24GB	~200	~$0.04	7-13B, fast
A100 40GB	~400	~$0.08	13-34B models
A100 80GB	~600	~$0.12	34-70B models

Prices vary by provider. Check CLORE.AI Marketplace for current rates.

Troubleshooting

HTTP 502 for a long time

1. Check RAM: Server must have 16GB+ RAM

2. Check VRAM: Must fit the model

3. Model downloading: First run downloads from HuggingFace (5-15 min)

4. HF Token: Gated models require authentication

Out of Memory

# Reduce memory usage
--gpu-memory-utilization 0.8
--max-model-len 4096
--max-num-seqs 64

# Or use quantization
--quantization awq

Model Download Fails

# Check HF token
echo $HUGGING_FACE_HUB_TOKEN

# Pre-download model
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.2

vLLM vs Others

Feature	vLLM	llama.cpp	Ollama
Throughput	Best	Good	Good
VRAM Usage	High	Low	Medium
Ease of Use	Medium	Medium	Easy
Startup Time	5-15 min	1-2 min	30 sec
Multi-GPU	Native	Limited	Limited

Use vLLM when:

• High throughput is priority

• Serving multiple users

• Have enough VRAM and RAM

• Production deployment

Use Ollama when:

• Quick setup needed

• Single user

• Less resources available

Next Steps

• Ollama - Simpler alternative with faster startup

• DeepSeek-V3 - Best reasoning model

• Qwen2.5 - Multilingual models

• Llama.cpp - Lower VRAM option