Qwen2.5

Run Alibaba's Qwen2.5 family of models - powerful multilingual LLMs with excellent code and math capabilities on CLORE.AI GPUs.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

Why Qwen2.5?

• Versatile sizes - 0.5B to 72B parameters

• Multilingual - 29 languages including Chinese

• Long context - Up to 128K tokens

• Specialized variants - Coder, Math editions

• Open source - Apache 2.0 license

Quick Deploy on CLORE.AI

Docker Image:

vllm/vllm-openai:latest

Ports:

22/tcp
8000/http

Command:

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 \
    --port 8000

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

Verify It's Working

# Check if service is ready
curl https://your-http-pub.clorecloud.net/health

# List available models
curl https://your-http-pub.clorecloud.net/v1/models

If you get HTTP 502, wait 5-15 minutes - the model is still downloading from HuggingFace.

Qwen3 Reasoning Mode

New in Qwen3: Some Qwen3 models support a reasoning mode that shows the model's thought process in <think> tags before the final answer.

When using Qwen3 models via vLLM, responses may include reasoning:

{
  "content": "<think>\nLet me think about this step by step...\n</think>\n\nThe answer is..."
}

To use Qwen3 with reasoning:

vllm serve Qwen/Qwen3-0.6B --host 0.0.0.0 --port 8000

Model Variants

Base Models

Model	Parameters	VRAM (FP16)	Context
Qwen2.5-0.5B	0.5B	2GB	32K
Qwen2.5-1.5B	1.5B	4GB	32K
Qwen2.5-3B	3B	8GB	32K
Qwen2.5-7B	7B	16GB	128K
Qwen2.5-14B	14B	32GB	128K
Qwen2.5-32B	32B	70GB	128K
Qwen2.5-72B	72B	150GB	128K

Specialized Variants

Model	Focus	Best For
Qwen2.5-Coder	Code	Programming, debugging
Qwen2.5-Math	Mathematics	Calculations, proofs
Qwen2.5-Instruct	Chat	General assistant

Hardware Requirements

Model	Minimum GPU	Recommended
0.5B-3B	RTX 3060 12GB	RTX 3080
7B	RTX 3090 24GB	RTX 4090
14B	A100 40GB	A100 80GB
32B	A100 80GB	2x A100 40GB
72B	2x A100 80GB	4x A100 80GB

Installation

Using vLLM (Recommended)

pip install vllm

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 \
    --port 8000

Using Ollama

# Pull model
ollama pull qwen2.5:7b

# Run chat
ollama run qwen2.5:7b

Using Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "Qwen/Qwen2.5-7B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

API Usage

OpenAI-Compatible API

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain machine learning in simple terms."}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Write a poem about AI"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

cURL

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-7B-Instruct",
        "messages": [
            {"role": "user", "content": "What is Python?"}
        ]
    }'

Qwen2.5-Coder

Optimized for code generation:

# Using vLLM
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-Coder-7B-Instruct \
    --host 0.0.0.0

# Using Ollama
ollama run qwen2.5-coder:7b

prompt = """Write a Python function that:
1. Takes a list of numbers
2. Returns the median value
3. Handles empty lists gracefully
Include type hints and docstrings."""

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Coder-7B-Instruct",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.2
)

print(response.choices[0].message.content)

Qwen2.5-Math

Specialized for mathematical reasoning:

# Using vLLM
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-Math-7B-Instruct \
    --host 0.0.0.0

prompt = """Solve step by step:
Find all values of x where: x^3 - 6x^2 + 11x - 6 = 0"""

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Math-7B-Instruct",
    messages=[{"role": "user", "content": prompt}],
    temperature=0.1
)

print(response.choices[0].message.content)

Multilingual Support

Qwen2.5 supports 29 languages:

# Chinese
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "用中文解释什么是人工智能"}]
)

# Japanese
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "人工知能について日本語で説明してください"}]
)

# Korean
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "인공지능에 대해 한국어로 설명해주세요"}]
)

Long Context (128K)

# Read a long document
with open("long_document.txt", "r") as f:
    document = f.read()

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": f"Summarize this document:\n\n{document}"}
    ],
    max_tokens=2000
)

Quantization

GGUF with Ollama

# 4-bit quantized
ollama pull qwen2.5:7b-instruct-q4_K_M

# 8-bit quantized
ollama pull qwen2.5:7b-instruct-q8_0

AWQ with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-72B-Instruct-AWQ \
    --quantization awq \
    --tensor-parallel-size 2

GGUF with llama.cpp

# Download GGUF
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf

# Run server
./llama-server -m qwen2.5-7b-instruct-q4_k_m.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 35

Multi-GPU Setup

Tensor Parallelism

# 72B on 4 GPUs
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-72B-Instruct \
    --tensor-parallel-size 4 \
    --max-model-len 32768

# 32B on 2 GPUs
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-32B-Instruct \
    --tensor-parallel-size 2

Performance

Throughput (tokens/sec)

Model	RTX 3090	RTX 4090	A100 40GB	A100 80GB
Qwen2.5-0.5B	250	320	380	400
Qwen2.5-3B	150	200	250	280
Qwen2.5-7B	75	100	130	150
Qwen2.5-7B Q4	110	140	180	200
Qwen2.5-14B	-	55	70	85
Qwen2.5-32B	-	-	35	50
Qwen2.5-72B	-	-	20 (2x)	40 (2x)

Time to First Token (TTFT)

Model	RTX 4090	A100 40GB	A100 80GB
7B	60ms	40ms	35ms
14B	120ms	80ms	60ms
32B	-	200ms	140ms
72B	-	400ms (2x)	280ms (2x)

Context Length vs VRAM (7B)

Context	FP16	Q8	Q4
8K	16GB	10GB	6GB
32K	24GB	16GB	10GB
64K	40GB	26GB	16GB
128K	72GB	48GB	28GB

Benchmarks

Model	MMLU	HumanEval	GSM8K	MATH
Qwen2.5-7B	74.2%	75.6%	85.4%	55.2%
Qwen2.5-14B	79.7%	81.1%	89.5%	65.8%
Qwen2.5-32B	83.3%	84.2%	91.2%	72.1%
Qwen2.5-72B	86.1%	86.2%	93.2%	79.5%

Docker Compose

version: '3.8'

services:
  qwen:
    image: vllm/vllm-openai:latest
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model Qwen/Qwen2.5-7B-Instruct
      --host 0.0.0.0
      --port 8000
      --gpu-memory-utilization 0.9
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Cost Estimate

Typical CLORE.AI marketplace rates:

GPU	Hourly Rate	Best For
RTX 3090 24GB	~$0.06	7B models
RTX 4090 24GB	~$0.10	7B-14B models
A100 40GB	~$0.17	14B-32B models
A100 80GB	~$0.25	32B models
2x A100 80GB	~$0.50	72B models

Prices vary by provider. Check CLORE.AI Marketplace for current rates.

Save money:

• Use Spot market for flexible workloads

• Pay with CLORE tokens

• Start with smaller models (7B) for testing

Troubleshooting

Out of Memory

# Reduce context
--max-model-len 8192

# Enable memory optimization
--gpu-memory-utilization 0.85

# Use quantized model
ollama pull qwen2.5:7b-instruct-q4_K_M

Slow Generation

# Enable flash attention
pip install flash-attn

# Use vLLM for better throughput
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --enable-prefix-caching

Chinese Characters Display

# Ensure UTF-8 encoding
import sys
sys.stdout.reconfigure(encoding='utf-8')

Model Not Found

# Check model name
huggingface-cli search Qwen/Qwen2.5

# Common names:
# Qwen/Qwen2.5-7B-Instruct
# Qwen/Qwen2.5-Coder-7B-Instruct
# Qwen/Qwen2.5-Math-7B-Instruct

Qwen2.5 vs Others

Feature	Qwen2.5-7B	Llama 3.1 8B	Mistral 7B
Context	128K	128K	32K
Multilingual	Excellent	Good	Good
Code	Excellent	Good	Good
Math	Excellent	Good	Good
Chinese	Excellent	Poor	Poor
License	Apache 2.0	Llama 3.1	Apache 2.0

Use Qwen2.5 when:

• Chinese language support needed

• Math/code tasks are priority

• Long context is required

• Want Apache 2.0 license

Next Steps

• vLLM - Production deployment

• Ollama - Easy local setup

• DeepSeek-V3 - Larger reasoning model

• Fine-tune LLM - Custom training