LMDeploy

Efficient LLM deployment toolkit by Shanghai AI Lab — production-grade inference, quantization, and serving for large language models with continuous batching and PagedAttention.

🏛️ Developed by OpenMMLab / Shanghai AI Lab | Apache-2.0 License | 4,000+ GitHub stars

What is LMDeploy?

LMDeploy is a comprehensive toolkit for compressing, deploying, and serving Large Language Models in production. Built by the same team behind OpenMMLab (MMDetection, MMSeg), it brings research-grade optimizations to practical deployment:

TurboMind engine — high-performance C++ inference backend with CUDA optimizations
PyTorch engine — flexible Python-based engine for broad model compatibility
Continuous batching — maximizes GPU utilization across concurrent requests
PagedAttention — efficient KV cache management (similar to vLLM)
4-bit / 8-bit quantization — AWQ and SmoothQuant support
Vision-Language Models — InternVL, LLaVA, Qwen-VL support

Compared to vLLM, LMDeploy's TurboMind engine delivers ~1.36× higher throughput on Llama 3 8B at batch=32, and its AWQ quantization is first-class — not an afterthought. For VLMs (especially InternVL2), LMDeploy is the reference deployment stack.

Why LMDeploy?

Feature

LMDeploy

vLLM

TGI

Continuous batching

✅

AWQ quantization

✅

❌

Speculative decoding

✅

Vision-Language

✅

Limited

OpenAI API

✅

TurboMind (custom engine)

✅

❌

Quick Start on Clore.ai

Step 1: Select a GPU Server

On clore.ai marketplace:

Minimum: NVIDIA GPU with 8GB VRAM (for 7B models)
Recommended: RTX 3090/4090 (24GB) or A100 (40/80GB)
CUDA: 11.8 or 12.x required

Step 2: Deploy LMDeploy Docker

Docker Image: openmmlab/lmdeploy

Port mappings:

Container Port

Purpose

22

SSH access

23333

LMDeploy API server

Environment variables:

HUGGING_FACE_HUB_TOKEN=your_hf_token_here  # For gated models

Step 3: SSH and Verify

ssh root@<clore-node-ip> -p <ssh-port>

# Verify installation
python -c "import lmdeploy; print(lmdeploy.__version__)"
lmdeploy --help

Starting the API Server

OpenAI-Compatible Server (Recommended)

# Serve Llama 3 8B with TurboMind engine
lmdeploy serve api_server \
  meta-llama/Meta-Llama-3-8B-Instruct \
  --server-port 23333 \
  --server-name 0.0.0.0 \
  --model-name llama3-8b

# With explicit engine selection
lmdeploy serve api_server \
  meta-llama/Meta-Llama-3-8B-Instruct \
  --backend turbomind \
  --server-port 23333 \
  --server-name 0.0.0.0 \
  --tp 1 \
  --max-batch-size 128 \
  --cache-max-entry-count 0.8

PyTorch Engine (Broader Compatibility)

# Use PyTorch engine for models not supported by TurboMind
lmdeploy serve api_server \
  mistralai/Mistral-7B-Instruct-v0.2 \
  --backend pytorch \
  --server-port 23333 \
  --server-name 0.0.0.0

Server Startup Output

[2024-01-01 12:00:00,000] INFO: Loading model: meta-llama/Meta-Llama-3-8B-Instruct
[2024-01-01 12:00:20,000] INFO: TurboMind engine initialized
[2024-01-01 12:00:20,000] INFO: Server started at http://0.0.0.0:23333
[2024-01-01 12:00:20,000] INFO: API docs: http://0.0.0.0:23333/docs

Once started, LMDeploy exposes interactive API docs at http://<your-ip>:23333/docs — useful for testing endpoints directly from the browser.

Supported Models

Text Models

# Llama 3
meta-llama/Meta-Llama-3-8B-Instruct
meta-llama/Meta-Llama-3-70B-Instruct

# Mistral / Mixtral
mistralai/Mistral-7B-Instruct-v0.2
mistralai/Mixtral-8x7B-Instruct-v0.1

# Qwen
Qwen/Qwen2-7B-Instruct
Qwen/Qwen2-72B-Instruct

# InternLM
internlm/internlm2-chat-7b
internlm/internlm2-chat-20b

# Yi
01-ai/Yi-1.5-9B-Chat
01-ai/Yi-1.5-34B-Chat

# Gemma
google/gemma-7b-it
google/gemma-2b-it

Vision-Language Models

# InternVL (recommended VLM)
OpenGVLab/InternVL2-8B
OpenGVLab/InternVL2-26B

# LLaVA
llava-hf/llava-1.5-7b-hf

# Qwen-VL
Qwen/Qwen-VL-Chat

Quantization

AWQ 4-bit Quantization

LMDeploy's AWQ (Activation-aware Weight Quantization) produces excellent quality at 4-bit:

# Quantize a model to AWQ 4-bit
lmdeploy lite auto_awq \
  meta-llama/Meta-Llama-3-8B-Instruct \
  --calib-dataset ptb \
  --calib-samples 128 \
  --calib-seqlen 2048 \
  --w-bits 4 \
  --w-group-size 128 \
  --work-dir ./quantized/llama3-8b-awq

# Serve the quantized model
lmdeploy serve api_server \
  ./quantized/llama3-8b-awq \
  --server-port 23333 \
  --server-name 0.0.0.0

SmoothQuant W8A8

8-bit weight and activation quantization (better for throughput-critical deployments):

lmdeploy lite smooth_quant \
  meta-llama/Meta-Llama-3-8B-Instruct \
  --work-dir ./quantized/llama3-8b-sq \
  --calib-dataset ptb \
  --calib-samples 512

Quantization Impact

Quantization

VRAM (7B)

Quality Loss

Throughput Gain

None (bf16)

~14GB

None

Baseline

SmoothQuant W8A8

~8GB

Minimal

+20%

AWQ W4A16

~4GB

Low

+15%

GPTQ W4A16

~4GB

Low

+10%

AWQ recommendation: For most use cases, AWQ 4-bit is the best balance of quality and VRAM savings. Use --w-group-size 128 for better quality at slightly higher memory usage.

API Usage Examples

Python Client

from openai import OpenAI

client = OpenAI(
    base_url="http://<clore-node-ip>:<api-port>/v1",
    api_key="none"
)

# Chat completion
response = client.chat.completions.create(
    model="llama3-8b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize the history of AI in 3 sentences."}
    ],
    temperature=0.7,
    max_tokens=512
)
print(response.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
    model="llama3-8b",
    messages=[{"role": "user", "content": "Write a poem about space."}],
    stream=True
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
print()

LMDeploy Native Python Client

from lmdeploy import pipeline, TurbomindEngineConfig

# Direct pipeline (no server needed)
pipe = pipeline(
    'meta-llama/Meta-Llama-3-8B-Instruct',
    backend_config=TurbomindEngineConfig(max_batch_size=16)
)

# Single inference
response = pipe("What is the capital of France?")
print(response.text)

# Batch inference
responses = pipe([
    "Explain gravity",
    "What is DNA?",
    "How does Bitcoin work?"
])
for r in responses:
    print(r.text)
    print("---")

Vision-Language Model

from lmdeploy import pipeline
from lmdeploy.vl import load_image

pipe = pipeline('OpenGVLab/InternVL2-8B')

image = load_image('https://example.com/photo.jpg')
response = pipe(('Describe this image in detail', image))
print(response.text)

Multi-GPU Deployment

Tensor Parallelism

# Distribute a 70B model across 4 GPUs
lmdeploy serve api_server \
  meta-llama/Meta-Llama-3-70B-Instruct \
  --backend turbomind \
  --server-port 23333 \
  --server-name 0.0.0.0 \
  --tp 4 \
  --max-batch-size 64

from lmdeploy import pipeline, TurbomindEngineConfig

pipe = pipeline(
    'meta-llama/Meta-Llama-3-70B-Instruct',
    backend_config=TurbomindEngineConfig(tp=4)
)

Advanced Configuration

TurboMind Engine Config

from lmdeploy import pipeline, TurbomindEngineConfig

engine_config = TurbomindEngineConfig(
    max_batch_size=64,          # Maximum concurrent requests
    cache_max_entry_count=0.8,  # KV cache ratio (0.0-1.0)
    quant_policy=0,             # 0=no quant, 4=4bit KV cache, 8=8bit KV cache
    rope_scaling_factor=1.0,    # For extended context
    num_tokens_per_iter=4096,   # Prefill chunk size
    max_prefill_token_num=8192, # Max prefill length
)

pipe = pipeline('meta-llama/Meta-Llama-3-8B-Instruct', backend_config=engine_config)

Generation Config

from lmdeploy import GenerationConfig

gen_config = GenerationConfig(
    temperature=0.7,
    top_p=0.9,
    top_k=40,
    repetition_penalty=1.1,
    max_new_tokens=1024,
    stop_words=['<|eot_id|>', '<|end_of_text|>'],
)

response = pipe("Hello, world!", gen_config=gen_config)

Monitoring & Metrics

Check Server Health

# Health check endpoint
curl http://localhost:23333/health

# List available models
curl http://localhost:23333/v1/models

# Server statistics
curl http://localhost:23333/stats

GPU Monitoring

# Real-time GPU stats
watch -n 1 'nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu --format=csv'

Docker Compose Example

version: '3.8'
services:
  lmdeploy:
    image: openmmlab/lmdeploy:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    ports:
      - "23333:23333"
      - "22:22"
    volumes:
      - hf-cache:/root/.cache/huggingface
      - ./models:/models
    command: >
      lmdeploy serve api_server
      meta-llama/Meta-Llama-3-8B-Instruct
      --server-port 23333
      --server-name 0.0.0.0
      --model-name llama3-8b
      --max-batch-size 64
    restart: unless-stopped
    shm_size: '2g'

volumes:
  hf-cache:

Benchmarking

# Built-in benchmark tool
lmdeploy benchmark \
  meta-llama/Meta-Llama-3-8B-Instruct \
  --backend turbomind \
  --concurrency 1 4 8 16 32 \
  --num-prompts 1000 \
  --prompt-len 128 \
  --output-len 256

Sample output (RTX 4090, TurboMind, bf16):

concurrency=1:  throughput=42.3 tokens/s, latency_p50=23ms
concurrency=8:  throughput=287.1 tokens/s, latency_p50=156ms
concurrency=32: throughput=412.6 tokens/s, latency_p50=621ms

On A100 80GB, expect ~2.2× higher throughput vs RTX 4090 at high concurrency due to HBM2e memory bandwidth (2 TB/s vs 1 TB/s).

Clore.ai GPU Recommendations

Choose based on your target model size and serving load:

Use Case

GPU

VRAM

Why

7–13B models, dev/staging

RTX 3090

24 GB

Best $/VRAM ratio; handles 7B bf16 or 13B AWQ

7–13B models, production

RTX 4090

24 GB

~40% faster than 3090 at same VRAM; 412 tok/s on Llama 3 8B

70B models, team serving

A100 40GB

40 GB

Fits 70B AWQ; ECC memory for reliability

70B models, high throughput

A100 80GB

80 GB

Fits 70B bf16; 2× throughput vs A100 40GB at batch=32

Budget pick: RTX 3090 + AWQ 4-bit — serves Llama 3 8B at ~280 tok/s batch=8, covers most API use cases.

Speed pick: RTX 4090 — fastest per-dollar for 7–13B models; TurboMind squeezes out every GB/s of its 1 TB/s bandwidth.

Production pick: A100 80GB — run Qwen2-72B or Llama 3 70B in full bf16 without quantization quality tradeoffs; fits easily into multi-instance GPU serving.

Troubleshooting

Model Not Loading

# Check HuggingFace token is set
echo $HUGGING_FACE_HUB_TOKEN

# Manually download model
pip install huggingface_hub
huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct --local-dir ./llama3-8b

# Use local path instead
lmdeploy serve api_server ./llama3-8b --server-port 23333

CUDA Out of Memory

# Reduce KV cache allocation
lmdeploy serve api_server MODEL \
  --cache-max-entry-count 0.5  # Reduce from 0.8

# Use quantized KV cache
lmdeploy serve api_server MODEL \
  --quant-policy 8  # 8-bit KV cache

Port Already in Use

# Check what's using port 23333
ss -tlnp | grep 23333
fuser 23333/tcp

# Kill existing process
kill -9 $(fuser 23333/tcp)

Docker network mode: When running in Docker, ensure the container uses --network host or proper port mapping (-p 23333:23333) so the API is reachable from outside.

Clore.ai GPU Recommendations

LMDeploy's TurboMind engine and W4A16 quantization deliver best-in-class throughput — especially on Ampere/Hopper GPUs.

GPU

VRAM

Clore.ai Price

Llama 3 8B Throughput

Llama 3 70B Q4

RTX 3090

24 GB

~$0.12/hr

~120 tok/s (fp16)

❌ Too large

RTX 4090

24 GB

~$0.70/hr

~200 tok/s (fp16)

❌ Too large

A100 40GB

40 GB

~$1.20/hr

~160 tok/s (fp16)

~55 tok/s (W4A16)

A100 80GB

80 GB

~$2.00/hr

~175 tok/s (fp16)

~80 tok/s (fp16)

2× RTX 4090

48 GB

~$1.40/hr

~380 tok/s (tensor parallel)

~60 tok/s

RTX 3090 at ~$0.12/hr is the top choice for 7B–13B models. LMDeploy's TurboMind engine extracts near-maximum throughput from consumer GPUs. A single RTX 3090 serving Llama 3 8B handles 120 tok/s — sufficient for production APIs with 10–20 concurrent users.

For 70B models: A100 40GB (~$1.20/hr) with W4A16 quantization delivers ~55 tok/s — more cost-effective than two RTX 4090s.

Resources

📦 Docker Hub: hub.docker.com/r/openmmlab/lmdeploy
🐙 GitHub: github.com/InternLM/lmdeploy
📚 Documentation: lmdeploy.readthedocs.io
💬 Discord: discord.gg/xa29JuW84p
🤗 Pre-quantized Models: huggingface.co/lmdeploy

PreviousPowerInfer NextMistral.rs

Last updated 23 hours ago

Was this helpful?

hashtagWhat is LMDeploy?

hashtagWhy LMDeploy?

hashtagQuick Start on Clore.ai

hashtagStep 1: Select a GPU Server

hashtagStep 2: Deploy LMDeploy Docker

hashtagStep 3: SSH and Verify

hashtagStarting the API Server

hashtagOpenAI-Compatible Server (Recommended)

hashtagPyTorch Engine (Broader Compatibility)

hashtagServer Startup Output

hashtagSupported Models

hashtagText Models

hashtagVision-Language Models

hashtagQuantization

hashtagAWQ 4-bit Quantization

hashtagSmoothQuant W8A8

hashtagQuantization Impact

hashtagAPI Usage Examples

hashtagPython Client

hashtagStreaming

hashtagLMDeploy Native Python Client

hashtagVision-Language Model

hashtagMulti-GPU Deployment

hashtagTensor Parallelism

hashtagAdvanced Configuration

hashtagTurboMind Engine Config

hashtagGeneration Config

hashtagMonitoring & Metrics

hashtagCheck Server Health

hashtagGPU Monitoring

hashtagDocker Compose Example

hashtagBenchmarking

hashtagClore.ai GPU Recommendations

hashtagTroubleshooting

hashtagModel Not Loading

hashtagCUDA Out of Memory

hashtagPort Already in Use

hashtagClore.ai GPU Recommendations

hashtagResources

What is LMDeploy?

Why LMDeploy?

Quick Start on Clore.ai

Step 1: Select a GPU Server

Step 2: Deploy LMDeploy Docker

Step 3: SSH and Verify

Starting the API Server

OpenAI-Compatible Server (Recommended)

PyTorch Engine (Broader Compatibility)

Server Startup Output

Supported Models

Text Models

Vision-Language Models

Quantization

AWQ 4-bit Quantization

SmoothQuant W8A8

Quantization Impact

API Usage Examples

Python Client

Streaming

LMDeploy Native Python Client

Vision-Language Model

Multi-GPU Deployment

Tensor Parallelism

Advanced Configuration

TurboMind Engine Config

Generation Config

Monitoring & Metrics

Check Server Health

GPU Monitoring

Docker Compose Example

Benchmarking

Clore.ai GPU Recommendations

Troubleshooting

Model Not Loading

CUDA Out of Memory

Port Already in Use

Clore.ai GPU Recommendations

Resources