MLC-LLM

Universal LLM deployment through ML Compilation — run any large language model on any hardware with maximum performance using machine learning compilation.

🌟 20,000+ GitHub stars | Maintained by the MLC AI team | Apache-2.0 License

---

What is MLC-LLM?

MLC-LLM (Machine Learning Compilation for Large Language Models) is a universal framework that enables efficient deployment of large language models across diverse hardware backends. By leveraging TVM (Tensor Virtual Machine) as its compilation backend, MLC-LLM compiles LLM models directly to native hardware code — achieving near-optimal performance without hardware-specific engineering.

Key Capabilities

• Universal hardware support — NVIDIA CUDA, AMD ROCm, Apple Metal, Vulkan, WebGPU

• OpenAI-compatible REST API — drop-in replacement for existing workflows

• Multiple model formats — Llama, Mistral, Gemma, Phi, Qwen, Falcon, and more

• 4-bit / 8-bit quantization — run large models on consumer GPUs

• Chat interface — built-in web UI for immediate testing

• Python & CLI tools — flexible integration options

Why Use MLC-LLM on Clore.ai?

Clore.ai GPU marketplace gives you access to high-performance NVIDIA GPUs at competitive rental rates. MLC-LLM's compilation approach squeezes maximum throughput from every GPU — making it ideal for:

• Production API inference at scale

• Research and benchmarking across model sizes

• Cost-efficient serving with quantized models

• Multi-model deployment on a single GPU instance

---

Quick Start on Clore.ai

Step 1: Find a GPU Server

1. Go to clore.ai marketplace

2. Filter servers: NVIDIA GPU, minimum 8GB VRAM (16GB+ recommended for 7B+ models)

3. For optimal performance: RTX 3090, RTX 4090, A100, or H100

Step 2: Deploy MLC-LLM

Note: MLC-LLM does not publish an official pre-built Docker image to Docker Hub. The recommended deployment approach is to use an NVIDIA CUDA base image and install MLC-LLM via pip. Use nvidia/cuda:12.1.0-devel-ubuntu22.04 as your base image on Clore.ai.

Use an NVIDIA CUDA base image in your Clore.ai order configuration:

Docker Image: nvidia/cuda:12.1.0-devel-ubuntu22.04

Port mappings:

Container Port	Purpose
22	SSH access
8000	REST API server

Recommended environment variables:

MLC_MODEL=HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
MLC_HOST=0.0.0.0
MLC_PORT=8000

Startup script (run after SSH):

pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu121 mlc-ai-nightly-cu121

Step 3: Connect via SSH

ssh root@<clore-node-ip> -p <assigned-ssh-port>

---

Installation & Setup

Option A: Use Pre-compiled Models (Fastest)

MLC-AI maintains a library of pre-compiled models on Hugging Face. No compilation needed:

# Pull and run a pre-compiled Llama 3 8B (4-bit quantized)
python -m mlc_llm serve HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC \
  --host 0.0.0.0 \
  --port 8000

Option B: Compile Your Own Model

For custom models or specific quantization requirements:

# Step 1: Convert model weights
python -m mlc_llm convert_weight \
  ./path/to/model \
  --quantization q4f16_1 \
  --output ./compiled/model-q4f16_1

# Step 2: Generate model configuration
python -m mlc_llm gen_config \
  ./path/to/model \
  --quantization q4f16_1 \
  --conv-template llama-3 \
  --output ./compiled/model-q4f16_1

# Step 3: Compile the model
python -m mlc_llm compile \
  ./compiled/model-q4f16_1/mlc-chat-config.json \
  --device cuda \
  --output ./compiled/model-q4f16_1/lib.so

Compilation time: Compiling a 7B model typically takes 10–30 minutes on first run. Compiled artifacts are cached and reused on subsequent launches.

---

Running the API Server

Start the OpenAI-Compatible Server

python -m mlc_llm serve \
  HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC \
  --host 0.0.0.0 \
  --port 8000 \
  --max-batch-size 4 \
  --max-total-sequence-length 8192

Server Startup Output

[2024-01-01 12:00:00] INFO: Loading model from HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
[2024-01-01 12:00:15] INFO: Model loaded successfully
[2024-01-01 12:00:15] INFO: Starting server on 0.0.0.0:8000
[2024-01-01 12:00:15] INFO: OpenAI-compatible API available at http://0.0.0.0:8000/v1

Available API Endpoints

Endpoint	Method	Description
/v1/chat/completions	POST	Chat completions (OpenAI format)
/v1/completions	POST	Text completions
/v1/models	GET	List available models
/v1/debug/dump_event_trace	GET	Performance debugging

---

API Usage Examples

Chat Completions (Python)

from openai import OpenAI

# Point to your Clore.ai server
client = OpenAI(
    base_url="http://<clore-node-ip>:<api-port>/v1",
    api_key="none"  # MLC-LLM doesn't require auth by default
)

response = client.chat.completions.create(
    model="Llama-3-8B-Instruct-q4f16_1-MLC",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    temperature=0.7,
    max_tokens=512
)

print(response.choices[0].message.content)

Streaming Response

stream = client.chat.completions.create(
    model="Llama-3-8B-Instruct-q4f16_1-MLC",
    messages=[{"role": "user", "content": "Write a short story about AI."}],
    stream=True,
    max_tokens=1024
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

cURL Example

curl http://<clore-node-ip>:<api-port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Llama-3-8B-Instruct-q4f16_1-MLC",
    "messages": [
      {"role": "user", "content": "What is 2+2?"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'

---

Available Pre-compiled Models

MLC-AI provides ready-to-use compiled models on Hugging Face:

Llama 3 Series

# 8B Instruct (recommended for most use cases)
HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC

# 70B Instruct (requires 40GB+ VRAM or multi-GPU)
HF://mlc-ai/Llama-3-70B-Instruct-q4f16_1-MLC

Mistral / Mixtral

HF://mlc-ai/Mistral-7B-Instruct-v0.3-q4f16_1-MLC
HF://mlc-ai/Mixtral-8x7B-Instruct-v0.1-q4f16_1-MLC

Gemma

HF://mlc-ai/gemma-2b-it-q4f16_1-MLC
HF://mlc-ai/gemma-7b-it-q4f16_1-MLC

Phi

HF://mlc-ai/phi-2-q4f16_1-MLC
HF://mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC

Full model list: Browse all pre-compiled models at huggingface.co/mlc-ai

---

Quantization Options

MLC-LLM supports multiple quantization schemes. Choose based on your VRAM budget:

Quantization	Bits	Quality	VRAM (7B)	VRAM (13B)
q4f16_1	4-bit	★★★★☆	~4GB	~7GB
q4f32_1	4-bit (f32 accum)	★★★★☆	~4GB	~7GB
q8f16_1	8-bit	★★★★★	~8GB	~14GB
q0f16	16-bit (no quant)	★★★★★	~14GB	~26GB
q0f32	32-bit (no quant)	★★★★★	~28GB	~52GB

VRAM recommendation: Always leave 2–3GB headroom for CUDA overhead and KV cache. A 7B model with q4f16_1 needs ~6–7GB total on a typical workload.

---

Multi-GPU Deployment

For large models (70B+) requiring multiple GPUs:

# Enable tensor parallelism across 2 GPUs
python -m mlc_llm serve \
  HF://mlc-ai/Llama-3-70B-Instruct-q4f16_1-MLC \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-shards 2

Check GPU topology before deploying:

nvidia-smi topo -m  # Check NVLink/PCIe connectivity

Best performance: Multi-GPU works best with NVLink-connected cards (e.g., A100 80GB SXM pairs). PCIe-connected GPUs will show bottlenecks on large models.

---

Web Chat Interface

MLC-LLM includes a built-in web UI accessible once the server is running:

# Start server with web UI enabled
python -m mlc_llm serve \
  HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-debug  # Optional: enables debug endpoint

Access the UI at: http://<clore-node-ip>:<api-port>

---

Performance Tuning

Optimize Batch Size

# Increase batch size for higher throughput (requires more VRAM)
python -m mlc_llm serve \
  HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC \
  --host 0.0.0.0 \
  --port 8000 \
  --max-batch-size 8 \
  --max-total-sequence-length 16384 \
  --prefill-chunk-size 2048

Monitor GPU Utilization

# In a separate terminal
watch -n 1 nvidia-smi

# More detailed monitoring
nvidia-smi dmon -s u  # Streaming utilization metrics

Benchmark Throughput

import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

start = time.time()
response = client.chat.completions.create(
    model="Llama-3-8B-Instruct-q4f16_1-MLC",
    messages=[{"role": "user", "content": "Count from 1 to 100"}],
    max_tokens=512
)
elapsed = time.time() - start

tokens = response.usage.completion_tokens
print(f"Throughput: {tokens/elapsed:.1f} tokens/sec")

---

Docker Compose Setup

For a production-ready deployment on Clore.ai using an NVIDIA CUDA base image with MLC-LLM installed via pip:

version: '3.8'
services:
  mlc-llm:
    image: nvidia/cuda:12.1.0-devel-ubuntu22.04
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    ports:
      - "8000:8000"
    volumes:
      - ./models:/root/models
      - mlc-cache:/root/.cache/mlc_llm
    command: >
      bash -c "pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu121 mlc-ai-nightly-cu121 &&
      python -m mlc_llm serve
      HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
      --host 0.0.0.0
      --port 8000
      --max-batch-size 4"
    restart: unless-stopped

volumes:
  mlc-cache:

---

Troubleshooting

Model Download Fails

# Check internet connectivity
curl -I https://huggingface.co

# Manually download with huggingface-cli
pip install huggingface_hub
huggingface-cli download mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC

Out of Memory (OOM)

# Reduce context length
python -m mlc_llm serve MODEL \
  --max-total-sequence-length 4096  # Reduce from default

# Use more aggressive quantization
# Switch from q8f16_1 to q4f16_1

CUDA Version Mismatch

# Check CUDA version
nvcc --version
nvidia-smi | grep CUDA

# For CUDA 12.1 servers, install:
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu121 mlc-ai-nightly-cu121

# For CUDA 12.2+ servers, install:
pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu122 mlc-ai-nightly-cu122

Common pitfall: MLC-LLM pip wheels are CUDA-version specific. Make sure to install the correct variant matching your server's CUDA version. Check available wheels at mlc.ai/wheels.

Server Not Accessible

# Verify port is listening
ss -tlnp | grep 8000

# Check firewall
iptables -L -n | grep 8000

# Test locally first
curl http://localhost:8000/v1/models

---

Clore.ai GPU Recommendations

MLC-LLM's compilation approach delivers near-optimal throughput on every GPU tier. Pick based on model size and budget:

GPU	VRAM	Clore.ai Price	Best For	Throughput (Llama 3 8B Q4)
RTX 3090	24 GB	~$0.12/hr	7B–13B models, budget serving	~85 tok/s
RTX 4090	24 GB	~$0.70/hr	7B–34B models, fast serving	~140 tok/s
A100 40GB	40 GB	~$1.20/hr	34B–70B, production API	~110 tok/s
A100 80GB	80 GB	~$2.00/hr	70B+, multi-model serving	~130 tok/s
H100 SXM	80 GB	~$3.50/hr	Maximum throughput, FP8	~280 tok/s

Recommended starting point: RTX 3090 at ~$0.12/hr is the best price-performance ratio for Llama 3 8B and Mistral 7B serving via MLC-LLM. The compiled kernels extract near-maximum utilization from consumer GPUs.

For 70B models (e.g., Llama 3 70B Q4): use A100 40GB (~$1.20/hr) or two RTX 3090s via tensor parallelism.

---

Resources

• 📦 Pip Wheels: mlc.ai/wheels (install via pip, no Docker Hub image available)

• 🐙 GitHub: github.com/mlc-ai/mlc-llm

• 📚 Documentation: llm.mlc.ai/docs

• 🤗 Pre-compiled Models: huggingface.co/mlc-ai

• 💬 Discord: discord.gg/9Xpy2HGBuD