> For the complete documentation index, see [llms.txt](https://docs.clore.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.clore.ai/guides/guides_v2-zh/yu-yan-mo-xing/exllamav2-fast.md).

# ExLlamaV2

在 Clore.ai GPU 上使用 ExLlamaV2 实现最快速度的 LLM 推理

使用 ExLlamaV2 以最大速度运行 LLM。

{% hint style="success" %}
所有示例都可以在通过以下方式租用的 GPU 服务器上运行 [CLORE.AI 市场](https://clore.ai/marketplace).
{% endhint %}

## 在 CLORE.AI 上租用

1. 访问 [CLORE.AI 市场](https://clore.ai/marketplace)
2. 按 GPU 类型、VRAM 和价格筛选
3. 选择 **按需** （固定费率）或 **竞价** （出价）
4. 配置你的订单：
   * 选择 Docker 镜像
   * 设置端口（SSH 用 TCP，Web UI 用 HTTP）
   * 如有需要，添加环境变量
   * 输入启动命令
5. 选择支付方式： **CLORE**, **BTC**，或 **USDT/USDC**
6. 创建订单并等待部署

### 访问你的服务器

* 在以下位置查找连接信息 **我的订单**
* Web 界面：使用 HTTP 端口 URL
* SSH： `ssh -p <port> root@<proxy-address>`

## 什么是 ExLlamaV2？

ExLlamaV2 是大语言模型最快的推理引擎：

* 比其他引擎快 2-3 倍
* 出色的量化（EXL2）
* 显存占用低
* 支持推测解码

## 需求

| 模型大小 | 最低显存 | 推荐       |
| ---- | ---- | -------- |
| 7B   | 6GB  | RTX 3060 |
| 13B  | 10GB | RTX 3090 |
| 34B  | 20GB | RTX 4090 |
| 70B  | 40GB | A100     |

## 快速部署

**Docker 镜像：**

```
pytorch/pytorch:2.11.0-cuda12.8-cudnn9-devel
```

**端口：**

```
22/tcp
8080/http
```

**命令：**

```bash
pip install exllamav2 && \\
huggingface-cli download turboderp/Llama2-7B-exl2 --local-dir ./model && \\
python -m exllamav2.server --model_dir ./model --host 0.0.0.0 --port 8080
```

## 访问你的服务

部署后，找到你的 `http_pub` URL 在 **我的订单**:

1. 前往 **我的订单** 页面
2. 点击你的订单
3. 找到 `http_pub` URL（例如， `abc123.clorecloud.net`)

使用 `https://YOUR_HTTP_PUB_URL` 替代 `localhost` 在下面的示例中。

## 安装

```bash

# 从 PyPI 安装
pip install exllamav2

# 或从源码安装（最新功能）
git clone https://github.com/turboderp/exllamav2
cd exllamav2
pip install .
```

## 下载模型

### EXL2 量化模型

```bash

# Llama 3.1 8B（4.0 bpw）
huggingface-cli download turboderp/Llama2-7B-exl2 \\
    --revision 4.0bpw \\
    --local-dir ./llama2-7b-exl2

# Llama 3.1 8B（4.0 bpw）
huggingface-cli download turboderp/Llama2-13B-exl2 \\
    --revision 4.0bpw \\
    --local-dir ./llama2-13b-exl2

# Mistral 7B（4.0 bpw）
huggingface-cli download turboderp/Mistral-7B-instruct-exl2 \\
    --revision 4.0bpw \\
    --local-dir ./mistral-7b-exl2

# Mixtral 8x7B
huggingface-cli download turboderp/Mixtral-8x7B-instruct-exl2 \\
    --revision 4.0bpw \\
    --local-dir ./mixtral-exl2
```

### 每权重比特数（bpw）

| BPW | 质量      | VRAM（7B） |
| --- | ------- | -------- |
| 2.0 | 低       | \~3GB    |
| 3.0 | 好       | 约 4GB    |
| 4.0 | 很高      | \~5GB    |
| 5.0 | 优秀      | \~6GB    |
| 6.0 | 接近 FP16 | \~7GB    |

## Python API

### 基础生成

```python
from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache, ExLlamaV2Tokenizer
from exllamav2.generator import ExLlamaV2StreamingGenerator, ExLlamaV2Sampler

# 加载模型
config = ExLlamaV2Config()
config.model_dir = "./llama2-7b-exl2"
config.prepare()

model = ExLlamaV2(config)
model.load()

tokenizer = ExLlamaV2Tokenizer(config)
cache = ExLlamaV2Cache(model, lazy=True)

# 创建生成器
generator = ExLlamaV2StreamingGenerator(model, cache, tokenizer)

# 设置采样参数
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.7
settings.top_k = 50
settings.top_p = 0.9

# 生成
prompt = "人工智能的未来是"
output = generator.generate_simple(prompt, settings, num_tokens=200)
print(output)
```

### 流式生成

```python
from exllamav2.generator import ExLlamaV2StreamingGenerator

generator = ExLlamaV2StreamingGenerator(model, cache, tokenizer)

prompt = "写一个关于机器人的短篇故事："
input_ids = tokenizer.encode(prompt)

generator.set_stop_conditions([tokenizer.eos_token_id])
generator.begin_stream(input_ids, settings)

while True:
    chunk, eos, _ = generator.stream()
    if eos:
        break
    print(chunk, end="", flush=True)
```

### 聊天格式

```python
def format_chat(messages):
    text = ""
    for msg in messages:
        role = msg["role"]
        content = msg["content"]
        if role == "system":
            text += f"[INST] <<SYS>>\n{content}\n<</SYS>>\n\n"
        elif role == "user":
            text += f"{content} [/INST]"
        elif role == "assistant":
            text += f" {content}</s><s>[INST] "
    return text

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "什么是 Python？"}
]

prompt = format_chat(messages)
output = generator.generate_simple(prompt, settings, num_tokens=300)
```

## 服务器模式

### 启动服务器

```bash
python -m exllamav2.server \\
    --model_dir ./llama2-7b-exl2 \\
    --host 0.0.0.0 \\
    --port 8080 \
    --max_seq_len 4096 \\
    --cache_size 4096
```

### API 使用

```python
import requests

response = requests.post(
    "http://localhost:8080/v1/completions",
    json={
        "prompt": "你好，你怎么样？",
        "max_tokens": 100,
        "temperature": 0.7
    }
)

print(response.json()["choices"][0]["text"]))
```

### 聊天补全

```python
import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="llama2-7b",
    messages=[{"role": "user", "content": "你好！"}],
    temperature=0.7
)

print(response.choices[0].message.content)
```

## TabbyAPI（推荐的服务器）

TabbyAPI 提供功能丰富的 ExLlamaV2 服务器：

```bash

# 克隆 TabbyAPI
git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI

# 安装
pip install -r requirements.txt

# 配置

# 使用你的模型路径编辑 config.yml

# 运行
python main.py
```

### TabbyAPI 功能

* 兼容 OpenAI 的 API
* 支持多个模型
* LoRA 热切换
* 流式输出
* 函数调用
* 管理 API

## 推测解码

使用更小的模型加速生成：

```python
from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Cache

# 加载主模型（13B）
main_config = ExLlamaV2Config()
main_config.model_dir = "./llama2-13b-exl2"
main_config.prepare()
main_model = ExLlamaV2(main_config)
main_model.load()

# 加载草稿模型（7B）
draft_config = ExLlamaV2Config()
draft_config.model_dir = "./llama2-7b-exl2"
draft_config.prepare()
draft_model = ExLlamaV2(draft_config)
draft_model.load()

# 创建推测生成器
from exllamav2.generator import ExLlamaV2DraftGenerator

generator = ExLlamaV2DraftGenerator(
    main_model, draft_model,
    cache_main, cache_draft,
    tokenizer
)

# 生成（推测更快）
output = generator.generate_simple(prompt, settings, num_tokens=500)
```

## 量化你自己的模型

### 转换为 EXL2

```python
from exllamav2 import ExLlamaV2, ExLlamaV2Config
from exllamav2.conversion import convert_model

# 来源：HuggingFace 模型

# 目标：EXL2 量化模型

convert_model(
    input_dir="./llama-3.1-8b-hf",
    output_dir="./llama-3.1-8b-exl2-4bpw",
    cal_dataset="wikitext",  # 校准数据集
    bits=4.0,  # 每权重比特数
    head_bits=6,  # 注意力部分更高精度
)
```

### 命令行

```bash
python convert.py \\
    -i ./llama-3.1-8b-hf \\
    -o ./llama-3.1-8b-exl2 \\
    -cf ./llama-3.1-8b-exl2 \\
    -b 4.0 \\
    -hb 6
```

## 内存管理

### 缓存分配

```python

# 固定缓存大小
cache = ExLlamaV2Cache(model, max_seq_len=4096)

# 动态缓存
cache = ExLlamaV2Cache(model, lazy=True)
cache.current_seq_len = 0  # 按需增长
```

### 多 GPU

```python
config = ExLlamaV2Config()
config.model_dir = "./large-model"

# 在多个 GPU 之间切分
config.set_auto_split([0.5, 0.5])  # 每个 GPU 50%

model = ExLlamaV2(config)
model.load()
```

## 性能对比

| 模型           | 引擎        | GPU      | 每秒 Token 数 |
| ------------ | --------- | -------- | ---------- |
| Llama 3.1 8B | ExLlamaV2 | RTX 3090 | \~150      |
| Llama 3.1 8B | llama.cpp | RTX 3090 | \~100      |
| Llama 3.1 8B | vLLM      | RTX 3090 | \~120      |
| Llama 3.1 8B | ExLlamaV2 | RTX 3090 | \~90       |
| Mixtral 8x7B | ExLlamaV2 | A100     | \~70       |

## 高级设置

### 采样参数

```python
settings = ExLlamaV2Sampler.Settings()
settings.temperature = 0.7
settings.top_k = 50
settings.top_p = 0.9
settings.token_repetition_penalty = 1.1
settings.token_frequency_penalty = 0.0
settings.token_presence_penalty = 0.0
settings.mirostat = False
settings.mirostat_tau = 5.0
settings.mirostat_eta = 0.1
```

### 批量生成

```python
prompts = [
    "生命的意义是",
    "人工智能将会",
    "气候变化是"
]

outputs = []
for prompt in prompts:
    output = generator.generate_simple(prompt, settings, num_tokens=100)
    outputs.append(output)
```

## 故障排查

### CUDA 显存不足

```python

# 使用更小的缓存
cache = ExLlamaV2Cache(model, max_seq_len=2048)

# 或使用更低 bpw 的模型（3.0 而不是 4.0）
```

### 加载缓慢

```python

# 启用快速加载
config.fasttensors = True
```

### 未找到模型

```bash

# 检查模型文件是否存在
ls ./model/

# 应包含：config.json、*.safetensors、tokenizer.json
```

## 与 LangChain 集成

```python
from langchain.llms.base import LLM
from typing import Optional, List

class ExLlamaV2LLM(LLM):
    model: ExLlamaV2
    tokenizer: ExLlamaV2Tokenizer
    generator: ExLlamaV2StreamingGenerator
    settings: ExLlamaV2Sampler.Settings

    @property
    def _llm_type(self) -> str:
        return "exllamav2"

    def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
        return self.generator.generate_simple(prompt, self.settings, num_tokens=500)

# 用法
llm = ExLlamaV2LLM(model=model, tokenizer=tokenizer, generator=generator, settings=settings)
result = llm("什么是量子计算？")
```

## 成本估算

CLORE.AI 市场常见费率（截至 2024 年）：

| GPU       | 小时费率    | 日费率     | 4 小时会话  |
| --------- | ------- | ------- | ------- |
| RTX 3060  | \~$0.03 | \~$0.70 | \~$0.12 |
| RTX 3090  | \~$0.06 | \~$1.50 | \~$0.25 |
| RTX 4090  | \~$0.10 | \~$2.30 | \~$0.40 |
| A100 40GB | \~$0.17 | \~$4.00 | \~$0.70 |
| A100 80GB | \~$0.25 | \~$6.00 | \~$1.00 |

*价格因提供商和需求而异。请查看* [*CLORE.AI 市场*](https://clore.ai/marketplace) *以获取当前费率。*

**节省费用：**

* 使用 **竞价** 可中断工作市场——约三分之一的服务器将现货价格定得低于按需价格（中位数约优惠 13%），其余则与按需价格持平
* 使用 **CLORE** 代币支付
* 比较不同提供商的价格

## 下一步

* vLLM 推理 - 高吞吐量服务
* [llama.cpp 服务器](/guides/guides_v2-zh/yu-yan-mo-xing/llamacpp-server.md) - 跨平台
* [Text Generation WebUI](/guides/guides_v2-zh/yu-yan-mo-xing/text-generation-webui.md) - Web 界面


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.clore.ai/guides/guides_v2-zh/yu-yan-mo-xing/exllamav2-fast.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.