> For the complete documentation index, see [llms.txt](https://docs.clore.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.clore.ai/guides/guides_v2-zh/gpu-devops/tensorrt-llm.md). # TensorRT-LLM > **借助 NVIDIA TensorRT 优化实现最大 LLM 推理吞吐量——通过 Triton Inference Server 部署** TensorRT-LLM 是 NVIDIA 的开源库，用于优化 NVIDIA GPU 上的大语言模型推理。它通过算子融合、量化（INT4、INT8、FP8）、动态批处理和分页 KV 缓存，提供业界领先的性能。与 Triton Inference Server 结合后，你将获得生产级的服务基础设施。 **GitHub：** [NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) — 1万+ ⭐ *** ## 为什么选择 TensorRT-LLM？ | 功能 | vLLM | TensorRT-LLM | | ------------ | ---- | ------------ | | 吞吐量 | 优秀 | 一流 | | 延迟 | 好 | 优秀 | | INT4/INT8 量化 | 部分 | 原生 | | FP8 支持 | 有限 | 完整 | | 多 GPU 张量并行 | 是 | 是 | | 设置复杂度 | 低 | 中-高 | {% hint style="success" %} **TensorRT-LLM 通常可提供 2–4 倍更高的吞吐量** 相比标准 HuggingFace transformers 推理，在批量服务场景下也比 vLLM 提供高 30–50% 的吞吐量。 {% endhint %} *** ## 前提条件 * 带 GPU 租赁的 Clore.ai 账户 * **采用 Ampere 架构或更新架构的 NVIDIA GPU** （RTX 3090、A100、RTX 4090、H100） * 具备基础的 Linux 和 Docker 知识 * 为你所选模型提供足够的 VRAM *** ## 各模型的 VRAM 要求 | 模型 | FP16 | INT8 | INT4 | | ------------- | ----- | ---- | ---- | | Llama-3.1 8B | 16GB | 8GB | 4GB | | Llama-3.1 70B | 140GB | 70GB | 35GB | | Mistral 7B | 14GB | 7GB | 4GB | | Mixtral 8x7B | 90GB | 45GB | 24GB | | Qwen2.5 72B | 144GB | 72GB | 36GB | *** ## 步骤 1 — 在 Clore.ai 上选择你的 GPU 1. 登录到 [clore.ai](https://clore.ai) → **市场** 2. **用于单 GPU 服务（7B–13B 模型）：** RTX 4090 24GB 或 RTX 3090 24GB 3. **用于大模型（70B+）：** 多块 A100 80GB 或 H100 {% hint style="info" %} **多 GPU 策略：** * 2x A100 80GB → FP16 版 Llama 3.1 70B 或 Qwen2.5 72B * 4x A100 80GB → INT8 版 Llama 3.1 405B * 在 Clore.ai 市场中选择列出了多块 GPU 的服务器 {% endhint %} *** ## 步骤 2 — 使用 TRT-LLM 后端部署 Triton Inference Server **Docker 镜像：** ``` nvcr.io/nvidia/tritonserver:24.01-trtllm-python-py3 ``` {% hint style="warning" %} 使用 `-trtllm-python-py3` 变体——其中已预装 TensorRT-LLM 后端。该标签对应 NVIDIA 容器发布版本（24.01 = 2024 年 1 月）。查看 [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags) 以获取最新标签。 {% endhint %} **暴露端口：** ``` 22 8000 ``` **环境变量：** ``` NVIDIA_VISIBLE_DEVICES=all NVIDIA_DRIVER_CAPABILITIES=compute,utility TRANSFORMERS_CACHE=/workspace/hf_cache HF_HOME=/workspace/hf_cache ``` **卷/磁盘：** 建议至少 100GB *** ## 步骤 3 — 连接并验证安装 ```bash ssh root@ -p # 检查 GPU nvidia-smi # 检查 TensorRT 版本 python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" # 检查 Triton 是否可用 tritonserver --version ``` *** ## 步骤 4 — 下载并准备模型我们将以 Llama 3.1 8B 为例。请根据你选择的模型调整路径。 ### 安装 HuggingFace CLI ```bash pip install huggingface_hub huggingface-cli login # 提示时输入你的 HuggingFace 令牌 ``` ### 下载模型权重 ```bash mkdir -p /workspace/models/llama-3.1-8b huggingface-cli download \ meta-llama/Llama-3.1-8B-Instruct \ --local-dir /workspace/models/llama-3.1-8b \ --local-dir-use-symlinks False # 或使用 snapshot_download python3 << 'EOF' from huggingface_hub import snapshot_download snapshot_download( repo_id="meta-llama/Llama-3.1-8B-Instruct", local_dir="/workspace/models/llama-3.1-8b", local_dir_use_symlinks=False ) EOF ``` *** ## 步骤 5 — 构建 TensorRT 引擎这是关键步骤——将模型编译为优化后的 TensorRT 引擎。 ### FP16 引擎（最佳质量） ```bash cd /workspace # 将 HuggingFace 权重转换为 TRT-LLM 格式 python3 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/examples/llama/convert_checkpoint.py \ --model_dir /workspace/models/llama-3.1-8b \ --output_dir /workspace/trt_checkpoints/llama-3.1-8b-fp16 \ --dtype float16 \\ --tp_size 1 # 构建 TensorRT 引擎 trtllm-build \ --checkpoint_dir /workspace/trt_checkpoints/llama-3.1-8b-fp16 \ --output_dir /workspace/trt_engines/llama-3.1-8b-fp16 \ --gemm_plugin float16 \ --max_batch_size 32 \ --max_input_len 4096 \ --max_seq_len 8192 \ --max_num_tokens 16384 \ --use_paged_context_fmha enable ``` ### INT8 SmoothQuant 引擎（更高吞吐量） ```bash # 使用 SmoothQuant 量化进行转换 python3 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/examples/llama/convert_checkpoint.py \ --model_dir /workspace/models/llama-3.1-8b \ --output_dir /workspace/trt_checkpoints/llama-3.1-8b-int8 \ --dtype float16 \\ --smoothquant 0.5 \ --per_channel \ --per_token trtllm-build \ --checkpoint_dir /workspace/trt_checkpoints/llama-3.1-8b-int8 \ --output_dir /workspace/trt_engines/llama-3.1-8b-int8 \ --gemm_plugin float16 \ --smoothquant_plugin float16 \ --max_batch_size 64 \ --max_input_len 4096 \ --max_seq_len 8192 ``` ### INT4 AWQ 引擎（最高吞吐量 / 最低内存） ```bash # 安装用于量化的 auto-gptq pip install autoawq # 量化为 INT4 AWQ python3 << 'EOF' from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = "/workspace/models/llama-3.1-8b" quant_path = "/workspace/models/llama-3.1-8b-awq-int4" model = AutoAWQForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } model.quantize(tokenizer, quant_config=quant_config) model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) EOF # 将 AWQ 转换为 TRT-LLM python3 /usr/local/lib/python3.10/dist-packages/tensorrt_llm/examples/llama/convert_checkpoint.py \ --model_dir /workspace/models/llama-3.1-8b-awq-int4 \ --output_dir /workspace/trt_checkpoints/llama-3.1-8b-int4 \ --dtype float16 \\ --quant_ckpt_path /workspace/models/llama-3.1-8b-awq-int4 \ --use_weight_only \ --weight_only_precision int4_awq \ --per_group trtllm-build \ --checkpoint_dir /workspace/trt_checkpoints/llama-3.1-8b-int4 \ --output_dir /workspace/trt_engines/llama-3.1-8b-int4 \ --gemm_plugin float16 \ --max_batch_size 128 \ --max_input_len 4096 \ --max_seq_len 8192 ``` {% hint style="info" %} **引擎构建时间：** 根据 GPU 和模型大小而定，约 10–30 分钟。这是一次性操作——一旦构建完成，加载引擎只需几秒。 {% endhint %} *** ## 步骤 6 — 使用 TRT-LLM Python API 进行快速测试在设置 Triton 之前，请先验证引擎是否可用： ```bash python3 << 'EOF' import tensorrt_llm from tensorrt_llm.runtime import ModelRunner from transformers import AutoTokenizer engine_dir = "/workspace/trt_engines/llama-3.1-8b-fp16" tokenizer_dir = "/workspace/models/llama-3.1-8b" tokenizer = AutoTokenizer.from_pretrained(tokenizer_dir) runner = ModelRunner.from_dir( engine_dir=engine_dir, rank=0 ) prompt = "法国的首都是哪里？" input_ids = tokenizer.encode(prompt, return_tensors="pt") output = runner.generate( batch_input_ids=[input_ids[0].tolist()], max_new_tokens=200, temperature=0.7, top_p=0.9 ) output_ids = output[0][0][len(input_ids[0]):] response = tokenizer.decode(output_ids, skip_special_tokens=True) print(f"Response: {response}") EOF ``` *** ## 步骤 7 — 设置 Triton Inference Server ### 创建模型仓库结构 ```bash mkdir -p /workspace/triton_model_repo/llama/1 # 创建模型配置 cat > /workspace/triton_model_repo/llama/config.pbtxt << 'EOF' backend: "tensorrtllm" name: "llama" max_batch_size: 64 model_transaction_policy { decoupled: true } dynamic_batching { preferred_batch_size: [1, 2, 4, 8, 16, 32, 64] max_queue_delay_microseconds: 1000 } input [ { name: "input_ids" data_type: TYPE_INT32 dims: [-1] }, { name: "input_lengths" data_type: TYPE_INT32 dims: [1] reshape: { shape: [] } }, { name: "request_output_len" data_type: TYPE_INT32 dims: [1] reshape: { shape: [] } }, { name: "temperature" data_type: TYPE_FP32 dims: [1] reshape: { shape: [] } optional: true } ] output [ { name: "output_ids" data_type: TYPE_INT32 dims: [-1, -1] }, { name: "sequence_length" data_type: TYPE_INT32 dims: [1] } ] instance_group [ { 数量：1 kind: KIND_GPU gpus: [0] } ] parameters: { key: "gpt_model_type" value: { string_value: "inflight_fused_batching" } } parameters: { key: "gpt_model_path" value: { string_value: "/workspace/trt_engines/llama-3.1-8b-fp16" } } parameters: { key: "max_tokens_in_paged_kv_cache" value: { string_value: "8192" } } parameters: { key: "batch_scheduler_policy" value: { string_value: "guaranteed_no_evict" } } EOF ``` ### 创建引擎符号链接 ```bash ln -s /workspace/trt_engines/llama-3.1-8b-fp16 \ /workspace/triton_model_repo/llama/1/ ``` ### 启动 Triton Server ```bash tritonserver \ --model-repository=/workspace/triton_model_repo \ --http-port=8000 \ --grpc-port=8001 \ --metrics-port=8002 \ --log-verbose=0 & # 等待服务器启动 sleep 30 # 检查服务器健康状态 curl -s http://localhost:8000/v2/health/ready ``` *** ## 步骤 8 — 查询 API ### 兼容 OpenAI 的客户端 ```python import requests import json def generate(prompt: str, max_tokens: int = 200) -> str: url = "http://localhost:8000/v2/models/llama/generate" payload = { "text_input": prompt, "parameters": { "max_tokens": max_tokens, "temperature": 0.7, "top_p": 0.9 } } response = requests.post(url, json=payload) result = response.json() return result.get("text_output", "") # 测试 print(generate("用简单的话解释量子计算：")) ``` ### 基准测试吞吐量 ```bash # 安装 tritonclient pip install tritonclient[all] # 运行性能基准测试 perf_analyzer \ -m llama \ -u localhost:8001 \ --protocol grpc \ --input-data /workspace/sample_inputs.json \ --concurrency-range 1:32:2 \ --measurement-interval 10000 \ --shape input_ids:512 \ --shape input_lengths:1 \ --shape request_output_len:1 ``` *** ## 步骤 9 — 添加兼容 OpenAI 的 API 封装为了更方便集成，请添加一个 FastAPI 封装： ```bash pip install fastapi uvicorn tritonclient[all] cat > /workspace/openai_server.py << 'EOF' from fastapi import FastAPI from pydantic import BaseModel import tritonclient.http as httpclient import numpy as np from transformers import AutoTokenizer app = FastAPI() tokenizer = AutoTokenizer.from_pretrained("/workspace/models/llama-3.1-8b") client = httpclient.InferenceServerClient("localhost:8000") class ChatRequest(BaseModel): model: str = "llama" messages: list max_tokens: int = 512 temperature: float = 0.7 @app.post("/v1/chat/completions") async def chat(req: ChatRequest): prompt = tokenizer.apply_chat_template( req.messages, tokenize=False, add_generation_prompt=True ) input_ids = tokenizer.encode(prompt) inputs = [ httpclient.InferInput("input_ids", [len(input_ids)], "INT32"), httpclient.InferInput("input_lengths", [1], "INT32"), httpclient.InferInput("request_output_len", [1], "INT32"), ] inputs[0].set_data_from_numpy(np.array(input_ids, dtype=np.int32)) inputs[1].set_data_from_numpy(np.array([len(input_ids)], dtype=np.int32)) inputs[2].set_data_from_numpy(np.array([req.max_tokens], dtype=np.int32)) result = client.infer("llama", inputs) output_ids = result.as_numpy("output_ids")[0][len(input_ids):] text = tokenizer.decode(output_ids, skip_special_tokens=True) return { "choices": [{"message": {"role": "assistant", "content": text}}] } if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=8080) EOF python3 /workspace/openai_server.py & ``` *** ## 故障排查 ### 引擎构建 OOM ```bash # 降低 max_batch_size 和 max_num_tokens trtllm-build \ --checkpoint_dir /workspace/trt_checkpoints/llama-3.1-8b-fp16 \ --output_dir /workspace/trt_engines/llama-3.1-8b-fp16 \ --gemm_plugin float16 \ --max_batch_size 8 \ # 从 32 降低 --max_input_len 2048 \ # 从 4096 降低 --max_seq_len 4096 # 从 8192 降低 ``` ### Triton Server 未启动 ```bash # 检查日志 cat /workspace/triton.log # 验证引擎文件是否存在 ls -la /workspace/trt_engines/llama-3.1-8b-fp16/ # 检查 GPU 内存 nvidia-smi ``` ### 吞吐量较低 ```bash # 启用动态批处理并增加并发 # 根据可用 VRAM 调整 max_tokens_in_paged_kv_cache ``` *** ## Clore.ai GPU 上的性能基准 | 模型 | GPU | 量化 | 吞吐量（tokens/秒） | | ------------- | ----------- | -------- | ------------- | | Llama 3.1 8B | RTX 4090 | FP16 | \~3,500 | | Llama 3.1 8B | RTX 4090 | INT4 AWQ | \~6,200 | | Llama 3.1 70B | 2x A100 80G | FP16 | \~1,800 | | Mixtral 8x7B | 2x RTX 4090 | INT8 | \~2,400 | *** ## 更多资源 * [TensorRT-LLM GitHub](https://github.com/NVIDIA/TensorRT-LLM) * [Triton Inference Server](https://github.com/triton-inference-server/server) * [NGC 容器注册表](https://catalog.ngc.nvidia.com/) * [TRT-LLM 文档](https://nvidia.github.io/TensorRT-LLM/) * [AWQ 量化](https://github.com/mit-han-lab/llm-awq) *** *在 Clore.ai 上使用 TensorRT-LLM 是生产级 LLM 服务的最佳选择，尤其适用于吞吐量和延迟至关重要的场景。对于更简单的部署，可以考虑 vLLM 指南。* *** ## Clore.ai GPU 推荐 {% hint style="warning" %} **Clore.ai 市场上未列出多 GPU 的 80GB 级机型。** 目前列出的最大配置是 4× RTX PRO 6000 Blackwell（每张 96GB，共 380GB）以及 8–11× RTX 5090（每张 32GB）。A100 / H200 / B200 容量可按 [裸机](https://clore.ai/bare-metal) 需求提供。部署前请查看 [GPU 价格与可用性](/guides/guides_v2-zh/ru-men-zhi-nan/pricing.md) 。 {% endhint %} | 使用场景 | 推荐 GPU | Clore.ai 预计成本 | | --------- | -------------- | --------------------------------- | | 开发/测试 | RTX 3090（24GB） | $0.07–0.21/gpu/hr | | 生产推理 | RTX 4090（24GB） | $0.14–0.42/gpu/hr | | 大模型（70B+） | A100 80GB | [裸机](https://clore.ai/bare-metal) | > 💡 本指南中的所有示例都可以部署在 [Clore.ai](https://clore.ai/marketplace) GPU 服务器上。浏览可用 GPU 并按小时租用——无需承诺，拥有完整 root 访问权限。 --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.clore.ai/guides/guides_v2-zh/gpu-devops/tensorrt-llm.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.