> For the complete documentation index, see [llms.txt](https://docs.clore.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.clore.ai/guides/guides_v2-zh/xun-lian/deepspeed-training.md).

# DeepSpeed 训练

在 Clore.ai GPU 上使用 DeepSpeed 高效训练大模型

使用 Microsoft DeepSpeed 高效训练大型模型。

{% hint style="success" %}
所有示例都可以在通过以下方式租用的 GPU 服务器上运行 [CLORE.AI 市场](https://clore.ai/marketplace).
{% endhint %}

## 在 CLORE.AI 上租用

{% hint style="warning" %}
**Clore.ai 市场上未列出多 GPU 的 80GB 级机型。** 目前列出的最大配置是 4× RTX PRO 6000 Blackwell（每张 96GB，共 380GB）以及 8–11× RTX 5090（每张 32GB）。A100 / H200 / B200 容量可按 [裸机](https://clore.ai/bare-metal) 需求提供。部署前请查看 [GPU 价格与可用性](/guides/guides_v2-zh/ru-men-zhi-nan/pricing.md) 。
{% endhint %}

1. 访问 [CLORE.AI 市场](https://clore.ai/marketplace)
2. 按 GPU 类型、VRAM 和价格筛选
3. 选择 **按需** （固定费率）或 **竞价** （出价）
4. 配置你的订单：
   * 选择 Docker 镜像
   * 设置端口（SSH 用 TCP，Web UI 用 HTTP）
   * 如有需要，添加环境变量
   * 输入启动命令
5. 选择支付方式： **CLORE**, **BTC**，或 **USDT/USDC**
6. 创建订单并等待部署

### 访问你的服务器

* 在以下位置查找连接信息 **我的订单**
* Web 界面：使用 HTTP 端口 URL
* SSH： `ssh -p <port> root@<proxy-address>`

## 什么是 DeepSpeed？

DeepSpeed 支持：

* 训练无法放入 GPU 内存的模型
* 多 GPU 和多节点训练
* ZeRO 优化（内存效率）
* 混合精度训练

## ZeRO 阶段

| 阶段            | 内存节省        | 速度     |
| ------------- | ----------- | ------ |
| ZeRO-1        | 优化器状态分区     | 快      |
| ZeRO-2        | + 梯度分区      | 平衡     |
| ZeRO-3        | + 参数分区      | 最大节省   |
| ZeRO-Infinity | CPU/NVMe 卸载 | 最大规模模型 |

## 快速部署

**Docker 镜像：**

```
pytorch/pytorch:2.11.0-cuda12.8-cudnn9-devel
```

**端口：**

```
22/tcp
```

**命令：**

```bash
pip install deepspeed transformers datasets accelerate
```

## 安装

```bash
pip install deepspeed

# 验证安装
ds_report
```

## 基础训练

### DeepSpeed 配置

**ds\_config.json：**

```json
{
    "train_batch_size": 32,
    "gradient_accumulation_steps": 4,
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": 1e-4,
            "betas": [0.9, 0.999],
            "eps": 1e-8,
            "weight_decay": 0.01
        }
    },
    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": 0,
            "warmup_max_lr": 1e-4,
            "warmup_num_steps": 100
        }
    },
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "initial_scale_power": 16
    },
    "zero_optimization": {
        "stage": 2,
        "contiguous_gradients": true,
        "overlap_comm": true
    }
}
```

### 训练脚本

```python
import torch
import deepspeed
from transformers import AutoModelForCausalLM, AutoTokenizer

# 初始化
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# DeepSpeed 初始化
model_engine, optimizer, _, _ = deepspeed.initialize(
    model=model,
    model_parameters=model.parameters(),
    config="ds_config.json"
)

# 训练循环
for epoch in range(num_epochs):
    for batch in dataloader:
        inputs = tokenizer(batch["text"], return_tensors="pt", padding=True, truncation=True)
        inputs = {k: v.to(model_engine.device) for k, v in inputs.items()}

        outputs = model_engine(**inputs, labels=inputs["input_ids"])
        loss = outputs.loss

        model_engine.backward(loss)
        model_engine.step()
```

## ZeRO 第 2 阶段配置

```json
{
    "train_batch_size": "auto",
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": 1.0,
    "fp16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 2,
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "reduce_scatter": true,
        "reduce_bucket_size": 2e8,
        "overlap_comm": true
    }
}
```

## ZeRO 第 3 阶段配置

适用于大型模型：

```json
{
    "train_batch_size": "auto",
    "gradient_accumulation_steps": "auto",
    "fp16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "cpu",
            "pin_memory": true
        },
        "offload_param": {
            "device": "cpu",
            "pin_memory": true
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true
    }
}
```

## 结合 Hugging Face Transformers

### Trainer 集成

```python
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=1e-4,
    num_train_epochs=3,
    fp16=True,
    deepspeed="ds_config.json",
    logging_steps=10,
    save_steps=500,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)

trainer.train()
```

## 多 GPU 训练

### 启动命令

```bash

# 单节点，4 块 GPU
deepspeed --num_gpus=4 train.py --deepspeed ds_config.json

# 指定 GPU
deepspeed --include="localhost:0,1,2,3" train.py --deepspeed ds_config.json
```

### 使用 torchrun

```bash
torchrun --nproc_per_node=4 train.py --deepspeed ds_config.json
```

## 多节点训练

### 主机文件

**hostfile：**

```
node1 slots=4
node2 slots=4
```

### 启动

```bash
deepspeed --hostfile=hostfile train.py --deepspeed ds_config.json
```

### SSH 设置

```bash

# 确保节点之间可免密 SSH
ssh-keygen -t rsa
ssh-copy-id user@node2
```

## 内存高效配置

### 24GB GPU 上的 7B 模型

```json
{
    "bf16": {"enabled": true},
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {"device": "cpu"},
        "offload_param": {"device": "cpu"}
    },
    "gradient_checkpointing": true,
    "train_micro_batch_size_per_gpu": 1,
    "gradient_accumulation_steps": 16
}
```

### 24GB GPU 上的 13B 模型

```json
{
    "bf16": {"enabled": true},
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {"device": "cpu"},
        "offload_param": {"device": "cpu"},
        "stage3_param_persistence_threshold": 0
    },
    "gradient_checkpointing": true,
    "train_micro_batch_size_per_gpu": 1,
    "gradient_accumulation_steps": 32
}
```

## 梯度检查点

通过重计算激活来节省内存：

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model.gradient_checkpointing_enable()
```

## 保存和加载检查点

### 保存

```python

# DeepSpeed 负责检查点处理
model_engine.save_checkpoint("./checkpoints", tag="step_1000")
```

### 加载

```python
model_engine.load_checkpoint("./checkpoints", tag="step_1000")
```

### 保存为 HuggingFace 格式

```python

# 将 DeepSpeed 检查点转换为 HF 格式
from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint

state_dict = get_fp32_state_dict_from_zero_checkpoint("./checkpoints/step_1000")
model.load_state_dict(state_dict)
model.save_pretrained("./hf_model")
```

## 监控

### TensorBoard

```json
{
    "tensorboard": {
        "enabled": true,
        "output_path": "./logs",
        "job_name": "training_run"
    }
}
```

### Weights & Biases

```json
{
    "wandb": {
        "enabled": true,
        "project": "my_project"
    }
}
```

## 常见问题

### 内存不足

```json
// 尝试：
{
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {"device": "cpu"},
        "offload_param": {"device": "cpu"}
    },
    "train_micro_batch_size_per_gpu": 1
}
```

### 训练缓慢

* 减少 CPU 卸载
* 增大批大小
* 使用 ZeRO 第 2 阶段而不是第 3 阶段

### NCCL 错误

```bash

# 设置环境变量
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1
```

## 性能提示

| 提示                 | 效果     |
| ------------------ | ------ |
| 优先使用 bf16 而不是 fp16 | 更好的稳定性 |
| 启用梯度检查点            | 更少内存   |
| 调优批大小              | 更高吞吐量  |
| 使用 NVMe 卸载         | 更大型模型  |

## 性能对比

| 模型  | GPU     | ZeRO 阶段 | 训练速度            |
| --- | ------- | ------- | --------------- |
| 7B  | 1x A100 | ZeRO-3  | \~1000 tokens/s |
| 7B  | 4x A100 | ZeRO-2  | \~4000 tokens/s |
| 13B | 4x A100 | ZeRO-3  | \~2000 tokens/s |
| 70B | 8x A100 | ZeRO-3  | \~800 tokens/s  |

## 故障排查

## 成本估算

CLORE.AI 市场常见费率（截至 2024 年）：

| GPU       | 小时费率    | 日费率     | 4 小时会话  |
| --------- | ------- | ------- | ------- |
| RTX 3060  | \~$0.03 | \~$0.70 | \~$0.12 |
| RTX 3090  | \~$0.06 | \~$1.50 | \~$0.25 |
| RTX 4090  | \~$0.10 | \~$2.30 | \~$0.40 |
| A100 40GB | \~$0.17 | \~$4.00 | \~$0.70 |
| A100 80GB | \~$0.25 | \~$6.00 | \~$1.00 |

*价格因提供商和需求而异。请查看* [*CLORE.AI 市场*](https://clore.ai/marketplace) *以获取当前费率。*

**节省费用：**

* 使用 **竞价** 可中断工作市场——约三分之一的服务器将现货价格定得低于按需价格（中位数约优惠 13%），其余则与按需价格持平
* 使用 **CLORE** 代币支付
* 比较不同提供商的价格

## 下一步

* [微调 LLM](/guides/guides_v2-zh/xun-lian/finetune-llm.md) - LoRA 训练
* vLLM 推理 - 部署训练好的模型
* [Hugging Face 指南](/guides/guides_v2-zh/xun-lian/huggingface-transformers.md) - Transformers 库


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.clore.ai/guides/guides_v2-zh/xun-lian/deepspeed-training.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.