> For the complete documentation index, see [llms.txt](https://docs.clore.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.clore.ai/guides/guides_v2-zh/shi-jue-mo-xing/llava-vision-language.md).

# LLaVA

在 Clore.ai 上使用 LLaVA 视觉语言模型与图像聊天

使用 LLaVA 与图像聊天——开源的 GPT-4V 替代方案。

{% hint style="success" %}
所有示例都可以在通过以下方式租用的 GPU 服务器上运行 [CLORE.AI 市场](https://clore.ai/marketplace).
{% endhint %}

## 在 CLORE.AI 上租用

1. 访问 [CLORE.AI 市场](https://clore.ai/marketplace)
2. 按 GPU 类型、VRAM 和价格筛选
3. 选择 **按需** （固定费率）或 **竞价** （出价）
4. 配置你的订单：
   * 选择 Docker 镜像
   * 设置端口（SSH 用 TCP，Web UI 用 HTTP）
   * 如有需要，添加环境变量
   * 输入启动命令
5. 选择支付方式： **CLORE**, **BTC**，或 **USDT/USDC**
6. 创建订单并等待部署

### 访问你的服务器

* 在以下位置查找连接信息 **我的订单**
* Web 界面：使用 HTTP 端口 URL
* SSH： `ssh -p <port> root@<proxy-address>`

## 什么是 LLaVA？

LLaVA（大型语言与视觉助手）可以：

* 理解并描述图像
* 回答有关视觉内容的问题
* 分析图表、示意图、截图
* OCR 和文档理解

## 模型变体

| 模型            | 大小    | 显存     | 质量 |
| ------------- | ----- | ------ | -- |
| LLaVA-1.5-7B  | 7B    | 8GB    | 好  |
| LLaVA-1.5-13B | 13B   | 16GB   | 更好 |
| LLaVA-1.6-34B | 34B   | 40GB   | 最佳 |
| LLaVA-NeXT    | 7-34B | 8-40GB | 最新 |

## 快速部署

**Docker 镜像：**

```
pytorch/pytorch:2.11.0-cuda12.8-cudnn9-devel
```

**端口：**

```
22/tcp
8000/http
```

**命令：**

```bash
pip install llava torch transformers accelerate gradio && \
python -m llava.serve.cli --model-path liuhaotian/llava-v1.5-7b --load-4bit
```

## 访问你的服务

部署后，找到你的 `http_pub` URL 在 **我的订单**:

1. 前往 **我的订单** 页面
2. 点击你的订单
3. 找到 `http_pub` URL（例如， `abc123.clorecloud.net`)

使用 `https://YOUR_HTTP_PUB_URL` 替代 `localhost` 在下面的示例中。

## 安装

```bash
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
pip install -e .
pip install flash-attn --no-build-isolation
```

## 基础用法

### Python API

```python
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
from llava.eval.run_llava import eval_model
from PIL import Image

model_path = "liuhaotian/llava-v1.5-7b"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)

# 简单推理
args = type('Args', (), {
    "model_path": model_path,
    "model_base": None,
    "model_name": get_model_name_from_path(model_path),
    "query": "详细描述这张图像",
    "conv_mode": None,
    "image_file": "photo.jpg",
    "sep": ",",
    "temperature": 0.2,
    "top_p": None,
    "num_beams": 1,
    "max_new_tokens": 512
})()

output = eval_model(args)
print(output)
```

### 使用 Transformers

```python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
model = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

# 加载图像
image = Image.open("photo.jpg")

# 创建对话
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "这张图里显示了什么？"}
        ]
    }
]

prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(prompt, image, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=200)
response = processor.decode(output[0], skip_special_tokens=True)
打印(response)
```

## Ollama 集成（推荐）

在 CLORE.AI 上运行 LLaVA 的最简单方法：

```bash
# 安装 Ollama
curl -fsSL https://ollama.com/install.sh | sh

# 拉取 LLaVA 模型
ollama pull llava:7b

# 使用图像运行（CLI）
ollama run llava:7b "描述这张图像：/path/to/image.jpg"
```

### 通过 Ollama 使用 LLaVA API

{% hint style="warning" %}
**重要：** LLaVA 视觉功能仅能 **仅** 通过 `/api/generate` 端点和 `图像` 参数。 `/api/chat` 和 OpenAI 兼容的端点 **不** 支持 LLaVA 的图像。
{% endhint %}

#### 可用方法：/api/generate

```bash
# 先将图像编码为 base64
BASE64_IMAGE=$(base64 -i photo.jpg | tr -d '\n')

# 发送视觉请求
curl https://your-http-pub.clorecloud.net/api/generate -d "{
  \"model\": \"llava:7b\",
  \"prompt\": \"这张图里你看到了什么？请详细描述。\",
  \"images\": [\"$BASE64_IMAGE\"],
  \"stream\": false
}"
```

响应：

```json
{
  "model": "llava:7b",
  "response": "这张图展示了山脉上美丽的日落……",
  "done": true
}
```

#### 不可用：/api/chat（视觉功能返回 null）

```bash
# 这对视觉查询并不起作用：
curl https://your-http-pub.clorecloud.net/api/chat -d '{
  "model": "llava:7b",
  "messages": [{"role": "user", "content": "描述", "images": ["..."]}]
}'
# 对图像相关响应返回 null
```

### 使用 Python 与 Ollama

```python
import requests
import base64

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode()

# 使用 /api/generate 进行视觉处理（不是 /api/chat！）
response = requests.post(
    "https://your-http-pub.clorecloud.net/api/generate",
    json={
        "model": "llava:7b",
        "prompt": "这张图里你看到了什么？",
        "images": [encode_image("photo.jpg")],
        "stream": False
    }
)

print(response.json()["response"])
```

### 完整可运行示例

```python
import requests
import base64
import sys

def analyze_image(ollama_url, image_path, question):
    """使用 Ollama 通过 LLaVA 分析图像"""

    # 编码图像
    with open(image_path, "rb") as f:
        image_base64 = base64.b64encode(f.read()).decode()

    # 使用 /api/generate（视觉功能唯一可用的端点）
    response = requests.post(
        f"{ollama_url}/api/generate",
        json={
            "model": "llava:7b",
            "prompt": question,
            "images": [image_base64],
            "stream": False
        }
    )

    return response.json()["response"]

# 用法
url = "https://your-http-pub.clorecloud.net"
result = analyze_image(url, "photo.jpg", "详细描述这张图像")
print(result)
```

## 应用场景

### 图像描述

```python
prompt = "详细描述这张图像，包括颜色、对象和氛围。"
```

### OCR / 文本提取

```python
prompt = "提取这张图中可见的所有文字。请清晰格式化。"
```

### 图表分析

```python
prompt = "分析这张图表。关键趋势和洞察是什么？"
```

### 截图中的代码

```python
prompt = "提取这张截图中显示的代码。只提供代码。"
```

### 目标检测

```python
prompt = "列出这张图中可见的所有对象及其大致位置。"
```

## Gradio 界面

```python
import gradio as gr
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
model = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

def analyze_image(image, question):
    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": question}
            ]
        }
    ]

    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(prompt, image, return_tensors="pt").to("cuda")

    output = model.generate(**inputs, max_new_tokens=500)
    response = processor.decode(output[0], skip_special_tokens=True)

    # 提取助手回复
    return response.split("[/INST]")[-1].strip()

demo = gr.Interface(
    fn=analyze_image,
    inputs=[
        gr.Image(type="pil", label="图像"),
        gr.Textbox(label="问题", value="详细描述这张图像")
    ],
    outputs=gr.Textbox(label="响应"),
    title="LLaVA 视觉助手"
)

demo.launch(server_name="0.0.0.0", server_port=8000)
```

## API 服务器

```python
from fastapi import FastAPI, UploadFile, File, Form
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import io

app = FastAPI()

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
model = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

@app.post("/analyze")
async def analyze(
    image: UploadFile = File(...),
    question: str = Form(default="描述这张图像")
):
    img = Image.open(io.BytesIO(await image.read()))

    conversation = [
        {
            "role": "user",
            "content": [
                {"type": "image"},
                {"type": "text", "text": question}
            ]
        }
    ]

    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(prompt, img, return_tensors="pt").to("cuda")

    output = model.generate(**inputs, max_new_tokens=500)
    response = processor.decode(output[0], skip_special_tokens=True)

    return {"response": response.split("[/INST]")[-1].strip()}

# 运行：uvicorn server:app --host 0.0.0.0 --port 8000
```

## 批量处理

```python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
import torch
from PIL import Image
import os

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
model = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto"
)

def analyze_image(image_path, question):
    image = Image.open(image_path)

    conversation = [
        {"role": "user", "content": [
            {"type": "image"},
            {"type": "text", "text": question}
        ]}
    ]

    prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
    inputs = processor(prompt, image, return_tensors="pt").to("cuda")

    output = model.generate(**inputs, max_new_tokens=300)
    return processor.decode(output[0], skip_special_tokens=True).split("[/INST]")[-1].strip()

# 处理图像文件夹
image_folder = "./images"
results = []

for filename in os.listdir(image_folder):
    if filename.endswith(('.jpg', '.png', '.jpeg')):
        path = os.path.join(image_folder, filename)
        description = analyze_image(path, "简要描述这张图像")
        results.append({"file": filename, "description": description})
        print(f"{filename}: {description[:100]}...")

# 保存结果
import json
with open("descriptions.json", "w") as f:
    json.dump(results, f, indent=2)
```

## 内存优化

### 4位量化

```python
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf",
    quantization_config=quantization_config,
    device_map="auto"
)
```

### CPU 卸载

```python
model = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto",
    offload_folder="offload"
)
```

## 性能

| 模型            | GPU      | 每秒 Token 数 |
| ------------- | -------- | ---------- |
| LLaVA-1.5-7B  | RTX 3090 | \~30       |
| LLaVA-1.5-7B  | RTX 4090 | \~45       |
| LLaVA-1.6-7B  | RTX 4090 | \~40       |
| LLaVA-1.5-13B | A100     | \~35       |

## 故障排查

### 内存不足

```python

# 使用 4-bit 量化

# 或使用更小的模型（7B 而不是 13B）

# 或处理更小的图像
image = image.resize((336, 336))
```

### 生成缓慢

* 使用 Flash Attention
* 减少 max\_new\_tokens
* 使用量化模型

### 质量较差

* 使用更大的模型
* 带上下文的更好提示词
* 更高分辨率的图像

## 成本估算

CLORE.AI 市场常见费率（截至 2024 年）：

| GPU       | 小时费率    | 日费率     | 4 小时会话  |
| --------- | ------- | ------- | ------- |
| RTX 3060  | \~$0.03 | \~$0.70 | \~$0.12 |
| RTX 3090  | \~$0.06 | \~$1.50 | \~$0.25 |
| RTX 4090  | \~$0.10 | \~$2.30 | \~$0.40 |
| A100 40GB | \~$0.17 | \~$4.00 | \~$0.70 |
| A100 80GB | \~$0.25 | \~$6.00 | \~$1.00 |

*价格因提供商和需求而异。请查看* [*CLORE.AI 市场*](https://clore.ai/marketplace) *以获取当前费率。*

**节省费用：**

* 使用 **竞价** 可中断工作市场——约三分之一的服务器将现货价格定得低于按需价格（中位数约优惠 13%），其余则与按需价格持平
* 使用 **CLORE** 代币支付
* 比较不同提供商的价格

## 下一步

* Ollama LLM - 使用 Ollama 运行 LLaVA
* RAG + LangChain - 视觉 + RAG
* vLLM 推理 - 生产环境服务


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.clore.ai/guides/guides_v2-zh/shi-jue-mo-xing/llava-vision-language.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.