# Fine-tune LLM

Train your own custom LLM using efficient fine-tuning techniques on CLORE.AI GPUs.

{% hint style="success" %}
All examples can be run on GPU servers rented through [CLORE.AI Marketplace](https://clore.ai/marketplace).
{% endhint %}

## Renting on CLORE.AI

1. Visit [CLORE.AI Marketplace](https://clore.ai/marketplace)
2. Filter by GPU type, VRAM, and price
3. Choose **On-Demand** (fixed rate) or **Spot** (bid price)
4. Configure your order:
   * Select Docker image
   * Set ports (TCP for SSH, HTTP for web UIs)
   * Add environment variables if needed
   * Enter startup command
5. Select payment: **CLORE**, **BTC**, or **USDT/USDC**
6. Create order and wait for deployment

### Access Your Server

* Find connection details in **My Orders**
* Web interfaces: Use the HTTP port URL
* SSH: `ssh -p <port> root@<proxy-address>`

## What is LoRA/QLoRA?

* **LoRA** (Low-Rank Adaptation) - Train small adapter layers instead of full model
* **QLoRA** - LoRA with quantization for even less VRAM
* Train 7B model on single RTX 3090
* Train 70B model on single A100

## Requirements

| Model | Method    | Min VRAM | Recommended |
| ----- | --------- | -------- | ----------- |
| 7B    | QLoRA     | 12GB     | RTX 3090    |
| 13B   | QLoRA     | 20GB     | RTX 4090    |
| 70B   | QLoRA     | 48GB     | A100 80GB   |
| 7B    | Full LoRA | 24GB     | RTX 4090    |

## Quick Deploy

**Docker Image:**

```
pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel
```

**Ports:**

```
22/tcp
8888/http
6006/http
```

**Command:**

```bash
pip install "transformers>=4.45" "datasets>=2.20" accelerate "peft>=0.14" \
    bitsandbytes "trl>=0.12" wandb jupyterlab && \
jupyter lab --ip=0.0.0.0 --port=8888 --allow-root
```

## Accessing Your Service

After deployment, find your `http_pub` URL in **My Orders**:

1. Go to **My Orders** page
2. Click on your order
3. Find the `http_pub` URL (e.g., `abc123.clorecloud.net`)

Use `https://YOUR_HTTP_PUB_URL` instead of `localhost` in examples below.

## Dataset Preparation

### Chat Format (Recommended)

```json
[
  {
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is Python?"},
      {"role": "assistant", "content": "Python is a programming language..."}
    ]
  }
]
```

### Instruction Format

```json
[
  {
    "instruction": "Translate to French",
    "input": "Hello, how are you?",
    "output": "Bonjour, comment allez-vous?"
  }
]
```

### Alpaca Format

```json
[
  {
    "instruction": "Give three tips for staying healthy.",
    "input": "",
    "output": "1. Eat balanced meals..."
  }
]
```

## Supported Modern Models (2025)

| Model                       | HF ID                                     | Min VRAM (QLoRA) |
| --------------------------- | ----------------------------------------- | ---------------- |
| Llama 3.1 / 3.3 8B          | `meta-llama/Llama-3.1-8B-Instruct`        | 12GB             |
| Qwen 2.5 7B / 14B           | `Qwen/Qwen2.5-7B-Instruct`                | 12GB / 20GB      |
| DeepSeek-R1-Distill (7B/8B) | `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` | 12GB             |
| Mistral 7B v0.3             | `mistralai/Mistral-7B-Instruct-v0.3`      | 12GB             |
| Gemma 2 9B                  | `google/gemma-2-9b-it`                    | 14GB             |
| Phi-4 14B                   | `microsoft/phi-4`                         | 20GB             |

## QLoRA Fine-tuning Script

Modern example with PEFT 0.14+, Flash Attention 2, DoRA support, and Qwen2.5 / DeepSeek-R1 compatibility:

```python
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

# === Configuration ===
# Choose one of: Qwen2.5, DeepSeek-R1-Distill, Llama 3.1, Mistral, etc.
MODEL_NAME = "Qwen/Qwen2.5-7B-Instruct"
# MODEL_NAME = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
# MODEL_NAME = "meta-llama/Llama-3.1-8B-Instruct"

DATASET = "your_dataset.json"  # or HuggingFace dataset name
OUTPUT_DIR = "./output"
MAX_SEQ_LENGTH = 4096           # Qwen2.5 supports up to 32K context
USE_DORA = True                 # DoRA improves quality over standard LoRA
USE_FLASH_ATTN = True           # Flash Attention 2 saves VRAM & speeds up

# === Load Model with 4-bit Quantization ===
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,  # Required for Qwen2.5 and DeepSeek
    # Flash Attention 2: requires Ampere+ GPU (RTX 30/40, A100)
    attn_implementation="flash_attention_2" if USE_FLASH_ATTN else "eager",
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# === Configure LoRA with optional DoRA ===
# DoRA (Weight-Decomposed Low-Rank Adaptation) — PEFT >= 0.14 required
# use_dora=True decomposes weights into magnitude + direction for better quality
lora_config = LoraConfig(
    r=64,                    # Rank (higher = more capacity, more VRAM)
    lora_alpha=16,           # Scaling factor (keep equal to or half of r)
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",  # Attention layers
        "gate_proj", "up_proj", "down_proj",      # MLP layers
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    use_dora=USE_DORA,        # DoRA: improved quality (PEFT 0.14+)
    # use_rslora=True,        # Optional: Rank-Stabilized LoRA
)

# Prepare model for QLoRA training
model = prepare_model_for_kbit_training(
    model,
    use_gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
)
model = get_peft_model(model, lora_config)

# Print trainable parameters summary
model.print_trainable_parameters()
# Example output: trainable params: 42,991,616 || all params: 7,284,891,648 || trainable%: 0.59

# === Load Dataset ===
dataset = load_dataset("json", data_files=DATASET)
# Or use a public dataset:
# dataset = load_dataset("HuggingFaceH4/ultrachat_200k")

# === Format Dataset for Qwen2.5 / ChatML format ===
def format_chat_qwen(example):
    """Format for Qwen2.5 using ChatML template."""
    messages = example.get("messages", [])
    if not messages:
        # Handle alpaca-style data
        text = f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
        text += f"<|im_start|>user\n{example['instruction']}"
        if example.get("input"):
            text += f"\n{example['input']}"
        text += f"<|im_end|>\n<|im_start|>assistant\n{example['output']}<|im_end|>"
    else:
        # Handle messages format (ChatML)
        text = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=False,
        )
    return {"text": text}

dataset = dataset.map(format_chat_qwen, remove_columns=dataset["train"].column_names)

# === Training Arguments (PEFT 0.14+ / TRL 0.12+) ===
training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,         # Effective batch = 2 * 8 = 16
    learning_rate=2e-4,
    weight_decay=0.001,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_steps=100,
    save_total_limit=3,
    bf16=True,                             # Use bf16 for modern GPUs (A100, RTX 30/40)
    # fp16=True,                           # Use fp16 for older GPUs
    optim="paged_adamw_8bit",
    max_grad_norm=0.3,
    group_by_length=True,
    report_to="wandb",                     # or "tensorboard"
    # SFTConfig-specific:
    max_seq_length=MAX_SEQ_LENGTH,
    dataset_text_field="text",
    packing=True,                          # Pack multiple examples for efficiency
)

# === Train ===
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    args=training_args,
)

trainer.train()

# === Save LoRA adapter ===
trainer.save_model(f"{OUTPUT_DIR}/final")
tokenizer.save_pretrained(f"{OUTPUT_DIR}/final")
print(f"Model saved to {OUTPUT_DIR}/final")
```

## Flash Attention 2

Flash Attention 2 reduces VRAM usage and speeds up training significantly. Requires Ampere+ GPU (RTX 3090, RTX 4090, A100).

```bash
# Install Flash Attention 2
pip install flash-attn --no-build-isolation
```

```python
# Enable in model loading:
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    attn_implementation="flash_attention_2",  # <-- add this
    torch_dtype=torch.bfloat16,               # FA2 requires bf16 or fp16
    device_map="auto",
)
```

| Setting                   | VRAM (7B) | Speed    |
| ------------------------- | --------- | -------- |
| Standard attention (fp16) | \~22GB    | baseline |
| Flash Attention 2 (bf16)  | \~16GB    | +30%     |
| Flash Attention 2 + QLoRA | \~12GB    | +30%     |

## DoRA (Weight-Decomposed LoRA)

DoRA (PEFT >= 0.14) decomposes pre-trained weights into magnitude and direction components. It improves fine-tuning quality, especially for smaller ranks.

```python
from peft import LoraConfig

# Standard LoRA
lora_config = LoraConfig(r=64, lora_alpha=16, use_dora=False, ...)

# DoRA — same parameters, better quality
lora_config = LoraConfig(r=64, lora_alpha=16, use_dora=True, ...)
# Note: DoRA adds ~5-10% VRAM overhead vs standard LoRA
# Note: Not compatible with quantized (4-bit/8-bit) models in all cases
```

## Qwen2.5 & DeepSeek-R1-Distill Examples

### Qwen2.5 Fine-tuning

```python
MODEL_NAME = "Qwen/Qwen2.5-7B-Instruct"
# For 14B: "Qwen/Qwen2.5-14B-Instruct" (needs 20GB+ VRAM with QLoRA)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,          # Required for Qwen2.5
    attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)

# Qwen2.5 uses ChatML format — use apply_chat_template
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there! How can I help?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
```

### DeepSeek-R1-Distill Fine-tuning

DeepSeek-R1-Distill models (Qwen-7B, Qwen-14B, Llama-8B, Llama-70B) are reasoning-focused. Fine-tune to adapt their chain-of-thought style to your domain.

```python
# DeepSeek-R1-Distill variants
MODEL_NAME = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"   # 7B on Qwen2.5 base
# MODEL_NAME = "deepseek-ai/DeepSeek-R1-Distill-Llama-8B" # 8B on Llama3 base
# MODEL_NAME = "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B" # 14B (needs A100)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
)

# DeepSeek-R1 uses <think>...</think> tags for reasoning
# Keep this in training data to preserve chain-of-thought capability
example_format = """<|im_start|>user
Solve: What is 15 * 23?<|im_end|>
<|im_start|>assistant
<think>
15 * 23 = 15 * 20 + 15 * 3 = 300 + 45 = 345
</think>
The answer is 345.<|im_end|>"""

# LoRA target modules for DeepSeek-R1-Distill (Qwen2.5 base)
lora_config = LoraConfig(
    r=32,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    use_dora=True,
    task_type="CAUSAL_LM",
)
```

## Using Axolotl (Easier)

Axolotl simplifies fine-tuning with YAML configs:

```bash
pip install axolotl

# Create config
cat > config.yml << 'EOF'
base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer

load_in_4bit: true
adapter: qlora
lora_r: 32
lora_alpha: 16

datasets:
  - path: your_data.json
    type: alpaca

sequence_len: 4096
sample_packing: true

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 3
learning_rate: 2e-4

output_dir: ./output
EOF

# Train
accelerate launch -m axolotl.cli.train config.yml
```

## Axolotl Config Examples

### Chat Model

```yaml
base_model: mistralai/Mistral-7B-Instruct-v0.2
load_in_4bit: true
adapter: qlora

datasets:
  - path: data.json
    type: sharegpt

chat_template: mistral
```

### Code Model

```yaml
base_model: codellama/CodeLlama-7b-hf
load_in_4bit: true
adapter: qlora

datasets:
  - path: code_data.json
    type: alpaca

sequence_len: 8192  # Longer context for code
```

## Merging LoRA Weights

After training, merge LoRA back into base model:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    torch_dtype=torch.float16,
    device_map="auto",
)

# Load LoRA
model = PeftModel.from_pretrained(base_model, "./output/final")

# Merge
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")
```

## Convert to GGUF

For use with llama.cpp/Ollama:

```bash

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Convert
python convert.py ../merged_model --outtype f16 --outfile model-f16.gguf

# Quantize
./quantize model-f16.gguf model-q4_k_m.gguf q4_k_m
```

## Monitoring Training

### Weights & Biases

```python
import wandb
wandb.init(project="llm-finetune", name="mistral-7b-lora")
```

### TensorBoard

```python

# In training args
report_to="tensorboard"
logging_dir="./logs"

# View
tensorboard --logdir ./logs --port 6006 --bind_all
```

## Best Practices

### Hyperparameters

| Parameter   | 7B Model | 13B Model | 70B Model |
| ----------- | -------- | --------- | --------- |
| batch\_size | 4        | 2         | 1         |
| grad\_accum | 4        | 8         | 16        |
| lr          | 2e-4     | 1e-4      | 5e-5      |
| lora\_r     | 64       | 32        | 16        |
| epochs      | 3        | 2-3       | 1-2       |

### Dataset Size

* Minimum: 1,000 examples
* Good: 10,000+ examples
* Quality > Quantity

### Avoiding Overfitting

```python
training_args = TrainingArguments(
    ...
    weight_decay=0.01,
    warmup_ratio=0.03,
    save_total_limit=3,
    load_best_model_at_end=True,
    evaluation_strategy="steps",
    eval_steps=100,
)
```

## Multi-GPU Training

```bash

# With accelerate
accelerate launch --multi_gpu --num_processes 4 train.py

# With DeepSpeed
accelerate launch --use_deepspeed --num_processes 4 train.py
```

DeepSpeed config:

```json
{
  "bf16": {"enabled": true},
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {"device": "cpu"}
  }
}
```

## Saving & Exporting

```bash

# Save LoRA adapter
trainer.save_model("./lora_adapter")

# Save merged model
merged_model.save_pretrained("./full_model")

# Upload to HuggingFace
huggingface-cli login
merged_model.push_to_hub("username/my-model")
```

## Troubleshooting

### OOM Errors

* Reduce batch size
* Increase gradient accumulation
* Use `gradient_checkpointing=True`
* Reduce lora\_r

### Training Loss Not Decreasing

* Check data format
* Increase learning rate
* Check for data issues

### NaN Loss

* Reduce learning rate
* Use fp32 instead of fp16
* Check for corrupted data

## Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

| GPU       | Hourly Rate | Daily Rate | 4-Hour Session |
| --------- | ----------- | ---------- | -------------- |
| RTX 3060  | \~$0.03     | \~$0.70    | \~$0.12        |
| RTX 3090  | \~$0.06     | \~$1.50    | \~$0.25        |
| RTX 4090  | \~$0.10     | \~$2.30    | \~$0.40        |
| A100 40GB | \~$0.17     | \~$4.00    | \~$0.70        |
| A100 80GB | \~$0.25     | \~$6.00    | \~$1.00        |

*Prices vary by provider and demand. Check* [*CLORE.AI Marketplace*](https://clore.ai/marketplace) *for current rates.*

> 📚 See also: [How to Fine-Tune LLaMA 3 on a Cloud GPU — Step-by-Step Guide](https://blog.clore.ai/how-to-fine-tune-llama-3-cloud-gpu/)

**Save money:**

* Use **Spot** market for flexible workloads (often 30-50% cheaper)
* Pay with **CLORE** tokens
* Compare prices across different providers