LitGPT

LitGPT is a high-performance library for pretraining, finetuning, and deploying 20+ large language models built on PyTorch Lightning. With 12K+ GitHub stars, it's a go-to toolkit for engineers who need clean, hackable LLM training code without the abstraction overhead of HuggingFace Transformers.

Each model in LitGPT is ~1,000 lines of clean PyTorch — no inheritance chains 10 levels deep, no magic. You can read the Llama 3 implementation end-to-end in an afternoon and modify it confidently.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

---

What is LitGPT?

LitGPT provides production-ready implementations of state-of-the-art LLMs with a unified training interface:

• 20+ supported models — Llama 3, Gemma 2, Mistral, Phi-3, Falcon, StableLM, and more

• Pretrain from scratch — full pretraining with Flash Attention, FSDP, and gradient checkpointing

• Finetune efficiently — full finetuning, LoRA, QLoRA, and Adapter methods

• Serve with confidence — built-in inference server with quantization

• Multi-GPU support — DDP, FSDP, tensor parallelism out of the box

• Memory efficient — 4-bit quantization, gradient checkpointing, activation checkpointing

---

Server Requirements

Component	Minimum	Recommended
GPU	RTX 3090 (24 GB)	A100 80 GB / H100
VRAM	16 GB (7B LoRA)	80 GB+ (70B full)
RAM	32 GB	64 GB+
CPU	8 cores	16+ cores
Storage	100 GB	500 GB+
OS	Ubuntu 20.04+	Ubuntu 22.04
Python	3.10+	3.11
CUDA	11.8+	12.1+

VRAM Requirements by Task

Task	Model	VRAM
Inference (4-bit)	Llama-3 8B	~6 GB
LoRA finetune	Llama-3 8B	~16 GB
Full finetune	Llama-3 8B	~80 GB
LoRA finetune	Llama-3 70B	~48 GB (2×A100)
Full finetune	Llama-3 70B	~640 GB (8×A100)
QLoRA finetune	Llama-3 8B	~8 GB

---

Ports

Port	Service	Notes
22	SSH	Terminal access & file transfer
8000	LitGPT Inference Server	REST API for model serving

---

Quick Start with Docker

# Pull the official LitGPT image
docker pull pytorchlightning/litgpt:latest

# Run interactive container with GPU
docker run -it --gpus all \
  -p 8000:8000 \
  -v $(pwd)/checkpoints:/checkpoints \
  -v $(pwd)/data:/data \
  pytorchlightning/litgpt:latest \
  bash

# Or run a specific command directly
docker run --gpus all \
  -v $(pwd)/checkpoints:/checkpoints \
  pytorchlightning/litgpt:latest \
  litgpt download --repo_id meta-llama/Llama-3.2-3B-Instruct

---

Installation on Clore.ai

Step 1 — Rent a Server

1. Go to Clore.ai Marketplace

2. Filter for VRAM ≥ 24 GB (RTX 3090 or better)

3. Choose a PyTorch or CUDA 12.1 base image

4. Open ports 22 and 8000 in your order settings

5. Select storage ≥ 200 GB for model weights

Step 2 — Connect via SSH

ssh root@<server-ip> -p <ssh-port>

Step 3 — Install LitGPT

# Install via pip (recommended)
pip install litgpt

# With all extras (quantization, server, etc.)
pip install 'litgpt[all]'

# Or install from source for latest features
git clone https://github.com/Lightning-AI/litgpt.git
cd litgpt
pip install -e '.[all]'

Step 4 — Verify Installation

litgpt --help

Expected output:

Usage: litgpt [OPTIONS] COMMAND [ARGS]...
  
Commands:
  chat       Chat with a model
  convert    Convert model weights
  download   Download model weights
  evaluate   Evaluate a model
  finetune   Finetune a model
  generate   Generate text
  pretrain   Pretrain a model
  serve      Serve a model for inference

---

Downloading Models

LitGPT downloads models from Hugging Face:

# List available models
litgpt download --list

# Download Llama 3.2 3B (requires HF token for gated models)
litgpt download \
  --repo_id meta-llama/Llama-3.2-3B-Instruct \
  --checkpoint_dir checkpoints/

# Download Mistral 7B (open access)
litgpt download \
  --repo_id mistralai/Mistral-7B-Instruct-v0.3

# Download Gemma 2 2B
litgpt download \
  --repo_id google/gemma-2-2b-it \
  --access_token your-hf-token

# Download Phi-3 (small but powerful)
litgpt download \
  --repo_id microsoft/Phi-3-mini-4k-instruct

Set HuggingFace Token

# For gated models (Llama, Gemma)
export HF_TOKEN=hf_your-token-here

# Or authenticate via CLI
pip install huggingface_hub
huggingface-cli login

---

Inference (Chat & Generate)

# Interactive chat
litgpt chat \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct

# Single generation
litgpt generate \
  --prompt "Explain GPU computing in simple terms" \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct \
  --max_new_tokens 200

# With temperature and sampling
litgpt generate \
  --prompt "Write a Python function to sort a list" \
  --checkpoint_dir checkpoints/mistralai/Mistral-7B-Instruct-v0.3 \
  --temperature 0.7 \
  --top_p 0.9 \
  --max_new_tokens 500

---

Finetuning

LoRA Finetuning (Recommended)

LoRA trains a small set of adapter parameters (typically 0.1–1% of total weights) while the base model stays frozen. Llama 3 8B LoRA on 10K examples takes ~2 hours on an RTX 3090 with r=16.

# Prepare your dataset
# Format: JSON lines with {"instruction": "...", "input": "...", "output": "..."}
cat > data/train.json << 'EOF'
{"instruction": "What is GPU cloud computing?", "input": "", "output": "GPU cloud computing provides on-demand access to GPU hardware through the internet, enabling AI training and inference without owning physical hardware."}
{"instruction": "How do I rent a GPU on Clore.ai?", "input": "", "output": "Visit clore.ai/marketplace, filter by GPU specs, select a server, configure ports, and click rent. SSH access is provided immediately."}
EOF

# Finetune with LoRA
litgpt finetune lora \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct \
  --data JSON \
  --data.json_path data/train.json \
  --train.epochs 3 \
  --train.micro_batch_size 4 \
  --lora_r 8 \
  --lora_alpha 16 \
  --out_dir out/llama-lora-finetuned

# Monitor training
# LitGPT outputs logs with loss, learning rate, and ETA

QLoRA (4-bit + LoRA)

Use QLoRA to finetune large models on limited VRAM. Llama 3 8B fits on a single RTX 3090 at 24 GB:

litgpt finetune lora \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-8B-Instruct \
  --quantize bnb.nf4 \
  --train.epochs 3 \
  --train.micro_batch_size 2 \
  --lora_r 16 \
  --lora_alpha 32 \
  --out_dir out/llama-qlora

Full Finetuning

litgpt finetune full \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct \
  --data JSON \
  --data.json_path data/train.json \
  --train.epochs 2 \
  --train.micro_batch_size 2 \
  --train.accumulate_gradients 8 \
  --out_dir out/llama-full-finetuned

Multi-GPU Training

# Use FSDP across multiple GPUs
litgpt finetune full \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-8B-Instruct \
  --devices 4 \
  --strategy fsdp \
  --train.epochs 3 \
  --out_dir out/llama-multigpu

---

Serving Models (REST API)

# Start inference server
litgpt serve \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct \
  --host 0.0.0.0 \
  --port 8000

# Test the API
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What is the capital of France?",
    "max_new_tokens": 100,
    "temperature": 0.7
  }'

Python Client

import requests

response = requests.post(
    "http://<server-ip>:8000/predict",
    json={
        "prompt": "Explain reinforcement learning",
        "max_new_tokens": 500,
        "temperature": 0.8,
        "top_p": 0.9,
    }
)
print(response.json()["output"])

---

Pretraining from Scratch

For training a custom LLM from scratch on your own data:

# Prepare pretraining data (tokenized and chunked)
python scripts/prepare_redpajama.py \
  --source_path /data/raw_text \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct \
  --destination_path /data/tokenized

# Start pretraining
litgpt pretrain \
  --model_name Llama-3.2 \
  --data /data/tokenized \
  --train.micro_batch_size 4 \
  --train.max_tokens 10_000_000_000 \
  --devices 8 \
  --strategy fsdp \
  --out_dir out/my-pretrained-llm

---

Converting and Exporting Models

# Merge LoRA weights into base model
litgpt merge_lora \
  --checkpoint_dir out/llama-lora-finetuned

# Convert to HuggingFace format for distribution
litgpt convert to_hf \
  --checkpoint_dir out/llama-lora-finetuned/final \
  --output_dir hf_model/

# Export to GGUF format (for Ollama/LlamaCpp)
# Use llama.cpp conversion script after HF export
python llama.cpp/convert.py hf_model/ --outfile model.gguf

---

Evaluating Models

# Run MMLU benchmark
litgpt evaluate \
  --checkpoint_dir checkpoints/meta-llama/Llama-3.2-3B-Instruct \
  --tasks mmlu \
  --num_fewshot 5

# Run multiple benchmarks
litgpt evaluate \
  --checkpoint_dir out/llama-lora-finetuned/final \
  --tasks "mmlu,hellaswag,truthfulqa_mc"

---

Clore.ai GPU Recommendations

LitGPT covers three distinct workloads — inference, LoRA finetuning, and full pretraining — each with different GPU requirements.

Workload	GPU	VRAM	Notes
Inference / chat (7–8B models)	RTX 3090	24 GB	Fits Llama 3 8B in bf16; ~95 tok/s generation
LoRA finetune (7–8B models)	RTX 3090	24 GB	Budget pick; QLoRA keeps VRAM under 10 GB
LoRA finetune (7–8B), fast iteration	RTX 4090	24 GB	~35% faster than 3090; reduces 2hr job to ~1.4hr
Full finetune (7B) or QLoRA (70B)	A100 40 GB	40 GB	40 GB fits 7B full-precision or 70B 4-bit
Full finetune (13B+) or pretrain runs	A100 80 GB	80 GB	Highest throughput; ~2,800 tok/sec training on 8B

Recommended for most users: RTX 3090 pair (2×24 GB = 48 GB effective with FSDP). Handles QLoRA on 70B models, or full finetune on 7B models with tensor parallelism. Cost on Clore.ai: ~$0.25/hr for two 3090s.

For pretraining or >70B finetuning: Use 4×A100 80GB with FSDP. LitGPT's FSDP integration handles sharding transparently — just pass --devices 4 --strategy fsdp.

---

Troubleshooting

CUDA Out of Memory

# Reduce batch size
--train.micro_batch_size 1

# Enable gradient checkpointing
--train.gradient_checkpointing true

# Use QLoRA instead of LoRA
--quantize bnb.nf4

# Check GPU memory
nvidia-smi

Download fails / HuggingFace 401

# Set HF token
export HF_TOKEN=hf_your-token-here
huggingface-cli login

# Or pass directly
litgpt download \
  --repo_id meta-llama/Llama-3.2-3B-Instruct \
  --access_token hf_your-token

Training loss doesn't decrease

# Check your data format — must be valid JSON Lines
python -c "
import json
with open('data/train.json') as f:
    for i, line in enumerate(f):
        json.loads(line)
        if i < 3: print(f'Line {i}: OK')
print('All lines valid')
"

# Reduce learning rate
--train.lr 1e-5  # Default is often too high for small datasets

# Check data size — LoRA needs at least 100-1000 examples
wc -l data/train.json

Server port 8000 not accessible

# Verify server is listening
ss -tlnp | grep 8000

# Open firewall
ufw allow 8000/tcp

# Restart server with explicit host
litgpt serve \
  --checkpoint_dir checkpoints/... \
  --host 0.0.0.0 \
  --port 8000

Multi-GPU training hangs

# Check NCCL connectivity
python -c "import torch; print(torch.cuda.device_count())"

# Try DDP instead of FSDP for smaller models
--strategy ddp

# Set NCCL environment variables
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1  # If InfiniBand is not available

---

Useful Links

• GitHub: https://github.com/Lightning-AI/litgpt ⭐ 12K+

• Documentation: https://lightning.ai/docs/litgpt

• PyTorch Lightning: https://lightning.ai

• HuggingFace Models: https://huggingface.co/models

• Discord: https://discord.gg/lightning-ai

• Clore.ai Marketplace: https://clore.ai/marketplace