Overview

Run large language models (LLMs) on CLORE.AI GPUs for inference and chat applications.

Popular Tools

Tool	Use Case	Difficulty
Ollama	Easiest LLM setup	Beginner
Open WebUI	ChatGPT-like interface	Beginner
vLLM	High-throughput production serving	Medium
Llama.cpp Server	Efficient GGUF inference	Easy
Text Generation WebUI	Full-featured chat UI	Easy
ExLlamaV2	Fastest EXL2 inference	Medium
LocalAI	OpenAI-compatible API	Medium
SGLang	Fast structured generation	Medium
Text Generation Inference (TGI)	HuggingFace serving solution	Medium
LMDeploy	MMlab serving toolkit	Medium
Aphrodite Engine	vLLM fork with extra features	Medium
MLC-LLM	Machine learning compilation	Hard
LiteLLM	Unified API proxy	Medium
PowerInfer	Sparse model inference	Hard
Mistral.rs	Rust-based inference engine	Medium

Model Guides

Latest & Best Models

Model	Parameters	Best For
DeepSeek-V3	671B MoE	Reasoning, code, math
DeepSeek-R1	671B MoE	Advanced reasoning
DeepSeek V4	TBA	Next-generation DeepSeek
Qwen2.5	0.5B-72B	Multilingual, code
Qwen3.5	TBA	Latest Qwen generation
Llama 3.3	70B	Meta's latest 70B
Llama 4	TBA	Scout & Maverick variants

Specialized Models

Model	Parameters	Best For
DeepSeek Coder	6.7B-33B	Code generation
CodeLlama	7B-34B	Code completion
GLM-4.7-Flash	4.7B	Fast Chinese/English
GLM-5	TBA	Zhipu AI latest
Kimi K2.5	TBA	Moonshot AI model
Ling-2.5-1T	1T	Massive open-source LLM
LFM2-24B	24B	Liquid AI model
MiMo-V2-Flash	TBA	Fast inference model

Efficient Models

Model	Parameters	Best For
Gemma 2	2B-27B	Efficient inference
Gemma 3	TBA	Google's latest compact
Phi-4	14B	Small but capable
Mistral/Mixtral	7B / 8x7B	General purpose
Mistral Large 3	675B MoE	Enterprise-grade
Mistral Small 3.1	TBA	Efficient Mistral variant

GPU Recommendations

Model Size	Minimum GPU	Recommended
7B (Q4)	RTX 3060 12GB	RTX 3090
13B (Q4)	RTX 3090 24GB	RTX 4090
34B (Q4)	2x RTX 3090	A100 40GB
70B (Q4)	A100 80GB	2x A100

Quantization Guide

Format	VRAM Usage	Quality	Speed
Q2_K	Lowest	Poor	Fastest
Q4_K_M	Low	Good	Fast
Q5_K_M	Medium	Great	Medium
Q8_0	High	Excellent	Slower
FP16	Highest	Best	Slowest

See Also

• Training & Fine-tuning

• Vision-Language Models