Overview

Run large language models (LLMs) on CLORE.AI GPUs for inference and chat applications.

Popular Tools

Tool	Use Case	Difficulty
Ollama	Easiest LLM setup	Beginner
Open WebUI	ChatGPT-like interface	Beginner
vLLM	High-throughput production serving	Medium
Llama.cpp Server	Efficient GGUF inference	Easy
Text Generation WebUI	Full-featured chat UI	Easy
ExLlamaV2	Fastest EXL2 inference	Medium
LocalAI	OpenAI-compatible API	Medium
SGLang	Fast structured generation	Medium
Text Generation Inference (TGI)	HuggingFace serving solution	Medium
LMDeploy	MMlab serving toolkit	Medium
Aphrodite Engine	vLLM fork with extra features	Medium
MLC-LLM	Machine learning compilation	Hard
LiteLLM	Unified API proxy	Medium
PowerInfer	Sparse model inference	Hard
Mistral.rs	Rust-based inference engine	Medium

Model Guides

Latest & Best Models

Model	Parameters	License	Fits on
Qwen3.8-27B	27B dense	Apache 2.0	1× RTX 4090 (Q4)
Mistral Small 4	119B / 6.5B active	Apache 2.0	2× RTX 4090 (Q2)
GLM-5.2	~750B / 40B active	MIT	10× RTX 5090 (Q2)
DeepSeek V4	Flash 167GB · Pro 893GB	MIT	6–8× RTX 5090 (Flash)
MiniMax M3	428B / 23B active	minimax-community	6–8× RTX 5090 (Q2–Q3)
Nemotron 3 Ultra	550B / 55B active	OpenMDW-1.1	4× RTX PRO 6000 (NVFP4)
Kimi K3	2.8T	Kimi K3 licence	Nothing on the marketplace
GLM-5.1	744B / 40B active	MIT	Large rigs
Llama 4	Scout & Maverick	Llama Community	1× RTX 4090 (Scout Q4)
DeepSeek-R1	671B MoE	MIT	Large rigs
Qwen2.5	0.5B–72B	Apache 2.0	Any card

Specialized Models

Model	Parameters	Best For
DeepSeek Coder	6.7B-33B	Code generation
CodeLlama	7B-34B	Code completion
GLM-4.7-Flash	4.7B	Fast Chinese/English
GLM-5	TBA	Zhipu AI latest
Kimi K2.5	TBA	Moonshot AI model
Ling-2.5-1T	1T	Massive open-source LLM
LFM2-24B	24B	Liquid AI model
MiMo-V2-Flash	TBA	Fast inference model

Efficient Models

Model	Parameters	Best For
Gemma 2	2B-27B	Efficient inference
Gemma 3	TBA	Google's latest compact
Phi-4	14B	Small but capable
Mistral/Mixtral	7B / 8x7B	General purpose
Mistral Large 3	675B MoE	Enterprise-grade
Mistral Small 3.1	TBA	Efficient Mistral variant

GPU Recommendations

Multi-GPU 80GB-class rigs are not listed on the Clore.ai marketplace. The biggest boxes listed today are 4× RTX PRO 6000 Blackwell (96GB each, 380GB total) and 8–11× RTX 5090 (32GB each). A100 / H200 / B200 capacity is sold as bare metal on request. Check GPU Pricing & Availability before sizing a deployment.

Model Size	Minimum GPU	Recommended	Clore.ai cost
7B (Q4)	RTX 3060 12GB	RTX 3090	$0.03–0.21/hr
13B (Q4)	RTX 3090 24GB	RTX 4090	$0.07–0.42/hr
27B (Q4)	RTX 3090 24GB	RTX 4090 / 5090	$0.07–0.77/hr
34B (Q4)	2× RTX 3090	1× RTX 5090	$0.14–0.77/hr
70B (Q4)	2× RTX 5090	1× RTX PRO 6000 96GB	$0.50–1.54/hr
120B+ (Q2)	2× RTX 4090	2× RTX 5090	$0.28–1.54/hr
400B+ (Q2)	8× RTX 5090	4× RTX PRO 6000	~$2.00–5.00/hr

Quantization Guide

Format	VRAM Usage	Quality	Speed
Q2_K	Lowest	Poor	Fastest
Q4_K_M	Low	Good	Fast
Q5_K_M	Medium	Great	Medium
Q8_0	High	Excellent	Slower
FP16	Highest	Best	Slowest

See Also

• Training & Fine-tuning

• Vision-Language Models