Mistral Small 4 (119B MoE, 6.5B Active)

Deploy Mistral Small 4 (119B MoE, 6.5B active) on two consumer GPUs from the Clore.ai marketplace — Apache 2.0, 256K context, vision and reasoning in one model

Status (August 2026): Mistral released Small 4 on 16 March 2026 under Apache 2.0. Weights: mistralai/Mistral-Small-4-119B-2603. 119B total parameters with only 6.5B active (128 experts, 4 active per token), 256K context, native vision, and a per-request reasoning toggle. It folds Mistral's three previous lines — Instruct, Magistral reasoning and Devstral coding — into a single checkpoint.

"Small" is a misnomer by parameter count and exactly right by cost. Only 6.5B parameters activate per token, so decode speed feels like a 7B model while quality comes from a 119B pool. Quantized to Q2 it is 40GB — two RTX 4090s, which the Clore.ai marketplace has 322 servers' worth of free capacity for. At Q3–Q4 it is 54–59GB, comfortable on two RTX 5090s.

If you want one open model that answers instantly for easy prompts, thinks harder when told to, reads images, and writes code, this is currently the best fit for hardware you can actually rent by the minute.

Key Specs

Property	Value
Parameters	119B total, 6.5B active (128 experts, 4 active)
Modality	Text + image in → text out
Context	262,144 tokens (256K)
License	Apache 2.0
Release Date	16 March 2026
Weights	Official ~242GB · Q3_K_M GGUF 54GB · Q2_K_XL GGUF 40GB
Reasoning	reasoning_effort: none (instant) or high (test-time compute)
Primary Tooling	vLLM, llama.cpp, Transformers

---

Requirements

Build	Size	Clore.ai setup
UD-IQ1_M	32GB	1× RTX PRO 6000 96GB, or 2× RTX 4090
UD-Q2_K_XL	40GB	2× RTX 4090 / 3090 (48GB) — 322 servers free at ≥48GB
UD-Q3_K_M	54GB	2× RTX 5090 (64GB)
UD-IQ4_NL	59GB	2× RTX 5090 (64GB)
Official checkpoint	~242GB	8× RTX 5090 (248GB)

Add headroom for the KV cache: at 256K context it is substantial even with an MoE this sparse. Start at 32–64K and raise it once you see actual usage.

---

Deploy with llama.cpp (two consumer cards)

huggingface-cli download unsloth/Mistral-Small-4-119B-2603-GGUF \
  --include "*UD-Q2_K_XL*" --local-dir /workspace/ms4

llama-server -m /workspace/ms4/*UD-Q2_K_XL*-00001-of-*.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 999 --split-mode layer \
  -c 65536 --flash-attn

Deploy with vLLM

vllm serve mistralai/Mistral-Small-4-119B-2603 \
  --tensor-parallel-size 8 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --enable-expert-parallel

An official NVFP4 build (mistralai/Mistral-Small-4-119B-2603-NVFP4) targets Blackwell cards — useful on RTX 50-series and RTX PRO 6000 rigs.

Reasoning on demand

client.chat.completions.create(
    model="mistral-small-4",
    messages=[{"role": "user", "content": "Find the deadlock in this scheduler."}],
    extra_body={"reasoning_effort": "high"},   # "none" for instant replies
    temperature=0.7,                            # 0.0–0.7 when reasoning is off
)

Mistral recommends temperature 0.7 with reasoning on. With reasoning off, stay between 0.0 and 0.7 depending on how deterministic you need the output.

---

Clore.ai GPU Recommendations

Setup	VRAM	Build	Context	Clore.ai cost
2× RTX 4090	48GB	Q2_K_XL	~64K	$0.28–0.84/hr
2× RTX 3090	48GB	Q2_K_XL	~64K	$0.14–0.42/hr
2× RTX 5090	64GB	Q3_K_M / IQ4_NL	~96K	$0.50–1.54/hr
1× RTX PRO 6000	96GB	Q3_K_M, single card	~131K	$0.92–1.38/hr
8× RTX 5090	248GB	official checkpoint	full 256K	~$2.00–3.50/hr

A pair of RTX 3090s is the value play — 48GB of VRAM for around $0.14–0.42/hr total, running a 119B model. Use partial GPU rental to take two cards out of a larger rig instead of hunting for a dedicated dual-3090 box.

---

Use Cases

• One model for a whole product — chat, coding and reasoning without three deployments

• Document and image extraction — vision plus 256K context handles long scanned documents

• Coding agents on a budget — Devstral's lineage, at 6.5B active parameters per token

• Latency-tiered serving — reasoning_effort: none for the fast path, high for the hard path, same endpoint

• Commercial products — Apache 2.0, no revenue clause, no regional restriction

---

Troubleshooting

Issue	Fix
vLLM tokenizer errors	Mistral models need --tokenizer-mode mistral --config-format mistral --load-format mistral
OOM at 256K	The KV cache, not the weights — lower --max-model-len
Reasoning never triggers	reasoning_effort is per request; check your client forwards extra_body
Slow first token on 2 cards	Layer-split GGUF across PCIe adds latency; prefer rigs with x8/x16 links
NVFP4 build fails to load	Needs Blackwell — see CUDA & PyTorch Compatibility

---

Next Steps

• Smaller sibling: Mistral Small 3.1 — 24B dense, one card

• Bigger sibling: Mistral Large 3 — 675B MoE

• Dense alternative: Qwen3.8-27B — 27B, one card, Apache 2.0

• Fine-tune it: Unsloth · LLaMA-Factory

Links

• Mistral Small 4 on Hugging Face · GGUF

• Mistral announcement

• Rent a GPU: RTX 4090 · RTX 5090 · Marketplace