> For the complete documentation index, see [llms.txt](https://docs.clore.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.clore.ai/guides/language-models/mistral-small4.md). # Mistral Small 4 (119B MoE, 6.5B Active) Deploy Mistral Small 4 (119B MoE, 6.5B active) on two consumer GPUs from the Clore.ai marketplace — Apache 2.0, 256K context, vision and reasoning in one model {% hint style="info" %} **Status (August 2026):** Mistral released **Small 4** on **16 March 2026** under **Apache 2.0**. Weights: [mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603). **119B total parameters with only 6.5B active** (128 experts, 4 active per token), **256K context**, native **vision**, and a per-request reasoning toggle. It folds Mistral's three previous lines — Instruct, Magistral reasoning and Devstral coding — into a single checkpoint. {% endhint %} "Small" is a misnomer by parameter count and exactly right by cost. Only **6.5B parameters activate per token**, so decode speed feels like a 7B model while quality comes from a 119B pool. Quantized to Q2 it is **40GB** — two RTX 4090s, which the Clore.ai marketplace has 322 servers' worth of free capacity for. At Q3–Q4 it is 54–59GB, comfortable on two RTX 5090s. If you want one open model that answers instantly for easy prompts, thinks harder when told to, reads images, and writes code, this is currently the best fit for hardware you can actually rent by the minute. ### Key Specs | Property | Value | | --------------- | ------------------------------------------------------------------ | | Parameters | 119B total, 6.5B active (128 experts, 4 active) | | Modality | Text + image in → text out | | Context | 262,144 tokens (256K) | | License | Apache 2.0 | | Release Date | 16 March 2026 | | Weights | Official \~242GB · Q3\_K\_M GGUF 54GB · Q2\_K\_XL GGUF 40GB | | Reasoning | `reasoning_effort`: `none` (instant) or `high` (test-time compute) | | Primary Tooling | vLLM, llama.cpp, Transformers | *** ## Requirements | Build | Size | Clore.ai setup | | ------------------- | ------- | --------------------------------------------------------- | | `UD-IQ1_M` | 32GB | 1× RTX PRO 6000 96GB, or 2× RTX 4090 | | `UD-Q2_K_XL` | 40GB | **2× RTX 4090 / 3090 (48GB)** — 322 servers free at ≥48GB | | `UD-Q3_K_M` | 54GB | 2× RTX 5090 (64GB) | | `UD-IQ4_NL` | 59GB | 2× RTX 5090 (64GB) | | Official checkpoint | \~242GB | 8× RTX 5090 (248GB) | Add headroom for the KV cache: at 256K context it is substantial even with an MoE this sparse. Start at 32–64K and raise it once you see actual usage. *** ## Deploy with llama.cpp (two consumer cards) ```bash huggingface-cli download unsloth/Mistral-Small-4-119B-2603-GGUF \ --include "*UD-Q2_K_XL*" --local-dir /workspace/ms4 llama-server -m /workspace/ms4/*UD-Q2_K_XL*-00001-of-*.gguf \ --host 0.0.0.0 --port 8080 \ -ngl 999 --split-mode layer \ -c 65536 --flash-attn ``` ## Deploy with vLLM ```bash vllm serve mistralai/Mistral-Small-4-119B-2603 \ --tensor-parallel-size 8 \ --tokenizer-mode mistral \ --config-format mistral \ --load-format mistral \ --max-model-len 262144 \ --enable-expert-parallel ``` An official NVFP4 build ([`mistralai/Mistral-Small-4-119B-2603-NVFP4`](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)) targets Blackwell cards — useful on RTX 50-series and RTX PRO 6000 rigs. ## Reasoning on demand ```python client.chat.completions.create( model="mistral-small-4", messages=[{"role": "user", "content": "Find the deadlock in this scheduler."}], extra_body={"reasoning_effort": "high"}, # "none" for instant replies temperature=0.7, # 0.0–0.7 when reasoning is off ) ``` Mistral recommends temperature 0.7 with reasoning on. With reasoning off, stay between 0.0 and 0.7 depending on how deterministic you need the output. *** ## Clore.ai GPU Recommendations | Setup | VRAM | Build | Context | Clore.ai cost | | --------------- | ----- | --------------------- | --------- | ----------------- | | **2× RTX 4090** | 48GB | Q2\_K\_XL | \~64K | **$0.28–0.84/hr** | | 2× RTX 3090 | 48GB | Q2\_K\_XL | \~64K | $0.14–0.42/hr | | **2× RTX 5090** | 64GB | Q3\_K\_M / IQ4\_NL | \~96K | **$0.50–1.54/hr** | | 1× RTX PRO 6000 | 96GB | Q3\_K\_M, single card | \~131K | $0.92–1.38/hr | | 8× RTX 5090 | 248GB | official checkpoint | full 256K | \~$2.00–3.50/hr | {% hint style="success" %} **A pair of RTX 3090s is the value play** — 48GB of VRAM for around $0.14–0.42/hr total, running a 119B model. Use [partial GPU rental](/guides/getting-started/partial-gpu-rental.md) to take two cards out of a larger rig instead of hunting for a dedicated dual-3090 box. {% endhint %} *** ## Use Cases * **One model for a whole product** — chat, coding and reasoning without three deployments * **Document and image extraction** — vision plus 256K context handles long scanned documents * **Coding agents on a budget** — Devstral's lineage, at 6.5B active parameters per token * **Latency-tiered serving** — `reasoning_effort: none` for the fast path, `high` for the hard path, same endpoint * **Commercial products** — Apache 2.0, no revenue clause, no regional restriction *** ## Troubleshooting | Issue | Fix | | --------------------------- | ----------------------------------------------------------------------------------------------------------- | | vLLM tokenizer errors | Mistral models need `--tokenizer-mode mistral --config-format mistral --load-format mistral` | | OOM at 256K | The KV cache, not the weights — lower `--max-model-len` | | Reasoning never triggers | `reasoning_effort` is per request; check your client forwards `extra_body` | | Slow first token on 2 cards | Layer-split GGUF across PCIe adds latency; prefer rigs with x8/x16 links | | NVFP4 build fails to load | Needs Blackwell — see [CUDA & PyTorch Compatibility](/guides/getting-started/cuda-pytorch-compatibility.md) | *** ## Next Steps * **Smaller sibling:** [Mistral Small 3.1](/guides/language-models/mistral-small.md) — 24B dense, one card * **Bigger sibling:** [Mistral Large 3](/guides/language-models/mistral-large3.md) — 675B MoE * **Dense alternative:** [Qwen3.8-27B](/guides/language-models/qwen38-27b.md) — 27B, one card, Apache 2.0 * **Fine-tune it:** [Unsloth](/guides/training/unsloth-finetune.md) · [LLaMA-Factory](/guides/training/llama-factory.md) ### Links * [Mistral Small 4 on Hugging Face](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) · [GGUF](https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF) * [Mistral announcement](https://mistral.ai/news/mistral-small-4/) * **Rent a GPU:** [RTX 4090](https://clore.ai/rent-4090.html) · [RTX 5090](https://clore.ai/rent-5090.html) · [Marketplace](https://clore.ai/marketplace) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.clore.ai/guides/language-models/mistral-small4.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.