Mistral Small 3.1

Deploy Mistral Small 3.1 (24B) on Clore.ai — the ideal single-GPU production model

Mistral Small 3.1, released March 2025 by Mistral AI, is a 24-billion parameter dense model that punches way above its weight. With a 128K context window, native vision capabilities, best-in-class function calling, and an Apache 2.0 license, it's arguably the best model you can run on a single RTX 4090. It outperforms GPT-4o Mini and Claude 3.5 Haiku on most benchmarks while fitting comfortably on consumer hardware when quantized.

Key Features

• 24B dense parameters — no MoE complexity, straightforward deployment

• 128K context window — RULER 128K score of 81.2%, beats GPT-4o Mini (65.8%)

• Native vision — analyze images, charts, documents, and screenshots

• Apache 2.0 license — fully open for commercial and personal use

• Elite function calling — native tool use with JSON output, ideal for agentic workflows

• Multilingual — 25+ languages including CJK, Arabic, Hindi, and European languages

Requirements

Component	Quantized (Q4)	Full Precision (BF16)
GPU	1× RTX 4090 24GB	2× RTX 4090 or 1× H100
VRAM	~16GB	~55GB
RAM	32GB	64GB
Disk	20GB	50GB
CUDA	11.8+	12.0+

Clore.ai recommendation: RTX 4090 (~$0.5–2/day) for quantized inference — best price/performance ratio

Quick Start with Ollama

The fastest way to get Mistral Small 3.1 running:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Run Mistral Small 3.1 (auto-downloads ~14GB Q4 quantization)
ollama run mistral-small3.1

# Or specify a specific quantization
ollama run mistral-small3.1:24b-instruct-2503-q4_K_M

Ollama as OpenAI-Compatible API

# Start Ollama server
ollama serve &

# Pull the model
ollama pull mistral-small3.1

# Query via API
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-small3.1",
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant."},
      {"role": "user", "content": "Write a Python decorator for rate limiting"}
    ],
    "temperature": 0.15
  }'

Ollama with Vision

# Send an image for analysis
curl http://localhost:11434/api/chat -d '{
  "model": "mistral-small3.1",
  "messages": [{
    "role": "user",
    "content": "What does this image show?",
    "images": ["/path/to/image.jpg"]
  }]
}'

vLLM Setup (Production)

For production workloads with high throughput and concurrent requests:

# Install vLLM (v0.8.1+ required)
pip install -U vllm

# Verify mistral_common is installed (should be automatic)
python -c "import mistral_common; print(mistral_common.__version__)"

Serve on Single GPU (Text Only)

vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Serve with Vision (2 GPUs Recommended)

vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --limit-mm-per-prompt 'image=10' \
  --tensor-parallel-size 2 \
  --max-model-len 65536

Query the Server

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-3.1-24B-Instruct-2503",
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Today is 2026-02-20."},
        {"role": "user", "content": "Write a complete REST API in FastAPI with CRUD operations for a blog"}
    ],
    temperature=0.15,
    max_tokens=4096
)
print(response.choices[0].message.content)

HuggingFace Transformers

For direct Python integration and experimentation:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    load_in_4bit=True  # 4-bit quantization — fits on 24GB GPU
)

messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Implement a binary search tree in Python with insert, delete, and search methods"}
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=2048,
    temperature=0.15,
    do_sample=True
)
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))

Function Calling Example

Mistral Small 3.1 is one of the best small models for tool use:

import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_stock_price",
            "description": "Get the current stock price for a given ticker symbol",
            "parameters": {
                "type": "object",
                "required": ["ticker"],
                "properties": {
                    "ticker": {"type": "string", "description": "Stock ticker symbol (e.g., AAPL)"}
                }
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "calculate_portfolio_value",
            "description": "Calculate total portfolio value given holdings",
            "parameters": {
                "type": "object",
                "required": ["holdings"],
                "properties": {
                    "holdings": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "ticker": {"type": "string"},
                                "shares": {"type": "number"}
                            }
                        }
                    }
                }
            }
        }
    }
]

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-3.1-24B-Instruct-2503",
    messages=[{"role": "user", "content": "What's the current price of AAPL and MSFT?"}],
    tools=tools,
    tool_choice="auto",
    temperature=0.15
)

for tool_call in response.choices[0].message.tool_calls:
    print(f"Call: {tool_call.function.name}({tool_call.function.arguments})")

Docker Quick Start

# Single GPU deployment
docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model mistralai/Mistral-Small-3.1-24B-Instruct-2503 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --max-model-len 32768

# With vision support (2 GPUs)
docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model mistralai/Mistral-Small-3.1-24B-Instruct-2503 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --limit-mm-per-prompt 'image=10' \
  --tensor-parallel-size 2

Tips for Clore.ai Users

• RTX 4090 is the sweet spot: At $0.5–2/day, a single RTX 4090 runs Mistral Small 3.1 quantized with room to spare. Best cost/performance ratio on Clore.ai for a general-purpose LLM.

• Use low temperature: Mistral AI recommends temperature=0.15 for most tasks. Higher temps cause inconsistent output with this model.

• RTX 3090 works too: At $0.3–1/day, RTX 3090 (24GB) runs Q4 quantized with Ollama just fine. Slightly slower than 4090 but half the price.

• Ollama for quick setups, vLLM for production: Ollama gives you a working model in 60 seconds. For concurrent API requests and higher throughput, switch to vLLM.

• Function calling makes it special: Many 24B models can chat — few can reliably call tools. Mistral Small 3.1's function calling is on par with GPT-4o Mini. Build agents, API backends, and automation pipelines with confidence.

Troubleshooting

Issue	Solution
OutOfMemoryError on RTX 4090	Use quantized model via Ollama or load_in_4bit=True in Transformers. Full BF16 needs ~55GB.
Ollama model not found	Use ollama run mistral-small3.1 (official library name).
vLLM tokenizer errors	Always pass --tokenizer-mode mistral --config-format mistral --load-format mistral.
Poor output quality	Set temperature=0.15. Add a system prompt. Mistral Small is sensitive to temperature.
Vision not working on 1 GPU	Vision features need more VRAM. Use --tensor-parallel-size 2 or reduce --max-model-len.
Function calls return empty	Add --tool-call-parser mistral --enable-auto-tool-choice to vLLM serve.

Further Reading

• Mistral Small 3.1 on HuggingFace

• Mistral AI Blog Post

• Ollama Model Page

• vLLM Documentation

• Mistral Common Library

• Mistral AI Platform