Mistral & Mixtral

Run Mistral and Mixtral models on Clore.ai GPUs

Newer versions available! Check out Mistral Small 3.1 (24B, Apache 2.0, fits on RTX 4090) and Mistral Large 3 (675B MoE, frontier-class).

Run Mistral and Mixtral models for high-quality text generation.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

Renting on CLORE.AI

1. Visit CLORE.AI Marketplace

2. Filter by GPU type, VRAM, and price

3. Choose On-Demand (fixed rate) or Spot (bid price)

4. Configure your order:

   • Select Docker image

   • Set ports (TCP for SSH, HTTP for web UIs)

   • Add environment variables if needed

   • Enter startup command

5. Select payment: CLORE, BTC, or USDT/USDC

6. Create order and wait for deployment

Access Your Server

• Find connection details in My Orders

• Web interfaces: Use the HTTP port URL

• SSH: ssh -p <port> root@<proxy-address>

Model Overview

Model	Parameters	VRAM	Specialty
Mistral-7B	7B	8GB	General purpose
Mistral-7B-Instruct	7B	8GB	Chat/instruction
Mixtral-8x7B	46.7B (12.9B active)	24GB	MoE, best quality
Mixtral-8x22B	141B	80GB+	Largest MoE

Quick Deploy

Docker Image:

pytorch/pytorch:2.11.0-cuda12.8-cudnn9-runtime

Ports:

22/tcp
8000/http

Command:

pip install vllm && \
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --port 8000

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

Installation Options

Using Ollama (Easiest)

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Run Mistral
ollama run mistral

# Run Mixtral
ollama run mixtral

Using vLLM

pip install vllm

# Start server
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --dtype float16

Using Transformers

pip install transformers accelerate

Mistral-7B with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "mistralai/Mistral-7B-Instruct-v0.2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms"}
]

inputs = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    inputs,
    max_new_tokens=500,
    do_sample=True,
    temperature=0.7,
    top_p=0.95
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Mixtral-8x7B

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

messages = [
    {"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

outputs = model.generate(
    inputs,
    max_new_tokens=1000,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quantized Models (Lower VRAM)

4-bit Quantization

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mixtral-8x7B-Instruct-v0.1",
    quantization_config=quantization_config,
    device_map="auto"
)

GGUF with llama.cpp

# Download GGUF model
wget https://huggingface.co/bartowski/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf

# Run with llama.cpp
./main -m Mistral-7B-Instruct-v0.3-Q4_K_M.gguf \
    -p "Explain machine learning" \
    -n 500

vLLM Server (Production)

python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.2 \
    --dtype float16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

OpenAI-Compatible API

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[
        {"role": "user", "content": "What is the capital of France?"}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)

Streaming

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

stream = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[{"role": "user", "content": "Write a story about a robot"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Function Calling

Mistral supports function calling:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools
)

print(response.choices[0].message.tool_calls)

Gradio Interface

import gradio as gr
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "mistralai/Mistral-7B-Instruct-v0.2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

def chat(message, history, temperature, max_tokens):
    messages = []
    for h in history:
        messages.append({"role": "user", "content": h[0]})
        messages.append({"role": "assistant", "content": h[1]})
    messages.append({"role": "user", "content": message})

    inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")

    outputs = model.generate(
        inputs,
        max_new_tokens=max_tokens,
        temperature=temperature,
        do_sample=True
    )

    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    # Extract assistant response
    return response.split("[/INST]")[-1].strip()

demo = gr.ChatInterface(
    fn=chat,
    additional_inputs=[
        gr.Slider(0.1, 2.0, value=0.7, label="Temperature"),
        gr.Slider(100, 2000, value=500, step=100, label="Max Tokens")
    ],
    title="Mistral-7B Chat"
)

demo.launch(server_name="0.0.0.0", server_port=7860)

Performance Comparison

Throughput (tokens/sec)

Model	RTX 3060	RTX 3090	RTX 4090	A100 40GB
Mistral-7B FP16	45	80	120	150
Mistral-7B Q4	70	110	160	200
Mixtral-8x7B FP16	-	-	30	60
Mixtral-8x7B Q4	-	25	50	80
Mixtral-8x22B Q4	-	-	-	25

Time to First Token (TTFT)

Model	RTX 3090	RTX 4090	A100
Mistral-7B	80ms	50ms	35ms
Mixtral-8x7B	-	150ms	90ms
Mixtral-8x22B	-	-	200ms

Context Length vs VRAM (Mistral-7B)

Context	FP16	Q8	Q4
4K	15GB	9GB	5GB
8K	18GB	11GB	7GB
16K	24GB	15GB	9GB
32K	36GB	22GB	14GB

VRAM Requirements

Model	FP16	8-bit	4-bit
Mistral-7B	14GB	8GB	5GB
Mixtral-8x7B	90GB	45GB	24GB
Mixtral-8x22B	180GB	90GB	48GB

Use Cases

Code Generation

prompt = """
Write a Python class for a REST API client with:
- Authentication handling
- Retry logic
- Error handling
"""

Data Analysis

prompt = """
Analyze this data and provide insights:
Sales Q1: $100K
Sales Q2: $150K
Sales Q3: $120K
Sales Q4: $200K
"""

Creative Writing

prompt = """
Write a short story about an AI that becomes self-aware,
in the style of Isaac Asimov.
"""

Troubleshooting

Out of Memory

• Use 4-bit quantization

• Use Mistral-7B instead of Mixtral

• Reduce max_model_len

Slow Generation

• Use vLLM for production

• Enable flash attention

• Use tensor parallelism for multi-GPU

Poor Output Quality

• Adjust temperature (0.1-0.9)

• Use instruct variant

• Better system prompts

Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

GPU	Hourly Rate	Daily Rate	4-Hour Session
RTX 3060	~$0.03	~$0.70	~$0.12
RTX 3090	~$0.06	~$1.50	~$0.25
RTX 4090	~$0.10	~$2.30	~$0.40
A100 40GB	~$0.17	~$4.00	~$0.70
A100 80GB	~$0.25	~$6.00	~$1.00

Prices vary by provider and demand. Check CLORE.AI Marketplace for current rates.

Save money:

• Use the Spot market for interruptible work — about a third of servers price spot below on-demand (median ~13% off), the rest match it

• Pay with CLORE tokens

• Compare prices across different providers

Next Steps

• vLLM - Production serving

• Ollama - Easy deployment

• DeepSeek-V3 - Best reasoning model

• Qwen2.5 - Multilingual alternative