Ollama

The easiest way to run LLMs locally on CLORE.AI GPUs.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

Server Requirements

Parameter	Minimum	Recommended
RAM	8GB	16GB+
VRAM	6GB	8GB+
Network	100Mbps	500Mbps+
Startup Time	~30 seconds	-

Ollama is lightweight and works on most GPU servers. For larger models (13B+), choose servers with 16GB+ RAM and 12GB+ VRAM.

Why Ollama?

• One-command setup - No Python, no dependencies

• Model library - Download models with ollama pull

• OpenAI-compatible API - Drop-in replacement

• GPU acceleration - Automatic CUDA detection

• Multi-model - Run multiple models simultaneously

Quick Deploy on CLORE.AI

Docker Image:

ollama/ollama

Ports:

22/tcp
11434/http

Command:

ollama serve

Verify It's Working

After deployment, find your http_pub URL in My Orders and test:

# Replace with your actual http_pub URL
curl https://your-http-pub.clorecloud.net/

# Expected response: "Ollama is running"

If you get HTTP 502, wait 30-60 seconds - the service is still starting.

Accessing Your Service

When deployed on CLORE.AI, access your Ollama instance via the http_pub URL:

# Find your http_pub in My Orders, then:
curl https://your-http-pub.clorecloud.net/api/tags

# For API calls, use your http_pub URL:
curl https://your-http-pub.clorecloud.net/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

All localhost:11434 examples below work when connected via SSH. For external access, replace with your https://your-http-pub.clorecloud.net/ URL.

Installation

Using Docker (Recommended)

docker run -d --gpus all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Manual Installation

curl -fsSL https://ollama.com/install.sh | sh

Running Models

Pull and Run

# Pull model
ollama pull llama3.2

# Run interactive chat
ollama run llama3.2

# Run with prompt
ollama run llama3.2 "Explain quantum computing"

Popular Models

Model	Size	Use Case
llama3.2	3B	Fast, general purpose
llama3.1	8B	Better quality
llama3.1:70b	70B	Best quality
mistral	7B	Fast, good quality
mixtral	47B	MoE, high quality
codellama	7-34B	Code generation
deepseek-coder-v2	16B	Best for code
qwen2.5	7B	Multilingual
phi4	14B	Microsoft's latest
gemma2	9B	Google's model

Model Variants

# Quantization variants
ollama pull llama3.1:8b-instruct-q4_K_M   # 4-bit (smaller, faster)
ollama pull llama3.1:8b-instruct-q8_0     # 8-bit (better quality)
ollama pull llama3.1:8b-instruct-fp16     # Full precision

# Size variants
ollama pull llama3.1:8b    # 8 billion parameters
ollama pull llama3.1:70b   # 70 billion parameters

API Usage

Chat Completion

# Via http_pub (external access):
curl https://your-http-pub.clorecloud.net/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

# Via SSH tunnel (localhost):
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

Add "stream": false to get the complete response at once instead of streaming.

OpenAI-Compatible Endpoint

from openai import OpenAI

# For external access, use your http_pub URL:
client = OpenAI(
    base_url="https://your-http-pub.clorecloud.net/v1",
    api_key="ollama"  # any string works
)

# Or via SSH tunnel:
# client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ]
)

print(response.choices[0].message.content)

Streaming

stream = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write a poem"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Embeddings

curl http://localhost:11434/api/embeddings -d '{
  "model": "llama3.2",
  "prompt": "Hello world"
}'

Text Generation (Non-Chat)

curl https://your-http-pub.clorecloud.net/api/generate -d '{
  "model": "llama3.2",
  "prompt": "The meaning of life is",
  "stream": false
}'

Complete API Reference

All endpoints work with both http://localhost:11434 (via SSH) and https://your-http-pub.clorecloud.net (external).

Model Management

Endpoint	Method	Description
/api/tags	GET	List all downloaded models
/api/show	POST	Get model details
/api/pull	POST	Download a model
/api/delete	DELETE	Remove a model
/api/ps	GET	List currently running models
/api/version	GET	Get Ollama version

List Models

curl https://your-http-pub.clorecloud.net/api/tags

Response:

{
  "models": [
    {"name": "llama3.2:latest", "size": 2019393189, "digest": "...", "modified_at": "..."}
  ]
}

Show Model Details

curl https://your-http-pub.clorecloud.net/api/show -d '{"name": "llama3.2"}'

Pull Model via API

curl https://your-http-pub.clorecloud.net/api/pull -d '{
  "name": "mistral:7b",
  "stream": false
}'

Response:

{"status": "success"}

Large models may take several minutes to download. For very large models (30GB+), consider using SSH and the CLI: ollama pull model-name

Delete Model

curl -X DELETE https://your-http-pub.clorecloud.net/api/delete -d '{"name": "mistral:7b"}'

List Running Models

curl https://your-http-pub.clorecloud.net/api/ps

Response:

{
  "models": [
    {"name": "llama3.2:latest", "size": 2019393189, "expires_at": "2025-01-25T12:00:00Z"}
  ]
}

Get Version

curl https://your-http-pub.clorecloud.net/api/version

Response:

{"version": "0.6.8"}

Inference Endpoints

Endpoint	Method	Description
/api/generate	POST	Text completion
/api/chat	POST	Chat completion
/api/embeddings	POST	Generate embeddings
/v1/chat/completions	POST	OpenAI-compatible chat

Custom Model Creation

Create custom models with specific system prompts via API:

curl https://your-http-pub.clorecloud.net/api/create -d '{
  "name": "my-assistant",
  "modelfile": "FROM llama3.2\nSYSTEM You are a helpful coding assistant."
}'

GPU Configuration

Check GPU Usage

# In container or server
nvidia-smi

# Ollama shows GPU in logs
ollama run llama3.2 --verbose

Multi-GPU

Ollama automatically uses available GPUs. For specific GPU:

CUDA_VISIBLE_DEVICES=0 ollama serve

Memory Management

# Set GPU memory limit
OLLAMA_GPU_MEMORY=8GiB ollama serve

# Keep model loaded
OLLAMA_KEEP_ALIVE=24h ollama serve

Custom Models (Modelfile)

Create custom models with system prompts:

# Modelfile
FROM llama3.2

SYSTEM You are a helpful coding assistant. Always provide code examples.

PARAMETER temperature 0.7
PARAMETER top_p 0.9

ollama create coding-assistant -f Modelfile
ollama run coding-assistant

Running as Service

Systemd

# /etc/systemd/system/ollama.service
[Unit]
Description=Ollama Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/ollama serve
Restart=always
Environment="OLLAMA_HOST=0.0.0.0"

[Install]
WantedBy=multi-user.target

systemctl enable ollama
systemctl start ollama

Performance Tips

1. Use appropriate quantization

   • Q4_K_M for speed

   • Q8_0 for quality

   • fp16 for maximum quality

2. Match model to VRAM

   • 8GB: 7B models (Q4)

   • 16GB: 13B models or 7B (Q8)

   • 24GB: 34B models (Q4)

   • 48GB+: 70B models

3. Keep model loaded

   OLLAMA_KEEP_ALIVE=1h ollama serve

4. Fast SSD improves performance

   • Model loading and KV cache benefit from fast storage

   • Servers with NVMe SSD can achieve 2-3x better performance

Benchmarks

Generation Speed (tokens/sec)

Model	RTX 3060	RTX 3090	RTX 4090	A100 40GB
Llama 3.2 3B (Q4)	120	160	200	220
Llama 3.1 8B (Q4)	60	100	130	150
Llama 3.1 8B (Q8)	45	80	110	130
Mistral 7B (Q4)	70	110	140	160
Mixtral 8x7B (Q4)	-	35	55	75
Llama 3.1 70B (Q4)	-	-	18	35

Benchmarks updated January 2026. Actual speeds may vary based on server configuration.

Time to First Token (ms)

Model	RTX 3090	RTX 4090	A100
3B	50	35	25
7-8B	120	80	60
13B	250	150	100
34B	600	350	200
70B	-	1200	500

Context Length vs VRAM (Q4)

Model	2K ctx	4K ctx	8K ctx	16K ctx
7B	5GB	6GB	8GB	12GB
13B	8GB	10GB	14GB	22GB
34B	20GB	24GB	32GB	48GB
70B	40GB	48GB	64GB	96GB

GPU Requirements

Model	Q4 VRAM	Q8 VRAM
3B	3GB	5GB
7-8B	5GB	9GB
13B	8GB	15GB
34B	20GB	38GB
70B	40GB	75GB

Cost Estimate

Typical CLORE.AI marketplace rates:

GPU	CLORE/day	Approx USD/hr	Good For
RTX 3060 12GB	~100	~$0.02	7B models
RTX 3090 24GB	~150	~$0.03	13B-34B models
RTX 4090 24GB	~200	~$0.04	34B models, fast
A100 40GB	~400	~$0.08	70B models

Prices vary by provider. Check CLORE.AI Marketplace for current rates. Pay with CLORE tokens for best value.

Troubleshooting

Model won't load

# Check available memory
nvidia-smi

# Try smaller quantization
ollama pull llama3.1:8b-q4_0

Slow generation

# Check if GPU is used
ollama run llama3.2 --verbose

# Ensure CUDA is available
nvidia-smi

Connection refused

# Make sure server is running
ollama serve

# Check if binding to all interfaces
OLLAMA_HOST=0.0.0.0 ollama serve

HTTP 502 on http_pub URL

This means the service is still starting. Wait 30-60 seconds and retry:

# Check if service is ready
curl https://your-http-pub.clorecloud.net/

# Expected: "Ollama is running"
# If 502: wait and retry

Next Steps

• Open WebUI - Beautiful chat interface for Ollama

• vLLM - High-throughput production serving

• DeepSeek-V3 - Best reasoning model

• Qwen2.5 - Multilingual alternative

• Text Generation WebUI - Advanced features