Continue.dev AI Coding

Power Continue.dev with Clore.ai GPUs — run CodeLlama 34B, DeepSeek Coder, and Qwen2.5-Coder locally on cheap GPU rentals for private AI coding assistance.

Continue.dev is an open-source AI coding assistant for VS Code and JetBrains with 25K+ GitHub stars. The extension runs on your local machine (or in your IDE), but it connects to a backend model server for inference. By pointing Continue.dev at a powerful GPU rented from Clore.ai, you get:

• Top-tier coding models (34B+ parameters) that won't fit on your laptop

• Full privacy — code stays on infrastructure you control

• Flexible costs — pay only while you're coding (~$0.20–0.50/hr vs. $19/mo for Copilot)

• OpenAI-compatible API — Continue.dev connects to Ollama, vLLM, or TabbyML seamlessly

This guide focuses on setting up the Clore.ai GPU backend (Ollama or vLLM) that your local Continue.dev extension connects to.

All GPU server examples use servers rented through the CLORE.AI Marketplace.

Architecture: Your IDE (with Continue.dev extension) → Internet → Clore.ai GPU server (running Ollama / vLLM / TabbyML) → local model inference. No code ever touches a third-party API.

Overview

Property	Details
Project	continuedev/continue
License	Apache 2.0
GitHub Stars	25K+
IDE Support	VS Code, JetBrains (IntelliJ, PyCharm, WebStorm, GoLand, etc.)
Config File	~/.continue/config.json
Backend Options	Ollama, vLLM, TabbyML, LM Studio, llama.cpp, OpenAI-compatible APIs
Difficulty	Easy (extension install) / Medium (self-hosted backend)
GPU Required?	On the Clore.ai server (yes); on your laptop (no)
Key Features	Autocomplete, chat, edit mode, codebase context (RAG), custom slash commands

Recommended Models for Coding

Model	VRAM	Strength	Notes
codellama:7b	~6 GB	Fast autocomplete	Good starting point
codellama:13b	~10 GB	Balanced	Best quality/speed for autocomplete
codellama:34b	~22 GB	Best CodeLlama quality	Needs RTX 3090 / A100
deepseek-coder:6.7b	~5 GB	Python/JS specialist	Excellent for web dev
deepseek-coder:33b	~22 GB	Top-tier open source	Rivals GPT-4 on code
qwen2.5-coder:7b	~6 GB	Multilingual code	Strong on 40+ languages
qwen2.5-coder:32b	~22 GB	State-of-the-art	Best open coding model 2024
starcoder2:15b	~12 GB	Code completion specialist	FIM (fill-in-the-middle) support

Requirements

Clore.ai Server Requirements

Tier	GPU	VRAM	RAM	Disk	Price	Models
Budget	RTX 3060	12 GB	16 GB	40 GB	$0.03–0.07/hr	CodeLlama 7B, DeepSeek 6.7B, Qwen2.5-Coder 7B
Recommended	RTX 3090	24 GB	32 GB	80 GB	$0.07–0.21/hr	CodeLlama 34B, DeepSeek 33B, Qwen2.5-Coder 32B
Performance	RTX 4090	24 GB	32 GB	80 GB	$0.14–0.42/hr	Same models as above, faster inference
Power	A100 40GB	40 GB	64 GB	120 GB	bare metal	Multiple 34B models concurrently
Maximum	A100 80GB	80 GB	80 GB	200 GB	bare metal	70B models (CodeLlama 70B)

Local Requirements (Your Machine)

• VS Code or any JetBrains IDE

• Continue.dev extension installed

• Stable internet connection to your Clore.ai server

• No local GPU needed — all inference happens on Clore.ai

Quick Start

Part 1: Set Up the Clore.ai Backend

Option A — Ollama Backend (Recommended for Most Users)

Ollama is the easiest backend for Continue.dev — simple setup, excellent model management, OpenAI-compatible API.

# 1. SSH into your Clore.ai server
ssh root@<clore-server-ip> -p <port>

# 2. Start Ollama with GPU support
docker run -d \
  --name ollama \
  --gpus all \
  -p 11434:11434 \
  -v /workspace/ollama:/root/.ollama \
  --restart unless-stopped \
  ollama/ollama

# 3. Verify Ollama is running
curl http://localhost:11434/

# 4. Pull your coding model (choose based on your VRAM)
# For 12GB VRAM (RTX 3060):
docker exec ollama ollama pull codellama:13b

# For 24GB VRAM (RTX 3090 / RTX 4090):
docker exec ollama ollama pull qwen2.5-coder:32b
# or:
docker exec ollama ollama pull deepseek-coder:33b

# 5. Pull a fast autocomplete model (separate from chat model)
docker exec ollama ollama pull starcoder2:3b   # Very fast, great for FIM autocomplete

# 6. Verify models are available
docker exec ollama ollama list

# 7. Test inference
docker exec ollama ollama run qwen2.5-coder:32b "Write a Python function to binary search a sorted list"

To expose Ollama externally (so your local IDE can connect):

# Restart Ollama with external access enabled
docker stop ollama && docker rm ollama

docker run -d \
  --name ollama \
  --gpus all \
  -p 11434:11434 \
  -v /workspace/ollama:/root/.ollama \
  -e OLLAMA_HOST=0.0.0.0 \
  --restart unless-stopped \
  ollama/ollama

# Test from your LOCAL machine:
curl http://<clore-server-ip>:11434/api/tags

Exposing port 11434 publicly has no authentication by default. For production use, set up an SSH tunnel instead (see Tips & Best Practices).

Option B — vLLM Backend (High-Throughput / OpenAI-Compatible)

vLLM offers faster inference and multi-user support. Ideal if multiple developers share one Clore.ai server.

# Start vLLM with OpenAI-compatible API
docker run -d \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  -v /workspace/hf-models:/root/.cache/huggingface \
  -e HF_TOKEN="your-huggingface-token" \
  --restart unless-stopped \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-Coder-32B-Instruct \
  --dtype auto \
  --max-model-len 32768 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --served-model-name qwen2.5-coder-32b

# For multi-GPU (e.g., two RTX 3090s):
docker run -d \
  --name vllm \
  --gpus all \
  -p 8000:8000 \
  -v /workspace/hf-models:/root/.cache/huggingface \
  -e HF_TOKEN="your-huggingface-token" \
  vllm/vllm-openai:latest \
  --model deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct \
  --tensor-parallel-size 2 \
  --dtype auto \
  --max-model-len 65536 \
  --served-model-name deepseek-coder-v2

# Test the API
curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder-32b",
    "messages": [{"role": "user", "content": "Write a hello world in Rust"}],
    "max_tokens": 200
  }'

Option C — TabbyML Backend (FIM Autocomplete Specialist)

TabbyML provides superior fill-in-the-middle (FIM) autocomplete — the inline ghost-text suggestions. See the TabbyML documentation for full setup details.

# Quick TabbyML setup for Continue.dev autocomplete
docker run -d \
  --name tabby \
  --gpus all \
  -p 8080:8080 \
  -v /workspace/tabby-data:/data \
  --restart unless-stopped \
  tabbyml/tabby serve \
  --model StarCoder2-7B \
  --chat-model Mistral-7B \
  --device cuda

# Verify
curl http://localhost:8080/v1/health

Part 2: Install Continue.dev Extension

VS Code:

1. Open the Extensions panel (Ctrl+Shift+X / Cmd+Shift+X)

2. Search "Continue" — install the official extension by Continue (continuedev)

3. Click the Continue icon in the sidebar (or Ctrl+Shift+I)

JetBrains (IntelliJ, PyCharm, WebStorm, GoLand):

1. File → Settings → Plugins → Marketplace

2. Search "Continue" and install

3. Restart the IDE; the Continue panel appears on the right sidebar

Part 3: Configure Continue.dev to Use Clore.ai

Edit ~/.continue/config.json on your local machine:

{
  "models": [
    {
      "title": "Clore.ai — Qwen2.5-Coder 32B",
      "provider": "ollama",
      "model": "qwen2.5-coder:32b",
      "apiBase": "http://<clore-server-ip>:11434",
      "contextLength": 32768,
      "completionOptions": {
        "temperature": 0.1,
        "topP": 0.95,
        "maxTokens": 4096
      }
    },
    {
      "title": "Clore.ai — CodeLlama 13B (fast)",
      "provider": "ollama",
      "model": "codellama:13b",
      "apiBase": "http://<clore-server-ip>:11434",
      "contextLength": 16384
    }
  ],
  "tabAutocompleteModel": {
    "title": "StarCoder2 3B (autocomplete)",
    "provider": "ollama",
    "model": "starcoder2:3b",
    "apiBase": "http://<clore-server-ip>:11434"
  },
  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-embed-text",
    "apiBase": "http://<clore-server-ip>:11434"
  },
  "contextProviders": [
    { "name": "code" },
    { "name": "docs" },
    { "name": "diff" },
    { "name": "terminal" },
    { "name": "problems" },
    { "name": "folder" },
    { "name": "codebase" }
  ],
  "slashCommands": [
    { "name": "edit", "description": "Edit selected code" },
    { "name": "comment", "description": "Add comments to code" },
    { "name": "share", "description": "Export conversation as markdown" },
    { "name": "cmd", "description": "Generate terminal command" },
    { "name": "commit", "description": "Generate git commit message" }
  ]
}

For vLLM backend instead of Ollama:

{
  "models": [
    {
      "title": "Clore.ai — DeepSeek Coder 33B (vLLM)",
      "provider": "openai",
      "model": "deepseek-coder-v2",
      "apiBase": "http://<clore-server-ip>:8000/v1",
      "apiKey": "not-required",
      "contextLength": 65536,
      "completionOptions": {
        "temperature": 0.0,
        "maxTokens": 8192
      }
    }
  ]
}

For TabbyML backend (autocomplete only):

{
  "tabAutocompleteModel": {
    "title": "Clore.ai — TabbyML StarCoder2",
    "provider": "openai",
    "model": "StarCoder2-7B",
    "apiBase": "http://<clore-server-ip>:8080/v1",
    "apiKey": "auth-token-if-set"
  }
}

Configuration

SSH Tunnel Setup (Secure Remote Access)

Instead of exposing ports publicly, use an SSH tunnel from your local machine:

# Open SSH tunnel: local port 11434 → Clore.ai server port 11434
ssh -N -L 11434:localhost:11434 root@<clore-server-ip> -p <clore-ssh-port>

# Keep the tunnel alive (add to ~/.ssh/config):
Host clore-coding
  HostName <clore-server-ip>
  Port <clore-ssh-port>
  User root
  LocalForward 11434 localhost:11434
  LocalForward 8000 localhost:8000
  ServerAliveInterval 60
  ServerAliveCountMax 3

# Connect with:
ssh -N clore-coding

# Then in config.json use localhost:
# "apiBase": "http://localhost:11434"

Persistent Tunnel with autossh

# Install autossh on your local machine (Linux/macOS)
sudo apt install autossh   # Ubuntu/Debian
brew install autossh       # macOS

# Run persistent tunnel that auto-reconnects
autossh -M 0 -N \
  -o "ServerAliveInterval 30" \
  -o "ServerAliveCountMax 3" \
  -L 11434:localhost:11434 \
  root@<clore-server-ip> -p <clore-ssh-port>

# Add to systemd for automatic start on boot (Linux)
cat > ~/.config/systemd/user/clore-tunnel.service << 'EOF'
[Unit]
Description=SSH tunnel to Clore.ai coding server
After=network.target

[Service]
ExecStart=autossh -M 0 -N \
  -o StrictHostKeyChecking=accept-new \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -L 11434:localhost:11434 \
  root@CLORE_IP -p CLORE_PORT
Restart=always
RestartSec=10

[Install]
WantedBy=default.target
EOF

systemctl --user enable clore-tunnel
systemctl --user start clore-tunnel

Load Multiple Models for Different Tasks

For an RTX 3090 (24 GB), you can run a large chat model and a small autocomplete model simultaneously:

# On the Clore.ai server:

# Pull the models
docker exec ollama ollama pull qwen2.5-coder:32b      # Chat (22 GB)
docker exec ollama ollama pull starcoder2:3b           # Autocomplete (2 GB)
docker exec ollama ollama pull nomic-embed-text        # Embeddings (0.5 GB)

# Ollama handles model swapping automatically
# All three fit within 24 GB VRAM with smart caching

# Monitor VRAM usage
nvidia-smi --query-gpu=memory.used,memory.free --format=csv -l 5

Codebase Indexing (RAG for Your Repo)

Continue.dev can index your codebase for context-aware suggestions. Pull an embedding model:

# On Clore.ai server — add embedding model to Ollama
docker exec ollama ollama pull nomic-embed-text

# In config.json (local), embeddings are already configured above.
# Continue.dev will index your open workspace automatically.
# Trigger manual re-index: Ctrl+Shift+P → "Continue: Index Codebase"

GPU Acceleration

Monitor Inference Performance

# On your Clore.ai server — watch GPU during coding sessions
watch -n 1 nvidia-smi

# Check tokens per second (Ollama logs)
docker logs ollama --tail 20 -f

# Detailed GPU stats
nvidia-smi dmon -s u -d 2

# Memory breakdown
nvidia-smi --query-gpu=name,memory.used,memory.free,utilization.gpu \
  --format=csv,noheader -l 5

Expected Performance by GPU

GPU	Model	Context	Tokens/sec (approx.)
RTX 3060 12GB	CodeLlama 7B	8K	~40–60 t/s
RTX 3060 12GB	DeepSeek-Coder 6.7B	8K	~45–65 t/s
RTX 3090 24GB	Qwen2.5-Coder 32B (Q4)	16K	~15–25 t/s
RTX 3090 24GB	DeepSeek-Coder 33B (Q4)	16K	~15–22 t/s
RTX 4090 24GB	Qwen2.5-Coder 32B (Q4)	16K	~25–40 t/s
A100 40GB	Qwen2.5-Coder 32B (FP16)	32K	~35–50 t/s
A100 80GB	CodeLlama 70B (Q4)	32K	~20–30 t/s

For autocomplete (fill-in-the-middle), starcoder2:3b or codellama:7b achieve 50–100 t/s — fast enough to feel instant in the IDE.

Tune Ollama for Better Performance

# On the Clore.ai server — optimize Ollama settings
docker stop ollama && docker rm ollama

docker run -d \
  --name ollama \
  --gpus all \
  -p 11434:11434 \
  -v /workspace/ollama:/root/.ollama \
  -e OLLAMA_HOST=0.0.0.0 \
  -e OLLAMA_NUM_PARALLEL=2 \
  -e OLLAMA_MAX_LOADED_MODELS=2 \
  -e OLLAMA_FLASH_ATTENTION=1 \
  --restart unless-stopped \
  ollama/ollama

# OLLAMA_NUM_PARALLEL=2: serve 2 requests simultaneously
# OLLAMA_MAX_LOADED_MODELS=2: keep 2 models in GPU memory
# OLLAMA_FLASH_ATTENTION=1: enable flash attention (faster, less memory)

Tips & Best Practices

Use Different Models for Different Tasks

Configure Continue.dev with specialized models per task type — the UI lets you switch models mid-conversation:

{
  "models": [
    {
      "title": "Chat — Qwen2.5-Coder 32B",
      "provider": "ollama",
      "model": "qwen2.5-coder:32b",
      "apiBase": "http://localhost:11434",
      "contextLength": 32768,
      "description": "Best for complex questions, code review, architecture decisions"
    },
    {
      "title": "Fast — CodeLlama 7B",
      "provider": "ollama",
      "model": "codellama:7b",
      "apiBase": "http://localhost:11434",
      "contextLength": 8192,
      "description": "Quick answers, simple completions, low latency"
    },
    {
      "title": "Autocomplete — StarCoder2 3B",
      "provider": "ollama",
      "model": "starcoder2:3b",
      "apiBase": "http://localhost:11434",
      "contextLength": 4096,
      "description": "Inline ghost-text suggestions"
    }
  ]
}

Cost Comparison

Solution	Monthly Cost (8hr/day usage)	Privacy	Model Quality
GitHub Copilot	$19/user/mo	❌ Microsoft cloud	GPT-4o (closed)
Cursor Pro	$20/user/mo	❌ Cursor cloud	Claude 3.5 (closed)
RTX 3060 on Clore.ai	~$24/mo	✅ Your server	CodeLlama 13B
RTX 3090 on Clore.ai	~$48/mo	✅ Your server	Qwen2.5-Coder 32B
RTX 4090 on Clore.ai	~$84/mo	✅ Your server	Qwen2.5-Coder 32B
A100 80GB on Clore.ai	~$264/mo	✅ Your server	CodeLlama 70B

For a team of 3+ developers sharing one Clore.ai RTX 3090 (~$48/mo total), the per-user cost beats Copilot while providing a larger, private model.

Shut Down When Not Coding

Clore.ai charges per hour. Use a simple script to start/stop the server:

# Save these as local scripts

# start-coding-server.sh
#!/bin/bash
echo "Opening SSH tunnel to Clore.ai..."
ssh -N -f -L 11434:localhost:11434 clore-coding
echo "Tunnel open. Continue.dev is ready."

# stop-coding-server.sh
#!/bin/bash
echo "Closing SSH tunnel..."
pkill -f "ssh.*clore-coding"
echo "Tunnel closed. Remember to stop your Clore.ai order to stop billing!"

Use Continue.dev Custom Commands

Add custom slash commands to config.json for common coding workflows:

{
  "customCommands": [
    {
      "name": "review",
      "prompt": "Review this code for bugs, security issues, and performance problems. Be specific and actionable.",
      "description": "Code review"
    },
    {
      "name": "test",
      "prompt": "Write comprehensive unit tests for this code. Include edge cases. Use the same language/framework as the code.",
      "description": "Generate tests"
    },
    {
      "name": "docstring",
      "prompt": "Add clear, comprehensive docstrings/comments to this code following best practices for the language.",
      "description": "Add documentation"
    },
    {
      "name": "optimize",
      "prompt": "Optimize this code for performance. Explain what you changed and why.",
      "description": "Optimize code"
    }
  ]
}

Troubleshooting

Problem	Likely Cause	Solution
Continue.dev shows "Connection refused"	Ollama not reachable	Check SSH tunnel is active; verify curl http://localhost:11434/ works
Autocomplete not triggering	Tab autocomplete model not set	Add tabAutocompleteModel to config.json; enable in Continue settings
Very slow responses (>30s first token)	Model loading from disk	First request loads model into VRAM — subsequent requests are fast
"Model not found" error	Model not pulled	Run docker exec ollama ollama pull <model-name> on Clore.ai server
High latency between tokens	Network lag or model too large	Use SSH tunnel; switch to smaller model; check server GPU utilization
Codebase context not working	Embeddings model missing	Pull nomic-embed-text via Ollama; check embeddingsProvider in config.json
SSH tunnel drops frequently	Unstable connection	Use autossh for persistent reconnection; add ServerAliveInterval 30
Context window exceeded	Long files/conversations	Reduce contextLength in config.json; use a model with longer context
JetBrains plugin not loading	IDE version incompatibility	Update JetBrains IDE to latest; check Continue.dev plugin compatibility matrix
vLLM OOM during loading	Not enough VRAM	Add --gpu-memory-utilization 0.85; use smaller model or quantized version

Debug Commands

# On your LOCAL machine — test connectivity
curl http://localhost:11434/api/tags          # if using SSH tunnel
curl http://<clore-ip>:11434/api/tags        # if port is open directly

# On the CLORE.AI server — check Ollama
docker logs ollama --tail 30 -f
docker exec ollama ollama list
docker exec ollama ollama ps                  # show currently loaded models

# Test model response time
time curl http://localhost:11434/api/generate \
  -d '{"model": "codellama:7b", "prompt": "def hello():", "stream": false}'

# Check GPU memory
nvidia-smi --query-gpu=memory.used,memory.free --format=csv

# Check vLLM logs
docker logs vllm --tail 50 -f

# Restart Ollama without losing models
docker restart ollama

Continue.dev Config Validation

# Validate config.json syntax on your local machine
python3 -c "
import json, sys
try:
    config = json.load(open(sys.argv[1]))
    print('✅ Config is valid JSON')
    print(f'Models: {[m[\"title\"] for m in config.get(\"models\", [])]}')
except Exception as e:
    print(f'❌ Error: {e}')
" ~/.continue/config.json

Further Reading

• Continue.dev Documentation — official docs for all IDE integrations and config options

• Continue.dev GitHub — source code, issues, model compatibility

• Continue.dev Config Reference — full config.json schema

• Ollama on Clore.ai — detailed Ollama setup guide (recommended backend)

• vLLM on Clore.ai — high-performance alternative backend for teams

• TabbyML — specialized autocomplete backend with FIM optimization

• GPU Comparison Guide — choose the right GPU for your coding workload

• Model Compatibility — which models fit in which VRAM sizes

• Qwen2.5-Coder — currently the best open coding model

• DeepSeek-Coder-V2 — strong alternative with long context

• CLORE.AI Marketplace — rent GPU servers