For the complete documentation index, see llms.txt. This page is also available as Markdown.

Deploying a REST API for Model Inference

Deploy Model as REST API on Clore.ai

What We're Building

A complete production-ready pipeline to:

  1. Download or train a machine learning model

  2. Rent a Clore.ai GPU programmatically

  3. Deploy Flask/FastAPI inference endpoint

  4. Handle requests with proper authentication

  5. Monitor health, performance, and costs

By the end, you'll have a live REST API serving your model on a rented GPU — for a fraction of cloud provider costs.

Prerequisites

  • Clore.ai account with 20+ CLORE balance

  • Python 3.10+

  • A model to deploy (we'll use Hugging Face models)

pip install requests fastapi uvicorn torch transformers flask gunicorn pydantic prometheus-client

┌─────────────────────────────────────────────────────────────┐ │ Your Application │ └─────────────────────────────────────┬───────────────────────┘ │ HTTPS ▼ ┌─────────────────────────────────────────────────────────────┐ │ Clore.ai Rented GPU Server │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Docker Container │ │ │ │ ┌───────────┐ ┌────────────┐ ┌──────────────┐ │ │ │ │ │ FastAPI │──│ Model │──│ GPU (CUDA) │ │ │ │ │ │ Server │ │ Pipeline │ │ RTX 4090 │ │ │ │ │ └─────┬─────┘ └────────────┘ └──────────────┘ │ │ │ │ │ │ │ │ │ ┌─────┴─────┐ ┌────────────┐ │ │ │ │ │ Auth │ │ Metrics │ │ │ │ │ │ Middleware│ │ Prometheus │ │ │ │ │ └───────────┘ └────────────┘ │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ Ports: 22 (SSH), 8000 (API), 9090 (Metrics) │ └─────────────────────────────────────────────────────────────┘

clore_client.py

import requests import time from typing import Dict, Any, List, Optional from dataclasses import dataclass

@dataclass class RentalInfo: """Information about an active rental.""" order_id: int server_id: int status: str ssh_host: str ssh_port: int http_endpoint: str cost_per_hour: float started_at: int

Step 2: FastAPI Inference Server

Step 3: Docker Configuration

Step 4: Deployment Script

Step 5: Monitoring Dashboard

Running the Deployment

Using Pre-built Images

Instead of building your own Docker image, use these pre-built inference servers:

Cost Comparison

Model Size
GPU
Clore.ai
AWS p3.2xlarge
Savings

7B params

RTX 3090

~$0.20/hr

$3.06/hr

93%

13B params

RTX 4090

~$0.35/hr

$3.06/hr

89%

70B params

A100 40GB

~$1.20/hr

$4.10/hr

71%

70B params (Q4)

RTX 4090

~$0.35/hr

N/A

∞

Monthly savings for a 70B model: ~$2,100/month compared to AWS.

Next Steps

Last updated

Was this helpful?