SAM2 Video

Track and segment objects in video with Meta's SAM2 on Clore.ai

Track and segment any object through video with Meta's SAM2.1 — the improved version of SAM2 with enhanced video accuracy.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

All examples in this guide can be run on GPU servers rented through CLORE.AI Marketplace marketplace.

Renting on CLORE.AI

1. Visit CLORE.AI Marketplace

2. Filter by GPU type, VRAM, and price

3. Choose On-Demand (fixed rate) or Spot (bid price)

4. Configure your order:

   • Select Docker image

   • Set ports (TCP for SSH, HTTP for web UIs)

   • Add environment variables if needed

   • Enter startup command

5. Select payment: CLORE, BTC, or USDT/USDC

6. Create order and wait for deployment

Access Your Server

• Find connection details in My Orders

• Web interfaces: Use the HTTP port URL

• SSH: ssh -p <port> root@<proxy-address>

What is SAM2?

SAM2 (Segment Anything Model 2) by Meta AI enables:

• Real-time video object segmentation

• Click-to-track any object

• Consistent tracking through occlusions

• Memory-efficient video processing

What's New in SAM2.1

SAM2.1 brings significant improvements over the original SAM2:

• Improved video accuracy — Better tracking through occlusions and fast motion

• Enhanced memory module — More consistent long-range tracking

• New checkpoints — sam2.1_hiera_* series with better performance

• Official pip package — Install with pip install sam-2 (no manual build required)

• Faster inference — Optimized CUDA kernels

Resources

• GitHub: facebookresearch/sam2

• Paper: SAM2 Paper

• Demo: SAM2 Demo

• Model Weights: SAM2.1 Checkpoints

Recommended Hardware

Component	Minimum	Recommended	Optimal
GPU	RTX 3060 12GB	RTX 4080 16GB	RTX 4090 24GB
VRAM	8GB	16GB	24GB
CPU	4 cores	8 cores	16 cores
RAM	16GB	32GB	64GB
Storage	30GB SSD	50GB NVMe	100GB NVMe
Internet	100 Mbps	500 Mbps	1 Gbps

Quick Deploy on CLORE.AI

Docker Image:

pytorch/pytorch:2.11.0-cuda12.8-cudnn9-devel

Ports:

22/tcp
7860/http

Command:

cd /workspace && \
pip install sam-2 && \
python -c "from sam2.build_sam import build_sam2; print('SAM2.1 ready!')"

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

Installation

# Official pip package (recommended for SAM2.1)
pip install sam-2

# Download SAM2.1 checkpoints
python -c "
from sam2.utils.misc import download_file_with_progress

checkpoints = [
    ('https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_tiny.pt', 'checkpoints/sam2.1_hiera_tiny.pt'),
    ('https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_small.pt', 'checkpoints/sam2.1_hiera_small.pt'),
    ('https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_base_plus.pt', 'checkpoints/sam2.1_hiera_base_plus.pt'),
    ('https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt', 'checkpoints/sam2.1_hiera_large.pt'),
]
"

# Or use the download script
mkdir -p checkpoints && cd checkpoints
wget https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt

Alternative: From Source (for development)

git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip install -e ".[demo]"

# Download SAM2.1 checkpoints
cd checkpoints
bash download_ckpts.sh

What You Can Create

Video Editing

• Remove objects from videos

• Replace backgrounds seamlessly

• Create video masks for compositing

Sports Analysis

• Track players through games

• Analyze movement patterns

• Generate highlight reels

Medical Imaging

• Segment organs in CT/MRI videos

• Track cell movement in microscopy

• Measure growth over time

Surveillance & Security

• Track objects across cameras

• Count people/vehicles

• Anomaly detection

Creative Projects

• Rotoscoping for VFX

• Interactive video installations

• AR/VR content creation

Basic Usage

Image Segmentation

import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
from PIL import Image
import numpy as np

# Load SAM2.1 model (improved accuracy over SAM2)
checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"

sam2 = build_sam2(model_cfg, checkpoint, device="cuda")
predictor = SAM2ImagePredictor(sam2)

# Load image
image = np.array(Image.open("image.jpg"))
predictor.set_image(image)

# Segment with point prompt
point_coords = np.array([[500, 375]])  # x, y coordinates
point_labels = np.array([1])  # 1 = foreground

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True
)

# Get best mask
best_mask = masks[scores.argmax()]

Video Object Tracking

import torch
from sam2.build_sam import build_sam2_video_predictor
import numpy as np

# Initialize SAM2.1 video predictor (improved tracking accuracy)
checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"

predictor = build_sam2_video_predictor(model_cfg, checkpoint, device="cuda")

# Initialize with video
video_path = "./video_frames"  # Directory with frame images
inference_state = predictor.init_state(video_path=video_path)

# Add point on first frame
predictor.reset_state(inference_state)
frame_idx = 0
obj_id = 1  # Object ID for tracking

points = np.array([[400, 300]], dtype=np.float32)
labels = np.array([1], dtype=np.int32)

# Add object to track
_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=frame_idx,
    obj_id=obj_id,
    points=points,
    labels=labels
)

# Propagate through video
video_segments = {}
for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
    video_segments[out_frame_idx] = {
        obj_id: (out_mask_logits[i] > 0.0).cpu().numpy()
        for i, obj_id in enumerate(out_obj_ids)
    }

Multi-Object Tracking

import torch
from sam2.build_sam import build_sam2_video_predictor
import numpy as np

predictor = build_sam2_video_predictor(
    "configs/sam2.1/sam2.1_hiera_l.yaml",
    "./checkpoints/sam2.1_hiera_large.pt",
    device="cuda"
)

video_path = "./video_frames"
inference_state = predictor.init_state(video_path=video_path)

# Track multiple objects
objects_to_track = [
    {"id": 1, "point": [200, 150], "frame": 0},  # Person 1
    {"id": 2, "point": [400, 200], "frame": 0},  # Person 2
    {"id": 3, "point": [600, 300], "frame": 0},  # Ball
]

for obj in objects_to_track:
    predictor.add_new_points_or_box(
        inference_state=inference_state,
        frame_idx=obj["frame"],
        obj_id=obj["id"],
        points=np.array([obj["point"]], dtype=np.float32),
        labels=np.array([1], dtype=np.int32)
    )

# Propagate all objects
all_masks = {}
for frame_idx, obj_ids, mask_logits in predictor.propagate_in_video(inference_state):
    all_masks[frame_idx] = {}
    for i, obj_id in enumerate(obj_ids):
        all_masks[frame_idx][obj_id] = (mask_logits[i] > 0.0).cpu().numpy()

Box Prompt Segmentation

from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
import numpy as np
from PIL import Image

sam2 = build_sam2(
    "configs/sam2.1/sam2.1_hiera_l.yaml",
    "./checkpoints/sam2.1_hiera_large.pt",
    device="cuda"
)
predictor = SAM2ImagePredictor(sam2)

image = np.array(Image.open("image.jpg"))
predictor.set_image(image)

# Segment with bounding box
box = np.array([100, 100, 400, 400])  # x1, y1, x2, y2

masks, scores, _ = predictor.predict(
    box=box,
    multimask_output=False
)

Gradio Interface

import gradio as gr
import numpy as np
from PIL import Image
import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

sam2 = build_sam2(
    "configs/sam2.1/sam2.1_hiera_l.yaml",
    "./checkpoints/sam2.1_hiera_large.pt",
    device="cuda"
)
predictor = SAM2ImagePredictor(sam2)

def segment_image(image, x, y):
    predictor.set_image(np.array(image))

    masks, scores, _ = predictor.predict(
        point_coords=np.array([[x, y]]),
        point_labels=np.array([1]),
        multimask_output=True
    )

    best_mask = masks[scores.argmax()]

    # Create overlay
    overlay = np.array(image).copy()
    overlay[best_mask] = overlay[best_mask] * 0.5 + np.array([255, 0, 0]) * 0.5

    return Image.fromarray(overlay.astype(np.uint8))

demo = gr.Interface(
    fn=segment_image,
    inputs=[
        gr.Image(type="pil", label="Input Image"),
        gr.Number(label="X coordinate"),
        gr.Number(label="Y coordinate")
    ],
    outputs=gr.Image(label="Segmented Image"),
    title="SAM2 - Segment Anything",
    description="Click coordinates to segment objects. Running on CLORE.AI GPU servers."
)

demo.launch(server_name="0.0.0.0", server_port=7860)

Export Masks as Video

import cv2
import numpy as np
from sam2.build_sam import build_sam2_video_predictor

predictor = build_sam2_video_predictor(
    "configs/sam2.1/sam2.1_hiera_l.yaml",
    "./checkpoints/sam2.1_hiera_large.pt",
    device="cuda"
)

# ... (tracking code from above)

# Export to video
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter('output_masks.mp4', fourcc, 30.0, (width, height))

for frame_idx in sorted(video_segments.keys()):
    frame = cv2.imread(f"./video_frames/{frame_idx:05d}.jpg")

    # Apply mask overlay
    for obj_id, mask in video_segments[frame_idx].items():
        color = [0, 255, 0] if obj_id == 1 else [0, 0, 255]
        frame[mask.squeeze()] = frame[mask.squeeze()] * 0.5 + np.array(color) * 0.5

    out.write(frame.astype(np.uint8))

out.release()

Performance

Task	Resolution	GPU	Speed
Image segmentation	1024x1024	RTX 3090	50ms
Image segmentation	1024x1024	RTX 4090	30ms
Video (per frame)	720p	RTX 4090	45ms
Video (per frame)	1080p	A100	35ms

Model Variants (SAM2.1)

SAM2.1 introduces new sam2.1_hiera_* checkpoints with improved video tracking accuracy:

Model	Parameters	VRAM	Speed	Quality	Checkpoint
sam2.1_hiera_tiny	38M	4GB	Fastest	Good	sam2.1_hiera_tiny.pt
sam2.1_hiera_small	46M	5GB	Fast	Better	sam2.1_hiera_small.pt
sam2.1_hiera_base_plus	80M	8GB	Medium	Great	sam2.1_hiera_base_plus.pt
sam2.1_hiera_large	224M	12GB	Slower	Best	sam2.1_hiera_large.pt

Note: SAM2.1 models consistently outperform their SAM2 counterparts on video benchmarks, especially for fast-moving objects and long occlusions.

Common Problems & Solutions

Out of Memory

Problem: CUDA out of memory on long videos

Solutions:

# Process in chunks
chunk_size = 100  # frames per chunk

for start_frame in range(0, total_frames, chunk_size):
    end_frame = min(start_frame + chunk_size, total_frames)
    # Process chunk...
    torch.cuda.empty_cache()  # Clear memory between chunks

Tracking Lost

Problem: Object tracking fails mid-video

Solutions:

• Add correction points when tracking drifts

• Use box prompts for better initial segmentation

• Choose clearer initial frames

# Add correction point
predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=lost_frame,
    obj_id=obj_id,
    points=np.array([[new_x, new_y]], dtype=np.float32),
    labels=np.array([1], dtype=np.int32)
)

Slow Processing

Problem: Video processing is too slow

Solutions:

• Use smaller model variant (tiny/small)

• Reduce video resolution

• Enable half-precision (fp16)

• Process on A100 GPU

# Use smaller SAM2.1 model for speed
predictor = build_sam2_video_predictor(
    "configs/sam2.1/sam2.1_hiera_t.yaml",
    "./checkpoints/sam2.1_hiera_tiny.pt",
    device="cuda"
)

Poor Mask Quality

Problem: Segmentation edges are rough

Solutions:

• Use larger model (large instead of tiny)

• Add more point prompts

• Combine point and box prompts

Troubleshooting

Segmentation inaccurate

• Click more precisely on target object

• Add multiple positive/negative points

• Use box prompt for large objects

Video memory error

• Process fewer frames at once

• Reduce video resolution

• Use streaming mode for long videos

Tracking lost

• Add more prompts when object changes

• Use memory bank feature

• Check object isn't occluded

Slow processing

• SAM2 is compute-heavy

• Use A100 for long videos

• Consider frame skipping

Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

GPU	Hourly Rate	Daily Rate	4-Hour Session
RTX 3060	~$0.03	~$0.70	~$0.12
RTX 3090	~$0.06	~$1.50	~$0.25
RTX 4090	~$0.10	~$2.30	~$0.40
A100 40GB	~$0.17	~$4.00	~$0.70
A100 80GB	~$0.25	~$6.00	~$1.00

Prices vary by provider and demand. Check CLORE.AI Marketplace for current rates.

Save money:

• Use the Spot market for interruptible work — about a third of servers price spot below on-demand (median ~13% off), the rest match it

• Pay with CLORE tokens

• Compare prices across different providers

Next Steps

• GroundingDINO - Auto-detect objects to segment

• Florence-2 - Vision-language understanding

• Depth Anything - Depth estimation