SAM2 Video

Track and segment any object through video with Meta's SAM2.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

All examples in this guide can be run on GPU servers rented through CLORE.AI Marketplace marketplace.

Renting on CLORE.AI

1. Visit CLORE.AI Marketplace

2. Filter by GPU type, VRAM, and price

3. Choose On-Demand (fixed rate) or Spot (bid price)

4. Configure your order:

   • Select Docker image

   • Set ports (TCP for SSH, HTTP for web UIs)

   • Add environment variables if needed

   • Enter startup command

5. Select payment: CLORE, BTC, or USDT/USDC

6. Create order and wait for deployment

Access Your Server

• Find connection details in My Orders

• Web interfaces: Use the HTTP port URL

• SSH: ssh -p <port> root@<proxy-address>

What is SAM2?

SAM2 (Segment Anything Model 2) by Meta AI enables:

• Real-time video object segmentation

• Click-to-track any object

• Consistent tracking through occlusions

• Memory-efficient video processing

Resources

• GitHub: facebookresearch/sam2

• Paper: SAM2 Paper

• Demo: SAM2 Demo

• Model Weights: SAM2 Checkpoints

Recommended Hardware

Component	Minimum	Recommended	Optimal
GPU	RTX 3060 12GB	RTX 4080 16GB	RTX 4090 24GB
VRAM	8GB	16GB	24GB
CPU	4 cores	8 cores	16 cores
RAM	16GB	32GB	64GB
Storage	30GB SSD	50GB NVMe	100GB NVMe
Internet	100 Mbps	500 Mbps	1 Gbps

Quick Deploy on CLORE.AI

Docker Image:

pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel

Ports:

22/tcp
7860/http

Command:

cd /workspace && \
git clone https://github.com/facebookresearch/sam2.git && \
cd sam2 && \
pip install -e . && \
python demo/app.py

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

Installation

git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip install -e .

# Download checkpoints
cd checkpoints
./download_ckpts.sh

What You Can Create

Video Editing

• Remove objects from videos

• Replace backgrounds seamlessly

• Create video masks for compositing

Sports Analysis

• Track players through games

• Analyze movement patterns

• Generate highlight reels

Medical Imaging

• Segment organs in CT/MRI videos

• Track cell movement in microscopy

• Measure growth over time

Surveillance & Security

• Track objects across cameras

• Count people/vehicles

• Anomaly detection

Creative Projects

• Rotoscoping for VFX

• Interactive video installations

• AR/VR content creation

Basic Usage

Image Segmentation

import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
from PIL import Image
import numpy as np

# Load model
checkpoint = "./checkpoints/sam2_hiera_large.pt"
model_cfg = "sam2_hiera_l.yaml"

sam2 = build_sam2(model_cfg, checkpoint, device="cuda")
predictor = SAM2ImagePredictor(sam2)

# Load image
image = np.array(Image.open("image.jpg"))
predictor.set_image(image)

# Segment with point prompt
point_coords = np.array([[500, 375]])  # x, y coordinates
point_labels = np.array([1])  # 1 = foreground

masks, scores, logits = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True
)

# Get best mask
best_mask = masks[scores.argmax()]

Video Object Tracking

import torch
from sam2.build_sam import build_sam2_video_predictor
import numpy as np

# Initialize video predictor
checkpoint = "./checkpoints/sam2_hiera_large.pt"
model_cfg = "sam2_hiera_l.yaml"

predictor = build_sam2_video_predictor(model_cfg, checkpoint, device="cuda")

# Initialize with video
video_path = "./video_frames"  # Directory with frame images
inference_state = predictor.init_state(video_path=video_path)

# Add point on first frame
predictor.reset_state(inference_state)
frame_idx = 0
obj_id = 1  # Object ID for tracking

points = np.array([[400, 300]], dtype=np.float32)
labels = np.array([1], dtype=np.int32)

# Add object to track
_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=frame_idx,
    obj_id=obj_id,
    points=points,
    labels=labels
)

# Propagate through video
video_segments = {}
for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
    video_segments[out_frame_idx] = {
        obj_id: (out_mask_logits[i] > 0.0).cpu().numpy()
        for i, obj_id in enumerate(out_obj_ids)
    }

Multi-Object Tracking

import torch
from sam2.build_sam import build_sam2_video_predictor
import numpy as np

predictor = build_sam2_video_predictor("sam2_hiera_l.yaml", "./checkpoints/sam2_hiera_large.pt", device="cuda")

video_path = "./video_frames"
inference_state = predictor.init_state(video_path=video_path)

# Track multiple objects
objects_to_track = [
    {"id": 1, "point": [200, 150], "frame": 0},  # Person 1
    {"id": 2, "point": [400, 200], "frame": 0},  # Person 2
    {"id": 3, "point": [600, 300], "frame": 0},  # Ball
]

for obj in objects_to_track:
    predictor.add_new_points_or_box(
        inference_state=inference_state,
        frame_idx=obj["frame"],
        obj_id=obj["id"],
        points=np.array([obj["point"]], dtype=np.float32),
        labels=np.array([1], dtype=np.int32)
    )

# Propagate all objects
all_masks = {}
for frame_idx, obj_ids, mask_logits in predictor.propagate_in_video(inference_state):
    all_masks[frame_idx] = {}
    for i, obj_id in enumerate(obj_ids):
        all_masks[frame_idx][obj_id] = (mask_logits[i] > 0.0).cpu().numpy()

Box Prompt Segmentation

from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor
import numpy as np
from PIL import Image

sam2 = build_sam2("sam2_hiera_l.yaml", "./checkpoints/sam2_hiera_large.pt", device="cuda")
predictor = SAM2ImagePredictor(sam2)

image = np.array(Image.open("image.jpg"))
predictor.set_image(image)

# Segment with bounding box
box = np.array([100, 100, 400, 400])  # x1, y1, x2, y2

masks, scores, _ = predictor.predict(
    box=box,
    multimask_output=False
)

Gradio Interface

import gradio as gr
import numpy as np
from PIL import Image
import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

sam2 = build_sam2("sam2_hiera_l.yaml", "./checkpoints/sam2_hiera_large.pt", device="cuda")
predictor = SAM2ImagePredictor(sam2)

def segment_image(image, x, y):
    predictor.set_image(np.array(image))

    masks, scores, _ = predictor.predict(
        point_coords=np.array([[x, y]]),
        point_labels=np.array([1]),
        multimask_output=True
    )

    best_mask = masks[scores.argmax()]

    # Create overlay
    overlay = np.array(image).copy()
    overlay[best_mask] = overlay[best_mask] * 0.5 + np.array([255, 0, 0]) * 0.5

    return Image.fromarray(overlay.astype(np.uint8))

demo = gr.Interface(
    fn=segment_image,
    inputs=[
        gr.Image(type="pil", label="Input Image"),
        gr.Number(label="X coordinate"),
        gr.Number(label="Y coordinate")
    ],
    outputs=gr.Image(label="Segmented Image"),
    title="SAM2 - Segment Anything",
    description="Click coordinates to segment objects. Running on CLORE.AI GPU servers."
)

demo.launch(server_name="0.0.0.0", server_port=7860)

Export Masks as Video

import cv2
import numpy as np
from sam2.build_sam import build_sam2_video_predictor

predictor = build_sam2_video_predictor("sam2_hiera_l.yaml", "./checkpoints/sam2_hiera_large.pt", device="cuda")

# ... (tracking code from above)

# Export to video
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter('output_masks.mp4', fourcc, 30.0, (width, height))

for frame_idx in sorted(video_segments.keys()):
    frame = cv2.imread(f"./video_frames/{frame_idx:05d}.jpg")

    # Apply mask overlay
    for obj_id, mask in video_segments[frame_idx].items():
        color = [0, 255, 0] if obj_id == 1 else [0, 0, 255]
        frame[mask.squeeze()] = frame[mask.squeeze()] * 0.5 + np.array(color) * 0.5

    out.write(frame.astype(np.uint8))

out.release()

Performance

Task	Resolution	GPU	Speed
Image segmentation	1024x1024	RTX 3090	50ms
Image segmentation	1024x1024	RTX 4090	30ms
Video (per frame)	720p	RTX 4090	45ms
Video (per frame)	1080p	A100	35ms

Model Variants

Model	Parameters	VRAM	Speed	Quality
sam2_hiera_tiny	38M	4GB	Fastest	Good
sam2_hiera_small	46M	5GB	Fast	Better
sam2_hiera_base	80M	8GB	Medium	Great
sam2_hiera_large	224M	12GB	Slower	Best

Common Problems & Solutions

Out of Memory

Problem: CUDA out of memory on long videos

Solutions:

# Process in chunks
chunk_size = 100  # frames per chunk

for start_frame in range(0, total_frames, chunk_size):
    end_frame = min(start_frame + chunk_size, total_frames)
    # Process chunk...
    torch.cuda.empty_cache()  # Clear memory between chunks

Tracking Lost

Problem: Object tracking fails mid-video

Solutions:

• Add correction points when tracking drifts

• Use box prompts for better initial segmentation

• Choose clearer initial frames

# Add correction point
predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=lost_frame,
    obj_id=obj_id,
    points=np.array([[new_x, new_y]], dtype=np.float32),
    labels=np.array([1], dtype=np.int32)
)

Slow Processing

Problem: Video processing is too slow

Solutions:

• Use smaller model variant (tiny/small)

• Reduce video resolution

• Enable half-precision (fp16)

• Process on A100 GPU

# Use smaller model for speed
predictor = build_sam2_video_predictor(
    "sam2_hiera_t.yaml",
    "./checkpoints/sam2_hiera_tiny.pt",
    device="cuda"
)

Poor Mask Quality

Problem: Segmentation edges are rough

Solutions:

• Use larger model (large instead of tiny)

• Add more point prompts

• Combine point and box prompts

Troubleshooting

Segmentation inaccurate

• Click more precisely on target object

• Add multiple positive/negative points

• Use box prompt for large objects

Video memory error

• Process fewer frames at once

• Reduce video resolution

• Use streaming mode for long videos

Tracking lost

• Add more prompts when object changes

• Use memory bank feature

• Check object isn't occluded

Slow processing

• SAM2 is compute-heavy

• Use A100 for long videos

• Consider frame skipping

Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

GPU	Hourly Rate	Daily Rate	4-Hour Session
RTX 3060	~$0.03	~$0.70	~$0.12
RTX 3090	~$0.06	~$1.50	~$0.25
RTX 4090	~$0.10	~$2.30	~$0.40
A100 40GB	~$0.17	~$4.00	~$0.70
A100 80GB	~$0.25	~$6.00	~$1.00

Prices vary by provider and demand. Check CLORE.AI Marketplace for current rates.

Save money:

• Use Spot market for flexible workloads (often 30-50% cheaper)

• Pay with CLORE tokens

• Compare prices across different providers

Next Steps

• GroundingDINO - Auto-detect objects to segment

• Florence-2 - Vision-language understanding

• Depth Anything - Depth estimation