Stable Video Diffusion

Generate videos from images with Stable Video Diffusion on Clore.ai

Newer alternatives available! Consider FramePack (only 6GB VRAM!), Wan2.1 (higher quality), or LTX-2 (video with native audio).

Generate videos from images using Stability AI's SVD model.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

What is Stable Video Diffusion?

SVD (Stable Video Diffusion) generates short video clips from a single image:

• 14 or 25 frame outputs

• 576x1024 resolution

• Smooth motion generation

• Open source weights

Resources

• HuggingFace: stabilityai/stable-video-diffusion-img2vid-xt

• GitHub: Stability-AI/generative-models

• Paper: SVD Paper

Hardware Requirements

Model	VRAM	Recommended GPU
SVD (14 frames)	16GB	RTX 4090
SVD-XT (25 frames)	24GB	RTX 4090 / A100

Quick Deploy

Docker Image:

pytorch/pytorch:2.11.0-cuda12.8-cudnn9-devel

Ports:

22/tcp
7860/http

Command:

pip install diffusers transformers accelerate && \
pip install gradio && \
python -c "
import gradio as gr
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import export_to_video
import torch

pipe = StableVideoDiffusionPipeline.from_pretrained(
    'stabilityai/stable-video-diffusion-img2vid-xt',
    torch_dtype=torch.float16,
    variant='fp16'
).to('cuda')

def generate(image, seed, fps):
    generator = torch.manual_seed(seed)
    frames = pipe(image, num_frames=25, generator=generator).frames[0]
    export_to_video(frames, 'output.mp4', fps=fps)
    return 'output.mp4'

gr.Interface(
    fn=generate,
    inputs=[gr.Image(type='pil'), gr.Number(value=42, label='Seed'), gr.Slider(6, 30, value=7, label='FPS')],
    outputs=gr.Video(),
    title='Stable Video Diffusion'
).launch(server_name='0.0.0.0', server_port=7860)
"

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

Installation

pip install diffusers transformers accelerate torch

# For video export
pip install imageio[ffmpeg]

Basic Usage

import torch
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video

# Load pipeline
pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")

# Load and resize image
image = load_image("input.jpg")
image = image.resize((1024, 576))

# Generate video
generator = torch.manual_seed(42)
frames = pipe(image, num_frames=25, generator=generator).frames[0]

# Save video
export_to_video(frames, "output.mp4", fps=7)

SVD vs SVD-XT

Feature	SVD	SVD-XT
Frames	14	25
Duration	~2 sec	~3.5 sec
VRAM	16GB	24GB
Quality	Good	Better

Memory Optimization

# Enable memory efficient attention
pipe.enable_model_cpu_offload()

# Or use attention slicing
pipe.enable_attention_slicing()

# For very low VRAM
pipe.enable_sequential_cpu_offload()

Batch Processing

import os
from pathlib import Path

input_dir = Path("./images")
output_dir = Path("./videos")
output_dir.mkdir(exist_ok=True)

for img_path in input_dir.glob("*.jpg"):
    image = load_image(str(img_path)).resize((1024, 576))
    frames = pipe(image, num_frames=25).frames[0]
    export_to_video(frames, str(output_dir / f"{img_path.stem}.mp4"), fps=7)
    print(f"Generated: {img_path.stem}.mp4")

ComfyUI Integration

SVD works great in ComfyUI:

1. Install ComfyUI

2. Download SVD model to models/checkpoints/

3. Use SVD nodes for img2vid workflow

Troubleshooting

Out of memory

• Use enable_model_cpu_offload()

• Reduce num_frames to 14

• Use fp16 variant

Video too short

• Use SVD-XT (25 frames) instead of SVD (14 frames)

• Interpolate with RIFE for smoother result

Poor motion quality

• Use high-quality input images

• Ensure image is 1024x576 (or 576x1024)

• Try different seeds

CUDA errors

• Update PyTorch and diffusers

• Check CUDA version compatibility

Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

GPU	Hourly Rate	Daily Rate	4-Hour Session
RTX 3060	~$0.03	~$0.70	~$0.12
RTX 3090	~$0.06	~$1.50	~$0.25
RTX 4090	~$0.10	~$2.30	~$0.40
A100 40GB	~$0.17	~$4.00	~$0.70
A100 80GB	~$0.25	~$6.00	~$1.00

Prices vary by provider. Check CLORE.AI Marketplace for current rates.

Next Steps

• AnimateDiff - Animate SD images

• RIFE Interpolation - Increase FPS

• Hunyuan Video - Text-to-video