AI Video Generation

Generate videos using Stable Video Diffusion, AnimateDiff, and other models.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

Renting on CLORE.AI

1. Visit CLORE.AI Marketplace

2. Filter by GPU type, VRAM, and price

3. Choose On-Demand (fixed rate) or Spot (bid price)

4. Configure your order:

   • Select Docker image

   • Set ports (TCP for SSH, HTTP for web UIs)

   • Add environment variables if needed

   • Enter startup command

5. Select payment: CLORE, BTC, or USDT/USDC

6. Create order and wait for deployment

Access Your Server

• Find connection details in My Orders

• Web interfaces: Use the HTTP port URL

• SSH: ssh -p <port> root@<proxy-address>

Available Models

Model	Type	VRAM	Duration
SVD	Image-to-Video	16GB	4 seconds
SVD-XT	Image-to-Video	20GB	4 seconds
AnimateDiff	Text-to-Video	12GB	2-4 seconds
CogVideoX	Text-to-Video	24GB	6 seconds

Stable Video Diffusion (SVD)

Quick Deploy

Docker Image:

pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel

Ports:

22/tcp
7860/http

Command:

pip install diffusers transformers accelerate gradio imageio && \
python svd_server.py

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

SVD Script

import torch
from diffusers import StableVideoDiffusionPipeline
from PIL import Image
import imageio

# Load model
pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16,
)
pipe.to("cuda")
pipe.enable_model_cpu_offload()

# Load and resize image
image = Image.open("input.png").resize((1024, 576))

# Generate video
frames = pipe(
    image,
    decode_chunk_size=8,
    num_frames=25,
    motion_bucket_id=127,
    noise_aug_strength=0.02
).frames[0]

# Save as GIF
imageio.mimsave("output.gif", frames, fps=6)

# Save as MP4
imageio.mimsave("output.mp4", frames, fps=6)

SVD with Gradio UI

import gradio as gr
import torch
from diffusers import StableVideoDiffusionPipeline
from PIL import Image
import imageio
import tempfile

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()

def generate_video(image, motion_bucket, fps, num_frames):
    image = image.resize((1024, 576))

    frames = pipe(
        image,
        decode_chunk_size=4,
        num_frames=num_frames,
        motion_bucket_id=motion_bucket,
    ).frames[0]

    with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as f:
        imageio.mimsave(f.name, frames, fps=fps)
        return f.name

demo = gr.Interface(
    fn=generate_video,
    inputs=[
        gr.Image(type="pil", label="Input Image"),
        gr.Slider(1, 255, value=127, label="Motion Amount"),
        gr.Slider(1, 30, value=6, label="FPS"),
        gr.Slider(14, 25, value=25, label="Frames")
    ],
    outputs=gr.Video(label="Generated Video"),
)

demo.launch(server_name="0.0.0.0", server_port=7860)

AnimateDiff

Installation

pip install diffusers transformers accelerate

Generate Video from Text

import torch
from diffusers import AnimateDiffPipeline, MotionAdapter, DDIMScheduler
import imageio

# Load motion adapter
adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")

# Load pipeline
pipe = AnimateDiffPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    motion_adapter=adapter,
    torch_dtype=torch.float16,
)
pipe.scheduler = DDIMScheduler.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    subfolder="scheduler",
    clip_sample=False,
    timestep_spacing="linspace",
    beta_schedule="linear",
    steps_offset=1,
)
pipe.to("cuda")
pipe.enable_model_cpu_offload()

# Generate
output = pipe(
    prompt="A cat walking through a garden, beautiful flowers, sunny day",
    negative_prompt="bad quality, blurry",
    num_frames=16,
    guidance_scale=7.5,
    num_inference_steps=25,
)

# Save
frames = output.frames[0]
imageio.mimsave("animatediff.gif", frames, fps=8)

AnimateDiff with Custom Model

from diffusers import AnimateDiffPipeline, MotionAdapter, EulerDiscreteScheduler

adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2")

# Use a custom checkpoint (e.g., RealisticVision)
pipe = AnimateDiffPipeline.from_pretrained(
    "SG161222/Realistic_Vision_V5.1_noVAE",
    motion_adapter=adapter,
    torch_dtype=torch.float16,
)

AnimateDiff in ComfyUI

Install Nodes

cd /workspace/ComfyUI/custom_nodes
git clone https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved.git
git clone https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite.git

Download Motion Models

cd /workspace/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/models
wget https://huggingface.co/guoyww/animatediff/resolve/main/mm_sd_v15_v2.ckpt

CogVideoX

Text-to-Video

import torch
from diffusers import CogVideoXPipeline
import imageio

pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-2b",
    torch_dtype=torch.float16
)
pipe.to("cuda")
pipe.enable_model_cpu_offload()

prompt = "A drone flying over a beautiful mountain landscape at sunset"

video = pipe(
    prompt=prompt,
    num_videos_per_prompt=1,
    num_inference_steps=50,
    num_frames=49,
    guidance_scale=6,
).frames[0]

imageio.mimsave("cogvideo.mp4", video, fps=8)

Video Upscaling

Real-ESRGAN for Video

import cv2
import torch
from basicsr.archs.rrdbnet_arch import RRDBNet
from realesrgan import RealESRGANer

model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
upsampler = RealESRGANer(
    scale=4,
    model_path='RealESRGAN_x4plus.pth',
    model=model,
    tile=400,
    tile_pad=10,
    pre_pad=0,
    half=True
)

# Process video frame by frame
cap = cv2.VideoCapture("input.mp4")

# ... upscale each frame

Interpolation (Smooth Videos)

FILM Frame Interpolation

# Install
pip install tensorflow tensorflow_hub

import tensorflow as tf
import tensorflow_hub as hub

model = hub.load("https://tfhub.dev/google/film/1")

def interpolate(frame1, frame2, num_interpolations=3):
    # Returns interpolated frames between frame1 and frame2
    ...

RIFE (Real-Time)

pip install rife-ncnn-vulkan-python

from rife_ncnn_vulkan import Rife
rife = Rife(gpu_id=0)

# Interpolate frames

Batch Video Generation

prompts = [
    "A rocket launching into space",
    "Ocean waves crashing on rocks",
    "A butterfly flying through flowers",
]

for i, prompt in enumerate(prompts):
    print(f"Generating {i+1}/{len(prompts)}")
    output = pipe(prompt, num_frames=16)
    imageio.mimsave(f"video_{i:03d}.mp4", output.frames[0], fps=8)

Memory Tips

For Limited VRAM

# Enable CPU offload
pipe.enable_model_cpu_offload()

# Enable VAE slicing
pipe.enable_vae_slicing()

# Enable attention slicing
pipe.enable_attention_slicing()

# Reduce frame count
num_frames = 14  # Instead of 25

Chunked Decoding

frames = pipe(
    image,
    decode_chunk_size=2,  # Decode 2 frames at a time
    num_frames=25,
).frames[0]

Converting Output

GIF to MP4

ffmpeg -i input.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" output.mp4

Frame Sequence to Video

ffmpeg -framerate 8 -i frame_%04d.png -c:v libx264 -pix_fmt yuv420p output.mp4

Add Audio

ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -shortest output_with_audio.mp4

Performance

Model	GPU	Frames	Time
SVD-XT	RTX 3090	25	~120s
SVD-XT	RTX 4090	25	~80s
SVD-XT	A100	25	~50s
AnimateDiff	RTX 3090	16	~30s
CogVideoX	A100	49	~180s

Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

GPU	Hourly Rate	Daily Rate	4-Hour Session
RTX 3060	~$0.03	~$0.70	~$0.12
RTX 3090	~$0.06	~$1.50	~$0.25
RTX 4090	~$0.10	~$2.30	~$0.40
A100 40GB	~$0.17	~$4.00	~$0.70
A100 80GB	~$0.25	~$6.00	~$1.00

Prices vary by provider and demand. Check CLORE.AI Marketplace for current rates.

Save money:

• Use Spot market for flexible workloads (often 30-50% cheaper)

• Pay with CLORE tokens

• Compare prices across different providers

Troubleshooting

OOM Error

• Reduce num_frames

• Enable CPU offload

• Use smaller decode_chunk_size

Flickering Video

• Increase num_inference_steps

• Try different motion_bucket_id

• Use frame interpolation

Poor Quality

• Use higher resolution input (SVD)

• Better prompts (AnimateDiff)

• Increase guidance_scale