# LivePortrait

Create realistic animated portraits from single images.

{% hint style="success" %}
All examples can be run on GPU servers rented through [CLORE.AI Marketplace](https://clore.ai/marketplace).
{% endhint %}

{% hint style="info" %}
All examples in this guide can be run on GPU servers rented through [CLORE.AI Marketplace](https://clore.ai/marketplace) marketplace.
{% endhint %}

## Renting on CLORE.AI

1. Visit [CLORE.AI Marketplace](https://clore.ai/marketplace)
2. Filter by GPU type, VRAM, and price
3. Choose **On-Demand** (fixed rate) or **Spot** (bid price)
4. Configure your order:
   * Select Docker image
   * Set ports (TCP for SSH, HTTP for web UIs)
   * Add environment variables if needed
   * Enter startup command
5. Select payment: **CLORE**, **BTC**, or **USDT/USDC**
6. Create order and wait for deployment

### Access Your Server

* Find connection details in **My Orders**
* Web interfaces: Use the HTTP port URL
* SSH: `ssh -p <port> root@<proxy-address>`

## What is LivePortrait?

LivePortrait by Kuaishou enables:

* Animate any portrait with driving video
* Single photo to video animation
* Expression and pose transfer
* Real-time capable inference

## Resources

* **GitHub:** [KwaiVGI/LivePortrait](https://github.com/KwaiVGI/LivePortrait)
* **Paper:** [LivePortrait Paper](https://arxiv.org/abs/2407.03168)
* **HuggingFace:** [KwaiVGI/LivePortrait](https://huggingface.co/KwaiVGI/LivePortrait)
* **Demo:** [HuggingFace Space](https://huggingface.co/spaces/KwaiVGI/LivePortrait)

## Recommended Hardware

| Component | Minimum      | Recommended   | Optimal       |
| --------- | ------------ | ------------- | ------------- |
| GPU       | RTX 3070 8GB | RTX 4080 16GB | RTX 4090 24GB |
| VRAM      | 8GB          | 16GB          | 24GB          |
| CPU       | 4 cores      | 8 cores       | 16 cores      |
| RAM       | 16GB         | 32GB          | 64GB          |
| Storage   | 30GB SSD     | 50GB NVMe     | 100GB NVMe    |
| Internet  | 100 Mbps     | 500 Mbps      | 1 Gbps        |

## Quick Deploy on CLORE.AI

**Docker Image:**

```
pytorch/pytorch:2.5.1-cuda12.4-cudnn9-devel
```

**Ports:**

```
22/tcp
7860/http
```

**Command:**

```bash
cd /workspace && \
git clone https://github.com/KwaiVGI/LivePortrait.git && \
cd LivePortrait && \
pip install -r requirements.txt && \
python app.py
```

## Accessing Your Service

After deployment, find your `http_pub` URL in **My Orders**:

1. Go to **My Orders** page
2. Click on your order
3. Find the `http_pub` URL (e.g., `abc123.clorecloud.net`)

Use `https://YOUR_HTTP_PUB_URL` instead of `localhost` in examples below.

## Installation

```bash
git clone https://github.com/KwaiVGI/LivePortrait.git
cd LivePortrait
pip install -r requirements.txt

# Download pretrained models
huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights
```

## What You Can Create

### Virtual Avatars

* AI influencers and virtual anchors
* Customer service avatars
* Educational presenters

### Content Creation

* Social media content
* Marketing materials
* Music video concepts

### Entertainment

* Animate historical photos
* Character animations
* Interactive experiences

### Professional Uses

* Video conferencing avatars
* Presentation assistants
* Training simulations

## Basic Usage

### Command Line

```bash
python inference.py \
    --source_image path/to/portrait.jpg \
    --driving_video path/to/driving.mp4 \
    --output_path output.mp4
```

### Python API

```python
from liveportrait import LivePortraitPipeline

# Initialize pipeline
pipeline = LivePortraitPipeline(
    device="cuda",
    model_path="./pretrained_weights"
)

# Animate portrait
result = pipeline.animate(
    source_image="portrait.jpg",
    driving_video="driving.mp4"
)

result.save("animated_portrait.mp4")
```

## Portrait with Expression Control

```python
from liveportrait import LivePortraitPipeline
import cv2

pipeline = LivePortraitPipeline(device="cuda")

# Control specific expressions
expressions = {
    "smile": 0.8,
    "eyebrow_raise": 0.3,
    "head_pitch": -5,  # degrees
    "head_yaw": 10
}

result = pipeline.animate_with_expression(
    source_image="portrait.jpg",
    expressions=expressions,
    num_frames=60,
    fps=30
)

result.save("expression_controlled.mp4")
```

## Batch Processing

```python
import os
from liveportrait import LivePortraitPipeline

pipeline = LivePortraitPipeline(device="cuda")

# Animate multiple portraits with same driving video
portraits = [
    "portrait1.jpg",
    "portrait2.jpg",
    "portrait3.jpg"
]

driving = "speech_driving.mp4"
output_dir = "./animated"
os.makedirs(output_dir, exist_ok=True)

for i, portrait in enumerate(portraits):
    print(f"Processing {i+1}/{len(portraits)}: {portrait}")

    result = pipeline.animate(
        source_image=portrait,
        driving_video=driving
    )

    result.save(f"{output_dir}/animated_{i:03d}.mp4")
```

## Gradio Interface

```python
import gradio as gr
from liveportrait import LivePortraitPipeline
import tempfile

pipeline = LivePortraitPipeline(device="cuda")

def animate(source_image, driving_video):
    with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as f:
        result = pipeline.animate(
            source_image=source_image,
            driving_video=driving_video
        )
        result.save(f.name)
        return f.name

demo = gr.Interface(
    fn=animate,
    inputs=[
        gr.Image(type="filepath", label="Portrait Image"),
        gr.Video(label="Driving Video")
    ],
    outputs=gr.Video(label="Animated Portrait"),
    title="LivePortrait - Animate Any Portrait",
    description="Upload a portrait and a driving video to create an animated video. Running on CLORE.AI GPU servers."
)

demo.launch(server_name="0.0.0.0", server_port=7860)
```

## Real-Time Webcam Animation

```python
import cv2
from liveportrait import LivePortraitPipeline

pipeline = LivePortraitPipeline(device="cuda")

# Load source portrait
source = cv2.imread("portrait.jpg")
pipeline.set_source(source)

# Open webcam
cap = cv2.VideoCapture(0)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Animate with current frame as driving
    animated = pipeline.animate_frame(frame)

    cv2.imshow("LivePortrait", animated)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()
```

## Integration with TTS

Create talking avatars with text-to-speech:

```python
from liveportrait import LivePortraitPipeline
from TTS.api import TTS

# Generate speech
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")
tts.tts_to_file(
    text="Hello! Welcome to our presentation.",
    file_path="speech.wav",
    speaker_wav="reference_voice.wav",
    language="en"
)

# Generate lip-sync driving video from audio

# (Using separate lip-sync tool or pre-made driving video)

# Animate portrait
pipeline = LivePortraitPipeline(device="cuda")
result = pipeline.animate(
    source_image="presenter.jpg",
    driving_video="lip_sync_driving.mp4"
)
result.save("talking_avatar.mp4")
```

## Performance

| Resolution | GPU      | FPS | Latency |
| ---------- | -------- | --- | ------- |
| 256x256    | RTX 3070 | 30  | 33ms    |
| 256x256    | RTX 4090 | 60+ | 16ms    |
| 512x512    | RTX 4090 | 30  | 33ms    |
| 512x512    | A100     | 45  | 22ms    |

## Common Problems & Solutions

### Face Not Detected

**Problem:** "No face detected in source image"

**Solutions:**

* Ensure face is clearly visible and front-facing
* Use good lighting in source image
* Crop image to focus on face
* Minimum face size: 128x128 pixels

### Motion Doesn't Match

**Problem:** Animation doesn't follow driving video

**Solutions:**

* Use driving videos with clear facial movements
* Ensure driving video has similar face orientation
* Try different driving videos

### Quality Issues

**Problem:** Output looks blurry or distorted

**Solutions:**

```python

# Use higher quality settings
result = pipeline.animate(
    source_image=source,
    driving_video=driving,
    output_size=512,  # Higher resolution
    enhance_face=True  # Enable face enhancement
)
```

### Real-Time Lag

**Problem:** Webcam animation is laggy

**Solutions:**

* Use smaller output resolution (256x256)
* Enable TensorRT optimization
* Use RTX 4090 or better for real-time

```python
pipeline = LivePortraitPipeline(
    device="cuda",
    use_tensorrt=True  # Enable TensorRT
)
```

### Audio Sync Issues

**Problem:** Lip movements don't match audio

**Solutions:**

* Use audio-to-driving video generation
* Adjust video timing in post-processing
* Use Wav2Lip for better lip sync

## Troubleshooting

### Face not detected

* Ensure face is clearly visible in source
* Use front-facing photos
* Check image resolution (512+ recommended)

### Animation looks unnatural

* Source and driving video should have similar face angles
* Avoid extreme expressions in driving video
* Use shorter driving clips

### Output video corrupted

* Install ffmpeg: `apt install ffmpeg`
* Check output format compatibility
* Ensure enough disk space

### CUDA errors

* Install compatible PyTorch version
* Check CUDA version matches requirements

## Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

| GPU       | Hourly Rate | Daily Rate | 4-Hour Session |
| --------- | ----------- | ---------- | -------------- |
| RTX 3060  | \~$0.03     | \~$0.70    | \~$0.12        |
| RTX 3090  | \~$0.06     | \~$1.50    | \~$0.25        |
| RTX 4090  | \~$0.10     | \~$2.30    | \~$0.40        |
| A100 40GB | \~$0.17     | \~$4.00    | \~$0.70        |
| A100 80GB | \~$0.25     | \~$6.00    | \~$1.00        |

*Prices vary by provider and demand. Check* [*CLORE.AI Marketplace*](https://clore.ai/marketplace) *for current rates.*

**Save money:**

* Use **Spot** market for flexible workloads (often 30-50% cheaper)
* Pay with **CLORE** tokens
* Compare prices across different providers

## Next Steps

* [SadTalker](https://docs.clore.ai/guides/talking-heads/sadtalker) - Alternative talking head
* [Wav2Lip](https://docs.clore.ai/guides/talking-heads/wav2lip) - Better lip sync
* [XTTS](https://docs.clore.ai/guides/audio-and-voice/xtts-coqui) - Voice generation