IP-Adapter

Use images as prompts for Stable Diffusion with IP-Adapter

Use images as prompts for Stable Diffusion generation.

All examples can be run on GPU servers rented through CLORE.AI Marketplace.

Renting on CLORE.AI

1. Visit CLORE.AI Marketplace

2. Filter by GPU type, VRAM, and price

3. Choose On-Demand (fixed rate) or Spot (bid price)

4. Configure your order:

   • Select Docker image

   • Set ports (TCP for SSH, HTTP for web UIs)

   • Add environment variables if needed

   • Enter startup command

5. Select payment: CLORE, BTC, or USDT/USDC

6. Create order and wait for deployment

Access Your Server

• Find connection details in My Orders

• Web interfaces: Use the HTTP port URL

• SSH: ssh -p <port> root@<proxy-address>

What is IP-Adapter?

IP-Adapter enables image prompting:

• Use reference images to guide generation

• Combine with text prompts

• Style transfer and content transfer

• Works with SD 1.5 and SDXL

Adapter Types

Adapter	Use Case	VRAM
IP-Adapter	General image prompting	8GB
IP-Adapter-Plus	Higher quality	10GB
IP-Adapter-Face	Face-focused	10GB
IP-Adapter-Full-Face	Full face details	12GB

Quick Deploy

Docker Image:

pytorch/pytorch:2.11.0-cuda12.8-cudnn9-devel

Ports:

22/tcp
7860/http

Command:

pip install diffusers transformers accelerate && \
python ip_adapter_app.py

Accessing Your Service

After deployment, find your http_pub URL in My Orders:

1. Go to My Orders page

2. Click on your order

3. Find the http_pub URL (e.g., abc123.clorecloud.net)

Use https://YOUR_HTTP_PUB_URL instead of localhost in examples below.

Installation

pip install diffusers transformers accelerate

Basic Image Prompting

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load IP-Adapter
pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter_sdxl.bin"
)

# Load reference image
ip_image = load_image("reference.jpg")

# Generate with image prompt
image = pipe(
    prompt="a cat in the same style",
    ip_adapter_image=ip_image,
    num_inference_steps=30
).images[0]

image.save("output.png")

Style Transfer

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="models",
    weight_name="ip-adapter_sd15.bin"
)

# Style reference (e.g., Van Gogh painting)
style_image = load_image("van_gogh_starry_night.jpg")

# Generate new content in that style
image = pipe(
    prompt="a modern city skyline",
    ip_adapter_image=style_image,
    num_inference_steps=30,
    guidance_scale=7.5
).images[0]

image.save("city_van_gogh.png")

Face Adapter

For face-focused generation:

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load face-specific adapter
pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter-plus-face_sdxl_vit-h.bin"
)

# Reference face
face_image = load_image("face_reference.jpg")

# Generate portrait
image = pipe(
    prompt="portrait painting, oil on canvas, museum quality",
    ip_adapter_image=face_image,
    num_inference_steps=30
).images[0]

image.save("portrait.png")

Combining Multiple Images

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter_sdxl.bin"
)

# Multiple reference images
images = [
    load_image("style1.jpg"),
    load_image("style2.jpg")
]

# Generate blending both
image = pipe(
    prompt="landscape painting",
    ip_adapter_image=images,
    num_inference_steps=30
).images[0]

Scale Control

# Set adapter strength
pipe.set_ip_adapter_scale(0.6)  # 0.0 to 1.0

# Low scale = more text prompt influence

# High scale = more image prompt influence

# Per-image scaling with multiple images
pipe.set_ip_adapter_scale([0.7, 0.3])

With ControlNet

from diffusers import (
    AutoPipelineForText2Image,
    ControlNetModel
)
from diffusers.utils import load_image
import torch

# Load ControlNet
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/control_v11p_sd15_canny",
    torch_dtype=torch.float16
)

pipe = AutoPipelineForText2Image.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# Load IP-Adapter
pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="models",
    weight_name="ip-adapter_sd15.bin"
)

# Style image
style_image = load_image("style.jpg")

# Control image (edge map)
control_image = load_image("edges.png")

image = pipe(
    prompt="detailed illustration",
    image=control_image,
    ip_adapter_image=style_image,
    num_inference_steps=30
).images[0]

Gradio Interface

import gradio as gr
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter_sdxl.bin"
)

def generate(reference_image, prompt, negative_prompt, scale, steps):
    pipe.set_ip_adapter_scale(scale)

    image = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        ip_adapter_image=reference_image,
        num_inference_steps=steps
    ).images[0]

    return image

demo = gr.Interface(
    fn=generate,
    inputs=[
        gr.Image(type="pil", label="Reference Image"),
        gr.Textbox(label="Prompt", value="high quality"),
        gr.Textbox(label="Negative Prompt", value="ugly, blurry"),
        gr.Slider(0.0, 1.0, value=0.6, label="IP-Adapter Scale"),
        gr.Slider(10, 50, value=30, step=1, label="Steps")
    ],
    outputs=gr.Image(label="Generated Image"),
    title="IP-Adapter Image Prompting"
)

demo.launch(server_name="0.0.0.0", server_port=7860)

Batch Style Transfer

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch
import os

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter_sdxl.bin"
)

# Style reference
style_image = load_image("art_style.jpg")

# Subjects to generate
subjects = [
    "a mountain landscape",
    "a city at night",
    "a forest in autumn",
    "an ocean sunset",
    "a snowy village"
]

output_dir = "./styled_outputs"
os.makedirs(output_dir, exist_ok=True)

for i, subject in enumerate(subjects):
    print(f"Generating {i+1}/{len(subjects)}: {subject}")

    image = pipe(
        prompt=subject,
        ip_adapter_image=style_image,
        num_inference_steps=30
    ).images[0]

    image.save(f"{output_dir}/styled_{i:03d}.png")

Use Cases

Product Photography Style

# Reference: professional product photo
style = load_image("product_photo_reference.jpg")

image = pipe(
    prompt="red sneakers on white background",
    ip_adapter_image=style,
    num_inference_steps=30
).images[0]

Artistic Style Transfer

# Reference: famous painting
style = load_image("monet_painting.jpg")

image = pipe(
    prompt="portrait of a woman in a garden",
    ip_adapter_image=style,
    num_inference_steps=30
).images[0]

Brand Consistency

# Reference: brand style guide image
style = load_image("brand_style.jpg")

prompts = [
    "website hero banner",
    "social media post",
    "email newsletter header"
]

for prompt in prompts:
    image = pipe(prompt=prompt, ip_adapter_image=style).images[0]

Memory Optimization

pipe.enable_model_cpu_offload()
pipe.enable_vae_slicing()

# For very limited VRAM
pipe.enable_sequential_cpu_offload()

Performance

Model	GPU	Time
SD 1.5 + IP-Adapter	RTX 3090	~5s
SDXL + IP-Adapter	RTX 3090	~12s
SDXL + IP-Adapter	RTX 4090	~8s
SDXL + IP-Adapter-Plus	RTX 4090	~10s

Troubleshooting

Style Not Applied

• Increase ip_adapter_scale

• Use clearer reference image

• Ensure adapter loaded correctly

Too Much Reference Influence

• Decrease ip_adapter_scale

• More specific text prompt

• Increase guidance_scale

Memory Issues

• Enable CPU offload

• Use SD 1.5 instead of SDXL

• Lower resolution

Cost Estimate

Typical CLORE.AI marketplace rates (as of 2024):

GPU	Hourly Rate	Daily Rate	4-Hour Session
RTX 3060	~$0.03	~$0.70	~$0.12
RTX 3090	~$0.06	~$1.50	~$0.25
RTX 4090	~$0.10	~$2.30	~$0.40
A100 40GB	~$0.17	~$4.00	~$0.70
A100 80GB	~$0.25	~$6.00	~$1.00

Prices vary by provider and demand. Check CLORE.AI Marketplace for current rates.

Save money:

• Use the Spot market for interruptible work — about a third of servers price spot below on-demand (median ~13% off), the rest match it

• Pay with CLORE tokens

• Compare prices across different providers

Next Steps

• InstantID - Face identity

• ControlNet - Structure control

• Stable Diffusion WebUI - IP-Adapter extension