Skip to content

Conversation

SahilCarterr
Copy link
Contributor

@SahilCarterr SahilCarterr commented Oct 12, 2024

What does this PR do?

Fixes #9610 Added Lora support to Stable Diffusion3 Image2Image Pipeline

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed.
@yiyixuxu


Usage

import os
import torch
from diffusers import StableDiffusion3Img2ImgPipeline
from diffusers.utils import load_image
from PIL import Image

# Set device and model path
device = "cuda"
model_id_or_path = "stabilityai/stable-diffusion-3-medium-diffusers"

# Initialize pipeline
pipe = StableDiffusion3Img2ImgPipeline.from_pretrained(model_id_or_path, torch_dtype=torch.float16)
pipe.load_lora_weights("zwloong/sd3-lora-training-rank16-v2")
pipe = pipe.to(device)

# Load the initial image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")

# Set prompt and negative prompt
prompt = "A design of a cute pokemon, featuring a single bird, water type pokemon, on a background"
negative_prompt = 'blurry, cropped, ugly'

# List of different scale values to apply
scale_values = [0.5, 0.7, 0.9, 1.1, 1.3]

# Directory to save images
save_dir = "./generated_images_scale"
os.makedirs(save_dir, exist_ok=True)

# Loop over each scale value
for i, scale in enumerate(scale_values):
    # Generate the image with the current scale value
    generator = torch.Generator(device='cuda').manual_seed(44)
    image = pipe(
        prompt=prompt,
        image=init_image,
        generator=generator,
        guidance_scale=5.0,
        negative_prompt=negative_prompt,
        joint_attention_kwargs={"scale": scale}
    ).images[0]
    
    # Save the image
    image_save_path = os.path.join(save_dir, f"image_scale_{scale}.png")
    image.save(image_save_path)
    print(f"Saved image with scale {scale} to {image_save_path}")

Results Table

Input Image Output without LoRA Weights With LoRA Weights Output at different lora scale
download (1) cat_without_lora image_scale_1 3 output

@a-r-r-o-w a-r-r-o-w requested a review from yiyixuxu October 14, 2024 12:39
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SahilCarterr
Copy link
Contributor Author

How to resolve the transformers Version error?
@a-r-r-o-w

@a-r-r-o-w
Copy link
Member

I think you just have to merge diffusers:main to here (it's been fixed in main)

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@yiyixuxu yiyixuxu merged commit 22ed39f into huggingface:main Oct 14, 2024
15 checks passed

def test_sd3_img2img_lora(self):
pipe = self.pipeline_class.from_pretrained(self.repo_id, torch_dtype=torch.float16)
pipe.load_lora_weights("nerijs/pixel-art-xl", weight_name="pixel-art-xl.safetensors")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SahilCarterr why are we using an SDXL LoRA to test SD3?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it can be replace by a sd3 lora hf-internal-testing/tiny-sd3-loras i will open a PR for the fix

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think because those LoRAs were meant for models that are much smaller in nature. So, configs won't match.

This should ideally be tested with "zwloong/sd3-lora-training-rank16-v2" as this is an integration test.

@SahilCarterr SahilCarterr mentioned this pull request Nov 2, 2024
6 tasks
sayakpaul pushed a commit that referenced this pull request Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add load_lora_weights to pipeline_stable_diffusion_3_img2img.py
5 participants