Load LoRAs for inference

There are many adapter types (with LoRAs being the most popular) trained in different styles to achieve different effects. You can even combine multiple adapters to create new and unique images.

In this tutorial, you’ll learn how to easily load and manage adapters for inference with the 🤗 PEFT integration in 🤗 Diffusers. You’ll use LoRA as the main adapter technique, so you’ll see the terms LoRA and adapter used interchangeably.

Let’s first install all the required libraries.

!pip install -q transformers accelerate peft diffusers

Now, load a pipeline with a Stable Diffusion XL (SDXL) checkpoint:

from diffusers import DiffusionPipeline
import torch

pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to("cuda")

Next, load a CiroN2022/toy-face adapter with the load_lora_weights() method. With the 🤗 PEFT integration, you can assign a specific adapter_name to the checkpoint, which let’s you easily switch between different LoRA checkpoints. Let’s call this adapter "toy".

pipe.load_lora_weights("CiroN2022/toy-face", weight_name="toy_face_sdxl.safetensors", adapter_name="toy")

Make sure to include the token toy_face in the prompt and then you can perform inference:

prompt = "toy_face of a hacker with a hoodie"

lora_scale= 0.9
image = pipe(
    prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
).images[0]
image

With the adapter_name parameter, it is really easy to use another adapter for inference! Load the nerijs/pixel-art-xl adapter that has been fine-tuned to generate pixel art images and call it "pixel".

The pipeline automatically sets the first loaded adapter ("toy") as the active adapter, but you can activate the "pixel" adapter with the set_adapters() method:

pipe.load_lora_weights("nerijs/pixel-art-xl", weight_name="pixel-art-xl.safetensors", adapter_name="pixel")
pipe.set_adapters("pixel")

Make sure you include the token pixel art in your prompt to generate a pixel art image:

prompt = "a hacker with a hoodie, pixel art"
image = pipe(
    prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
).images[0]
image

Merge adapters

You can also merge different adapter checkpoints for inference to blend their styles together.

Once again, use the set_adapters() method to activate the pixel and toy adapters and specify the weights for how they should be merged.

pipe.set_adapters(["pixel", "toy"], adapter_weights=[0.5, 1.0])

Remember to use the trigger words for CiroN2022/toy-face and nerijs/pixel-art-xl (these are found in their repositories) in the prompt to generate an image.

prompt = "toy_face of a hacker with a hoodie, pixel art"
image = pipe(
    prompt, num_inference_steps=30, cross_attention_kwargs={"scale": 1.0}, generator=torch.manual_seed(0)
).images[0]
image

Impressive! As you can see, the model generated an image that mixed the characteristics of both adapters.

To return to only using one adapter, use the set_adapters() method to activate the "toy" adapter:

pipe.set_adapters("toy")

prompt = "toy_face of a hacker with a hoodie"
lora_scale= 0.9
image = pipe(
    prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
).images[0]
image

Or to disable all adapters entirely, use the disable_lora() method to return the base model.

pipe.disable_lora()

prompt = "toy_face of a hacker with a hoodie"
lora_scale= 0.9
image = pipe(prompt, num_inference_steps=30, generator=torch.manual_seed(0)).images[0]
image

Manage active adapters

You have attached multiple adapters in this tutorial, and if you’re feeling a bit lost on what adapters have been attached to the pipeline’s components, use the get_active_adapters() method to check the list of active adapters:

active_adapters = pipe.get_active_adapters()
active_adapters
["toy", "pixel"]

You can also get the active adapters of each pipeline component with get_list_adapters():

list_adapters_component_wise = pipe.get_list_adapters()
list_adapters_component_wise
{"text_encoder": ["toy", "pixel"], "unet": ["toy", "pixel"], "text_encoder_2": ["toy", "pixel"]}