monkeypatch_lora finds CrossAttention blocks and looks for any Linears

• https://github.com/cloneofsimo/lora/blob/master/lora_diffusion/lora.py#L152

It uses named_modules which will find any descendants, either immediate children or subchildren, But then it sets the replacement LoraInjectedLinear on the CrossAttention block directly

• https://github.com/cloneofsimo/lora/blob/master/lora_diffusion/lora.py#L169

CrossAttention has a ModuleList for to_out that contains a Linear.

• https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention.py#L567

Because of the above, the to_out Linear is not replaced correctly. Instead, an (unused) LoraInjectedLinear model with the name of to_out.0 is set on the CrossAttention block.

You can tell by looking at the module names on the CrossAttention block after patching.

Before patching:

to_q
to_k
to_v
to_out

After patching:

to_q
to_k
to_v
to_out
to_out.0