Describe the bug

When calling load_ip_adapter() with a custom cache_dir param, if the specified model requires the image encoder (i.e. ip-adapter-plus_sd15.bin), then diffusers will install the ip adapter model (i.e. ip-adapter-plus_sd15.bin) in the specified cache_dir but the image encoder (i.e. image_encoder/model.safetensors) in the HF_HUB_CACHE dir.

The expected behaviour is that it should install both of the models in the specified cache_dir.

Similarly, when calling load_ip_adapter() with local_files_only=True and ip-adapter-plus_sd15.bin, if only that .bin file is already installed (either in HF_HUB_CACHE or in the specified cache_dir if given) but not the required image encoder, it will install that image encoder to the HF_HUB_CACHE dir. In this case, the expected behaviour is that it throws an exception since it can't find the file locally, like what it does if ip-adapter-plus_sd15.bin does not exist in cache_dir.

Reproduction

import diffusers, torch

pipe = diffusers.StableDiffusionImg2ImgPipeline.from_pretrained("SG161222/Realistic_Vision_V5.1_noVAE",torch_dtype=torch.float16).to("mps")  # Or any SD1.5-based model
                    
pipe.load_ip_adapter("h94/IP-Adapter", cache_dir="/Users/[username]/Desktop/test", subfolder="models", weight_name="ip-adapter-plus_sd15.bin") # , local_files_only=True)

Logs

No response

System Info

macOS 14
diffusers 0.30.0

Who can help?

@yiyixuxu @asomoza @sayakpaul @DN6