FLUX LoRA with CLIP text-encoder weights fails (empty rank -> IndexError) under transformers>=5

[christopher5106]

Describe the bug

Loading a kohya-style FLUX LoRA that contains CLIP text-encoder weights (lora_te1_*) fails with IndexError: list index out of range when transformers>=5 is installed.

The text-encoder half of the LoRA is never loaded; pipe.load_lora_weights(...) raises before any adapter is injected.

Root cause

transformers>=5 flattened CLIPTextModel: the text_model. wrapper module was removed, so text_encoder.named_modules() now yields names like encoder.layers.0.self_attn.k_proj instead of text_model.encoder.layers.0.self_attn.k_proj.

The kohya→diffusers conversion (_convert_kohya_flux_lora_to_diffusers) still produces text-encoder keys prefixed with text_model., e.g. text_encoder.text_model.encoder.layers.0.self_attn.k_proj.lora_B.weight.

In diffusers/loaders/lora_base.py::_load_lora_into_text_encoder, the rank dict is built by matching text_encoder.named_modules() against the (converted, PEFT-format) state-dict keys:

for name, _ in text_encoder.named_modules():
    if name.endswith((".q_proj", ".k_proj", ".v_proj", ".out_proj", ".fc1", ".fc2")):
        rank_key = f"{name}.lora_B.weight"   # e.g. "encoder.layers.0.self_attn.k_proj.lora_B.weight"
        if rank_key in state_dict:           # but keys are "text_model.encoder.layers.0...."
            rank[rank_key] = state_dict[rank_key].shape[1]

Under transformers 5 the module names no longer carry the text_model. prefix while the state-dict keys still do, so nothing matches, rank stays empty, and _create_lora_config → get_peft_kwargs does r = lora_alpha = list(rank_dict.values())[0] → IndexError.

Reproduction

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
# any kohya FLUX LoRA that includes CLIP text-encoder weights (lora_te1_*)
pipe.load_lora_weights("kohya_flux_lora_with_text_encoder.safetensors")

Minimal isolation of the empty-rank step:

from transformers import CLIPTextModel
te = CLIPTextModel.from_pretrained("black-forest-labs/FLUX.1-dev", subfolder="text_encoder")
print(hasattr(te, "text_model"))                       # False on transformers>=5 (was True on 4.x)
print([n for n, _ in te.named_modules()][:6])          # ['', 'embeddings', ..., 'encoder', 'encoder.layers']
# -> module names are now un-prefixed, but converted LoRA keys are still 'text_model.encoder....'

Traceback

File ".../diffusers/loaders/lora_base.py", line 390, in _load_lora_into_text_encoder
    lora_config = _create_lora_config(state_dict, network_alphas, metadata, rank, is_unet=False)
File ".../diffusers/utils/peft_utils.py", line 355, in _create_lora_config
    lora_config_kwargs = get_peft_kwargs(...)
File ".../diffusers/utils/peft_utils.py", line 158, in get_peft_kwargs
    r = lora_alpha = list(rank_dict.values())[0]
IndexError: list index out of range

Suggested fix

Reconcile the converted text-encoder key namespace with the model's named_modules() for transformers>=5 (drop/normalize the stale text_model. segment), or build rank in a prefix-tolerant way. A clearer error than a bare IndexError when rank ends up empty would also help.

System Info

• diffusers 0.38.0
• transformers 5.9.0
• peft 0.19.1
• torch 2.10–2.11 (+cu128), Python 3.11/3.12, Linux
• Model: black-forest-labs/FLUX.1-dev

Who can help?

@sayakpaul @BenjaminBossan

[BenjaminBossan]

I believe this should be fixed via huggingface/peft#3212. Could you please try again with PEFT installed from the main branch?

[christopher5106]

Thanks @BenjaminBossan. I tried it, but the IndexError still happens.

Setup: peft @ git+https://github.com/huggingface/peft@75b88f7 (the #3212 merge commit, verified installed from main), diffusers==0.38.0, transformers==5.9.0, with my local text_model.-stripping workaround removed.

The reason #3212 doesn't help here is that the crash happens before any PEFT state-dict injection. It's raised inside diffusers while building the LoraConfig rank, not inside set_peft_model_state_dict:

File ".../diffusers/loaders/lora_base.py", line 390, in _load_lora_into_text_encoder
    lora_config = _create_lora_config(state_dict, network_alphas, metadata, rank, is_unet=False)
File ".../diffusers/utils/peft_utils.py", line 355, in _create_lora_config
    lora_config_kwargs = get_peft_kwargs(...)
File ".../diffusers/utils/peft_utils.py", line 158, in get_peft_kwargs
    r = lora_alpha = list(rank_dict.values())[0]
IndexError: list index out of range

rank is built in _load_lora_into_text_encoder purely from string-matching, with no PEFT involvement:

for name, _ in text_encoder.named_modules():        # transformers>=5: "encoder.layers.0.self_attn.q_proj"
    if name.endswith((".q_proj", ".k_proj", ".v_proj", ".out_proj", ".fc1", ".fc2")):
        rank_key = f"{name}.lora_B.weight"          # "encoder.layers.0.self_attn.q_proj.lora_B.weight"
        if rank_key in state_dict:                  # but converted keys are "text_model.encoder.layers.0...."
            rank[rank_key] = state_dict[rank_key].shape[1]

convert_state_dict_to_diffusers / convert_state_dict_to_peft still leave the stale text_model. segment on the text-encoder keys, while text_encoder.named_modules() no longer carries it under transformers>=5, so rank stays empty and we never reach set_peft_model_state_dict (where #3212 applies).

So this looks like it needs a diffusers-side fix — reconciling the converted text-encoder key namespace with the model's named_modules() (drop/normalize text_model.), or building rank in a prefix-tolerant way. Re-adding the text_model. strip on the text-encoder state dict before _load_lora_into_text_encoder makes the same LoRA load correctly with this exact peft-from-main build, which confirms the mismatch is upstream of PEFT.