1. LoRA加载与使用

• https://llamafactory.cn/huggingface-docs/diffusers/using-diffusers/merge_loras.html
• https://huggingface.co/docs/diffusers/main/en/using-diffusers/merge_loras

import os
import torch
from PIL import Image
from diffusers import QwenImageEditPlusPipeline

pipeline = QwenImageEditPlusPipeline.from_pretrained(f"{model_hub}/Qwen/Qwen-Image-Edit-2509", torch_dtype=torch.bfloat16).to(0)
print("pipeline loaded")

# infer code ....
inputs = {
    "image": [Image.open("../" + data["control_path_0"])],
    "prompt": prompt, # data["caption"],
    "generator": torch.manual_seed(0),
    "true_cfg_scale": 4.0,
    "negative_prompt": " ",
    "num_inference_steps": 20,
    "guidance_scale": 1.0,
    "num_images_per_prompt": 1,
}
with torch.inference_mode():
    output = pipeline(**inputs)
    output_image = output.images[0]

1. 1Load LoRA no-fuse

pipeline.load_lora_weights(
    "xxx.safetensors",  # 本地 LoRA 文件夹路径
    adapter_name="xxx"
)

pipeline.set_adapters("xxx", adapter_weights=[1.0])

# infer code ....

1.2 fuse: 合并LoRA权重到主权重

pipeline.unfuse_lora()   # 移除LoRA权重
pipeline.load_lora_weights(
    "xxx.safetensors",  # 本地 LoRA 文件夹路径
    adapter_name="xxx"
)
pipeline.fuse_lora(adapter_names=["posterv3_lora"], lora_scale=1.0, safe_fusing=True)  # 合并lora权重

# infer code ....

2. Diffusers中使用LoRA的代码逻辑

2.1 Diffusers中使用LoRA的伪代码

在diffsuers阅读某个pipeline的代码（如pipeline_qwenimage_edit_plus.py），会发现代码中既没有创建LoRA层，也没有在推理的时候调用LoRA层，那到底是什么时候调用的LoRA参数的呢？

原来diffsuers使用lora的逻辑是：在调用load_lora_weights时，会将参数对应的层替换为
peft中的LoRA层对象（BaseTunerLayer的子类，比如Linear替换成peft.tuners.lora.layer.Linear），并将原本的Linear对象放到新对象的base_layer这个成员变量下，而peft.tuners.lora.layer.Linear调用时会根据激活的adapter自动应用lora。伪代码如下：

"""NOTE:: pesudo_code"""
class peft.tuners.lora.layer.Linear(...):
	def __init__(self, ...):
		self.active_adapter = []  # 可能包含多个lora的权重
		self.scaling = []            # 多个lora融合的权重 
	
	def _set_base_layer(self, layer):  # no such func, i assume for pesudo code
		self.base_layer = layer
	
	def forward(self, x):
		"""
		w = self.base_layer.weight.data + self.get_delta_weight(self.activate_adpater_names)
		w @ x + b
		"""
		result = self.base_layer(x)
		for active_adapter in self.active_adapters:
			lora_B, lora_A, dropout, scaling = get_lora_layers(active_adapter)
			result = result + lora_B(lora_A(dropout(x))) * scaling
		return result

def load_lora_weights(self, lora_path, adpater_name, ...):
	sd = load(lora_path)
	name_to_peft_module = defaultdict(list)
	for name, module in self.named_modules():
		w = get_weight(sd, name)
		if w is not None:
			if name not in name_to_peft_module:
				name_to_peft_module[name] = build_peft_layer(module, w)
				peft_module._set_base_layer(module)
				
			peft_module = name_to_peft_module[name]
			peft_module.add_lora_weight(w, adpater_name)

打印后可以获得：

from peft.tuners.tuners_utils import BaseTunerLayer
for name, module in pipeline.transformer.named_modules():
    if isinstance(module, BaseTunerLayer):
        print(name, type(module))
        break

print(module)

transformer_blocks.0.img_mod.1 <class 'peft.tuners.lora.layer.Linear'>
lora.Linear(
  (base_layer): Linear(in_features=3072, out_features=18432, bias=True)
  (lora_dropout): ModuleDict(
    (posterv3_lora): Identity()
  )
  (lora_A): ModuleDict(
    (posterv3_lora): Linear(in_features=3072, out_features=16, bias=False)
  )
  (lora_B): ModuleDict(
    (posterv3_lora): Linear(in_features=16, out_features=18432, bias=False)
  )
  (lora_embedding_A): ParameterDict()
  (lora_embedding_B): ParameterDict()
  (lora_magnitude_vector): ModuleDict()
)

2.2 Diffuser中LoRA层的一些有用的接口

• merge: fuse_lora会调用该代码。
• get_delta_weight: 将lora权重A、B转换成最终的delta_w，merge会调用到

a = module.get_base_layer().weight.data
b = module.get_delta_weight("posterv3_lora")
(a + b) - a
``

# 3. Lora权重的分析