add LoRA parameters into a unet model

[lyb369]

I use the code below to add LoRA parameters into a unet model, but I find the result is not so good, could someone help me to solve the problem or how can I add LoRA parameters into a unet model?

target_modules = [
    "to_q","to_k","to_v","to_out.0"
]


lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    lora_dropout=0,
    init_lora_weights="gaussian",
    target_modules=target_modules,
)

def apply_lora(unet):
    # return unet.add_adapter(lora_config)
    return LoraModel(unet, lora_config, adapter_name="default")