Make LoRA from two SDXL models doesn't work: "Error(s) in loading state_dict for CLIPTextModel"

[tsukimiya]

I seem to have the same problem as #329 when trying to generate a LoRA of the differences from the SDXL model.

building U-Net
loading U-Net from checkpoint
U-Net:  <All keys matched successfully>
building text encoders
loading text encoders from checkpoint
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "/notebooks/sd_reforge/extensions/sd-webui-supermerger/scripts/mergers/pluslora.py", line 333, in makelora
    result = ext.svd(args)
  File "/notebooks/sd_reforge/extensions/sd-webui-supermerger/scripts/kohyas/extract_lora_from_models.py", line 60, in svd
    text_encoder_o1, text_encoder_o2, _, unet_o, _, _ = sdxl_model_util.load_models_from_sdxl_checkpoint(
  File "/notebooks/sd_reforge/extensions/sd-webui-supermerger/scripts/kohyas/sdxl_model_util.py", line 265, in load_models_from_sdxl_checkpoint
    info1 = _load_state_dict_on_device(text_model1, te1_sd, device=map_location)  # remain fp32
  File "/notebooks/sd_reforge/extensions/sd-webui-supermerger/scripts/kohyas/sdxl_model_util.py", line 157, in _load_state_dict_on_device
    raise RuntimeError("Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	Unexpected key(s) in state_dict: "text_model.embeddings.position_ids". 

[hako-mikan]

Fixed.

[muooon]

ここで良いかわかりませんが、モデル間の差分取り出しでエラーになります。

make LoRA start
Loading weights [8766244cde] from E:\SdxlWebUi\stable-diffusion-webui\models\Stable-diffusion\muonXLmuse-v0110X-0110-ZtVP-PDX.fp16.safetensors
Creating model from config: E:\SdxlWebUi\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Applying attention optimization: xformers... done.
Traceback (most recent call last):
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "E:\SdxlWebUi\stable-diffusion-webui\extensions\sd-webui-supermerger\scripts\mergers\pluslora.py", line 296, in makelora
    load_model(checkpoint_info)
  File "E:\SdxlWebUi\stable-diffusion-webui\extensions\sd-webui-supermerger\scripts\mergers\pluslora.py", line 1581, in load_model
    sd_models.load_model(checkpoint_info)
  File "E:\SdxlWebUi\stable-diffusion-webui\modules\sd_models.py", line 869, in load_model
    sd_model.cond_stage_model_empty_prompt = get_empty_cond(sd_model)
  File "E:\SdxlWebUi\stable-diffusion-webui\modules\sd_models.py", line 728, in get_empty_cond
    d = sd_model.get_learned_conditioning([""])
  File "E:\SdxlWebUi\stable-diffusion-webui\modules\sd_models_xl.py", line 32, in get_learned_conditioning
    c = self.conditioner(sdxl_conds, force_zero_embeddings=['txt'] if force_zero_negative_prompt else [])
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 1557, in _call_impl
    args_result = hook(self, args)
  File "E:\SdxlWebUi\stable-diffusion-webui\modules\lowvram.py", line 55, in send_me_to_gpu
    module_in_gpu.to(cpu)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 1160, in to
    return self._apply(convert)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
    module._apply(fn)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
    module._apply(fn)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply
    module._apply(fn)
  [Previous line repeated 5 more times]
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply
    param_applied = fn(param)
  File "E:\SdxlWebUi\venv-a1111\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

A1111 / version: [v1.10.1 • python: 3.10.11 • torch: 2.1.2+cu121 • xformers: 0.0.23.post1 • gradio: 3.41.2

[muooon]

最新版｢f6bf9b5c24｣引き続き以下のエラーで止まります
NotImplementedError: Cannot copy out of meta tensor; no data!

[hako-mikan]

May be fixed.