Support PhotoMaker

[kelvincai522]

Trying to use compel on the photomaker pipeline. Got error: RuntimeError: CUDA error: device-side assert triggered

pipe = PhotoMakerStableDiffusionXLPipeline.from_single_file(

    "model.safetensors",
    torch_dtype=torch.bfloat16, 
    use_safetensors=True, 
    variant="fp16"
).to("cuda")

### Load PhotoMaker checkpoint
pipe.load_photomaker_adapter(
    folder,
    subfolder="",
    weight_name= weightname,

    trigger_word="img"  # define the trigger word
)     

pipe.scheduler = DDPMScheduler.from_config(pipe.scheduler.config)


# Weighted prompt with compel
from compel import Compel, ReturnedEmbeddingsType

compel = Compel(
  tokenizer=[pipe.tokenizer, pipe.tokenizer_2] ,
  text_encoder=[pipe.text_encoder, pipe.text_encoder_2],
  returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED,
  requires_pooled=[False, True]
)

id_img = Image.open(img.png")

prompt = "blahbla"

conditioning, pooled = compel(prompt)

# generation
generator = torch.Generator(device="cuda").manual_seed(4)

images = pipe(
    #prompt=prompt,
    prompt_embeds=conditioning,
    pooled_prompt_embeds=pooled,    
    input_id_images= [id_img],
    num_images_per_prompt=1,
    num_inference_steps=10,
    guidance_scale=4,
    start_merge_step=0, 
    generator=generator,
    height= 1024,
    width= 1024,    
).images[0]
images.show()

[bghira]

you need to adjust compel to index on cpu instead of cuda @damian0815 if an indexing error occurs when accessing nonexistent sequence positions then the CUDA context is corrupted and the interpreter must be restarted

[bghira]

@kelvincai522 you didnt use the trigger word