Strange behavior with latest Anim DIff with A1111_V1.6

[dkamajix]

using Toonyou beta6 and any of the latest motion models of Anim Diff
I get almost no motion, and then a cut of 2 diff shots within the same gif.
with 16 frames @8fps. everything else left at default.
image looks contrasted lo res and barely any motion and more importantly with 2 different shots within the same gif.
What can be happening? I have reinstalled many times, I have tried many other 1.5 checkpoints, and motion models.
Any help would be wonderful, because I see so many beautiful results, but for a strange reason, my set up is far from it.

[AiSeppli]

I can confirm, see video below. The video is "Model hash: 8635af1c8c, Model: epicphotogasm_x"
But I have found some models that work for me"Model hash: d193ccfee5, Model: duchaitenjourney_v56723" or "Model hash: 4e089bea6e, Model: realcartoon3d_v8" with animatediffMotion_v15V2.ckpt.
I have not much experience with this extension, it seems to be very buggy still. But when it works, it's great, thank you developers.
https://github.com/continue-revolution/sd-webui-animatediff/assets/143931321/e0938253-c10c-42b8-be98-3b5dd0d33804

[HansHunz]

I've had this issue as well but noticed that the prompt length on both (positive/negitive) have to be under 75. If it goes over it seems to cause this issue for me.

[continue-revolution]

#83

If you did it but the issue still exists, you should switch to another checkpoint.

[LIQUIDMIND111]

#83

If you did it but the issue still exists, you should switch to another checkpoint.

hello mate, prompt traveling is not working for me..... this is what i get

its nice, but not what i was expecting..... i was using the same sample as the FAQ here ----- https://github.com/continue-revolution/sd-webui-animatediff#faq

[continue-revolution]

@LIQUIDMIND111 You have to install ControlNet and update it to 1.1.410. You do not need to enable ControlNet

[LIQUIDMIND111]

@LIQUIDMIND111 You have to install ControlNet and update it to 1.1.410. You do not need to enable ControlNet

yes i have all that installed and updated everyday since always.... im gonna try this prompt again with this model mm_sd_v15_v2.safetensors

1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed,
0: closed mouth
8: open mouse
smile

[continue-revolution]

In the worst case you may want to do a fresh reinstall.

[LIQUIDMIND111]

@LIQUIDMIND111 You have to install ControlNet and update it to 1.1.410. You do not need to enable ControlNet

i do have control net buddy, look

2023-10-09 15:57:53,209 - AnimateDiff - INFO - Setting DDIM alpha.
2023-10-09 15:57:53,294 - AnimateDiff - INFO - Injection finished.
2023-10-09 15:57:53,294 - AnimateDiff - INFO - Hacking lora to support motion lora
2023-10-09 15:57:53,295 - AnimateDiff - INFO - Hacking CFGDenoiser forward function.
2023-10-09 15:57:53,296 - AnimateDiff - INFO - Hacking ControlNet.

it just wont do a prompt travel like the samples i see

[LIQUIDMIND111]

1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed,
0: closed mouth
8: open mouse
smile

ok lets say im doing 32 frames, does this prompt should read like this then:

1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed,
0: closed mouth
8: open mouth
16: funny mouth
24: large mouth
smile

[continue-revolution]

yes, your syntax is correct, but I have no idea why your webui is not working but many others work.

1. See if your terminal output "You are using prompt travel."
2. If so, add some print statements in animatediff_prompt.py, especially in multi_cond method at L91-L97 to see what's going on.

[LIQUIDMIND111]

yes, your syntax is correct, but I have no idea why your webui is not working but many others work.

1. See if your terminal output "You are using prompt travel."
2. If so, add some print statements in animatediff_prompt.py, especially in multi_cond method at L91-L97 to see what's going on.

i update my extension of animatediff yesterday, so its supposed to work..... but, what's this and how do i activate it?:

---> See if your terminal output "You are using prompt travel."

i can see that here-----> animatediff_prompt.py
if data['mapp_prompts']:
logger.info("You are using prompt travel.")
self.prompt_map = {}
prompt_list = []
last_frame = 0
current_prompt = ''

[continue-revolution]

2023-10-09 15:14:23,634 - AnimateDiff - INFO - AnimateDiff process start.
2023-10-09 15:14:23,635 - AnimateDiff - INFO - You are using mm_sd_v15_v2.ckpt, which has been tested and supported.
2023-10-09 15:14:23,635 - AnimateDiff - INFO - Loading motion module mm_sd_v15_v2.ckpt from /home/conrevo/stable-diffusion-webui/extensions/sd-webui-animatediff/model/mm_sd_v15_v2.ckpt
2023-10-09 15:14:25,680 - AnimateDiff - WARNING - Missing keys <All keys matched successfully>
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet middle block.
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet input blocks.
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet output blocks.
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Setting DDIM alpha.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Injection finished.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Hacking lora to support motion lora
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Hacking CFGDenoiser forward function.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Hacking ControlNet.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - You are using prompt travel.

This is the whole terminal log I can see before the generation starts. If you do not see "You are using prompt travel." then there might be some problem. You should check if you disabled ControlNet in Extensions panel.

[LIQUIDMIND111]

2023-10-09 15:14:23,634 - AnimateDiff - INFO - AnimateDiff process start.
2023-10-09 15:14:23,635 - AnimateDiff - INFO - You are using mm_sd_v15_v2.ckpt, which has been tested and supported.
2023-10-09 15:14:23,635 - AnimateDiff - INFO - Loading motion module mm_sd_v15_v2.ckpt from /home/conrevo/stable-diffusion-webui/extensions/sd-webui-animatediff/model/mm_sd_v15_v2.ckpt
2023-10-09 15:14:25,680 - AnimateDiff - WARNING - Missing keys <All keys matched successfully>
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet middle block.
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet input blocks.
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Injecting motion module mm_sd_v15_v2.ckpt into SD1.5 UNet output blocks.
2023-10-09 15:14:25,842 - AnimateDiff - INFO - Setting DDIM alpha.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Injection finished.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Hacking lora to support motion lora
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Hacking CFGDenoiser forward function.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - Hacking ControlNet.
2023-10-09 15:14:25,843 - AnimateDiff - INFO - You are using prompt travel.

This is the whole terminal log I can see before the generation starts. If you do not see "You are using prompt travel." then there might be some problem. You should check if you disabled ControlNet in Extensions panel.

got you, NO i dont get that statement.... let me check the controlnet thing

[continue-revolution]

[LIQUIDMIND111]

ahh ok, yes its activated THERE in that TAB and HERE.....

what else can it be? im using ControlNet v1.1.410

[LIQUIDMIND111]

ok so in order to activate PROMPT TRAVEL, do i need to click on something or just type a prompt like this
1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed,
0: closed mouth
8: open mouse
smile

and it will activate automatically?

[continue-revolution]

you do not need to click anything. it will be activated automatically.

https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_prompt.py#L47 add print("prompt debug: ", data) here and see what's going on. The indentation should be the same as the "if" below it.

https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_cn.py#L106 add print("prompt parsing start") here and see if you can see anything in your terminal. The indentation should be the same as the line below it.

You should remove them afterwards to receive future updates from me.\

Another possibility is that you did not enable padding cond uncond in Settings/Optimization

[LIQUIDMIND111]

you do not need to click anything. it will be activated automatically.

https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_prompt.py#L47 add print("prompt debug: ", data) here and see what's going on. The indentation should be the same as the "if" below it.

https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_cn.py#L106 add print("prompt parsing start") here and see if you can see anything in your terminal. The indentation should be the same as the line below it.

You should remove them afterwards to receive future updates from me.\

Another possibility is that you did not enable padding cond uncond in Settings/Optimization

i deleted and reinstalled the axtension and now i get this
2023-10-09 16:29:41,069 - AnimateDiff - INFO - Setting DDIM alpha.
2023-10-09 16:29:41,122 - AnimateDiff - INFO - Injection finished.
2023-10-09 16:29:41,123 - AnimateDiff - INFO - Hacking lora to support motion lora
2023-10-09 16:29:41,124 - AnimateDiff - INFO - Hacking CFGDenoiser forward function.
2023-10-09 16:29:41,124 - AnimateDiff - INFO - Hacking ControlNet.
2023-10-09 16:29:41,125 - AnimateDiff - INFO - You are using prompt travel.

seems it will work now, lets see, thanks a lot for your attention, will let you know...

[Seryogerr]

you do not need to click anything. it will be activated automatically.
https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_prompt.py#L47 add print("prompt debug: ", data) here and see what's going on. The indentation should be the same as the "if" below it.
https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_cn.py#L106 add print("prompt parsing start") here and see if you can see anything in your terminal. The indentation should be the same as the line below it.
You should remove them afterwards to receive future updates from me.
Another possibility is that you did not enable padding cond uncond in Settings/Optimization

i deleted and reinstalled the axtension and now i get this 2023-10-09 16:29:41,069 - AnimateDiff - INFO - Setting DDIM alpha. 2023-10-09 16:29:41,122 - AnimateDiff - INFO - Injection finished. 2023-10-09 16:29:41,123 - AnimateDiff - INFO - Hacking lora to support motion lora 2023-10-09 16:29:41,124 - AnimateDiff - INFO - Hacking CFGDenoiser forward function. 2023-10-09 16:29:41,124 - AnimateDiff - INFO - Hacking ControlNet. 2023-10-09 16:29:41,125 - AnimateDiff - INFO - You are using prompt travel.

seems it will work now, lets see, thanks a lot for your attention, will let you know...

did it work?

[LIQUIDMIND111]

you do not need to click anything. it will be activated automatically.
https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_prompt.py#L47 add print("prompt debug: ", data) here and see what's going on. The indentation should be the same as the "if" below it.
https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_cn.py#L106 add print("prompt parsing start") here and see if you can see anything in your terminal. The indentation should be the same as the line below it.
You should remove them afterwards to receive future updates from me.
Another possibility is that you did not enable padding cond uncond in Settings/Optimization

i deleted and reinstalled the axtension and now i get this 2023-10-09 16:29:41,069 - AnimateDiff - INFO - Setting DDIM alpha. 2023-10-09 16:29:41,122 - AnimateDiff - INFO - Injection finished. 2023-10-09 16:29:41,123 - AnimateDiff - INFO - Hacking lora to support motion lora 2023-10-09 16:29:41,124 - AnimateDiff - INFO - Hacking CFGDenoiser forward function. 2023-10-09 16:29:41,124 - AnimateDiff - INFO - Hacking ControlNet. 2023-10-09 16:29:41,125 - AnimateDiff - INFO - You are using prompt travel.
seems it will work now, lets see, thanks a lot for your attention, will let you know...

did it work?

yes it did...... works well

[LIQUIDMIND111]

you do not need to click anything. it will be activated automatically.

https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_prompt.py#L47 add print("prompt debug: ", data) here and see what's going on. The indentation should be the same as the "if" below it.

https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_cn.py#L106 add print("prompt parsing start") here and see if you can see anything in your terminal. The indentation should be the same as the line below it.

You should remove them afterwards to receive future updates from me.\

Another possibility is that you did not enable padding cond uncond in Settings/Optimization

works well now, but one question. MOTION LORAS wont work with prompt travel? if yes or no, how many motion loras can you use at once?

[Li1177]

1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed,
0: closed mouth
8: open mouth
16: funny mouth
24: large mouth
smile

I have a question. I saw in a tutorial that both prompt tokens and Negative Prompt tokens should ideally be kept within 75 tokens. Besides the main prompt token, will the content of the prompt travel increase the token count?

[continue-revolution]

@Li1177 For 75 token issue, there is already a pinned issue addressing this issue. I don't think this is a problem now. Regarding how token is counted for prompt travel, your actual token will be processed to be something like this internally:
0: 1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed, closed mouth, smile
8: 1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed, open mouth, smile
16: 1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed, funny mouth, smile
24: 1girl, yoimiya (genshin impact), origen, line, comet, wink, Masterpiece, BestQuality. UltraDetailed, large mouth, smile

and cond for frames between the index listed above will be interpolated. You can count token in this way.