1.11.1 (#289)

* better naming convention * add date * api return b64 video * hook encode_pil_to_base64 * use date instead of datetime * remove unnecessary code for ipadapter * vram optim * recover from assertion error such as OOM without the need to re-start * bugfix * add todo to pr a1111 * fix lllite, fix absolute path, fix infotext * fix api * readme
continue-revolution · Nov 7, 2023 · 0e6dc4d · 0e6dc4d
1 parent fd8a7f0
commit 0e6dc4d
Show file tree

Hide file tree

Showing 9 changed files with 177 additions and 161 deletions.
diff --git a/README.md b/README.md
@@ -53,6 +53,7 @@ You might also be interested in another extension I created: [Segment Anything f
 - `2023/10/21`: [v1.9.4](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.9.4): Save prompt travel to output images, `Reverse` merged to `Closed loop` (See [WebUI Parameters](#webui-parameters)), remove `TimestepEmbedSequential` hijack, remove `hints.js`, better explanation of several context-related parameters.
 - `2023/10/25`: [v1.10.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.10.0): Support img2img batch. You need ControlNet installed to make it work properly (you do not need to enable ControlNet). See [ControlNet V2V](#controlnet-v2v) for more information.
 - `2023/10/29`: [v1.11.0](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.11.0): Support [HotShot-XL](https://github.com/hotshotco/Hotshot-XL) for SDXL. See [HotShot-XL](#hotshot-xl) for more information.
+- `2023/11/06`: [v1.11.1](https://github.com/continue-revolution/sd-webui-animatediff/releases/tag/v1.11.1): optimize VRAM to support any number of control images for ControlNet V2V, patch [encode_pil_to_base64](https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/master/modules/api/api.py#L104-L133) to support api return a video, save frames to `AnimateDIff/yy-mm-dd/`, recover from assertion error without restart.
 
 For future update plan, please query [here](https://github.com/continue-revolution/sd-webui-animatediff/pull/224).
 
@@ -67,16 +68,16 @@ For future update plan, please query [here](https://github.com/continue-revoluti
 1. Go to txt2img if you want to try txt2gif and img2img if you want to try img2gif.
 1. Choose an SD1.5 checkpoint, write prompts, set configurations such as image width/height. If you want to generate multiple GIFs at once, please [change batch number, instead of batch size](#batch-size).
 1. Enable AnimateDiff extension, set up [each parameter](#webui-parameters), then click `Generate`.
-1. You should see the output GIF on the output gallery. You can access GIF output at `stable-diffusion-webui/outputs/{txt2img or img2img}-images/AnimateDiff`. You can also access image frames at `stable-diffusion-webui/outputs/{txt2img or img2img}-images/{date}`. You may choose to save frames for each generation into separate directories in `Settings/AnimateDiff`.
+1. You should see the output GIF on the output gallery. You can access GIF output at `stable-diffusion-webui/outputs/{txt2img or img2img}-images/AnimateDiff/{yy-mm-dd}`. You can also access image frames at `stable-diffusion-webui/outputs/{txt2img or img2img}-images/{yy-mm-dd}`. You may choose to save frames for each generation into separate directories in `Settings/AnimateDiff`.
 
 ### API
-Just like how you use ControlNet. Here is a sample. Due to the limitation of WebUI, you will not be able to get a video, but only a list of generated frames. You will have to view GIF in your file system, as mentioned at [WebUI](#webui) item 4. For most up-to-date parameters, please read [here](https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_ui.py#L26).
+It is quite similar to the way you use ControlNet. API will return a video in base64 format. In `format`, `PDF` means to save frames to your file system without returning all the frames. If you want your API to return all frames, please add `Frame` to `format` list. For most up-to-date parameters, please read [here](https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_ui.py#L26).
 ```
 'alwayson_scripts': {
   'AnimateDiff': {
     'args': [{
       'model': 'mm_sd_v15_v2.ckpt',   # Motion module
-      'format': ['GIF'],      # Save format, 'GIF' | 'MP4' | 'PNG' | 'WEBP' | 'TXT'
+      'format': ['GIF'],      # Save format, 'GIF' | 'MP4' | 'PNG' | 'WEBP' | 'TXT' | 'Frame'
       'enable': True,         # Enable AnimateDiff
       'video_length': 16,     # Number of frames
       'fps': 8,               # FPS
@@ -105,6 +106,7 @@ Just like how you use ControlNet. Here is a sample. Due to the limitation of Web
 1. **Save format** — Format of the output. Choose at least one of "GIF"|"MP4"|"WEBP"|"PNG". Check "TXT" if you want infotext, which will live in the same directory as the output GIF. Infotext is also accessible via `stable-diffusion-webui/params.txt` and outputs in all formats.
     1. You can optimize GIF with `gifsicle` (`apt install gifsicle` required, read [#91](https://github.com/continue-revolution/sd-webui-animatediff/pull/91) for more information) and/or `palette` (read [#104](https://github.com/continue-revolution/sd-webui-animatediff/pull/104) for more information). Go to `Settings/AnimateDiff` to enable them.
     1. You can set quality and lossless for WEBP via `Settings/AnimateDiff`. Read [#233](https://github.com/continue-revolution/sd-webui-animatediff/pull/233) for more information.
+    1. If you are using API, by adding "PNG" to `format`, you can save all frames to your file system without returning all the frames. If you want your API to return all frames, please add `Frame` to `format` list.
 1. **Number of frames** — Choose whatever number you like. 
 
     If you enter 0 (default):
@@ -238,12 +240,6 @@ Batch number is NOT the same as batch size. In A1111 WebUI, batch number is abov
 We are currently developing approach to support batch size on WebUI in the near future.
 
 
-## FAQ
-1.  Q: Will ADetailer be supported?
-
-    A: I'm not planning to support ADetailer. However, I plan to refactor my [Segment Anything](https://github.com/continue-revolution/sd-webui-segment-anything) to achieve similar effects.
-
-
 ## Demo
 
 ### Basic Usage
@@ -274,7 +270,7 @@ I thank researchers from [Shanghai AI Lab](https://www.shlab.org.cn/), especiall
 
 I also thank community developers, especially
 - [@zappityzap](https://github.com/zappityzap) who developed the majority of the [output features](https://github.com/continue-revolution/sd-webui-animatediff/blob/master/scripts/animatediff_output.py)
-- [@TDS4874](https://github.com/TDS4874) and [@opparco](https://github.com/opparco) for resolving the grey issue which significantly improve the performance of this extension
+- [@TDS4874](https://github.com/TDS4874) and [@opparco](https://github.com/opparco) for resolving the grey issue which significantly improve the performance
 - [@talesofai](https://github.com/talesofai) who developed i2v in [this forked repo](https://github.com/talesofai/AnimateDiff)
 - [@rkfg](https://github.com/rkfg) for developing GIF palette optimization
 

diff --git a/scripts/animatediff.py b/scripts/animatediff.py
@@ -1,5 +1,3 @@
-import os
-
 import gradio as gr
 from modules import script_callbacks, scripts, shared
 from modules.processing import (Processed, StableDiffusionProcessing,
@@ -39,12 +37,13 @@ def show(self, is_img2img):
 
 
     def ui(self, is_img2img):
-        model_dir = shared.opts.data.get("animatediff_model_path", os.path.join(script_dir, "model"))
-        return (AnimateDiffUiGroup().render(is_img2img, model_dir),)
+        return (AnimateDiffUiGroup().render(is_img2img, motion_module.get_model_dir()),)
 
 
     def before_process(self, p: StableDiffusionProcessing, params: AnimateDiffProcess):
-        if isinstance(params, dict): params = AnimateDiffProcess(**params)
+        if p.is_api and isinstance(params, dict):
+            self.ad_params = AnimateDiffProcess(**params)
+            params = self.ad_params
         if params.enable:
             logger.info("AnimateDiff process start.")
             params.set_p(p)
@@ -60,25 +59,25 @@ def before_process(self, p: StableDiffusionProcessing, params: AnimateDiffProces
 
 
     def before_process_batch(self, p: StableDiffusionProcessing, params: AnimateDiffProcess, **kwargs):
-        if isinstance(params, dict): params = AnimateDiffProcess(**params)
+        if p.is_api and isinstance(params, dict): params = self.ad_params
         if params.enable and isinstance(p, StableDiffusionProcessingImg2Img) and not hasattr(p, '_animatediff_i2i_batch'):
             AnimateDiffI2VLatent().randomize(p, params)
 
 
     def postprocess_batch_list(self, p: StableDiffusionProcessing, pp: PostprocessBatchListArgs, params: AnimateDiffProcess, **kwargs):
-        if isinstance(params, dict): params = AnimateDiffProcess(**params)
+        if p.is_api and isinstance(params, dict): params = self.ad_params
         if params.enable:
             self.prompt_scheduler.save_infotext_img(p)
 
 
     def postprocess_image(self, p: StableDiffusionProcessing, pp: PostprocessImageArgs, params: AnimateDiffProcess, *args):
-        if isinstance(params, dict): params = AnimateDiffProcess(**params)
+        if p.is_api and isinstance(params, dict): params = self.ad_params
         if params.enable and isinstance(p, StableDiffusionProcessingImg2Img) and hasattr(p, '_animatediff_paste_to_full'):
             p.paste_to = p._animatediff_paste_to_full[p.batch_index]
 
 
     def postprocess(self, p: StableDiffusionProcessing, res: Processed, params: AnimateDiffProcess):
-        if isinstance(params, dict): params = AnimateDiffProcess(**params)
+        if p.is_api and isinstance(params, dict): params = self.ad_params
         if params.enable:
             self.prompt_scheduler.save_infotext_txt(res)
             self.cn_hacker.restore()
@@ -94,7 +93,7 @@ def on_ui_settings():
     shared.opts.add_option(
         "animatediff_model_path",
         shared.OptionInfo(
-            os.path.join(script_dir, "model"),
+            None,
             "Path to save AnimateDiff motion modules",
             gr.Textbox,
             section=section,
@@ -144,7 +143,7 @@ def on_ui_settings():
         "animatediff_save_to_custom",
         shared.OptionInfo(
             False,
-            "Save frames to stable-diffusion-webui/outputs/{ txt|img }2img-images/AnimateDiff/{gif filename}/ "
+            "Save frames to stable-diffusion-webui/outputs/{ txt|img }2img-images/AnimateDiff/{gif filename}/{date} "
             "instead of stable-diffusion-webui/outputs/{ txt|img }2img-images/{date}/.",
             gr.Checkbox,
             section=section

diff --git a/scripts/animatediff_cn.py b/scripts/animatediff_cn.py
@@ -22,12 +22,11 @@
 
 
 class AnimateDiffControl:
+    original_processing_process_images_hijack = None
+    original_controlnet_main_entry = None
+    original_postprocess_batch = None
 
     def __init__(self, p: StableDiffusionProcessing, prompt_scheduler: AnimateDiffPromptSchedule):
-        self.original_processing_process_images_hijack = None
-        self.original_img2img_process_batch_hijack = None
-        self.original_controlnet_main_entry = None
-        self.original_postprocess_batch = None
         try:
             from scripts.external_code import find_cn_script
             self.cn_script = find_cn_script(p.scripts)
@@ -118,15 +117,19 @@ def hacked_processing_process_images_hijack(self, p: StableDiffusionProcessing,
             update_infotext(p, params)
             return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
 
-        self.original_processing_process_images_hijack = BatchHijack.processing_process_images_hijack
+        if AnimateDiffControl.original_processing_process_images_hijack is not None:
+            logger.info('BatchHijack already hacked.')
+            return
+
+        AnimateDiffControl.original_processing_process_images_hijack = BatchHijack.processing_process_images_hijack
         BatchHijack.processing_process_images_hijack = hacked_processing_process_images_hijack
         processing.process_images_inner = instance.processing_process_images_hijack
 
 
     def restore_batchhijack(self):
         from scripts.batch_hijack import BatchHijack, instance
-        BatchHijack.processing_process_images_hijack = self.original_processing_process_images_hijack
-        self.original_processing_process_images_hijack = None
+        BatchHijack.processing_process_images_hijack = AnimateDiffControl.original_processing_process_images_hijack
+        AnimateDiffControl.original_processing_process_images_hijack = None
         processing.process_images_inner = instance.processing_process_images_hijack
 
 
@@ -412,21 +415,21 @@ def set_numpy_seed(p: processing.StableDiffusionProcessing) -> Optional[int]:
 
                     if control_model_type == ControlModelType.IPAdapter:
                         if model_net.is_plus:
-                            controls_ipadapter['hidden_states'].append(control['hidden_states'][-2])
+                            controls_ipadapter['hidden_states'].append(control['hidden_states'][-2].cpu())
                         else:
-                            controls_ipadapter['image_embeds'].append(control['image_embeds'])
+                            controls_ipadapter['image_embeds'].append(control['image_embeds'].cpu())
                         if hr_control is not None:
                             if model_net.is_plus:
-                                hr_controls_ipadapter['hidden_states'].append(hr_control['hidden_states'][-2])
+                                hr_controls_ipadapter['hidden_states'].append(hr_control['hidden_states'][-2].cpu())
                             else:
-                                hr_controls_ipadapter['image_embeds'].append(hr_control['image_embeds'])
+                                hr_controls_ipadapter['image_embeds'].append(hr_control['image_embeds'].cpu())
                         else:
                             hr_controls_ipadapter = None
                             hr_controls = None
                     else:
-                        controls.append(control)
+                        controls.append(control.cpu())
                         if hr_control is not None:
-                            hr_controls.append(hr_control)
+                            hr_controls.append(hr_control.cpu())
                         else:
                             hr_controls = None
 
@@ -599,17 +602,21 @@ def hacked_postprocess_batch(self, p, *args, **kwargs):
                     images[i] = post_processor(images[i], i)
             return
 
-        self.original_controlnet_main_entry = self.cn_script.controlnet_main_entry
-        self.original_postprocess_batch = self.cn_script.postprocess_batch
+        if AnimateDiffControl.original_controlnet_main_entry is not None:
+            logger.info('ControlNet Main Entry already hacked.')
+            return
+
+        AnimateDiffControl.original_controlnet_main_entry = self.cn_script.controlnet_main_entry
+        AnimateDiffControl.original_postprocess_batch = self.cn_script.postprocess_batch
         self.cn_script.controlnet_main_entry = MethodType(hacked_main_entry, self.cn_script)
         self.cn_script.postprocess_batch = MethodType(hacked_postprocess_batch, self.cn_script)
 
 
     def restore_cn(self):
-        self.cn_script.controlnet_main_entry = self.original_controlnet_main_entry
-        self.original_controlnet_main_entry = None
-        self.cn_script.postprocess_batch = self.original_postprocess_batch
-        self.original_postprocess_batch = None
+        self.cn_script.controlnet_main_entry = AnimateDiffControl.original_controlnet_main_entry
+        AnimateDiffControl.original_controlnet_main_entry = None
+        self.cn_script.postprocess_batch = AnimateDiffControl.original_postprocess_batch
+        AnimateDiffControl.original_postprocess_batch = None
 
 
     def hack(self, params: AnimateDiffProcess):

diff --git a/scripts/animatediff_i2ibatch.py b/scripts/animatediff_i2ibatch.py
@@ -24,6 +24,7 @@
 class AnimateDiffI2IBatch:
 
     def hack(self):
+        # TODO: PR this hack to A1111
         logger.info("Hacking i2i-batch.")
         original_img2img_process_batch = img2img.process_batch
 
@@ -299,5 +300,4 @@ def cap_init_image(self, p: StableDiffusionProcessingImg2Img, params):
                 params.batch_size = len(p.init_images)
 
 
-
 animatediff_i2ibatch = AnimateDiffI2IBatch()