img2img Generations different from WebUI vs API

[bijanv]

Have successfully generated desired images using img2img masked inpainting, with ControlNet canny edge maps through the WebUI.

However when I take the same configs and try to run the generation through the API, I get extremely poor results (low quality / seemingly overfitted inpaints, and not using my custom checkpoint).

• I'm hitting /sdapi/v1/options - which confirms that my checkpoint is correct (is there another way to ensure it is loaded for the generations or is this good enough)?
• When running through the API , I get confirmation through the CLI that the canny model (control_sd15_canny [fef5e48e]) is loading and that the preprocessor is canny

I'm not sure why the results don't come out nearly the same as the WebUI, any ideas of where else I can check for discrepancies that could be causing this?

[ljleb]

When receiving control unit configs from the API, the script sees the json objects as python dicts. They are then converted to class instances, which means any extra property should raise an error instead of silently bypassing them. As the models are loading in memory, I assume the dicts are properly passed to the script.

This may be caused by default values not being the same. Can you make sure all (non-deprecated) properties are explicitly passed to the API? You can find them here: https://github.com/Mikubill/sd-webui-controlnet/wiki/API#controlnetunitrequest-json-object

[bijanv]

@ljleb I think you gave me a good hint though, there may also be unwanted defaults for the standard img2img, going to try and be explicit on every param and that might fix it.

[ljleb]

Perfect, let me know if you find anything in that territory.

I tried debugging the gradio values and it appears that the mask has by default a 4th dimension filled with 255 values when passed by the ui, which I expect to be the alpha channel of the mask. At the moment the API creates a mask with the same shape as the image and filled with 0s when it is omitted, which might cause problems if your input image has an alpha channel (i.e. transparency). If the route parameters seem to be fine, you could try experimenting in this direction maybe?

[bijanv]

Interesting, my input image does have an alpha channel. In this scenario what would you recommend as the ideal fix? To send through the API a "mask" that is just pure 255 or white image?

[ljleb]

I'm really not sure. I haven't wrapped my head around the code for mask yet, so I'm not sure what will happen if you put all 255. I suggest you try it and let us know if anything happens.

[Vespinian]

It does seem to be checking for the inpainting mask here L704 and is doing something with it assuming you are using inpaint_full_res (which seems to translate to the inpaint area setting in the webui, Whole picture = 0 and Only masked = 1), but I don't see why it wouldn't work with the API. What are the inpainting settings used in the webui?